You should never have to feel guilty about saying Yes or No.
Open yourself up to new possibilities. You may doubt yourself, or question your choice. Whatever you decide to do, own it and the consequences whether good or bad that comes along with it.
Give yourself permission, and open yourself up to new possibilities.
No is ok too. If something doesn’t work for you or doesn’t fit with your mission. Say no. Saying no means you can concentrate on the things that do work for you.