Will it be rude to cancel this appointment the same day of?

I have a first time appointment with a psychiatrist tomorrow. His fee for initial consultation is $380. I have been stressing over it for days trying to figure out if I should keep it or not. I think it's a universal unwritten rule that someone should cancel with a 24 hour notice so I don't know if I'd probably be charged even if I cancel it which sucks, but since it's a first time would it be rude to cancel and ask not to be billed? I looked at my bank account and I haven't sent my mortgage yet which will get a late fee if received after the 15th - i can be late on the mortgage if I go tomorrow and not cancel it bec i'm scared to call to cancel with such short notice. What do you think I should do? Plus my therapist referred him so I'm scared it'll make her look bad to have referred someone flakey who cancels last minute (appointment is for 2pm). Any opinions???