Hiya! I'm actually going through exactly the same thing you're experiencing! -- the same thoughts, & fears.  I'm planning to transfer to Calarts too!  It's nice to know other people go through the same thing, and makes it a bit less daunting.

A lot of people seem to think it's all the same, but I don't really think that's true.  It may be the same subject matter, but some schools just aren't a perfect match for everyone.  I guess you could think of it as a relationship -- you wouldn't want to stay in it if it wasn't working out at all;  it could do more harm than good.  I think with school it's hard to give it your all when you feel it's not a right match, or you're not getting what you expected.  Sometimes it even drains you. I think that's why there's transfers in the first place! 

I've spent awhile worrying and thinking things over as well.  I think that maybe from the perspective of others, it may be "stupid", but in the end it's your life.  Not everyone will understand how you feel or understand your needs.  If it's really in your heart to go to Calarts, you should!  I still worry about what other people think sometimes, but I've come to realize that I wouldn't want to go through life constantly wondering "what if".

Remember you're not alone! and good luck!!