Should I just switch barns immediately?

I would just leave the barn. Its very clear your not happy there and most likely your horse isnt either plus they can sense if someone doesnt like them. You and your horse will most likely feel much better after you leave.