Actually, if you use a gritt guard, the right mitt, wash your mitt and the right techniques, swirls will be minimal at best and almost non-existent.
I don't own an ISF yet, but i see people with Bently's let these high school kids wash their cars on the weekends at the local gas station, it sort of makes me laugh. I would personally give them the $$$$ and not let them touch my car.
OP: If you already have swirls, you'll need to get some assistance from someone who knows how to use an orbital polisher or a buffer (there is a difference) to get the swirls removed, then it is just a matter of not taking the car to these car washes or letting the dealer "detail" your car.
I've been detailing my STi for the past 5 years, and I don't have swirls at all. Mind you, Subaru paint isn't as good as Lexus.