castles: Dealing with multiple OS's isn't too bad. Where reasonable, the differences are abstracted (e.g., window creation, file system change detection), and for cases where one OS simply has different requirements (e.g., on OS X the application has to specify if it wants a light i-beam mouse cursor vs a dark one, while other Windows and Linux only have a single one) then the API is made wide enough to handle it, with dummy implementations on the platforms that don't care.
I mostly code on Linux, as compile times are by far the fastest there, partially because of better hardware, and partially because of the gold linker. Also, I'm happiest with the linux dev tool chain (objdump, valgrind, etc), although MSVC does have by far and away the best debugger. I'll generally only frequently switch between computers when working on features with platform specific hooks, like the recent drag and drop integration.
I use VMs for building the linux releases, but otherwise have 3 PCs on my desk.