From looking at screenshots it's fairly obvious that the Windows and OSX GUIs of μTorrent are not identical, so it's fairly unlikely that they used a multi-platform GUI library. μTorrent for Linux doesn't even have a conventional GUI, it offers a web-based interface, which would be programmed separately to a Windows GUI in any case.
That's the way to go: If you want to keep binary size down, you can't depend on libraries that aren't included in the operating system, because you'll have to bundle them.
Programming separate GUIs for separate platforms is, in my opinion, usually the correct approach: user interfaces on OSX and Windows follow significantly different paradigms, and if you port an app designed for Windows to OSX without changing the user interface, it will look quite alien to the user. Cross-platform GUI libraries can be very helpful tools, but they make it very easy to build software that looks out-of-place everywhere.