I love working in Ruby, and I don't have much opportunity or desire to do web development, so I tried doing exactly this. I grabbed ruby-gnome2 to give me an interface framework and got a good deal of (fairly basic) stuff.
Then I tried it out on a VM under Linux. And sending it to friends. And I learned how horribly difficult it is to distribute Ruby code. For example, the Ruby Windows Installer gives you a lot of basic libraries (such as YAML) for free, which don't come under Linux (and I never did find the right gem install for it). Some libraries work on one OS and not another. A lot of the graphical stuff, especially lower level* stuff, seems to be entirely broken as of 1.9, and isn't getting updated.
So I wouldn't recommend it really.
Ruby is great for scripting purposes, and if you're completely in control of the deployment environment (like a web server) you can do just fine. But distribution is a pain. You might be able to mitigate this though, by using JRuby or IronRuby, which run on the JVM and CLR respectively. They also give you access to graphical toolkits.
*I had a nasty experience trying to get OpenGL to work with Ruby 1.9.3 on Windows. It turned out to be essentially impossible, even after following several rather arcane guides by people who ran into the same problem.