I was browsing through the gem documentation today, and I found a command that I didn't know existed.

sudo gem cleanup

It removes old versions of your installed gems. You can of course specify the gems you'd like to cleanup.