One downside is that when pairing you can't swap the keyboard between you as fluently. Between mainstream IDEs this is probably not a huge problem, but if one person is used to Eclipse while the other is used to vim, there is going to be a mismatch. The Eclipse user may well be entirely unable to use vim, while the vim user (that's me ;) spends a lot of time cursing under their breath at the horrible slowness of using vanilla Eclipse.
That said, I'd still much rather use vim myself. Provided your pair are happy with just one of you "driving" for extended periods it works OK.
And I know there are plugins to make Eclipse work like vi, but I'm talking about pairing where I go and sit with someone who has Eclipse working as they like it, so they won't be installing that plugin.