The usual general questions would work.
- How do you think the project went?
- Would you continue to work with the software if you were given the opportunity? If no, how so?
I'd be probing the guy with all sorts of questions using things like the Joel test as a basis for self reflection. You only need to ask the external developer on how you should deal with all the problem points or give suggestions to them. The points being:
- How did source control work for you? Can we improve source control practices?
- Are you happy with our build process? Are there ways to improve it?
- Are you happy with the bug database? Did it work as intended?
- Did the project schedule work? Was there any overtime that was unneccesary?
- Was the spec enough? If no, what was missing?
- Did you get a satisfiable working condition in the office.
- Did you get all the things you needed to complete the project? Were you missing something?
- How was your interaction with the testers? (BAs? PMs? Anyone else?)
- What did you think of your co-workers, did you have any problem with any of them?
- Can we improve our testing in some way?
Print out the questions as a questionare for yourself, with enough space to write between the questions. Bring it to the exit meeting as you will use it to take notes. Make sure you stress that the information you'll give will be confidential if he or she desires it. Usually, people only give up feedback under confidence.
After some thinking, it seems like people take the concept of an exit interview to a hyperbole. The OP wanted to know how to get feedback from a contractor, thus the meeting shouldn't be called an exit interview or exit meeting due to the faulty connotation; it should instead be called a "feedback" interview instead.