If we have a large development team, say 100, and we would like to keep our encryption key hidden from developers who are not directly involved in the encryption module/algorithm, what are some best practices to keep this key as secret as possible while still allowing for development and debugging?
The Apache foundation has a pretty simple yet effective way to handle this. The encryption key (or code signing cert) lives in a separate repository only accessible by the trusted and the BuildBot. When building the bits that need the key, the bot checks it out -- or more properly just downloads it -- and includes it in the appropriate place in the solution. You get change control and security while not interrupting the modern development practice.
All that said, I would address the underlying problem of not being able to fob the encryption key off to infrastructure that dev never has to see.