Well, there's a couple of ways you can modify open source software without releasing it.
If you wrote it and are the one who applied the license then you can do whatever you want. It is fully possible to release code under the GPL and have your own, internal proprietary version.
If you are using and modifying the software for internal purposes, and not releasing the binaries, then under most licenses (even GPL) you don't have to release source.
If you are in the process of developing, but have not released to the public, you don't have to release source.
Some open source licenses do not require release of the source. (For instance, BSD or apache style licenses.)
If you are in one of these situations, and don't have to release source, then anything in the unreleased source can be a trade secret. Once you release of course that stops being the case.
My understanding is that even with the GPL you can certainly limit who you ship binaries to, which in turn limits who you have to release source to. So, for instance, if you say "you can't have the binaries unless you sign this NDA", then I don't think you have to give the source to anyone who has not signed the NDA.
In general, my understanding is that the GPL doesn't require giving source to anyone at all, just anyone with access to the binaries.
I am not a lawyer, so obviously consult a real one if it is an issue.