If the problem is a common one like, say, writing a compiler or a browser, the requirements are pretty much given in the form of language standards, target operating systems and target hardware, etc.
For things like GNU Emacs, which is many things to many besides excellently fulfilling its original goal of being a text editor, I think the requirements made sense because of the immense scope to extend it. Chats, emails, newsgroups, code editing, version control come to the mind. There is a research scientist working on Emacspeak. I think similar things can be said of browsers and other things that allow extensions.
If the software is catching up a function that is available only in non open-source software, the requirement is pretty much given again.
When the open source software goes on to maintenance and fewer original requirements remain unmet, most of the requirements can come from bugs, need to adapt to new platforms such as multi core CPUs and other hardware that offer better performance when exploited, and such.
In a fully research based project like the GNU Hurd, I'd think the requirements come from research results and papers.
To sum up,
when starting out, the requirements for software that attempts to solve common problems can come from standards documents
for software that is catching up to other existing software, the requirements are likely to be to produce all or most of the existing software's feature set and some other features that the developers/users find interesting to have
for research projects, papers and other publications could set the requirements
when in maintenance, bugs, need to adapt to newer environments can be major source for requirements