Your issue is the same as if you were asked to do an app which can use either .NET Framework or Java, or a website component which works either with Silverlight or with Flash, depending on what the user has.
It is possible technically, but it leads to duplicate code and maintenance difficulties. There is nothing to do with it.
That's also why most frameworks are able to work side by side: it allows to use several libraries which use several frameworks on a same website¹, avoiding collisions in method names like
Note about the advice given in two other answers: “don't build libraries on frameworks”. It is, well, too idealistic. This advice may be applied to maybe 1% of the libraries, probably much less: the libraries which are written by teams of developers able to build a library which will work on any browser, which will still be maintenable, and which require, for some reason, no framework at all.
Outside Amazon/Google/Apple scale websites, do never follow the “don't build libraries on frameworks” advice, unless your library is too basic to require a framework. Instead:
- Reuse, instead of reinventing the wheel,
- Spend time doing something useful and interesting, i.e. writing your library, instead of writing something already available in every framework,
- Trust the frameworks: they are written by skilled people who know things you may not know or forget,
- Rely on abstractions: when I use a JQuery animation, I know it will work in Chrome, Firefox, Safari, Opera and even Internet Explorer. When I'm creating my own, I have to test it in those five browsers (up to ten if you count their major versions), and I'm pretty sure I will fail the first time at least in one browser.
² Note that according to jQuery website, Google, Dell, NBC etc. are using it, so it's not because you're working on a large scale website that you cannot use a framework. In practice, large companies will end up inventing their own, which fits better their needs. This is the case for example for Google and its Closure Library.