I've tried to read this series of books many times. I recently bought Vol. 4 to add to my shelf. I do actually hope to one day read and understand it. I only seem to be able to get the most surface understanding of any of it though.
When I was in college I did one of those design your own courses. I decided for myself to get through the initial part of Vol. 1, hoping I'd come out with the ability to read the rest. Was the one class I ended up pestering professors about on a daily basis and I still didn't get it. The proofs blew me away for the most part and though I could look at an answer in the back and usually, through many hours of struggling and help, figure out how it was derived...I could never actually do it.
I know of the book Concrete Mathematics, and I've read parts of it. Still though, I think I need more foundation to put these topics in perspective and actually DO some of the exercises myself.
What other things can I do in order to better myself enough to begin facing this mountain?