I have heard that improving software efficiency and algorithms over the years has made
huge performance gains
I think what you have heard here is related to better algorithms, but that is what you explicitly excluded from your question.
At the level of operating systems, I don't think many applications run faster today than 10 years ago because the OS has improved (in most cases the opposite is true - most OSs getting bigger and bigger each year, producing more overhead). Only exception may be better support for multiple processors and parallel computing, but that cannot be seen independent of hardware improvements.
So it comes mainly down to hardware. In terms of 20 or 30 years ago, it was mainly the processor clock speed, where the increasement correlated roughly 1:1 in application performance for a lot of applications (that's not perfectly true, the real increasement was less since application performance depends also on other hardware components like memory access speed, hard drive speed, GPU speed etc). When you think about the last 10 years, processor clock speed did not increase much more, since processor vendors came to practical manufacturing limits. Instead, there was a paradigm shift to parallel computing. Multi-core processor, GPU computing (which is also a form of parallel computing), SIMD in the mainstream etc. have dominated the last decade.