My stance is it's QA's job to find bugs that we can't find,
Finding bugs is a joint responsibility.
QA's job is to find bugs that you don't find. There should not be such a thing as bugs that you can't find. (And if there is, then maybe you should ask for access to the tools / tests that they are using so that you can find them for yourself.
refactoring usually introduces breaking changes.
That is not true. Refactoring done properly doesn't break things. And you could argue that fixing the things that break should be part of the refactoring.
The problems arise in reality when:
the refactoring triggers latent bugs in other parts of the codebase that the tests you can run don't reveal, or
you can't do the full refactor because someone else "owns" some parts of the code that is affected by the refactor.
You shouldn't cop the blame for latent bugs (unless they were yours in the first place). But on the other hand, if your refactoring is intentionally changing APIs and or behaviours then you need to discuss this with all parties before you refactor, and have a plan for dealing with the effects. On the third hand1, if the breaking change is unexpected or accidental ... then you should have thought it through in more detail.
And finally, if the regressions were your fault (or partly your fault), then "man up" and take the blame. And aim to do better next time.
If he agrees that refactoring is necessary (which he does), then why does he have a problem when stuff breaks? Especially when it hasn't affected our schedule so far.
That is easy to answer if you put yourself into his shoes. He thinks you did the wrong thing in the way you refactored the code. This time it didn't impact on the schedule. But next time, it could. He is trying to prevent that eventuality.
1 - A Larry Niven allusion.