If you are truly doing agile, then your team should be fully empowered as to the implementation details and your PM shouldn't have a say in it.
On one hand, agile is all about delivering some value quickly, rather than waiting until every wanted feature is perfectly in place. This occasionally means reducing short term effort at the expense of additional long term work. That is one of the hardest habits to break. I know it has been for our team.
On the other hand, it is a myth that being agile requires foregoing all long term planning. It's perfectly fine, even encouraged, to think about "phase 3," as long as you recognize that business priorities may change to never implement it. That means if two weeks of extra work now could prevent two weeks of extra work in phase 3, no question you do it in phase 3. If one week of extra work now prevents two months of work in phase 3, there's little question about doing it now.
Between those extremes, your team must use their best judgement, but I would lean toward postponing the work when you could go either way. The reason is that by the time you get to phase 3, you will know a lot more about what you need. Perhaps you will need a large refactor anyway to incorporate some feature you haven't even thought about today, and all your work now is essentially wasted.
This is an Agile development project and I do not have the ability to change that.
Your attitude about Agile speaks worlds about how it is currently being practiced in your organization.