As questions I'd consider useful:
When there is a new story created, how much work is being done to make sure of its quality? Is there a lot of work for developers to do before working on something or is it good to go once the PO has written the request?
How available for questions and clarification has the PO been over the course of the evaluation period?
How receptive to feedback has the PO been in terms of the evolution of the process and team over the course of several sprints?
From a higher level, here's what I'd be watching:
Decision-making. From prioritizing the backlog to clarifying the requirements on a story, there are lots of decisions that may fall to the PO to handle. Was this an issue for the team in terms of delivering work? Were there frequent delays in getting things done?
Communication. How well were the priorities communicated? Were there demonstrations of functionality where the PO provided either feedback directly or brought in customers to review new functionality to ensure that it was the desired results?
Vision. This goes a bit in hand with communication but the idea here is to know what big dates are coming up, how is the work being reviewed and what direction is the team being driven.