Sounds like a motivational problem - why don't some people care about the project they're working on? Maybe its because the team is split into the 'organisers' and the 'left outs'.
So involve everyone, instead of 1 or 2 people taking charge of the planning sessions, you engage everyone - make different people take charge of each session, preferably make different people take charge during the session. Rotate it all around. I know this can seem difficult because there's always someone who wants to fuss about and organise everybody but they are the problem here.
Here's an idea: when planning choose a person at random to take charge of each story. At random. Record who was responsible for the planning of it too, so next sprint you can tell if they did a good job of getting a good consensus of estimates and task split. That'll make them pay attention, and will also give them a reason to engage with the project.
Remember, the problem isn't them, its you and the way your planning sessions are done. So when someone else takes over a story plan, they get to chose how to go about it there should be official way of proceeding. (ie don't sit back and continue to force your organisation upon them by proxy)