Message passing is a different way of handling the need in OO code for one object to get another object (or potentially itself) to do something.
In most modern languages that descend from the C++ approach we do that with method calls. In this case the called object (via its class definition) puts a big list of what method calls it accepts and then the coder of the calling object simply write the the call:
public void doSomething ( String input )
other_object.dosomething ( local )
For statically typed languages then compiler can then check the type of the thing being called and confirm that the method has been declared. For dynamically typed languages then that is carried out at runtime.
But in essence what happens is that a bundle of variables is sent to a specific block of code.
In message passing languages (such as Objective C) instead of methods there are recievers, but broadly the approach of defining them and calling them is much the same - the difference is the way its handled.
In a message passed language the compiler may check that the receiver that you've called exists, but at worst it will pop up a warning to say it's not sure that its there. This is because at run time what will happen is that a block of code on the receiving object will be called passing both the bundle of variables and the signature of the receiver you want to call. That block of code then looks for the receiver and calls it. However if the receiver doesn't exist then the code will simply return a default value.
As a result one of the oddities found when moving from C++/Java -> Objective C is understanding that you can "call a method" on an object that wasn't declared on the compile-time type, and didn't even exist on the run-time type ... and that the call would not result in an exception being thrown but in fact a result being passed back.
The advantages of this approach are that it flattens the subclass hierachy and avoids most of the needs for interfaces/multiple inheritance/duck types. It also allows objects to define default behaviour when asked to do something that they don't have a receiver for (commonly "if I don't do it, forward the request to this other object"). It can also simplify the linking to callbacks (e.g. for UI elements and timed events) particularly over statically typed languages such as Java (so you can have the button call the receiver "runTest" rather than call the "actionPerformed" method on the inner class "RunTestButtonListener" which does the call for you).
However it would seem to be at the cost of the need for additional checking by the developer that the call they think they are making is on the right object with the right type and passing the right parameters in the right order, because the compiler might not warn you and it will run perfectly well at runtime (just returning a default response). There is also arguably a performance hit from the extra look-up and paramter passing.
These days, dynamically typed languages can give a lot of the benefits of message passed OO with less of the issues.