The benefits of binary: smaller size, and faster (de)serialisation. The small size can also mean faster transmission, especially if you're using a slow network (eg mobile).
Sure, the programmer might have more work to do but that depends on the library used to perform the serialisation. If you're using a JSON or XML library to do this, I would expect that you could replace the internals of that library with a binary protocol and not even notice the change. (in fact, I know this - we did it once, our serialisation lib was XML, when we replaced the object with a binary version, overall performance of our program increased by 20%).
I would expect that the only true benefit of JSON or XML is that a human can read the data whilst it is serialised. Whether this really is useful depends if you're using the data for storage or just for communication. Even then, create a tool to show you the contents.
Incidentally, csv format is still just as bad - the split is fundamentally between some form of textual format, or a binary one. Given the choice between csv, xml or json ... I would have to say "what's the data stored", csv can be a very good choice for a lot of data, especially as you already have a tool to manipulate it (a spreadsheet) or to import it into a database. If the data is hierarchical in some way (eg a graph or tree) then csv isn't such a good choice.