I've recently went through manually editing some tables on a Wordpress website.
I've also had some experience with database internationalization so I know that serializing
is not the best (IMO) option to apply multiple languages.
So why is it done?
Because both systems started out as single-language projects, and the internationalization was added later. And as the projects were already widespread and change in DB structure would break the backward-compatibility, it would have seemed as only viable alternative.
But this is only speculations on part.
Internationalization was not and is not the only consideration given when deciding how to structure data within the database.
I can't speak for WordPress (although I'm relatively certain they face similar problems), but in Drupal, it's not strictly true that all data is serialized: core and contributed modules can access a standard Schema API to describe their data. However, many objects are non-trivial and can be modified at will. Serializing the entire object allows for less problems when changing the structure of objects.
Actually, the answer (at least for Drupal) has nothing to do with backward compatibility. Drupal has a policy of not maintaining backwards compatibility between major versions. Instead, we go ahead and make Drupal as much better as we can with each major version release, and provide a migration path so that the database can be converted for the next version during the upgrade process.
What you describe is simply what happens when you work in an SQL database: you have to decide between slowing down and complicating even fairly basic queries with a bunch of joins, or serializing data so that items with different fields can live in the same table (which slows down and complicates a bunch of other stuff).
Remember, we're working with tables here -- every row has to have the same columns, right down the line.