I'm implementing some automatic functional tests in an environenment that has about 20 applications talking to eachother using queues and/or restful interfaces.
The problem I'm facing is that I added some test to one of the application, lets call it Payments, and they all looked very nice and green. One of the changes on the Payments application added an extra field in a messages that is send to another application, lets call it CRM, and the CRM discarded the message, which meant that the applications started to become out of sync. The Payments application (as most other applications) send messages in a fire-and-forget way, which means that the producer never knows if a message is processed or discarded.
I'm wondering is how to test these dependencies between systems. One option (that I dislike from my gut, but no idea why) is to connect to the other system and verify that the record was updated. Most applications have a restful interface and also provide clients, so is not a lot of work. But I have the strong feeling that doing this will make the test more brittle. One of the reasons why I don't like this, is that most of the applications talk to another 4-10 other applications, so it's not just checking another app... and since lots of things are asyncronous, it adds even more uncertainity to the test.
So, how do you create functional tests for this type of environment?