Is there much thought towards internal diagnosis testing within an enterprise application?

Question

Imagine a discussion between four people - one a developer, one a manager, one a tester and one a deployment/operations type.

The Manager is very angry as a system has fallen over after a new version of code was deployed into production. He is looking to point fingers.

Operations person pleads ignorance - they followed the deployment steps exactly.

Tester pleads ignorance - they have functionally tested code and it worked fine in test environment.

Development person pleads ignorance - they provided good code, and claim that the system badly configured.

This situation may seem familiar to some. There's always an excuse, some mistake made along the chain - from design to deployment - that allows a system to be upgraded but doesn't work as intended.

What are the pros and cons of diagnosis testing - tests run built into the application (perhaps through a command line API or something) that would allow a system to check it's internal connections (database, middleware, filesystem etc) and confirm that everything it expects to work is correct?

This is rather than relying on an external monitoring system that may not be configured in the same fashion as the application? (new release, different properties, looking at the wrong database etc)

An example :

I'm writing a class that interacts with a database. Within this class, I create a method that interacts with the database without affecting it, say a simple select query.

In the constructor of this class, I pass to an observer a test based on the arguments passed into the constructor. This observer can be called to run all the tests it has at any time so as to test for the validity of the connection - even while the system is actually running.

Answer 1

What are the pros and cons of diagnosis testing - tests run built into the application (perhaps through a command line API or something) that would allow a system to check it's internal connections (database, middleware, filesystem etc) and confirm that everything it expects to work is correct?

Pros

• It allows the deployment people to run the tests rather than you waiting until off hours to smoke test the app.
• It gives you one more hurdle to clear before the boss is pissed.

Cons

• It's more maintenance.
• It's another exposed endpoint to input, which has its own security concerns.
• It's unnecessary.

Okay, I threw this last one in here. In my experience ...

• If your test environment followed best practices and mirrored production, this wouldn't be an issue.
• If someone smoke tested the deployment to production, this wouldn't be an issue.
• If you could write the tests to properly verify that stuff, why aren't you doing it in code already and including the cause of the failure in your errors?
• And really, how are the post-deploy checks going to save your system once you've broken it?

Answer 2

I can't comment, so I'll have to answer here. Robert is onto something with his comments. This seems to be a management and communication issue, and less one of testing. So management should point the finger right back at themselves or just get over it, pointing fingers does a lot of nothing in making the product better and causes people to form sides. /soapbox

As for the testing, there are many methodologies that match this. Methods like Test Driven Development even place this first, before your production code is written. Write the test that meets your specification, then write code to pass it. These tests can be left in and made accessible for future pushes/merges of your code as a base level of acceptance, providing they don't have security issues related to the data of course. I personally use this method in my org in our CI environment and it works great.

The main cons of using something like this when deployed to a production environment are mostly security (if you've exposed a new attack surface through one of your test hooks), or performance due to excessive use, loops, or excessive logging. The value of all of the additional information this can provide though is quite a big check in the "pro" category. Providing someone actually uses the info, of course. But that's a mess I won't get into.

Answer 3

You make it sound as though this enterprise application is a web application. Rarely, are test environments every exact mirrors of production. Even if there is a pre-prod environment between test and prod, it's a challenge to keep it as an exact mirror of prod, especially in an enterprise environment where there may be different teams responsible for the maintenance of each environment. All of the testing in the world may not have caught this before going to production.

What you are suggesting sounds much like a common health check page.

Cons:

• Many developers may feel that it is a waste of time
• People may forget that it exists and not use it
• Everything that Telastyn mentioned

Pro's:

• Allows for on the fly checking of all key config files and integration points
• Speeds up trouble-shooting immensely. Support staff can often diagnose issues before getting development team involved.
• Allows preliminary validation to be completed before production smoke testing is even started

Personally, I'm a huge fan of these kinds of tests. They make my life as a tester a whole lot easier. Support staff and developers also tend to really like these pages as well after needing them once. Can make the world's difference when trouble-shooting.

Answer 4

IMHO internal diagnosis can be a good thing - when it is done correctly and in a way that shows potential errors, which would otherwise stay undetected. But I think your example shows a case where this should normally not apply. Testing for validity of a database connection should by no means need additional programming effort by an application developer: that is a test the underlying database framework should do by itself every time a query is executed against the database - in case the connection is broken, this should result in throwing an exception or an automatic reconnect (proper error handling in any application assumed, which is a different thing).

However, a good diagnosis test is to check the database (schema) version or patch level, in case there are dependencies between the application version and the database version. That check could be run whenever a db connection is established. Such a test requires that you have something like a version number stored in your database, which is properly updated whenever you update your db schema.

When you are trying to build a fail safe system, internal diagnosis may be one thing, but IMHO more important is redundancy: your example

the Manager is very angry as a system has fallen over after a new version of code was deployed

shows a situation where someone did not expect failures when upgrading a system. This is a totally wrong approach - everyone must (!) expect failure in such a situation, and should have proper preventive measures taken beforehand, like a proper backup of all data and applications of the previous version or a simulation of the restore process to make sure it works and is fast enough. This includes mostly organizational measures like not sending all your admins into holidays right after such an upgrade, and running enough tests afterwards to make sure the system works as intended.

Any kind of diagnosis (internal or external) cannot prevent you from failure, but it may help to detect failure earlier.

Answer 5

I think enterprise grade applications or system should have monitoring and diagnostics as core design features. These non functional requirements are an absolute must. It can be achieved in many ways from passive (like writing to event logs, diagnostic logging, exposing performance counters etc.) to some kind of active Admin Dashboard and all the way to APIs as you mentioned. e.g. Microsoft's flagship StockTrader application has this kind of health monitoring built in and is a good reference architecture. Those practicing Continuous Delivery has concepts like immune system where if some predefined checks fail then an automated rollback is performed.