Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

Imagine a discussion between four people - one a developer, one a manager, one a tester and one a deployment/operations type.

The Manager is very angry as a system has fallen over after a new version of code was deployed into production. He is looking to point fingers.

Operations person pleads ignorance - they followed the deployment steps exactly.

Tester pleads ignorance - they have functionally tested code and it worked fine in test environment.

Development person pleads ignorance - they provided good code, and claim that the system badly configured.

This situation may seem familiar to some. There's always an excuse, some mistake made along the chain - from design to deployment - that allows a system to be upgraded but doesn't work as intended.

What are the pros and cons of diagnosis testing - tests run built into the application (perhaps through a command line API or something) that would allow a system to check it's internal connections (database, middleware, filesystem etc) and confirm that everything it expects to work is correct?

This is rather than relying on an external monitoring system that may not be configured in the same fashion as the application? (new release, different properties, looking at the wrong database etc)

An example :

I'm writing a class that interacts with a database. Within this class, I create a method that interacts with the database without affecting it, say a simple select query.

In the constructor of this class, I pass to an observer a test based on the arguments passed into the constructor. This observer can be called to run all the tests it has at any time so as to test for the validity of the connection - even while the system is actually running.

share|improve this question
2  
Testing methodologies are pretty well understood nowadays. This seems like a management problem; either failure to provide adequate leadership with respect to reliability testing, or failure to provide adequate resources, or both. –  Robert Harvey Aug 8 '12 at 19:27
    
@RobertHarvey I've added an example to perhaps help with what I'm trying to ask in my question. –  Spedge Aug 8 '12 at 19:35
    
Is the situation you posed hypothetical, or did this really happen? –  Robert Harvey Aug 8 '12 at 19:37
    
I'll say hypothetical... –  Spedge Aug 8 '12 at 19:39
1  
@Spedge How? This sounds like an entirely people-centric problem! –  Rig Aug 8 '12 at 19:50

5 Answers 5

I think enterprise grade applications or system should have monitoring and diagnostics as core design features. These non functional requirements are an absolute must. It can be achieved in many ways from passive (like writing to event logs, diagnostic logging, exposing performance counters etc.) to some kind of active Admin Dashboard and all the way to APIs as you mentioned. e.g. Microsoft's flagship StockTrader application has this kind of health monitoring built in and is a good reference architecture. Those practicing Continuous Delivery has concepts like immune system where if some predefined checks fail then an automated rollback is performed.

share|improve this answer

IMHO internal diagnosis can be a good thing - when it is done correctly and in a way that shows potential errors, which would otherwise stay undetected. But I think your example shows a case where this should normally not apply. Testing for validity of a database connection should by no means need additional programming effort by an application developer: that is a test the underlying database framework should do by itself every time a query is executed against the database - in case the connection is broken, this should result in throwing an exception or an automatic reconnect (proper error handling in any application assumed, which is a different thing).

However, a good diagnosis test is to check the database (schema) version or patch level, in case there are dependencies between the application version and the database version. That check could be run whenever a db connection is established. Such a test requires that you have something like a version number stored in your database, which is properly updated whenever you update your db schema.

When you are trying to build a fail safe system, internal diagnosis may be one thing, but IMHO more important is redundancy: your example

the Manager is very angry as a system has fallen over after a new version of code was deployed

shows a situation where someone did not expect failures when upgrading a system. This is a totally wrong approach - everyone must (!) expect failure in such a situation, and should have proper preventive measures taken beforehand, like a proper backup of all data and applications of the previous version or a simulation of the restore process to make sure it works and is fast enough. This includes mostly organizational measures like not sending all your admins into holidays right after such an upgrade, and running enough tests afterwards to make sure the system works as intended.

Any kind of diagnosis (internal or external) cannot prevent you from failure, but it may help to detect failure earlier.

share|improve this answer
    
Superb suggestion about the versions of the schema! This is exactly the sort of thing I'd be keen on. –  Spedge Aug 9 '12 at 8:49

What are the pros and cons of diagnosis testing - tests run built into the application (perhaps through a command line API or something) that would allow a system to check it's internal connections (database, middleware, filesystem etc) and confirm that everything it expects to work is correct?

Pros

  • It allows the deployment people to run the tests rather than you waiting until off hours to smoke test the app.
  • It gives you one more hurdle to clear before the boss is pissed.

Cons

  • It's more maintenance.
  • It's another exposed endpoint to input, which has its own security concerns.
  • It's unnecessary.

Okay, I threw this last one in here. In my experience ...

  • If your test environment followed best practices and mirrored production, this wouldn't be an issue.
  • If someone smoke tested the deployment to production, this wouldn't be an issue.
  • If you could write the tests to properly verify that stuff, why aren't you doing it in code already and including the cause of the failure in your errors?
  • And really, how are the post-deploy checks going to save your system once you've broken it?
share|improve this answer
1  
"And really, how are the post-deploy checks going to save your system once you've broken it?" Heh, a good point :) But it's more that you can confirm that things have gone well before you start allowing users onto the system. –  Spedge Aug 8 '12 at 19:57
    
+1 for error-handling. If you're not checking that your connection to the database is valid, you've written bad code that is bound to fail and be a challenge for trouble-shooting. –  David Navarre Aug 8 '12 at 20:13
    
@Telastyn Don't you think most of these cons - other than the security concern - can be applied to unit tests? –  Spedge Aug 9 '12 at 8:28
    
@DavidNavarre: if you have to invest any programming effort to check if the connection is valid, then you are using the wrong database framework. –  Doc Brown Aug 9 '12 at 10:48
    
@Spedge - Not exactly. Unit tests are more up-front development, but the maintenance is something you want to have. Instead of people being able to change behaviors willy-nilly, they also have to fix the test. And the unnecessary-ness doesn't really apply. The unit tests preclude errors in practice, the tests described in the OP don't prevent errors in production. –  Telastyn Aug 9 '12 at 12:21

You make it sound as though this enterprise application is a web application. Rarely, are test environments every exact mirrors of production. Even if there is a pre-prod environment between test and prod, it's a challenge to keep it as an exact mirror of prod, especially in an enterprise environment where there may be different teams responsible for the maintenance of each environment. All of the testing in the world may not have caught this before going to production.

What you are suggesting sounds much like a common health check page.

Cons:

  • Many developers may feel that it is a waste of time
  • People may forget that it exists and not use it
  • Everything that Telastyn mentioned

Pro's:

  • Allows for on the fly checking of all key config files and integration points
  • Speeds up trouble-shooting immensely. Support staff can often diagnose issues before getting development team involved.
  • Allows preliminary validation to be completed before production smoke testing is even started

Personally, I'm a huge fan of these kinds of tests. They make my life as a tester a whole lot easier. Support staff and developers also tend to really like these pages as well after needing them once. Can make the world's difference when trouble-shooting.

share|improve this answer
    
Smells to me like you have a better perspective on the full cycle, rather than a through-and-through developer :) –  Spedge Aug 9 '12 at 8:47
    
We have some similar tools, one that's very handy allows operation people to dynamically change the logging level of various parts of the system on the fly. So they can snapshot some debug or trace information for one part of the system, then turn it off and avoid the performance penalty. –  TMN Aug 9 '12 at 17:35

I can't comment, so I'll have to answer here. Robert is onto something with his comments. This seems to be a management and communication issue, and less one of testing. So management should point the finger right back at themselves or just get over it, pointing fingers does a lot of nothing in making the product better and causes people to form sides. /soapbox

As for the testing, there are many methodologies that match this. Methods like Test Driven Development even place this first, before your production code is written. Write the test that meets your specification, then write code to pass it. These tests can be left in and made accessible for future pushes/merges of your code as a base level of acceptance, providing they don't have security issues related to the data of course. I personally use this method in my org in our CI environment and it works great.

The main cons of using something like this when deployed to a production environment are mostly security (if you've exposed a new attack surface through one of your test hooks), or performance due to excessive use, loops, or excessive logging. The value of all of the additional information this can provide though is quite a big check in the "pro" category. Providing someone actually uses the info, of course. But that's a mess I won't get into.

share|improve this answer
    
I agree that procedure and policy have their place, but people can make mistakes. Several people can make mistakes. Assumptions are made, assurances are given without full knowledge of the situation etc. Is there not room for a technical solution within the code that checks assumptions about configuration are correct - allowing the users / operators of the solution to move on and focus on the more corner case possibilities for a problem? –  Spedge Aug 9 '12 at 8:45
    
There is definitely room for mistakes. We're all human after all. And we should be allowed to make mistakes, otherwise how would we learn. What you mentioned is exactly what I use my CI environment for. Code checks assumptions/config/specifications and providing those meet "current standards", development and test are free to move to the much more interesting corner and special case scenarios. –  maznika Aug 10 '12 at 22:27

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.