Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

Have you ever shipped test code (JUnit, NUnit etc) with your application that could be runnable by the end-user ?

Basic Idea is this :

The application can be deployed in a varied environment and it is hard to gather all the possible hardware - software combination or pre-requisites for the application. It is however possible to write a test suite that assert that the application has all that it needs to perform the task at hand. Unfortunately this must be done by mocking a few components so that the test does not affect the real production environment.

There are clear benefits to shipping these tests such as :

  • give the customer a means to verify the system is in order.
  • provide reports to help support diagnose issues.
  • provide means for implementation team to assert their setup should work
  • etc

Natural way to do this would be to have a subset of tests that are actually packaged with the application (either as a separate project or as a part of each components of the whole). Even more natural would be to use the tools and techniques we already use for our own unit tests namely Unit test library, mocking framework etc.

Have you ever done this ? How have you done it ? or why do you think this is a very bad idea ?

share|improve this question
    
If you do this, make sure you provide means to setup and cleanup after tests are run. –  Emmad Kareem Dec 19 '11 at 10:26
add comment

2 Answers

We used to call this smoke test. However, unit tests aren't suitable for this purpose, as they execute small pieces of code in isolation (how big is the chance that your tax calculation method suddenly stops working?) Rather, you need integration tests (also called system tests) to execute the system as a whole, and ensure that it is assembled and configured properly.

The smoke test suite may be a simple, small subset of your full integration / functional test suite, only to verify that the system as a whole can start up and can execute some of its most basic functionalities correctly. Preferably it should be

  • safe to run on the real, live system (i.e. make no changes to persistent data),
  • trivial to execute (with a single command, or push of a button),
  • fast, and
  • its output concise and unambiguous, like that of the unit tests (e.g. either "Success", or "Tests Foo and Bar failed with the following errors: ...").

And I agree with Thorbjörn in that this should rather be executed - or at least initiated - by the deployer / installer, or the support personnel in case of later system problems.

We have used, and are using such tests on several projects, although these all being server side developments, the end users executing smoke tests was usually not even an option. On the client side, of course, the picture is different. There you may consider making the smoke tests the final step of the installation process, and/or documenting its usage for troubleshooting purposes. Alternatively, if they are fast enough, they may even be run automatically at startup time, always.

share|improve this answer
1  
+1 Perhaps the most ubiquitous example of "(tests) run automatically at startup time, always", is the Power On Self Test when you switch on a machine. –  Craig Young Dec 19 '11 at 15:21
add comment

Why would the end-user do this?

To me this is part of the deployment pipe line (where it might be a very good idea). If anything, then you would run this either silently on a regular basis, or have the end-user do it on your request.

share|improve this answer
2  
An end-user would do this for trouble-shooting procedures (remember: end-user <> non-technical-user). If an option like this is available from Help|Diagnostics (or similar), an end-user could run a simple health-check. If he gets a clear error message like: "MS-XML not found", he can berate himself for uninstalling the component, and quickly fix the problem. –  Craig Young Dec 19 '11 at 15:27
    
@CraigYoung : I'm not even sure I would go so far as to give enough hints to the end-user to proceed by himself. Although some of our typical uses would deal with this easily others risk making a mess of things. –  Newtopian Dec 20 '11 at 2:01
    
Indeed perhaps the end-user is not the best target. A simpler version could be run in the application itself on start-up and yield a sort of status (green-yellow-red kinna thing). Hidden away in the help menu give the user the possibility to run the full diagnostics and provide the report to support on request. –  Newtopian Dec 20 '11 at 2:05
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.