Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

We have to organize a very long list of manually running tests. Currently we use Word documents, print them out check them off etc. Ugh-ly but works, with problems.

Problems with the current solution

  • Requesting and executing subsets, depending on the actual changes. (i.e. tests 153, 155-157 and 24325 need to run only if Module X changed)
  • Branching. Different releases (which are still maintained, up to a point) have different test protocols, but share a lot of contents. Manual copy & paste.
  • Tracking "Test Template" vs. "Test executed" documents. It's a bit old-style office with lots of forms & binders.
  • Tracking in source control together with the source changes. Ideally, the same way features are copied / moved between branches, i.e. as diffable document that is included in a commit.

Content requirements

Documents need images (usually, hardware setup specs and screenshots of "good" curves), "Pass / Fail" checkboxes and minor formatting (suc has emphasis, steps, distinction of "things to do" and "what to expect")

Current idea

One of my devs suggested we use HTML documents instead (Sharepoint Designer is deemed appropriate for editing), "normalize them somehow so they are diffable", and put them under version control together with the sources.

I am personally not wholly sold on this idea for various reasons, but I agree with the other guys that if it works it's better than Word documents.

The Question:

How do you / would manage the documentation for manual tests? Do you know any tools? Ideas?

We are open to other solutions, but only with minimal custom development efforts. Example: installing dedicated software, configuring some templates would be great, but we have no resources for "hack something together with XML and stylesheets and a database, that should be easy".

Some facts&figures

Source control is git, 6 software developers, ~1.5KLoc, one run through all manual tests takes about 4 man-weeks (order of magnitude) and roughly the same in wall time with all modifications and repetitions. (A full run only happens for major releases.)


Note: Please don't tell me to test automatically, thank you but we already do. The manual tests involve integration, visualization and external measurement equipment. In addition, the information produced is so multifold that the discoveries of a engineer who knows the whole system discovers a whole lot more with a single look at a chart.


Example: A typcial test setup would look like this, every question mark needs to be "checked", it's usually 5..15 tests per setup

Connect 10Ohm resistor to S1. Connect amplifier input to OUT 1. Run test Template 42.

  • Measure voltage over resistor. ~1Vrms ?
  • In Table 3 Uac = 1.0 V, Iac= 0.1 A, ISNR+D > 45 dB ?
  • U(f) Spectrum flat for f > 10 Hz?
  • ...
share|improve this question

closed as off-topic by Scant Roger, Bart van Ingen Schenau, GlenH7 Dec 9 '15 at 22:26

This question appears to be off-topic. The users who voted to close gave this specific reason:

  • "Questions asking us to find or recommend tools, libraries, programming languages, resources (including books, blogs, tutorials, and examples), or projects to undertake are off-topic on Programmers as they attract opinionated answers that won't have lasting value to others. You may be able to get help in The Whiteboard, our chat room." – Scant Roger, GlenH7
If this question can be reworded to fit the rules in the help center, please edit the question.

1  
I have a very similar problem due to auditing reasons we need the documentation and proof of manual testing, look forward to answers to this. – Jonno Aug 10 '11 at 10:19
    
It wouldn't hurt to make your actual question clearer - it is a bit lost in the text. – Péter Török Aug 10 '11 at 10:35
    
@Péter Török : I agree, I've tried to clarify a bit, is this better? – peterchen Aug 10 '11 at 11:34
    
@peterchen, absolutely :-) – Péter Török Aug 10 '11 at 11:39

How about moving your test documentation to a format like reStructuredText? You would be able to keep the documentation in git along with the source code and it's easier to diff than Word documents. It allows you to generate the documentation into other formats like HTML and PDF. Regarding your first point about subset testing, in reST it's possible to make documents include other documents so you could probably have "template" documents that pick "test case" documents from different places and generate complete test instruction documents for different modules.

share|improve this answer
    
I'd file that under "Markdown to HTML". Viable approach, but seems to lack a bit "infrastructure" to get up to speed quickly. (Such as interactively browsing tests and selecting them in a test run) – peterchen Aug 10 '11 at 11:54
    
@peterchen Yes, you're right, it's lacking a bit in the tools department. – fejd Aug 10 '11 at 14:57

We're using http://www.gurock.com/testrail/ to manage our manual testing. It allows us to create a bunch of different test/templates that can be combined into different test suites and the QAs execute the test suites depending on what type of test they need to do (smoke test, pre-release, post release) etc.

Each test run is tracked separately and you can create reports on different aspects. There's a lot of flexibility of how to structure the individual test, so you can fail either the whole test (e.g. Test 42) or an individual step in the test (e.g. "In Table 3 Uac = 1.0 V, Iac= 0.1 A, ISNR+D > 45 dB ?").

I wouldn't advice any process to rely on shared documents, even if they're source controlled, as you end up spending a lot of time on maintenance. Better go with a tool that's build for the purpose :)

share|improve this answer

Not the answer you're looking for? Browse other questions tagged or ask your own question.