Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.

We have to organize a very long list of manually running tests. Currently we use Word documents, print them out check them off etc. Ugh-ly but works, with problems.

Problems with the current solution

  • Requesting and executing subsets, depending on the actual changes. (i.e. tests 153, 155-157 and 24325 need to run only if Module X changed)
  • Branching. Different releases (which are still maintained, up to a point) have different test protocols, but share a lot of contents. Manual copy & paste.
  • Tracking "Test Template" vs. "Test executed" documents. It's a bit old-style office with lots of forms & binders.
  • Tracking in source control together with the source changes. Ideally, the same way features are copied / moved between branches, i.e. as diffable document that is included in a commit.

Content requirements

Documents need images (usually, hardware setup specs and screenshots of "good" curves), "Pass / Fail" checkboxes and minor formatting (suc has emphasis, steps, distinction of "things to do" and "what to expect")

Current idea

One of my devs suggested we use HTML documents instead (Sharepoint Designer is deemed appropriate for editing), "normalize them somehow so they are diffable", and put them under version control together with the sources.

I am personally not wholly sold on this idea for various reasons, but I agree with the other guys that if it works it's better than Word documents.

The Question:

How do you / would manage the documentation for manual tests? Do you know any tools? Ideas?

We are open to other solutions, but only with minimal custom development efforts. Example: installing dedicated software, configuring some templates would be great, but we have no resources for "hack something together with XML and stylesheets and a database, that should be easy".

Some facts&figures

Source control is git, 6 software developers, ~1.5KLoc, one run through all manual tests takes about 4 man-weeks (order of magnitude) and roughly the same in wall time with all modifications and repetitions. (A full run only happens for major releases.)


Note: Please don't tell me to test automatically, thank you but we already do. The manual tests involve integration, visualization and external measurement equipment. In addition, the information produced is so multifold that the discoveries of a engineer who knows the whole system discovers a whole lot more with a single look at a chart.


Example: A typcial test setup would look like this, every question mark needs to be "checked", it's usually 5..15 tests per setup

Connect 10Ohm resistor to S1. Connect amplifier input to OUT 1. Run test Template 42.

  • Measure voltage over resistor. ~1Vrms ?
  • In Table 3 Uac = 1.0 V, Iac= 0.1 A, ISNR+D > 45 dB ?
  • U(f) Spectrum flat for f > 10 Hz?
  • ...
share|improve this question
1  
I have a very similar problem due to auditing reasons we need the documentation and proof of manual testing, look forward to answers to this. –  Jonno Aug 10 '11 at 10:19
    
It wouldn't hurt to make your actual question clearer - it is a bit lost in the text. –  Péter Török Aug 10 '11 at 10:35
    
@Péter Török : I agree, I've tried to clarify a bit, is this better? –  peterchen Aug 10 '11 at 11:34
    
@peterchen, absolutely :-) –  Péter Török Aug 10 '11 at 11:39
add comment

1 Answer

How about moving your test documentation to a format like reStructuredText? You would be able to keep the documentation in git along with the source code and it's easier to diff than Word documents. It allows you to generate the documentation into other formats like HTML and PDF. Regarding your first point about subset testing, in reST it's possible to make documents include other documents so you could probably have "template" documents that pick "test case" documents from different places and generate complete test instruction documents for different modules.

share|improve this answer
    
I'd file that under "Markdown to HTML". Viable approach, but seems to lack a bit "infrastructure" to get up to speed quickly. (Such as interactively browsing tests and selecting them in a test run) –  peterchen Aug 10 '11 at 11:54
    
@peterchen Yes, you're right, it's lacking a bit in the tools department. –  fejd Aug 10 '11 at 14:57
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.