Is Living Documentation a good way to document?

What is a Documentation?

In software development, a documentation is something that tells you about the functionality of software. It tells you what functionalities it has, and how they are supposed to work. Can Living Documentation fulfill this role, and fulfill it well?

The single most common problem with a documentation is that it is never up-to-date. Living Documentation is always up-to-date because we generate it automatically from the latest acceptance criteria.

Additionally, if we include test results in the Living Documentation, we have an up-to-date status on the completeness of the system – something no other documentation will ever give you. Just imagine trying to determine the actual completeness of a system: you need to make sure the requirements are kept up-to-date, you may have to dig through old backlog items, you may need to review other documentation about the system (living documentation or otherwise) – and then check in the code what is really implemented.

High-quality scenarios will not only tell you what the system does, they will also explain why the system does it. Those are important qualities of a documentation: to tell you what a system does, and what the purpose of the features is. If we generate a Living Documentation from those scenarios, we have an artefact that fulfills the main goals of a documentation.

In my experience, a Living Documentation can be a good documentation of a system – provided you have good scenarios. What makes a scenario a good scenario is beyond the scope of this article, but I want to share some thoughts with you about how many scenarios you should include, and what kind of scenarios.

Which Scenarios Contribute to a Documentation?

Whenever we specify a feature, we can ask ourselves two questions. One: do we include a scenario for this feature? Two: Do we automate this scenario? In the next paragraphs, I will focus a lot on the first question. My guiding light for including scenarios is whether it helps the reader to understand the feature.

In my experience, it’s good practice to include at least one scenario for each feature – the so-called “happy path” scenario. It shows the intention of the feature, and – if you automate the scenario – you can automatically verify that the scenario works.

If a feature has one or more decision points, then I prefer to include a scenario for each branch in the decision tree. That shows the impact of each decision on the result. This can get out of hand quickly if there are multiple decision points in a feature. If you keep your features simple, then the number of scenarios will remain manageable.

How many scenarios should you include beyond that? In my experience, there comes a point when the reader of the scenarios says “OK, I get it – I understand how this is supposed to work”. That’s when I stop writing scenarios. My benchmark for scenarios is that they should help me to understand the functionality of the system. Once that is achieved, I can move on to the next feature.

That being said: if you automate the scenarios, and if your system can run a single scenario in a short time, then you can add more scenarios with the goal of increasing the test coverage of your system. These scenarios do not contribute towards the goal of understanding the system, they are simply there to improve the test coverage. I suggest that you mark these scenarios; if you use Gherkin (like SpecFlow does) you can use @tags for this purpose. I strongly discourage you from writing these “for test” scenarios if your system is slow, because it would rob you of the “fast feedback about whether development is on the right track” benefit of specification by example.

Not All Scenarios Contribute to a Good Documentation

A couple of years ago, I attended a workshop by Gojko Adzic about Specification by Example. He gave us several examples of acceptance criteria. Our task was to determine for each scenario whether it was written by a developer, a tester or a product owner. This was an easy task: each of these three roles has their own priorities, and they showed in the acceptance criteria. Sadly, each of those scenarios was lamentably unfit as a documentation of the system.

The scenarios written by a developer had a regrettable tendency to be highly technical. They focused on the internal steps of the algorithm under development. They used language and terminology that were understandable to a developer, but that a product owner is not expected to be familiar with. One of those scenarios even had a step like “when I execute this code fragment” followed by an actual C# snippet. Those scenarios were factually correct, but they did not contribute to understanding the system.

The acceptance criteria written by testers were no better. They tended to simply write their test cases in a Given-When-Then structure, including all the permutations they wanted to test. They were effectively writing test scripts. Most of all, they wrote so many acceptance criteria that I couldn’t see the forest anymore because of all the trees. Again, those scenarios were factually correct, but they did not contribute to understanding the system.

Lastly there were the scenarios written by a product owner. They were usually few in number and on a very high level, like “then the system produces the correct output”. That begs the question “what is the correct output?”, but there was no answer in the scenarios. In other features, there were so many edge cases left open that the specification had more holes than Swiss cheese. Those scenarios also did not help in understanding the system.

Clearly, these kinds of scenarios do not help in documenting the behavior of the system. The different viewpoints of developer, tester and product owner are important, no question about that. Equally important is the balance of those viewpoints: all three need to be represented, otherwise you end up with scenarios that do nothing to improve the understanding of the system. For more inputs about writing good scenarios, I recommend following the #GivenWhenThenWithStyle Challenge series by Gojko himself.

Not Everything Should Be Documented in Scenarios

One of the best examples of a feature that is extremely tedious to document using Given-When-Then scenarios is a configuration API. It can be done that way – of course. That doesn’t mean it should be done that way.

In my experience, documenting the configuration parameters of the system using scenarios makes for very boring reading. The good old-fashioned table-style documentation with each parameter’s name, possible values and their meanings works better here. It’s probably not too exciting to read either, but it’s more compact and more readable.

A way to describe those kinds of features without using Given-When-Then scenarios while remaining in a feature file context, is to create a feature file without scenarios. Use the description field of the feature itself to document the functionality. You can use MarkDown syntax: LivingDoc will pick it up and render it as a formatted page. Pickles (the open-source Living Documentation generator that I maintained for several years) also has the ability to include MarkDown files into the living documentation.

A Word About Automation

Automating your scenarios is – in my opinion – not the main goal of Behavior Driven Development. In my experience, you derive about 80% of the benefits of BDD from the improved understanding and communication that result from talking about features using examples. All the rest is a side effect.

As side effects go, automated verification of your features is a pretty neat one, of course. It provides a valuable way of communicating about the status of development. If you decide against automating one or more features, then the status of development becomes less clear. Technically, this can be done by tagging your scenario or feature with the @ignore tag. This special tag instructs SpecFlow to leave the scenario (or feature) alone, and to not complain about missing automation. Is it with caution, however: @ignore-ing scenarios can lead to lack of trust in the progress.  In my experience, automating features is a worthwhile investment: it gives the dual benefit of feedback while developing and maintaining the software, and of clearly communicating the status of development.

Conclusion

Can Living Documentation be a documentation of your system? It’s not going to be a complete documentation of the system, but then again, no documentation is ever complete. If you augment Living Documentation with some other documentation types (like the good old table-style documentation of parameters), then it is one of the best ways I know of to document the features of a system. With this article, I gave you some guidelines on how many and what kind of scenarios to write to make sure your Living Documentation gives a good explanation of your system.