There is an interesting tension between agile and documentation. I often hear people complain about the lack of documentation in agile projects. “Organizations use scrum as an excuse for not having to work on documentation”, was a familiar lament. Now that agile methods have become more common, this excuse is less heard. I see more and more organizations that adapt scrum. Not to get rid of the traditional documentation requirements, but because they want working software with a short time to market.
Yet documentation remains an issue. The agile manifesto says that working software is preferred above documentation, but adds to it that documentation still can have its value. How do teams balance this in practice? I have worked with several teams that each had their own documentation issues.
Team 1 is an experienced scrum team. I ask one of its testers if he ever has a problem with the lack of documentation, and he seems surprised by my question. “We are one team”, he says,” If I have a question, I ask my team members. The business analyst sits to the left of me, and the developer to my right. Within our team we have a continue exchange of information.” The code that the team delivers hardly contains bugs, but it remains unclear who defines the behaviour of the system. I guess the team does, but who checks whether the business requirements are fully met.
Team 2 has a different dynamic. Cooperation is less than within the first team and team members stick to their traditional roles. I sit down with the tester and the analyst. The analyst shows me some samples of the documentation he made. I think it looks great. As an experiment I translate his design into some tests, and find I have an issue. I try to explain to him, that in order to test the function, I need some additional details that are not in the design. I give him some examples of conditions that seem to matter, but are not found in his use case. The analyst admits, “That kind of details are never documented.” The tester adds to the discussion, that she really needs those details. “I spent a lot of my time in figuring out how the system is supposed to work,” she explains. “When I get hold of those details, I’ll document them in our test design. This way I know when my test is successful. Even if I have to do a regression test, later on”.
My meeting with team 3 starts with a lot of agreement. The Business Analysts agrees completely: Testers need detailed information. “But”, they add, “We are only responsible for the functional system descriptions. The developers decide on the technical implementation. They should add their choices and related details to the design”. As it turns out, the analysts occasionally do add technical details to their use cases. However, this data is incomplete and the testers are often frustrated by its inconsistency. During the session it becomes clear, that the business analysts try to help the testers and actually deliver more than they need to. So, that is one problem out of the way. But is it? How are the developer’s choices documented? There is no process for this, and a quick survey learns that, again, most details are captured in the regression test set.
From the above we can learn that all testers needed detailed information in order to do their job properly. Depending on the situation, they obtain these details by cooperating with other disciplines in the team or by doing some extensive shopping. It seems logical to include the obtained information in the tests, since this ensures that these can be reused for regression testing. This results in test scripts that are far more detailed than the original specifications. “Look in the tests, then you know what the system is supposed to do”, is a common heard advice.
I do have some concerns with situation. The details in the test set cannot be verified since they are nowhere in the design, and this will lead to maintenance issues. Besides, testers are not the only one that need to understand how the system works. Developers and Analysts and Administrators do too. Therefore I think it is better to have detailed specifications rather than detailed tests. Thus scrum-teams, please take the time to secure your knowledge and do it in the specifications, not in the tests.
Derk-Jan de Grood
Derk-Jan de Grood works for Valori as senior test manager and product manager. His drive is to improve the visibility of testing in both agile and traditional organizations, by sharing his knowledge and experience by means of training, presentations, workshops and publications. He is a regular speaker at conferences like EuroSTAR, wrote several successful books on software testing and publishes articles for the major testing magazines.
