COVID -19 has made us all look at the world in a different way. For many of us, this has been very challenging and we have been forced to change.
Agile testing was one of the many areas of the IT industry that was significantly affected by the pandemic. I recently reflected on this in a webinar, where I observed what was happening “yesterday”, explored what we can do “today”, and speculated on what the new “tomorrow” could look like.
I shared some experiments that you can try to accelerate your testing whilst still maintaining high quality and customer focus. Now is a great time to put into action your continuous learning plans and share your wisdom with our profession. We are all in this together.
This article will focus on five key points from the presentation, as well as answers to some of the questions asked by participants. If you are keen for additional insights, then I recommend watching the entire webinar.
Accelerating quality
It’s important to accelerate quality, but not at the expense of product quality. We have all seen some very public examples in the recent weeks where organisations have not invested appropriately in their cost of quality models, which has caused disastrous brand damage.
All of us in IT need to understand the “4 Rights” – the right product, right quality, right time, and right cost. After all, customers want value and will “vote with their feet” by going to a competitor if they do not get what they need.
The best way to accelerate quality is to build it in from the start. Here are five ways to do it:
Are your requirements testable?
If they contain words such as easily or quickly, then it is likely that they are not.
Static reviews are a great way to work with your Product Owners to support them in articulating their requirements in a way that the whole team understands what is needed. It also helps to remove ambiguities, which will cause rework in the form of test maintenance (waste) and may mean you build the wrong product.
A requirement should also be just that and not a solution. Too often requirements contain functional specifications which constrain the team and do not allow them to do what they do best, which is to design creative and innovative solutions.
Work closer with developers
Using a whole team responsibility for quality is critical. Testers are no longer the “quality police” but should adopt ways to become quality coaches within the team.
Duplicate testing is wasted effort, so testers pair with developers to understand what unit testing they are doing. They work with them so that they understand what levels of quality is expected from them.
Tests and their results should be visible to the whole team. Code should not be released without the associated unit tests, otherwise it is not complete.
Adopt test first approaches
Test-driven design and behaviour-driven design are two examples, that allow teams to discuss, at the design stage, how they are going to build quality in. If you cannot test it, then how to you provide confidence that you have built the right product to the right level of quality?
Replace some/all traditional test cases
Traditional test cases are time consuming to write and often require additional time to maintain.
By replacing these with session-based exploratory tests, you get the same levels of traceability, auditability, and coverage, but at a fraction of the cost in test preparation. They also allow flexibility in exploring the product whilst still recording what you have tested.
Automate tests
Automate where there is a return on investment. Not all tests are candidates for automation, however the team should be practising automation and have these skills within the team.
Up front discussions should happen on what can be automated, as early as the iteration planning stage. Why write a manual test when it can be automated?
Automation needs to happen within the current iteration and not be lagging in subsequent ones. Focus on the bottom of the automation pyramid on automated unit tests; API tests and services tests, and not on the UI tests where we will know they are harder and more costly to create and maintain.
The Agile factor
A couple of questions were raised about test cases, which I will address here:
Are test cases in Agile different to traditional methods?
From what I am seeing with our clients, they are not. That is why I am saying move some of testing away from test cases to using session-based testing.
A common theme I see when conducting Agile health checks across organisations is that testing is being blamed for user stories not completing within an iteration. Or even more worrying, that testing is being conducted in n+1 iteration, which is, in effect, Waterfall testing.
One of the reasons for this, although by no means the only reason (a discussion for another article), is that test preparation is taking too long. I.e. if code is delivered day 2 of the iteration, which is when it should start, testers do not have test cases written to execute this code. If the testers are not ready to execute as code is continuously delivered to them, then test debt is being accumulated.
Writing a test session takes significantly less time, in the order of 5 minutes vs 30 minutes or more for a traditional test case to be written. That is a time saving of 80%, which also translates to a significant cost saving.
How to measure the quality of test cases in Agile methods?
I work with my teams to move away from traditional test cases. Therefore, quality for me is do tests that:
- Reduce product risk (of course, you need to have identified the risks and prioritised them first)
- Identify residual risk
- Provide confidence in the system, rather than finding defects (quality should have been built in, not be tested out. See above.)
- And a lagging metric – reduce defects leaking to production
Other ways to accelerate quality are discussed in the next sections.
Agile test planning
Test planning, and for that matter, test strategy, need to be more lightweight within Agile product delivery. Often an organisation test strategy will be produced, with the teams then just needing to state any variance.
Process and practice detail need to be moved out of strategy and plan documentation, and put into a Wiki, where it can be updated based on feedback from retrospectives and chapter/guild/community of practice forums. Both the strategy and plans should be living documents, updated as the teams learn more.
One of the techniques that I use with my teams is brainstorming sessions within the iteration planning meetings, each iteration. This means that the whole team discusses building the quality in, and how they all (not just the testers) can help to meet the iteration goals to complete testing.
I use a mind map to capture this as our discussions happen. That means, by the end of the meeting, we have an agreed plan that the whole team has committed to, and any issues/risks have been identified. No additional work is then required to write this up – it is simply saved into our tool/Wiki, either the map itself (if using a tool), or a photo of the whiteboard.
Make sure that you speak with your compliance/audit team to ensure that any changes that you are making to your process, practices, and templates still fits with what they require. I have found that every time I have approached them to discuss, they have been more than happy to endorse these changes and, in fact, welcome them. Often, traditional test assets contain more than they need to see, so leaner assets help them.
This then means we have removed the obstacles for testing, and ensured that, as a team, all of the quality activities have capacity and capability within the team to be completed within the iteration (this was another question from the audience).
Session-based testing
I am particularly interested in how many teams are practising session-based testing as a way to optimise test preparation, whilst ensuring the same levels of coverage. Here is a poll I recently did to the question, “are you currently using session-based testing?
The results are very interesting, and very much in line with what I am seeing with clients I work with:
- For all tests – 9.5%
- For high risk/business critical tests only – 0%
- For low/medium risk business tests only – 19%
- Continuing with traditional test cases – 42.9%
- Some teams are using it, but not all – 28.6%
There is currently a split between teams using session-based testing, and those continuing with traditional test cases, which was shown in the poll and based on my experiences reflective of Agile teams. If your testing is blocking user story completion within the iteration, then I would encourage you to experiment with session-based testing.
In essence, it is a way to fully document your testing, whilst exploring and giving flexibility to add information as you learn. The session has a charter or test objective, and then a set of test ideas related to the charter.
A session should start with a plan of what you are going to test – session-based testing is definitely not ad hoc testing. As you continue throughout the session, you add more details and update your results.
This also cuts down on test case maintenance. Rather than the need to writing step by step actions and expected results, the tester will have a greater understanding of what has been delivered, as they have been included in requirements and design discussions. This means they only need document their test ideas, what they are planning to test.
I would recommend a simple session sheet be adopted to start, but then add to this as you discover what additional detail would help you. Remember to continuously reflect to ensure you are only recording information that helps deliver the best value to your customer, and no more. This may include pre/post conditions and some simple quality metrics.
I work with teams to store the session sheets within whatever tool the team choses as a configured session sheet. This allows the team to gain metrics and visibility across their session sheets through dashboards.
If you do not have access to such a configurable tool, you can use a template within Word and attach it to a task. Understand the constraints of this however as it will just not give you so much flexibility to view the data.
As with any test, these should be based on product risk, which could be one of the metrics you capture. I.e. coverage and/or residual risk.
Test techniques should be used to build the test ideas as would be expected by any good test preparation process. The test techniques are the only way to identify test coverage. Using a risk based approach, you then decide given your context how many to execute to give the required coverage. You should then use the gathered metrics to adjust your planning as you execute and learn more and as feedback for your session debrief.
Agile variances
The power of business agility is that you can adapt it and make it your own. The challenge is that same flexibility means you can get it very wrong as well.
Its simplicity, over four manifesto statements and twelve principles, is also its complexity. If your team stays true to these values, the implementation of your process, practice, and frameworks can fit your organisation, environment, and even provide autonomy for your teams within certain scalable company constraints. There are several of the methodologies that have similar practices.
The key is to focus on delivering value to your customers – the right product, right quality, right time, and right cost. If you chose to use a particular framework, such as Scrum, then I recommend that you should implement it, in the first instance, as stated.
Do several Sprints before you decide to make changes, so the processes/practices become a habit. Of course, how you conduct the testing, for example, should be up to the team.
I have provided some for both in this article and webinar. I would advise that you pick one item at a time to experiment with, and do that long enough to understand whether it works for your team or not.
There are some testing fundamentals that apply, regardless of your methodology:
- Build quality in, not test it out.
- Use testing techniques to identify coverage.
- Focus on testing product risk to reduce residual risk.
- Lightweight test planning that updates as you learn more.
- The whole team is responsible for quality, not just the testers.
- Start test execution on day 2. This should be discussed with the developers to ensure they are releasing at least daily.
- Automate as much as possible, where there is return on investment, and as early as you can (test first approaches), particularly unit testing, APIs, and services.
Different frameworks have differing benefits, so it is important that the team talks about which will be best suited to the way they want to work with each other, and within the wider organisational teams.
Agile continuum
This is an approach that I am using and adapting, which I presented at the webinar for the first time.
In this concept, there are three key areas of work in order to deliver a product: discover something, flexibly build it, and then study what we built. There are certain mindsets and principles that support these which are in the “cloud shape” in the middle to denote that they underpin everything.
Then there are activities to complete. I have put these within a circle to try to convey that they are not sequential, although some are more likely to be completed for certain areas.
For example, shared coding style would, more obviously, be carried out when flexibly building your product. However, when you study what we built, there may be a need to change or update your shared code style.
This is not supposed to be an exhaustive list, and, of course, given your context, there could be others that you would want to include, or some that you would exclude. I can see some I want to adjust based on using it several times now – watch out for version 2 to come out soon.
I’ve already had some feedback to include exploratory testing. I always do it from habit, so it should definitely be included.
It is always important to iterate back over processes, practices, and concepts like this to understand if they are still working for you, if they are then keep doing them or if there are improvements that you can make in order to optimise, then add them to your backlog to give visibility and ownership to make sure they get implemented.
Retrospectives, even if you are not doing Scrum, are mandatory for me as a good way for the team to continuously inspect their processes, celebrate success, and change those that are not working to make them better.
Turning adversity into positivity
Looking at areas of waste, following the Lean principles, are also a good practice for the teams to do. If you have regular handoffs, or are waiting for code to be delivered, these are some examples of waste. Try doing some value stream mapping to help identify these, and then prioritise the highest waste areas to improve first.
Alternatively, you could ask an independent expert to come in and perform an Agile health check on your team or across the organisation. We do this for our clients, and our health check is aligned to the Agile manifesto and 12 principles, back to the core fundamentals I mentioned earlier.
Our tomorrow needs to include “Better Ways of Working (BWOW)”. That takes our learnings from yesterday, our experiments and overnight changes forced on us for today and blending them to create a better way of working for tomorrow.
