Outside-in development
It seems to me that outside-in development, which I will refer to as OID from now on, is what confuses testers most. Essentially, OID works by using automation tools combined with scenarios from a collaborative session to create a guide for developers. This results in ensuring developers develop what the business wants. The process is formed of three states, which you can see in the model, but let’s look at each state in more detail:
- Red state
A developer starts on the ‘outer wheel’ by using a Gherkin-based tool to tie explicit user actions to the each step of a scenario that creates a failing automated check (because the feature doesn’t exist). For example, a developer may use Cucumber to trigger a series of Selenium WebDriver actions on a browser to simulate how a user would execute a scenario.
- Green state
The developer will then move into the ‘inner wheel’ and run a similar red, green and amber process on a lower level. The developer will use this pattern against individual methods, applying a different unit-level automation tool multiple times to get all the production code working. This, in turn, provides code that means the ‘outer wheel’ automated scenario can be executed without issue.
This process ensures that the developer has delivered what is expected from the business, as well as informing the developer when they are done.
- Amber state
The developer is now able to refactor their code and be informed if their changes are no longer delivering what the business expects.
Business guided development
“If I could change one word in BDD, it would be driven to guidance.” (Dan North)
I’ve been very careful not to mention testers, tests or pass/fail in the above description. That’s because OID is not about testing, it’s about guidance, and this is what testers get wrong. The assumption from testers is that because OID uses tools that are typically related to automated testing, that must mean OID is automated testing.
OID is an evolution of acceptance test driven development and test driven development, which means it has inherited some misconceptions from ATDD/TDD. Testers struggle with ATDD and TDD, believing it’s a testing approach and not a design process. This isn’t helped by the word ‘test’ existing in both names, which adds to the confusion.
However, testers should be aware that test automation tools are not exclusively used for test automation, they can be used for other things; much like you can use a screwdriver in its traditional way but also use it to open a can of paint. While these approaches use test automation tools, they are using them as a means to keep the developer honest. OID helps developers design good code and deliver what the business really wants; not deliver testing.
When I was struggling with BDD as a concept, more specifically it was OID that I struggled with the most. The assumptions and errors I’ve claimed testers make are all ones that I’ve made personally, thinking that OID, ATDD and TDD were testing approaches when they are not. However, as I created my model I began to realise that OID is a development practice, not testing. That said, testers can support developers by pairing with them during the OID process to ask questions, give information about the application and observe what is being created for future testing.
So what does that mean for test automation? Surely, we need good test automation in place to help testers deal with the mountains of work we face regularly? Yes, we do need test automation, but it should sit alongside the work done by OID and not be replaced by it. They each have a very different focus; OID focuses on guiding development whereas test automation focuses on supporting testing in terms of rapid feedback and executing actions. Attempting to cover all those actions with OID results in a process that can neither deliver guidance (due to the noise generated), nor can it provide robust test automation (because the tools being used aren’t the right ones for the job).
Well come on then, is ‘BDD’ testing?
The easiest option would be to simply say no, but it’s more nuanced than that. No, BDD is not solely a testing activity, it’s a process that is owned and carried out by the whole. However, there are aspects of BDD than can help facilitate testing activities, especially during the collaborative phase. The trick is to know which aspects of BDD are beneficial to testing, and which are not.
Leveraging BDD in a test strategy
For me, a good test strategy is a rich collage of different processes, techniques and tools. The various testing activities support one another whilst also having their own unique place with testing. To execute a lot of these activities. testers need the opportunity to work and share with teams. BDD offers this opportunity meaning a tester working in a BDD context will be able to ask questions, collect information and help identify risks to inform other testing activities. For example:
- A tester engaged in the collaborative phase of BDD can talk to and learn from the business and development sides of the team.
- Examples can be used as a heuristic to generate test ideas, much like a diagram or design document can be used to generate ideas.
- Testers can pair with developers during outside-in development to learn how a feature is being implemented technically, both to provide feedback to the developer and to inform testing at a later date.
Working in a BDD context can be invaluable to a tester but we have to be careful. A good strategy is balanced which means exploiting BDD but not relying on it solely. Testers have other testing activities to execute, in the same way that developers and business owners will carry out other activities outside the BDD process.
Using BDD as a test strategy
“It’s easy to lose sight of the fact that this is documentation. The whole point is executable specification, it’s documentation. And in the same way, because people think ‘oh no this is tests, more tests must be better, because tests are good’, more documentation isn’t better, the right documentation is better, the right level of documentation and the right focus is better.” (Dan North, Interview with Liz Keogh & Dan North at GOTO Chicago 2013)
Unfortunately, it’s currently popular for test teams to use BDD as the basis of their test strategy. There are teams ignoring the collaborative side of BDD, focusing too much on using a Gherkin syntax as means to build test cases and misunderstanding the purpose of outside-in development to focus on automating test coverage. It’s understandable why these mistakes are made. For a long time, testing has been too reliant on test case management-based strategies which has led teams to make incorrect connections between Gherkin examples and test cases. Both are step-based, concerned with asserting behaviour, and are written in plain language.
I’ve talked at length at conferences about how attempting to use Gherkin-based scenarios over test cases is a false economy. This results in teams following the same test case management strategy but with a different syntax. It’s important to remember that Gherkin is for development guidance, not for test coverage. Accepting that distinction means teams can begin to discover better processes, techniques and tools to improve testing.
Ultimately, for me, testing is much than asserting an application meets expectations that were written down so using BDD as the sole basis for a test strategy will never deliver a robust test strategy a team will need. BDD was never designed to be a solely testing process.
Beyond BDD
I encourage you to watch Dan North’s talk “BDD is not about testing” where he challenges testers to start focusing on approaches that solve problems that BDD was never designed to solve, such as test coverage or automation.
There is a lot of great work around exploratory testing which teams are already using successfully in conjunction with BDD, but what about test automation? As far as I am concerned, this is a question that hasn’t yet been answered. Work is required to understand and explain how quality automation in testing fits with BDD. We need to look at the current state of popular test automation, unpick it from BDD and ask, “What next?” because if BDD isn’t the answer to automation in testing, what takes its place?
About The Author
Mark is a tester, teacher, mentor, coach and international speaker, presenting workshops and talks on technical testing techniques. He has worked on award-winning projects across a wide variety of technology sectors ranging from broadcast, digital, financial and public sector working with various web, mobile and desktop technologies.
Mark is part of the Hindsight Software team, sharing his expertise in technical testing and test automation and advocating for risk-based automation and automation in testing. He regularly blogs at mwtestconsultancy.co.uk and is also the co-founder of the Software Testing Clinic in London, a regular workshop for new and junior testers to receive free mentoring and lessons in software testing. Mark has a keen interest in various technologies, regularly developing new apps and devices for the Internet of things. You can get in touch with Mark on twitter: @2bittester
