On Wednessday 6th, Bob Galen & Mary Thorn hosted a webinar entitled: ATDD And BDD The Great Beat Down…or…Debate
Slide notes and recording are available HERE
They got many good questions, and here are the answers:
Q. Why isn’t integration test and/or mocking not a part of TDD?
Mary: I am not sure we said that it was not, it can be
Q. I work on a remote team and getting together to have the conversations is difficult, how do you keep the conversation going?
Mary: Having consistent grooming sessions is one way. Also, having developer/test/po reviewing the AC together, reviewing the test cases together, testing in development environment as soon as any of the requirements is implemented to give quick feedback, reviewing the story together when the story is done are also additional ways to keep it going.
Bob: I think committing to agile collaboration in a co-located team is difficult as well. I think it starts with setting up the team and chartering it so that everyone understands the importance and value of collaborating (having ongoing conversations). Doing this makes it “easier” to be more effective in distributed teams.
Q. Comment as opposed to a question. We have a similar concept to 3 Amigos but we add the Architects and, on larger scale systems, the project managers. We refer to it as “The Smiley faces”. I use the idea to remind developers, product owners and architects not to disappear into a cosy huddle and forget the testers. Are there any others who could be in the conversation?
Mary: I have included the scrum masters and BA’s as well, as they often have valuable insite to the conversations.
Q. What is your definition of ready? Does the user story have to have 100% acceptance tests written before working on them? Is it OK for the PO to change the acceptence tests during the sprint?
Mary: No, if you have a 100% of the AC tests written before then where is the conversation in that? I think Bob’s rule is 75% ready. The 80/20 rule is that the PO should not change the tests during the sprint, but we all know discoveries happen all the time in implementation and if a test needs to be changed we should allow it. Bob I know but might disagree with me on this..
Bob: I do disagree with Mary. Ask yourself, can the story changed during the sprint? I would say, yes. If we discover a better/simpler way to implement it – then that’s the right thing to do. Point being, the conversations and iterations on the story don’t stop when it “enters” the sprint. So if you buy this, then by defnition the Acceptance Criteria can change or morph. But certainly NOT in drastic ways that change the fundamental scope of the story. More info here.
Q. What is the value of product owners writing executable requirements, when in order to make those requirements executable, they must live with a layer of obscurity around the QA team? What does the product owner really gain, when how the product is driven and how it is measured is obscured by an interpretive engine like Gherkin?
Mary: What is gained is the conversation around those tests influences building the right thing the first time. Which leads to productivity gain in the long time.
Q. So who writes the acceptance tests? The PO or the team, during the conversation?
Mary: Ideally the PO writes the acceptance criteria in gherkin, and then the team adds to them during the conversations in grooming.
Q. Mary – what ratio do you see between BDD integration tests versus UI tests? How do you decide what level you write the BDD test for?
Mary: I go by Mike Cohns automation pyramid. 50-60% of the tests should be at the unit test level, 30-40% at the integration and 10% at the UI. So how I decide typically is that they all should be integration tests, and if you need to write a UI test then let’s really make sure it needs to be.
Q. There was a point in the survey about organizational resistance – how can we overcome them? or overcome it 🙂
Mary: It really is around selling the Value of doing BDD. The only way to see the value is doing it at least once. So my recommendation is trying a small pilot scrum team, see if you see the benefits and that will help with any resistance.
Q. How hard is it to have an agile team implement BDD/ATDD when they have already been working for a period of time without BDD?
Mary: Any change is hard.Like I mentioned above my recommendation is trying a small pilot scrum team, see if you see the benefits and that will help with any resistance.
Q. In my experience, Business Owners should never be allowed to write testable specs alone… They lack the skills to write the most concise set of acceptance tests without Dev + test. Do you agree?
Mary: I have seen it both ways, but overal I think I disagree. The goal with Gherkin is an easy common language your team can understand. So writing acceptance tests in the common language is not hard, they can definitely give it a try and then the team can fill in the gaps. Remember NOT all the tests will be written in the AC of the story, the testers will add more scenarios when they test.
Bob: So, never say never to limit any role within an agile team. The view should not be on roles doing specific activities. It should be on the TEAM working together to get stories DONE. So I disagree.
Q. We work in a environment where enforcing a development method on multiple development teams is near impossible – therefore I am considering takin aspects of both approaches for different areas of testing – would you agree with this as a approach?
Mary: Agree, retrospectives are great place to introduce change like this. BDD can solve multiple problems, so putting a “TRY” of BDD is definitely a way to influence multiple scrum teams.
Bob: But don’t lose sight of the fact that “consistency” can be good in large, at-scale agile instances. Meaning, sometimes for the greater good, enforcing some “standards” is absolutely fine in agile contexts. It’s not the inmates running the asylum you know…
Q. I think you already spoke up about having a mini waterfall situations when you have testers testing behind developers, struggling with velocity agains developers and then simply there is no time to keep up with the current work and also invest into pre-work conversations and acceptance criteria specification. What would you advise to do in this situation?
Mary: The best solution I have for this is putting a WIP (work In progress) limit on the scrum or Kanban board. I typically do not allow more that 2-3 stories in progress at a time. If we all swarm to get the top 2-3 stories done and then pick up other stories then you can avoid this. It’s hard to influence b/c developers like working in silos on their own stories but ultimately you should have as many developers on a story until it does not make sense then they pick up the next.
Q. We are in the process of adopting agile practices. In your experience, what is the best way of learning cucumber and how long does it take to learn to write good tests?
Mary: What is best about cucumber is the ramp up time is fast. I would say you can learn it and have tests running in 4-6 weeks, and writing good tests within 3-6 months
Q. Can you talk about optimal Dev/QA ratio and how you see developers involved in testing?
Mary: My best ration is 1:2.5 or 1:3. I always say that the TEAM owns quality, so if all left in a sprint is testing or automation then the developers can help. Also, they can help in testing making sure that their is unit tests for every story.
Q. If you work on a multi tier project and only two of the tiers are automated, what sort of tagging would you recommend in the BDD stories to identify which acceptance tests are to be automation and which ones are manual tests?
Mary: I put an @manual tag on scenarios that are manual only, everything else assume automation
Q. Hi, not sure this is really a question. We are new to agaile and BDD. We have decided to automate as much testing as is possible. My concern is how much is enough and how much is too much BDD and what/where else do we apply automation to provide better covereage. This conversation seems to tackle this perfectly. Seems though there is no real answer? The more I cover with BDD the more I need to ensure this is driven through conversation and not individual thinking?
Mary: Not sure I understand this question
Bob: I agree. Inevitably it is up to each TEAM to decide on the mix of testing, coverage, automation, functional vs. non-functional testing, etc. for each story, sprint, and release they’re working on. I normally focus on giving them the SPACE to do what they think is “right”.
Mary: have you evaluated Fitnesse as well? Is it possible to write Cucumber tests outside the developer environment? I would like “all the amigos” to be able to access the tests, run them and edit them outside the meetings – is it possible with Cucumber?
Mary: I used Fitnesse in a previous company and then replaced it with Specflow(.Net version of Cucumber) The main reason is lack of good debugging, CI integration, and maintanability. You can write Cucumber tests in a Word doc and associate them to your user story. Tool like Jira and Rally have Cucumber plugins that let you do this with in your agile lifecycle management tool as well.
Q. I’ve seen organisations that assign the BA as a ‘proxy product owner’, is there a conflict there when it comes to writing stories if a BA has both roles?
Mary: None, this works that same.
Bob: I see no conflict as long as the PO + BA engages the team in writing / evolving the stories WITH them.
Q. Is there any benefit to user stories other than discovering requirements in a way that’s accessible to the entire team?
Bob: They are lean, if done well, they reinforce just-enough and just-in-time conversation and thinking. They are minimalist – meaning they potentially save you time over writing big-honking documents. They are unique per story – meaning, the team decides when a story is defined suffeciently (ready) to enter a sprint.
Q. How do you involve developers in the (functional) conversation?
Mary: Having consistent grooming sessions is one way. Also, having developer/test/po reviewing the AC together, reviewing the test cases together, testing in development environment as soon as any of the requirements is implemented to give quick feedback, reviewing the story together when the story is done are also additional ways to keep it going.
Q. Can the BDD be used only in test team or does it have to be a part of the user story creations and development process?
Mary: Yes ultimately I want it in the user story creation and development process.
Q. Assuming the product owner writes the specs for the team, the automation engineer writes the code for the tests. How effective the specs should be so that the tester need not change it while writing tests for feeding in data to the tests.
Mary: The PO’s acceptance tests are roughly only 50-60% of the tests that will be written, the testers will add all their additional test and data combinations in the testing phase.
Q. I don’t see any justification for taking on the overhead of a tool like Gherkin, relative to having effective requriements discovery and communication!
Mary: If you never have tried it how do you know, there are ton of benefits around increase productivity that you are missing without even giving it a chance.
Bob: One of the key aspects of an agile mindset is a focus on Continuous Improvement. An important element of that is the notion of experimentation – at an individual level and at a team level. To be inquisitive and curious. I’d encourage you to TRY an experiement with Gherkin – perhaps on a single story and see what you (and your team) think about it. How hard would that be?
Q. Any books or other resources you would encourage us to look at for BDD beside Ken Pugh’s book?
Mary: There is the Cucumber book by Matt Wynne and Aslak Hellesoy, blogs by Dan North and Liz Keogh.
Bob: I’d add Gojko Adzik to the mix as a great author.
