A New Model for Testing – Discussion

[Paul]

Hi, if you attended the webinar today (or even if you didn’t) please post your questions here and I’ll respond as soon as I can.

Paul.

The paper can be downloaded here: http://dev.sp.qa/download/newModel

The webinar recording & slides can be viewed here

[Maud]

Hi Paul,
“Agile, but …” directly corresponds to http://lafable.com/ – Take a look and enjoy (or cry because of it being so true?)
Maud

[Pauline]

Interesting so far. I already implement this “Shift – Left” in my company (this is the first time I’ve heard it called this, I base it on decades in a professional tester role). Our testers are involved from the start.

As for the “Job spec” at the start. That’s not a spec I recognise as a spec for a tester. The confusion between Developer in Test and a professional tester needs addressing in not only the industry, but with the recruitment consultancies too.

[Frank Pedersen]

Regarding Shift – Left, it is also something we are doing.
We started with embedded testers in our scrum teams years ago and in our effort to keep improving quality we have drifted more and more towards involving everyone in the test, not just the testers. And involve them as early as possible. Having testers on the team has involved the developers in the test effort.
Just recently we did away with our test department and made the testers part of the development department. All the people directly involved with testing are now from the development department. Now we have a QA department focusing on all the tools we have to improve quality, and not be as focused on test as previously.

I am not sure we could have done that before without having everyone live the experience themselves. Instead of it feeling like a big change (which it actually is) it just feels like normal evolution. Had anyone suggested that just 2 years ago, I am sure people would think of that person as a bit radical and weird.

[Pauline]

Paul,

Thank you for your time on this, it was interesting, but not “a New Model for Testing” – not for me anyway. It is nice to see that something I have been doing and implementing for a few years now, is being recognised by the testing community.

I will still disagree that Testers and Developers think alike. Devs focus on the positive path, where as testers are the “what happens if…” thinkers.

Automation is more akin to developers thinking. It is useful as part of the testing process, but should not be relied upon.

For the record – I don’t believe that testers need to code. I prefer testers who can’t/don’t code because they have a better understanding of early involvement testing.

[Maud]

“Testing is about measuring achievement, not quality.” – And when quality is an important goal that you try to achieve? Well, nitpicking of course. But nevertheless, don’t you define quality measures you are trying to achieve with the project? Not achieving through testing of course, that is only measuring the achievement…

[Maud]

Thanks for the very interesting webinar, Paul. Hopefully we are able to discuss this further …
Bye,
Maud

[Paul]

Lafable indeed! Unfortunately there are a lot of people finding Agile hard to implement in their world.

[Patrick Prill]

Paul,
that was an awesome webinar today. Thanks for showing me the missing links to so many single ideas that floated around my mind. I really love that model, and I can’t wait to hear more about it. I will definitley use several of those ideas to further improve our situation at work.

Cheers
Patrick

[Paul]

As everyone knows, ‘Nothing is new’!’

I did the same at a ‘dotcom’ some years ago and it worked nicely. Elsewhere – it hasn’t worked at all!

[Anne Mette]

Hi Paul,

Thanks for this interesting webinar. As I have mentioned before, the model precisely reflects how I work when I test. To me it is hence not new, but a synthesis of different models – very abstract. But (sorry that there is a “but” what is the exact purpose of the model?

Anne Mette

[Sarah]

Hi Paul,

A very interesting webina and I can relate to this very readily.

We are currently in early phases of a new release of software and I was in fact this morning scratching my head as to why I was considering using an altogether different approach to the test planning and test case development. The penny has dropped since your webinar – my points of reference (sources) are different and I’ve naturally adapted my approach. This time around I have a far better understanding of how the system will be used, due to a large customer adopting our solution and giving a lot of ‘feedback’.. The test team have customer interaction as well as testing responsibilities so we are far more in touch with the real world usage of the solution. We have improved development processes and the test team also work with developers when buddy checking code before check-in. This gives us a greater knowledge of how the software works from the inside, which has most definitely contributed to my shift in approach to testing.

It is also worth mentioning that as testers, my team are definitely challenging the specification more – in effect ‘testing’ it. I am therefore very hopeful that, when we get our hands on the release, we will have caught more issues early, making testing an altogether more straightforward phase with less re-work.

Thanks Paul!

Best wishes
Sarah Bilsborough (Saltzman)

[Paul]

@Pauline

Interesting so far. I already implement this “Shift – Left” in my company (this is the first time I’ve heard it called this, I base it on decades in a professional tester role). Our testers are involved from the start.

I started saying ‘redistributed testing’ 2-3 years ago but shift-left is what other peopleI have met use. I’m not sure it’s very well defines, and it may be a buzzword/phrase rather than a well-defined approach.

As for the “Job spec” at the start. That’s not a spec I recognise as a spec for a tester. The confusion between Developer in Test and a professional tester needs addressing in not only the industry, but with the recruitment consultancies too.

Not a tester? Perhaps. I mentioned it because the number of jobs like that (and SDET/SDiT etc) seem to be increasing and the number of jobs we would recognise as ‘a tester’ seem to be decreasing..

[Paul]

@Frank

Regarding Shift – Left, it is also something we are doing…

I think your example is increasingly common.

Just recently we did away with our test department and made the testers part of the development department. All the people directly involved with testing are now from the development department. Now we have a QA department focusing on all the tools we have to improve quality, and not be as focused on test as previously.

So your ‘testers’ don’t test? I would hope they have some form of test oversight/consulting role in your teams? A pattern I’ve seen quite a lot is testers shifting from being responsible for testing to being accountable. What I mean is they advise, consult, review, critique and occasionally help with testing, rather than be ‘the tester’

I am not sure we could have done that before without having everyone live the experience themselves. Instead of it feeling like a big change (which it actually is) it just feels like normal evolution. Had anyone suggested that just 2 years ago, I am sure people would think of that person as a bit radical and weird.

That makes a lot of sense. It is a big change It sounds like everyone was engaged and had a contribution to make. This is a universal change management principle

[Paul]

@Pauline

Thank you for your time on this, it was interesting, but not “a New Model for Testing” – not for me anyway. It is nice to see that something I have been doing and implementing for a few years now, is being recognised by the testing community.

If you say the model is not new – that’s the best compliment you could give me. If it maps closely to your experience, it isn’t too far off as a description of how you (and I) work.. Other people have said ‘not new’ – which I’m taking to mean it’s a helpful description.

Recognised by the testing community? I suspect there are people who disagree with this representation. We’ll see :O)

I will still disagree that Testers and Developers think alike. Devs focus on the positive path, where as testers are the “what happens if…” thinkers.

I won’t argue. But… Speaking as a developer, I cannot separate the exploration side of the job from the testing side of the job. Now, I might be a rubbish developer – it’s hard to say :O) But I would argue that all professional developers must think along two threads almost all the time.

1. How can I implement this?
2. How will I know it works?

Sure – number 2 could be limited to one or two positive tests, and a ‘tester’ might think of 100 negative tests. But the exploration and modelling, aspect I think is almost congruent. The outcome of developer v tester exploration/modelling are likely to be different models. I certainly agree.

Automation is more akin to developers thinking. It is useful as part of the testing process, but should not be relied upon.

Again, speaking as a developer, my first reaction to seeing a testing task is, “could I write a bit of code to do that?”. Now, the ‘do that’ part would inevitably be some form of check. Checks can be simple or complex and automated checks do not equate to what humans can do. Complex tests or tests that require human senses or insights can’t be coded. As you know, the general strategy for using test automation is to focus on low-level activities, probably API-based tests that in fact cannot be run by a human at all. This is where developers minds should be at.

Automation of (let’s say) UI end to end tests is difficult to justify. But still, many testers try to do it (encouraged by managers who don’t understand the ‘problem’ they want to solve). My argument is that automating existing functional tests is a) hard and b) might not be of much value. The goal of automation isn’t usually to demonstrate the system meets a requirement. Rather, we use automation to demonstrate ‘functional equivalence’. That is “having undergone some change, does the software do now, what it did before the change?” – this is a subtly different goal from the ‘does it work’ goal of functional tests.

So. We should use a different objective, and model for regression tests, especially if they are to be automated. We might then be more successful.

For the record – I don’t believe that testers need to code. I prefer testers who can’t/don’t code because they have a better understanding of early involvement testing.

I have posted a blog on this subject – which caused quite a debate here: http://www.ministryoftesting.com/2014/02/testers-coding-debate-can-move-now/

I would never say testers *must* learn to code. but to shift-left and/or embed testers with developers, it’s likely the tester will need to talk developer language – code. Code comprehension may be all that is required. I cannot see how learning a new/coding skill subtracts from other skills a tester has (or may need also to acquire).

Does learning to code push out other skills/knowledge from testers brains? That’s what I call the Homer Simpson argument – see the link above :O)

[Stuart]

The webinar recording & slides can be viewed here

“You do not have permission to preview drafts.”

Seems I don’t have the access rights to it, was the link tested? 🙂

This is a better link https://huddle.eurostarsoftwaretesting.com/resource/a-new-model-for-testing/

[Seija]

Hi Paul!

Thanks for the great webinar!

There are many points that I readily agree and have lived with and worked with without knowing those exist in explicit terms.

I am only worried of one thing, related to the similarities in the thinking of developers and testers. Maybe there are similarities but as tester there is typically one big difference I have faced when working with the developers. Namely, I want always to break the code they made. They seldom agree that they are thinking in the same lines. I on the other hand very seldom think how would I like to see it working ok. My job is not needed, when that would take the place, but not before that.

I do not believe, that the models would be the same, but very different, or at least, I’m using the same model in a different way and for different reasons. Maybe the model in this case is not so relevant, or the most relevant, good that the importance of those was recognized, but maybe the most important thing is what we do with the models, or what kind of new models we create out of those?

Christin Wiedemann has written very interestingly of the notion of software testing and of “scientific testing” and the nature of it in terms of falsification. We cannot ever show by testing that something works she’s saying in one of her blog posts, but instead we can only try to show that it does not work.
http://christintesting.wordpress.com/

And that is the point we start the actual dialogue with the developer because what I found s/he is about to fix. During this discussion I many times feel that we are in different worlds, not often has it occurred to me that we would be actually talking about the same model. I think I would need some more clarification of this issue, of course it can be my limitation not fully to grasp it. Concrete examples could be helpful too.

Thanks, Seija

[Paul]

@Patrick

Thanks for the kind words – I’m glad you like it :O)

For years, I’ve wanted to put some kind of picture together to link together the various bits and pieces of thought processes that I think were going on in my head as I worked. But I didn’t have names for the various thoughts, and didn’t know how they could be related. I had, for some years, thought that testers ‘explored to build models then used the models to build tests’ but that’s as far as it went.

Then one morning I just made a list of the these things and tried to draw a picture. It was very crude – and wrong of course. It came together much more when sketched out the picture in a group discussion at a Test Management Forum session in April. The picture was crude and wrong, but the group were very supportive and we refined it on the whiteboard. The most valuable thing was that people we so supportive as they thought ‘there was something in this’.

So I gave all the thinking processes consistent names as verbs and each verb maps to (what is usually) a physical activity (asking questions, sketching models, running tests…) The diagram doesnt’ represent a process or sequence. I think of them more as ‘modes of thought’. Perhaps there’s a different area of the brain for each one :O)

If you do take the model and try and use it practically – I would love ot hear what you discover!

[Paul]

@Ann Mette

Thanks for the kind words. ‘This is not new’ – of course the way we think isn’t new (to us as individuals). It is so familiar to us that we hardly think about it of course. The model may be a synthesis of other models – but I haven’t seen any that cover the end to end process – so it is new in that respect – I believe.

What is useful for? There are no buts – if it isn’t useful, it’s a rather pointless exercise.

(I’m reminded of the story of the laser. Einstein predicted it in 1917. When the first one was made people said it was a ‘solution looking for a problem’. I’m not comparing the model to the laser but the sense that the Test Axioms and the model might help with some as yet undefined problem is a view I have heard many times. But I think there are real opportunities).

• By identifying the thought processes, the model suggests what skills are needed in order to test. I haven’t provided a lot of explanation of each process (but there is some background in the paper). But the highly speculative list of skills suggests that we are teaching testers ‘the wrong things’. The overlap with the certification scheme syllabuses is almost zero. What does that say? It might justify the assertion that the certification schemes are not improving people’s testing capability.
• The separation of exploration and testing makes it clear where the creative activities lie and the mostly procedural activities lie. It also makes it plain that testing cannot succeed without the enquiring, modelling, predicting, challenging skills. It’s the exploration side of testing that is most neglect.
• I suggest that the exploration thought processes of testers may be the same as developers. If this is the case, there are definite bridges between testers and developers that we can build. In terms of embedding testers, pairing or just working together there is more in common than most people think.
• If developers explore like testers, then perhaps we have a possible route to helping developers to understand testing better and to test more effectively.
• The model suggests that whereas BDD fits as a test process, TDD does not. This might help us to understand the different value of BDD and TDD (and other emerging approaches).
• If we look at test automation using the model, and regard automated tests as following the same thought process and we recognise the goal of automation is different, we would build different models and end up with different tests. Perhaps this will help us to be more successful with test execution tools?
• The picture might be something you put on the wall and throw darts at :O)

Some of the ideas are more speculative than others – but I think all are worthy of some research. I have to say, the notion of stripping away what I have loosely called ‘logistics’ was a real eye-opener to me. It makes clear how off target the certifications schemes are (or how peripheral their target is).

In the paper (http://dev.sp.qa/download/newModel) I explore some of these issues a little more – but it is very early days.

[Paul]

@Sarah

Been a long time! I hope you are well. Thanks for saying ‘it relates’. That’s a very positive thing for me.

Your story fits with my other experiences of companies taking a fresh look at how they build systems and test.

• It still surprises me how many testers seek and expect all their information to come from one source (often a document). For as long as I’ve been working in testing, we have always had to scavenge for information, sift through the junk, discard the wrong and irrelevant and above all challenge the sources. The model makes those activities a little more explicit – but there’s a lot more work to be done.
• Not every tester needs (or is able) to work at a ‘code level’. But code comprehension – the most basic level of programming skill – has value for reviews but also gives insights into how to ‘test the inside from the outside’. Such skills only add, they never subtract.
• My very first EuroSTAR talk in 1993 was titled ‘A Unified Approach to System Functional Testing’. My second talk in 1994 was ‘Testing Requirements’. challenging requirements has always been part of the tester’s remit. In 20 years, the argument for doing it hasn’t changed. but only now are people beginning to take it on seriously.

You might find the DeFOSPAM approach helpful, particularly if your developers are working with user stories. It’s described in the Business Story Pocketbook that you can download (with the New Model paper) here http://dev.sp.qa/download

I’d be interested in how you get on :O)

[Paul]

@Seija

Thanks for the positive words :O)

I am only worried of one thing, related to the similarities in the thinking of developers and testers.

I’m not suggesting devs and testers think the same things – but that the process of exploration (the left hand side of the model) might be. The goals of devs and testers are different – so using the same thought process – the outcomes would be different. I’ve also suggested that when creating tests, a developer choose fewer tests (‘to show something works, mainly’) but they also take a different perspective – they might be looking directly at objects, services or components that a user (or tester) would not or could not see or test.

I’m not suggesting that ‘we can get rid of testers’. However, some companies have disbanded test teams. To compensate, their developers take testing much more seriously. Embedded/reassigned and perhaps refocused testers think about risk, they think about user-perspectives and use their knowledge/experience to create better tests. But they use these tests as challenges to users and requirements, to provide checks for developers to implement in code, but also know what test can only be applied to the system through it’s user interface and/or when integrated with other systems and so on and perhaps these tests are run too.

But the developer – tester silos are eliminated.

The ‘breaking the code’ goal is misnamed. As many have said before – if the code has bugs – it’s already broken. The tester exposes failures, and from that evidence perhaps the existence of bugs can be inferred. It’s the mindset of course and in most companies, a good developer-testing mindset is rare. But there is no inherent reason for this to be the case. Developers can break each other’s code. Developers have difficulty because they take the assumptions they made while coding into testing their own code (so are blind to some problems) and of course there’s the emotional difficulty in trying to ‘undo their own hard work’. So there will always be a role for independent thinking here.

The suggestion that devs explore the same way as testers is speculative. But I think it could be the basis of an interesting conversation between devs and testers. For testers, the purpose of exploration is to create trusted models as the basis of tests. For developers, The models are the basis for writing some code. If developers see how they can use the same or similar process to write good tests – perhaps they’ll test better. If testers see how developers explore to write code, perhaps testers will have better insights to create ‘tougher tests’?

I believe it was Edsger Dijkstra who said “Program testing can be used to show the presence of bugs, but never to show their absence!” in 1969. http://en.wikiquote.org/wiki/Edsger_W._Dijkstra

The ‘testing as falsification’ is of course correct. It is often presented as if the developers post the hypotheses and we are the tireless critics of the developers work. This make them look like they are creative and the testers are negative, destructive, even. I don’t think this is the right way to look at it.

Developers have their models, and we have ours as testers. It would make much more sense for us to talk about our models (from out different perspectives) before devs write code, or we write tests. We can challenge through example (and suggest behaviours) and see what behaviour the the devs suggest. Perhaps the user/product owner is part of the discussion (and we are ‘the three amigos’ as it has been called).

Dev and test in ‘Different worlds’…

By talking before committing to code or tests, we make suggestions the developer will not have considered (and improve their design). We will also hear more about the design and how some tests we imagine could be affected/changed by what we learn. It’s an interactive design process. If we go into testing having reconciled the user view, the developer models and the tester models, the code should be better designed, and our testing should be much easier, and the bugs we find are implementation bugs (usually easy ot fix), not requirements or design bugs (hard/expensive to fix).

Yes – examples are good! I think the advocates of BDD offer good examples through their stories. Take a look at Gojko Adzic’s site http://gojko.net/ or Dan North http://dannorth.net or Liz Keogh’s blog http://lizkeogh.com/. Liz probably provides the most/best examples of using stories to communicate.

I like BDD and think it has real value, and the collaboration it promotes fits the exploration part of the New Test Model really well. But the test-automation part of BDD is not a replacement for testing. James Bach makes a good point here: http://www.satisfice.com/blog/archives/638 “(Testing as compared to BDD) … is not the process of demonstrating that the product CAN work, but exploring if it WILL.”

[Prashant]

Agile is really hard to implement. But thanks to SDTimes article on “Testing in modern agile world” Read the article and loved a lot. Moreover they did considered TestingWhiz, a tool developed by me and my team as one of them. You can have a look at article here

[Author]

Posts