On Wednesday 14. March 2018 I presented my talk “Continuous Delivery using Crowdsourced Testing” at the Swiss Testing Day conference. It was a pleasure to talk about the topic which occupied lots of my time and thoughts in the last one and a half years. Even if presenting on the main stage at the Samsung Hall made me just a bit nervous at the beginning, I really enjoyed it.
I’d like to underscore the most important learnings I’ve made working intensively with crowdsourced testers and incorporating the crowdsourced manual test cycles into our overall agile testing strategy, while working in a fast-paced DevOps environment.
1.
It is important to understand what continuous testing is and where it fits into the DevOps culture and mindset.
Continuous testing is a process of executing many different types of tests — both manual and automated – continually throughout the delivery process
to obtain immediate feedback
on the business risks
associated with a software release candidate.
- Definition adapted from Jez Humble. I encourage you to read his Continuous Delivery book.
2.
Testing actually happens on every stage of the DevOps toolchain. Note that this article from Dan Ashby is from October 2016 but lots of the people still don’t know about it and how true it is.
@majaschreiner about continuous delivery using crowdsourced Testing: „Testing happens on every stage of the DevOps process“#SwissFusion18 pic.twitter.com/SMBIM70SRa
— Swiss Testing Day (@SwissTesting) March 14, 2018
3.
In our efforts to continuously deliver online storage solutions for our private customers we never forget the manual validations. Automated tests run continuously on our development, integration and test environment but how do you scale and get fast feedback on your UI (web, mobile apps) or usability tests? Also, you can’t automate all your customers flows.
4.
Ideally you should, but usually you don’t have an experienced tester in every cross-functional development team. Also, lots of the time it can happen that the team can’t finish all manual or automation tasks before the sprint ends (here I’m not talking about the other causes for sprint-spillover). And even if the team tested all sprint artefacts, how much time would they have to run regression, exploratory, usability or user beta tests?
To compensate the activities for which you’re not skilled or just don’t have enough people to get dirty (which IS part of the game) you can hire a testing service provider who maintains the crowd: community of experienced testers and IT specialists. Having that complementary testing service to internal (testing) teams with a focus on the end-user experience, will certainly help you reach your sprint goals and increase the quality of your products.
https://twitter.com/baseltnicole/status/973917138594430976
5.
Main advantages of crowdsourced testing
- Resources flexibility
- Recources scalability
- Fixed price contract normally includes the hours of CT company’s internal stuff (project manager, technical leads) as well as any possible compensations to the testers for the issues they find.
- Provides user point of view needed for testing user-centric apps
- Crowd is not confirmation biased
- User surveys can be included in every test cycle
- You can also test mocks and wireframes
- Crowd testers are rated based on their performance on past projects -> you get an access to a large pool of good testers
6.
Even if the crowd test execution cycle and the whole process might look simple there are a lot of challenges to solve on the way.
- depending on the product your developers might get overwhelmed with the number of found issues. Some of them will start rejecting them or the process in general.
- lots of the results found might lack the significance or impact, meaning they get approved as somewhat valuable issues. Remember we want to find those exceptionaly valuable issues that involve business risks.
- internal coordination & communication time needed can develop into another bottleneck, especially if you’re the only one capable or willing to do those tasks. Already after I started, I found myself spending the half of my working time “only” on preparing next cycles and analysing the results of the current or past ones. That didn’t leave me with enough time for other strategic activities or building up on my test-automation skills.
- even if the crowd is huge you still paid for some particular number of testing hours per month and having only good testers claiming their slots on a first come first served basis is pure luck. Of course you can invite the names from your “top 20” list but nobody guarantees you they will really claim their slot and test
- besides having the good testers it is very important to have a great testing team lead on the CT company side. Imagine having and then loosing a person who’s a brain of your test execution cycles who knows and understands your products, the known issues, remembers what was found in the last weeks and why. When such a valuable resource is not available anymore, it will take lots of luck and some weeks to onboard the other one (or you will have even more work).
7.
There are certain strategies to solve those problems but you need to know where to start and that is normally where it hurts the most.
- Stick to (your) timeboxed time and delegate all different preparation or follow-up tasks to your team -> be persistent in that and don’t give up
- Educate the team about the importance of “three amigos” principle, meaning that not only Product Owner is syncing with you on the issues found or on the tests planed to be run or updated, but the developers as well
- Start buddying and pairing with those developers who value the CT process and don’t forget to appreciate and thank them for their help and support
- Make sure that the issues found from crowd test cycles get into the next bug refinement
- Track improvements
- Continuously adjust your strategy and process and make it flexible
- Never forget to star and reward good testers and even send them a personal thank you note
- Never stop educating your team
- Never stop learning 🙂
- Also learn from other testers findings in terms of improving your testing skills and mind-maps
8.
Further things to keep in mind:
- preparation is the key ingredient to your success and good input creates the good output
- information on which the particular test cycle is based need to be clear and precise
- Test scope / products to test
- Software version / environment
- Test plan (if you want them to run regression based on test scripts)
- Known issues list
- Internal release notes
- Sprint review video demo & presentation slides if any
- be clear about how you’re using the CT report dashboard with a results summary per risk, devices, browsers, technical or functional etc.
- concentrate on testing with the devices your real users use the most and get those numbers from the analytics
- beware the internal costs needed for coordinating the whole process
9.
Summary:
- Know your CT goal & purpose and aim for the “highest” added value:
- What
- How
- When do you want to crowdsource your tests (in our project it’s normally towards the end of the sprint, every 2nd week)
- CT won’t solve all your testing, QA or software development problems, will though help with functional or user tests
- test internally as well
- every cycle should include explorative tests – the results are much better than with regression test scripts
- setup the two-way integration with your JIRA
- insist on keeping the good testers in your next cycle and try to attract them by giving new products and features to test, otherwise they might get bored or will stop testing your current product(s) since they know they won’t find any more valuable issues (less money earned for them)
10.
We are using Applause manual testing services (formerly: utest) and have been satisfied especially with the professionality of their internal stuff who really knows about testing. Check out their how-to pages to understand in more detail how bug verification, two-way JIRA integration and test management process work. If you have any questions where I could certainly help with my experience and tips, don’t hesitate to contact me.
