Complex testing activities were being performed by our testing team while being remote from Development team. We worked with all stakeholders within and outside the team to find out what information they are seeking and then took a methodological approach to make our testing public. That made us transparent, trustworthy and our job is appreciated more.
Your trust in a person, process or organization is based on few things with openness standing out in the list. As Testing is essentially a service that we provide to rest of the team, making things public about testing results in gaining trust of our consumers. It is as when you see an “Open Kitchen” policy of a restaurant, you feel more confident about the food they will serve.
Usually making testing public means making the test results visible through some dashboards. It is very important, however, you cannot build trust only by showing results. Testing is a complex process and by involving your consumers in every bit of it you build a trust worthy relationship that goes a long way. I’ll share a personal journey of how we made every testing related thing public and what approach we took for each type. And believe me, the CTO trusts our team more than anyone else in the company.
Test Schedule: We had this classic problem of juggling between various testing tasks (Smoke, Performance, Data Conversion etc.) on multiple releases (future, patching released versions). Our life became more difficult as we started supporting multiple platform (Android, iOS, UWP and Windows) and add to that multiple devices in our test lab. We created a color-coded chart of what testing is performed on what type on what schedule and put it on our landing page of internal team site. Everyone now knew the scope of our team’s work and became clear on when they can expect a feedback from us. This has become a de facto page in our organization and many other teams have followed the same pattern.
Illustration 1: Test dashboard showing schedule
Test Code: Our code is managed in Mercurial and each repository has a sub-folder called “Tests”. When someone pulls the code, they can see all tests not just tests written by a subset of people.
Illustration 2: Test code inside repo.
Similarly, when we started to write our Python based test scripts to automate testing tasks, we had them on our own boxes in scattered locations. We quickly realized the issue of visibility and moved them to a shared repository which is accessible to all. Yes, even people outside our team has a read-only access.
Test Data: This was easy as we made some basic data part of the repositories mentioned above. For large datasets, they were put on shared locations along with some meta data so that anyone can reuse that dataset for reproducing issues or running their tests with that data. This dramatically reduced the number of follow up requests from developers on reported defects which would basically say “what dataset you are using?”. Now everyone know what dataset is being used and how to get it.
Test Results: That goes without saying. All results of our testing go to a Microsoft SharePoint site that anyone in the organization can see. It starts with fancy charts and ends up with real information on what passed and what failed. We also learnt that the audience of these pages is very wide so we deigned it in a way that different level of viewers gets the info they need. For example, top management can view the overview of results to make decisions whereas individual developers can get insights into test failures to fix them through the same set of pages.
Below are some samples from our reports:
Illustration 3: Test results
Our motto for the results are:
“Everyone should be able to see it. Your in-laws included”
Test Jobs: We use Jenkins to run scheduled and triggered testing jobs. No surprise that Jenkins output can be viewed by anyone to see “live” progress on ongoing tasks. Also, we arranged for many people to be able to see the configuration of these jobs to suggest improvements.
I can write more about how we made public our Test documentation, Test planning, Test strategy, Test infrastructure, Testing devices and more but I hope you can have some idea from the above pattern so as what we did for each category.
By taking this approach of making our testing public, we moved from the confusions on various testing activities to a relationship built on trust. With that trust, our product delivery is smoother than ever and we get increasing demands of more testing because our customers like us. The journey has been full of challenges conquered through hard work and the results have been very rewarding and enjoyable. We can now say based on our experience that:
“Openness always builds trust. Testing has no exception”
About me:
My name is Majd Uddin and I am currently serving as Senior Manager Software Quality at Bentley Systems Pakistan. I have 20 years of Industry experience with most of that time spent in the pursuit of Quality. My core expertise include building test teams, testing platform/SDKs and devising test strategies. I am a “tester for life” and run a blog as http://knowledgetester.org. I also like to travel and explore the beauty of majestic northern areas of Pakistan. You can reach me on twitter at @majduddin.
