Joonas Palomäki was the EuroSTAR 2023 RisingSTAR Award winner for his idea of utilising AI to help in testing related tasks. The original title was about using AI to make test cases 50% faster, but the subject has been broadened to cover more areas where Artificial Intelligence can be used to help testers work more efficiently. This is a sneak peek into the evolving work.
It has been an interesting time! A lot has happened during a year. The word AI is everywhere. It is at the same time a buzzword that is added to toothbrushes to sell them, and at the same time the core AI services are developing so fast that people and companies have hard time learning to utilize them properly. We are just starting to see how they are incorporated into uses and really useful products. Depending on who we listen the AI is a futile gimmick that just creates trash content and is actually counterproductive, or it is a miracle that will do our work for us. I’ve been finding uses for LLM AIs that help software testers in their work, and I’ve found that the AIs can be used as very effective tools when used correctly. This is an update about the progress I’ve made so far.
What are these AIs made of? – How to Use them and Get the Benefits
To be clear, when I’m talking about AIs I’m talking about Large Language Models, LLMs. To use them as effectively as possible, it is needed to understand a little bit about AIs – how they work to understand what are their strengths and their weaknesses. In EuroSTAR I will delve much deeper into this, but here’s a few key points
- When you ask non-agent AI ( Like ChatGPT ) to do something, it works very differently than a human who is able to iterate on the work and has the ability to control the work itself. This makes a huge difference on what kind of tasks AI can solve.
- AIs are the most effective when the user outsources the parts of the tasks to AI where the machine is best and does the parts itself where the human brain is needed.
- AI is “Tabula Rasa” – it does not know anything about users or users’ needs. It is super knowledgeable about the world. It surpasses any human in general knowledge because it knows quite much about everything. It just does not know what the user wants. By being verbose about what the AI should be doing leads to much better results.
- Different LLMs perform differently. The difference is absolute on complex tasks – weak AIs always fail, strong AIs do perform.
- There are limits to what AIs can do. They can not do everything – know the limits!
- AIs do make mistakes. Expect that. It does not mean the whole AI is not useful if you get good answers often enough.
- You can expect at most as good results as how well you have defined the task for the AI. Basically this means you need to input quite a lot of information to the AI so it can take that into account in its answers.
Uses for AI in the Field of Testing
So the approach for utilizing AIs is to use it for small enough tasks that are
- Easy enough for user to define
- Easy enough for AI to handle
- Complex enough to benefit from the AIs answers
Another thing I’ve found effective to think of AI uses is to differentiate tasks where you use AI to give you insights on something, and when you also use AI to formulate data into more permanent format. When you do these steps separately, your workflow is more efficient.
Practical Uses
GPT-4 was the first strong enough LLM AI that was able to produce good enough test cases from the specification to be useful. By just giving GPT specification and asking for tests does produce tests, but hardly good enough – as too big portion of the tests are less than useful. By splitting the task – asking AI for ideas for testing – test factors – then reviewing those and finally formulating the test case yields much better results.
That is hardly the most interesting use for AI. There are multiple other uses where AI can be used to help in various testing related tasks. Here are some examples I’ve been using AI for
Doing Exploratory Testing with AI
-
- Test cases are one way to document the testing work and known to everyone, but on many cases they are not needed.
Writing Better Bug Reports
-
-
- AI can be used to double check if the user has missed something important.
-
Writing Better Improvement Ideas for / or change requests
-
- Pretty much the same idea than with bug reports but AI is given different information to work with.
Incorporating Change Requests into Specification
-
- Automatically finding the the specification items where the change request belongs
- Suggesting improvements to specification
- Helping formulate the specification based on suggestions and other input
Using AI works in all these cases – in certain circumstances. It all depends on the project, available data and also personal preferences. Some like to ideate with AI, some hate it.
These are not all traditional tester tasks, but then again the role of software testing has long been intertwined with other development tasks. In addition to these, there are numerous other uses for AI especially in test automation and data handling and generation that I would love to dive into, and plan to after EuroSTAR.
What about the Future – will there be a renaissance of the specifications? What will Testers do?
Related to utilising AI, a brief look at the future – my prediction about what we are likely to see in the future. As a tester I have always valued specification – especially a maintained one.
Maintaining up to date specification is often a bit overlooked – especially with agile methods the focus is usually in doing, not documenting. It is common to document the thing currently being done, but leaving the old stuff as is. This is problematic for the mentioned AI techniques, new people in the project and AI code generation tools as well. Companies that want to increase their efficiency by using AI code generation tools should also think about this – is the important specification data in a format that can be easily read and understood by people AND can it be sent to AI to help coders in their work. I see that very likely future as we are already seeing tools that do exactly that – make software from the specifications. They are still rudimentary, but are developing fast.
That will then open up a question – what will be testers’ role in the future? In a broad sense, the essence will probably be the same – Someone should be in control of the quality. The AI is just another tool – albeit a powerful one, that helps getting more done during the workday. Still, someone needs to use that or those tools – to make sure the tool did what it was supposed to. Testers’ role will probably intertwine more deeply with business and development – towards overseeing quality control by defining what software should be doing and using AI tools to automate the actual testing.
The 2024 RisingSTAR Award was won by Bart Van Raemdonck (Belgium), for his idea of creating a Quality Evolution Tracker. See more about Bart’s idea and all the 2024 finalists. Over the coming year, Bart will be mentored by the panel of RisingSTAR Supporters and will take to the EuroSTAR stage in June 2025 to share his project story and updates.
The 2025 RisingSTAR Award will open for submissions in the New Year. The RisingSTAR Award recognises innovation in the software testing and quality assurance industry. It is self-nominated and the winner is chosen by the RisingSTAR supporter group who provide mentorship and guidance to the winner as they develop their idea into being. This is part of the annual EuroSTAR Software Testing Conference.
