IoT is no longer a buzz word. IoT is here and it will stay for a while. Most of the expectations for a growth rate of the IoT market in the last few years were not met due to a lack of understanding in most organizations about the practical appliances of the smart devices. However the technological progress of the domain was tremendous. Such a fast pace always introduces a lot of challenges in ensuring the quality of the product. The purpose of this article is to outline the most common challenges when considering a test strategy for an IoT project and Testing IoT devices.
Starting from the required skills for test execution to the deep dive of test strategy definition, test design, choice of suitable tools and automation testing frameworks, the challenges in IoT seems to be different. This comes due the fact that IoT is not just a new software technology. It is the interworking of software and hardware in a new, lightweight, and extremely distributed way. Smart devices are considered not only computers, tablets and watches but also smoke detectors, hoods, fridges, cameras and lighting equipment. Some of these devices have a very specific Operating systems, in most cases do not communicate using the well-known networking protocols, their graphical user interface and API are vendor specific.
Despite the differences with the standard Software testing, the basic principles of Software testing are still applicable in the context of IoT, but the way how to follow those principles is completely different. Working in such projects does not mean you don’t have to define certain conditions so that your product meets the desired quality. The challenge here is to determine what those conditions have to be. A lot of aspects in the IoT field are hardly measurable and in order to be clearly defined, we need to know the current specifics of the setup. Is the smart device tested in the same physical conditions as it will be at the customer premises? Are all possible use cases of the device considered? How much of the Hardware/Firmware/Software compatibility matrix have to be covered?
Testing Conditions
In order to determine the best testing conditions for a device such as distance, walls, radio interferences, we are going to need a profound knowledge of how the different communication protocols work. Some of the already established technologies such as Wi-Fi, RFID, NFC and Bluetooth have been adapted to suite the purpose of exchanging data with most of the smart devices. On the other hand, there are a lot of developments with the sole purpose of providing fast and lightweight communication over the air. Technologies like Zigbee, Z-Wave and ULE introduced a whole new world of authentication, authorisation and data transfer mechanisms that have to considered. Understanding how those protocols work is crucial for determining the right physical conditions when testing an IoT product. Some protocols have the benefit of having long ranges, others are sustainable in environments with a lot of interferences such as highly as large amount of Wireless signals. At the end of the day, what is important is verifying the use case in the same conditions as it will be executed by the end customer. That could mean testing through walls and other physical obstacles, building an environment that is flooded with signals from other wireless devices or simply increasing the distance between the product under test and the smart device.
Testing Scenarios
Another aspect of the IoT testing is ensuring that all functional aspect of the product are covered. It is a rare case when a single device could fulfil a customer scenario by its own. What most products are trying to achieve is to provide the end-customer with the opportunity to interconnect a large amount of devices together. Thus enabling use cases that cover those devices working together as a whole system. Definition of testing scenarios that cover the big picture rather than only checking the functionalities of the gadgets separately is crucial. This means covering scenarios that make sure that not only the devices behave as expected, but they also work together as a software solution.
Test Strategy for IoT Devices
One of the big problems that have to be solved when considering a test strategy in IoT is the large amount of integration points. The main goal of the verification of your product will be to make sure that the software works properly with various different devices. The problem here is that the pace with which the device manufacturers produce their products is very fast. In fact the biggest players are already selling second or third generations of their gadgets. For a Quality Assurance engineer that means executing the same test scenario with different generations and firmwares of the device, which requires big effort, especially if you are supporting a large amount of devices. Automatic approach here is mandatory if you want to scale the testing process. In the IoT context this does not only mean automation testing of software, but of the hardware as well. This is indeed one the biggest differences and challenges – finding and adapting a suitable technology that will allow you to easily represent the physical interactions of the end-customer with the smart devices.
Learn, Network, Be Inspired – Attend Europe’s Biggest Software Testing Conference
Testing IoT
There are still no established good practices when considering a test strategy for an IoT. Understanding the specifics of IoT is crucial when defining the testing scenarios for the product. The most important technical challenges to consider before deciding where to focus the testing efforts are the device functionalities, the integration points and finding a suitable solution for test automation. Those topics are very complex themselves and separate article for each of them will follow-up in the near future.
