Testing in IoT with an IoT product requires a lot of innovative and out-of-the box thinking, due to the fact that the domain is still relatively new there are no established good practices when considering a test strategy. This article will discuss the five most effective testing types for IoT and why you can’t live without them.
Integration
The base for every IoT product is the smart devices. The market for the devices is a very diverse one which gives a lot of opportunities for achieving profound and extensive use cases. Having the opportunity to integrate the functions of lighting equipment, energy management installations, cameras and etc. into a single centralized solution allows you to have a much better sense of integrity of the product. One the other hand, this requires a lot of focus on integration testing as there won’t be a single generic integration mechanism with all the devices you will have to work with. And the diversity of integration mechanisms with the devices is quite big indeed – Cloud APIs that are different for each manufacturer, various over-the-air networking protocols with their specifics and so on. Each of those has a different authentication and authorization mechanisms that have to be tested very carefully. As of those facts, it is crucial to have a strategy for integration testing that scales early, so the testing effort for introducing a new device, technology and integration mechanism to the product is minimal.
Security
In general, Security is a hot topic for IoT especially after a few massive cyber-attacks took advantage of not very well protected devices. Those events increased the awareness among the end-customers and made them very cautious toward smart devices. The Security challenges in IoT project are tightly related to the topology of your solution and the devices that you are integrated with. That’s why it is important to have at least three levels of security checks: Device, Network and System. The Device level will ensure you are not introducing a device for which Hardware, Access control, APIs, Authentication, Updates and others are not a subject to a security breach. The Network security level, on the other hand, ensures that no potential attacks are possible on the network layer, i.e. no data modification, eavesdropping, spoofing and others are possible. Once the network and the devices are secured, what is left is to check that the System as a whole is properly secured. Verifying the System as a whole is the last step of the process. This includes ensuring the stored customer data is protected. Securing all web, mobile and cloud interfaces is also crucial. It does not really matter how safe the devices and the network are if the authorization and authentication are not secured. Luckily OWASP already defined a guide on how to check that your IoT solution is protected against the most common threats. It can be found here. Those instructions are suitable for people with no or little experience in security testing which makes them useful for anybody who would like to do at least a high level verification of a IoT product.
Usability
Simplicity and ease of use are crucial principles in the IT and electronics industry and this is especially true for IoT. A delay in a few seconds made a huge difference in the usability of an IoT product. The end-customers’ expectations are that they receive immediate information if the state of a device is changed and vice versa – if they interact with the device via the software, their interaction is immediately happening in the physical environment. On the other hand, they expect that the product will be easy to set-up and will have a working smart environment after a few configurations from their side. Generally UX is an important aspect of IoT and it is a challenge for testing. It is important to have a predefined certain usability metrics in advance and keep track of them during the development process. Having a direct relationship with the end-customers and receiving regular feedback could also be of big help as the metrics could be defined based on a direct customer feedback.
Performance
A crucial factor for every IoT product is its scalability in terms of number of devices. The more the devices, the more opportunities for increasingly complicated and extensive use cases for the customers. The great challenge in this area is that some wireless IoT protocols do not scale very well. Their range and response time are in some cases decreased when the number of connected devices is increased. This requires early focus on a mass type of testing where a lot of devices are connected together and their performance is measured. Measuring the response time and maximum range of the devices regularly in different conditions and storing this information as historical data is a good approach for solving the issue. This allows you to track the performance of the product over the time and see if a certain change has cause performance issues. Thus you will be able track which changes have introduced performance degradations, which have a great value for the product and the team.
Beta
A Beta or some sort of friendly user phase gives great value before an official rollout of the software. The reason behind this is that it is almost impossible to reproduce all the different kind of physical environments that the customers could have. No matter how much effort is spent for testing different physical aspects of the device interactions, the number of possible scenarios is too big. For some of them very specific equipment is required which is not always available. Another challenge is that in general the IoT products are highly customizable. If you wish to test all possible customizations and your releases are frequent, the effort you have to spend is huge. Having a dedicated customer base to which you deliver the new version of the product for a certain amount of time before releasing it to all customers gives a lot of crucial information. It provides feedback if an issue related to the physical environment or some specific customer configuration is introduced. Of course, it is important that the right people are hand-picked for the beta tests so that a bigger amount of customization and physical conditions are covered.
Testing in IoT
Precise planning and estimations of the testing efforts is always important success factor for each software project. Understanding the specifics of the domain before considering the needed testing types and how they can be executed gives a great value for this process as it allows for making informed decisions.
