The Challenges of software testing IOT devices

‘Susan Harris is alone in the house when, suddenly, doors lock, windows slam shut and the phone stops working. Susan is trapped by an intruder – but this is no ordinary thug. Instead, the intruder is a computer named Proteus, an artificial brain that has learned to reason. And to terrorise…’

The 1977 film Demon Seed, a story about an AI gone rogue may have been a tad melodramatic on the danger that artificial intelligence poses to man, but it did have some smart insight into home automation and its potential malfunctions.

From healthcare and home automation to transport and the oil and gas industry, the Internet of Things (IoT) is rapidly growing. Researchers Garner Inc predicted that by 2020, there will be 20.4 billion IoT devices connected to the internet.

So whilst you are watching films on your smartwatch, asking your home to warm up to a prefered temperature, and perhaps negotiating what constitutes as burnt with your toaster, spare a thought for the testers. The men and women toiling behind closed doors to ensure that these devices actually work as intended.

So what are the challenges software testers face when testing IoT devices?

Firstly, there’s the obvious – is it safe?

 Security

A quick search of IoT will bring a plethora of articles discussing security concerns with IoT. However, rather than worrying your kettle might be spying on you, there is the very real concern of adding new or breachable devices to your trusted networks. With a laptop or computer software can be installed to protect devices, but for IoT devices the support is slim. Many devices also have default usernames and passwords that the average user never changes, making them vulnerable.

Testers must also consider the wider affect IoT devices can have on internet security. One only needs to look back to the Mirai botnet attack that used items such as digital cameras and video recorders to cause huge disruption to large parts of America’s internet, bringing down sites such as Netflix, Twitter, CNN and Reddit.

For a tester then, issues such as data flow between devices, data encryption and integrity of testing software (if automating) are all key concerns. 

Sometimes testers must simply rely that users will protect their networks. However, with cheaper routers often containing less safety measures, there lies another issue for IoT testers, that of replicating the environment the item will operate in.

Replicating environments

When testing a website it’s easy for testers to run the pages through the mill. Open the page on the popular browsers, run it on laptops, tablets and smartphones and you’ve pretty much tested how it will be used by 90% of users.

However, with IoT devices, replicating a user’s environment is far harder. Will the device be used at home or at work? Will the internet speed be fast or slow? How many unknown devices are also connect to this network? Will the device need to leave and rejoin the network?

Forgetting the technical aspects, what type of furnishings will the device rest on? Apple’s HomePods have been documented as leaving white rings on wooden tables that have been treated or varnished. Whilst not strictly an issue for the humble software tester, it shows how the wide variety of environments means the job of replicating conditions for testers is challenging.

For this reason testers must ensure they have a strong understanding of the product and try to encompass as many different scenarios into their testing as possible, including how the device will work with others.

How it works with other devices

Compatibility testing is always a bugbear of testers, but in the IoT world it becomes a myriad of challenges. Just take the seemingly endless interactions that can be created on devices that support IFTTT – there are simply too many moves on the chess board to analyse and test every device for every release in every environment.

The only way testers can begin to address this task is to focus on the most popular operating systems and communication methods (ie bluetooth), on the most popular devices, so that efforts have the widest reach, before moving on to more niche items.

Costs

Another challenge is perhaps one of the most obvious – the cost. Testing something simpler such as a website is reasonably quick and cheap. But testing IoT devices in different environments with different integrations can be very costly. Time is a big factor, due to testers having to cover so many bases, but another issue is the possible need for specialist testing software. A few studies have alluded that the combination of a lack of developers failing to review their code for security flaws, and a lack of thorough testing of IoT apps mean that testing becomes very complex. And with complexity, comes cost. 

Power & Backup 

Unlike the most common IT solution of turning it off and on again, some IoT devices can struggle if there is a sudden loss of power. Testers need to ensure that they test how a device changes following a loss of power – particularly for items in the healthcare or industrial sectors. 

This will become less of a problem as wireless power continues to grow, but until then the variables need to be considered. Consider a smart intruder alarm or a security camera, if the system fails due to a bug or drop in power or connection, they are essentially useless.

How or even if a device backs up is another issue, especially when you consider how new or updated software will responded to older data.

Updates 

Firmware is an integral part of many IoT devices, and there’s nothing more developers love than pushing out new updates. But for the tester, updates can carry with them the monumental headache of bugs. If a missed bug gets through into a standard computer, then the issue can usually be simply tested and fixed before there is too much damage. But if a bug gets through into an IoT device, it can have far more physiological ramifications. 

Take Nest’s Thermostat issue back in January 2016. A software update the previous December had resulted in a bug draining the battery life of some Thermostats, causing the system to deactivate. Eventually, the company rolled out a nine-step-fix for users, but proper and thorough testing could have saved the user’s shivers and the company’s face.

So whilst we are not in danger of our homes becoming the Demon AI Proteus; if Software Testers do not take the proper precautions, they do risk being left in the cold.