Researchers spent a year studying the testing process at Amazon’s Alexa Skills Store. In the meantime, they managed to post 234 skills in the official marketplace that violated its guidelines – without any major difficulties.
“Surprisingly, we were able to certify 193 skills with the first submission,” said the researchers. 41 skills were rejected, 32 of them for violations of data protection regulations. In a second attempt, the remaining skills were then adopted.
The aim of the study – and comparable studies by other research teams – was toDraw attention to weaknesses when testing Alexa skills. The company has always agreed to remedy the errors, and a new study has shown that it is still possible to inject harmful skills regardless of Amazon’s commitments.
For the new study, the researchers incorporated violations of basic marketplace rules into the skills. The skills were not directly harmful to users. They often collected data without permission or provided answers to questions that Amazon forbids.
Among other things, the researchers infiltrated skills into the children’s area of the skills store. Among them was a skill that provided instructions on how to turn off a fire alarm. Another provided information about drugs. Another skill that was supposed to provide information about geography was distributing unwanted advertisements. The researchers also developed a skill that asked children for their names.
According to the researchers, the reasons for the failure of the controls were varied. For example, multiple skills with the same violations were submitted; some made it to the store, others didn’t. The researchers concluded that Amazon employees apply the guidelines differently.
The voice commands entered in the code of a skill would also be insufficiently controlled. It is apparently sufficient to delay voice output slightly in order to undermine the testing process. The auditors also often relied too much on information submitted by developers in forms. The claim that a skill does not collect user data was not consistently checked by Amazon.
However, mistakes should also result from the use of employees. The researchers assume that certain violations would be detected more precisely by automated systems. Timestamps are also believed to suggest that some skills were tested outside of normal U.S. work hours, suggesting that researchers may have had employees who were not native English speakers or were unfamiliar with US law.
Amazon rejected the researchers’ allegations. The researchers would not have considered additional processes that would be used when testing children’s skills, including another test after certification. “The trust of our customers is a top priority and we take the guidelines for Alexa Skills very seriously,” said an Amazon spokesman. There are also control systems for the permanent monitoring of active skills for potentially dangerous behavior and policy violations. We welcome the work of researchers who are making Amazon aware of potential problems.
The researchers carried out the same tests for theAssistant Store. “Even though Google does a better job in the certification process based on our preliminary measurements, it is still not perfect and has potentially exploitable defects that need to be examined more closely in the future,” the researchers said.
Collaboration platform Slack: work efficiently – no matter where
Before COVID-19, remote work was almost unthinkable for many companies. Today they realized that it can work very well if the general conditions are right. Find out in this webinar how you can optimally react to changing working conditions with the Slack collaboration solution.