As digital assistants continue to improve, more and more users are integrating them into their daily routines. In our Robot Fear Index, we found that 43% of Americans had used a digital assistant in the last three months. But we believe that we are still in the early innings of how natural language processing will improve our daily lives and our interactions with machines.

To see how far along our digital companions are, we chose to review the two most popular home assistants: Amazon Echo and Google Home.  We put the devices to the test by asking them 800 different everyday queries.  Google Home came out on top, answering 39.1% of the queries correctly vs. the Echo at 34.4%.

Methodology

For this experiment, we asked the same 800 queries of both the Echo and Home.  We graded the queries on two metrics:  First, did the device understand what we asked correctly?  Second, did the device answer the query correctly? In our study, Amazon Echo understood 94.4% of the queries we asked and answered 34.4% of all queries correctly. Google Home understood only 77.0% of the queries we asked, but was able to answer 39.1% correctly.

In our study, Amazon Echo understood 94.4% of the queries we asked and answered 34.4% correctly.  Google Home understood only 77.0% of the queries we asked, but was able to answer 39.1% correctly.

One reason that the Amazon Echo had a higher rate of understanding queries was due to our ability to confirm the data using the companion app.  This app gives the user a live feed of what Amazon Echo heard.  Google Home does not offer a transcript of what the device picked up.  Because of this, it was difficult to tell if Google Home understood the queries but couldn’t answer them, or if it truly had a harder time understanding queries. Since we were unable to see exactly how well Google Home understood our queries, we assumed that if Google Home responded that it was unable to perform a certain function, then it had understood the query correctly.  For example, if we asked “Hey Google, send a text to John” and received a response “Sorry, I can’t send texts yet,” then the query would be marked as understood correctly, but answered incorrectly.

Results

The queries that we asked divide into five buckets: local, commerce, navigation, information, and command.  When breaking down the comparison of correct answers between the Echo and Home, Amazon predictably performed better at commerce and Google at information, in-line with their stated missions.

Amazon Echo’s Advantage:  Commerce Assistance

As expected, Amazon Echo has the upper-hand when it comes to commerce-related queries.  Prime members have the ability to order products via voice control, while non-Prime members can add products to their cart for review and purchase later.

At this point, Alexa is best suited to help re-order simple, consumable products.  In an interview with Billboard, Jeff Bezos shared, “Voice interface is only going to take you so far on shopping.  It’s good for reordering consumables, where you don’t have to make a lot of choices, but most online shopping is going to be facilitated by having a display.”  Amazon is reportedly looking to introduce an Echo model with a touchscreen sometime in 2017, which would significantly expand Alexa’s capabilities when it comes to commerce.

Google Home’s commerce capabilities are limited to providing store recommendations or price quotes on specific products, much like core Google search.  Currently, Google Assistant can’t help users order products via voice control.

Google Home’s Advantage:  Information Accuracy

Google Home outperformed the Amazon Echo in the navigation, information, and command categories.  However, it’s clearest advantage is in information. Aside from Google’s Knowledge Graph, a big factor in the Home’s information advantage is Google Assistant’s ability to engage in two-way conversations.  By doing so, Google Assistant is able to answer follow-up questions, building on topics discussed in prior queries.  This conversational approach makes it much easier for users to get the information they want because they are able to add clarifying comments to questions if they don’t receive the answer they are looking for the first time. By comparison, Alexa must be asked complete, individual questions each time and can’t engage in two-way dialogue.

Strength of Home Assistants: Home Audio Control

While the Amazon Echo and Google Home have their differences, there is one thing they both do very well: home audio control. Google Home and Amazon Echo can each connect to Spotify, Pandora, and TuneIn.  In addition, Google Home has access to Google Play Music and YouTube Music, while Amazon Echo has access to Amazon Music and iHeartRadio.  One slight drawback is that Amazon Echo owners must be Amazon Prime Music or Spotify Premium subscribers in order to play specific songs.  If they are not, they are limited to user-created playlists or customized radio stations. For many, home audio control is a major selling point for home assistants and another reason why we expect to see continued investments in content across all the major Internet companies.

Weakness of Home Assistants: Navigation Support & Reminders

There are two basic categories at which home assistants performed poorly: setting reminders and providing navigation support.  However, we view both of these as minor issues that will be fixed in the future.

Reminders are a key function of phone-based personal assistants; however, neither the Echo nor Google Home have the ability to push reminders. Both devices can access the calendar associated with the signed-in user, but can’t proactively remind the user of an upcoming event. It’s up to the user to remember to look at their calendars on other devices. This is a minor downside to voice-based home assistants given that effectively all users also have a smartphone pushing them synced calendar alerts, but still an area where the assistants could improve.

Navigation is another area for potential improvement.  Neither the Echo nor Home could send directions to a smartphone efficiently.  Google Home in particular had noticeable differences between the reported distance of a location versus the actual distance. Navigation would seem to be a logical improvement in the future, although the prevalence of smartphones with rich built-in maps renders the issue minor in our opinion.

The Outlook for Home Assistants

We view home assistants as one of the core elements of moving beyond the computer as we know it.  While still early, the future is bright for these platforms as a companion to VR and AR. Adding third-party developers to the mix is the next logical step.

Amazon appears to be following a version of Apple’s App Store model by offering third-party add-ons, which it dubs “skills,” that can improve the user experience. Some of the most-downloaded skills are ambient noises, games, bedtime stories, or even daily historical facts. Amazon Echo’s skill store is populated with thousands of add-ons that can improve the user experience.

Google Home has not yet fully opened the door to third-party developers.  In December, Google announced its “conversation actions” platform, which would allow developers to create back-and-forth conversations with users through Google Assistant. Conversation actions allow users to ask third-party services a question by name, and then have a discussion about that initial question. Interestingly, Google is not requiring users to download or enable these skills, as Amazon does.

As companies like Amazon and Google continue to improve on the capabilities of home assistants, we would expect others, like Apple and Microsoft, to push further into the space. Our homes have just begun to be connected and we expect home assistants to be one of the centerpieces of managing our lives in the future.

 

Coming soon: We’ll have a three-way battle of digital assistants between Siri, Cortana, and Google Assistant.