We tested the Amazon Echo, Google Home, and Harman Kardon Invoke (powered by Microsoft’s Cortana) smart speakers, each with 800 queries, and found that Google Home answered 81% of the questions correctly vs. Echo at 64% and Cortana at 56%.
Did Santa bring you a smart speaker? Smart speakers are quickly becoming one of the most common consumer uses of AI thanks to aggressive pricing and marketing from companies like Amazon and Google. We’ve conducted our latest round of queries, designed to test the full range of abilities and accuracy of a given AI assistant. Our conclusion: home assistants aren’t just gaining popularity, they’re making drastic improvements quickly. Here is what you can expect from the smart speaker that you unwrapped yesterday.
Methodology. Just as we have in February and August of this year, we asked 800 questions to the Amazon Echo (2nd generation this time around) and Google Home. We added the Harman Kardon Invoke speaker, coupled with Microsoft’s Cortana assistant, to our evaluation. The queries covered five categories: Local, Commerce, Navigation, Information, and Command. The smart speakers were graded on two metrics: did the device understand what was asked? (this can be seen on the device’s companion app), and did it answer or execute correctly? It is important to note that we have slightly modified our question set to be more reflective of the changing abilities of AI assistants. As voice computing becomes more versatile and smart speakers become more capable, we will continue to update our question set to be reflective of those improvements going forward. Our changes included questions around the use of smart home devices. We tested each of the speakers with the Philips Hue smart lighting and Wemo Mini smart plugs.
This round, Google Home was the decisive winner, answering 81% of questions correctly vs. the Echo’s 64% and the Invoke’s 56%. On average, the speakers understood what was being asked 99% of the time. This is a jump up from 95% in August and means that these assistants’ natural language processing has improved to correctly understand, apart from a few anomalies, everything you say. Whether or not it correctly executes your request varies widely between devices and categories.
Results. Google Home continued its outperformance with the top score in each of the five categories. The Invoke, powered by Microsoft’s Cortana, scored more or less in line with the Amazon Echo – an impressive feat considering its tiny market share and comparatively short time in the hands of users. The Echo made considerable improvements in several categories, most notably navigation, while remaining relatively flat in areas like commerce and local.
Improvement across the board. We continue to be impressed with the rate at which these AI assistants are improving. Since our first test in February, the total number of correct responses has increased by 29% for the Echo and 42% for the Google Home. Cortana, which we tested in April on a Microsoft Surface Book, has improved 8%. In less than a year we have noted remarkable improvement in every category and can only expect this level of progress to continue as adoption grows and user base expands. Google Home, however seems to be on the steepest improvement curve. One would expect Google to dominate the information category with its unprecedented access to search data, but its outperformance in categories like commands, navigation, and even commerce furthers the case, in our minds, for Google Home to steal market share away from Alexa going forward.
New features emerge. During our testing, we noticed several minor, but encouraging enhancements to the voice computing experience, namely connectivity to both other devices and other services. Excluding the Echo, when you ask for navigation information, Google and Cortana will send directions to your phone either through a push notification or the companion app. Hailing a ride through Uber was smooth on both the Echo and Google Home. The Google Home could also hand you off to a designated EBay assistant to shop for used goods. We were impressed with improvements around media services like Spotify and smart home capabilities using both lights and smart switches.
Google takes a commanding lead in commerce. Google Home correctly answered 72% of commerce questions, as opposed to 42% and 15% for the Echo and Invoke respectively. This is an area that is thought to be dominated by Amazon. However, Google’s recent anti-Amazon partnerships with retailers like Target and Walmart to make voice-based commerce more accessible pushed the Google Home ahead of other assistants. This was the Google Home’s largest area of improvement, correctly answering 34% more queries than it did in August.
Sound quality. Overwhelmingly, smart speaker owners use them to call up and listen to music. According to Quartz data, 74% of owners play music through their smart speakers, more than asking for weather and basic info. Of the devices we tested, the Harman Kardon Invoke sounded the best; the Google Home and Echo are very similar, but are not focused on premium sound quality. For this mid-tier price range (sale prices – Google Home: $79, Echo: $79, Invoke: $99) the Invoke sounds best. Great sounding music, however, is less defensible than the AI itself. A useful assistant can be loaded onto a tiny speaker like the Echo Dot and Home Mini, or a premium speaker as we have seen with the Alexa-enabled Sonos One, Google Home Max, and the incoming Apple HomePod. It becomes clear that you are paying almost exclusively for sound quality.
Still not mainstream. Despite our comprehensive question set, the everyday user rarely extends beyond simple queries like asking for the weather, general knowledge questions, or requesting a song. In fact, 65% of Alexa users have not even enabled a skill. In order for voice computing to take hold in the home and beyond, there needs to be improvement in terms of connectivity with other devices like phones, laptops, and televisions. We did note much better connection to your phone via companion apps, your television via devices like a Chromecast of Fire TV, and better control of smart home applications, which is an encouraging sign. A short list of persistent issues keeps smart speakers in the camp of tech gadget instead of essential device (like a phone or computer), but at the rate of change we are observing, we remain confident that AI assistants, in time, will become an integral part of how we interact with computers.
Disclaimer: We actively write about the themes in which we invest: artificial intelligence, robotics, virtual reality, and augmented reality. From time to time, we will write about companies that are in our portfolio. Content on this site including opinions on specific themes in technology, market estimates, and estimates and commentary regarding publicly traded or private companies is not intended for use in making investment decisions. We hold no obligation to update any of our projections. We express no warranties about any estimates or opinions we make.