Annual smart speaker IQ test

LoupVentures put out their annual smart speaker IQ test. They tested Google Home, Amazon Echo, Apple HomePod and Microsoft Invoke (Cortana). Each system was asked 800 questions on a variety of topics and areas.

Here are the results:

And here are the results over time – they’ve been doing this test for 4 years now:

I personally use Alexa. I chose Alexa because it  was the only one on the market at the time and I like the integration with Amazon. It also works well with other services like Amazon and has a very mature skill ecosystem like integration with my Smartthings hub and Nest thermostats.

Apple’s product would never work for me. It’s too expensive for its purpose and too much tied to their ecosystem. Google would be my secondary option. It’s better in some regards but it’s ecosystem is still immature. I’m also too much invested in Alexa’s ecosystem that it will cost real money to switch. Plus I do like ordering my Amazon stuff through Alexa.

One of the things we use a lot now is the intercom function. Mostly because we went from an apartment to a multi-story house. It’s quite useful.

The questions they used in their research are not disclosed. I asssume this is to avoid test optimization by the various companies. I do see that Alexa is getting better at mundane questions and commands. But the product category still feels immature because the devices lack contextual awareness beyond the address they reside in. I can’t ask Alexa to switch on the lights in the bedroom because she’ll ask “which lights?”. Well obviously, not downstairs Alexa. Nor does one Alexa “knows” about other Alexa’s in your household. Each Alexa is separate. You cannot ask about a timer you set on another Alexa nor switch it off. This happens a lot when a timer of the kitchen Alexa goes off but I have to get up and walk over from the family room to the kitchen to switch it off because if I yell “Alexa, stop!” the family room Alexa is just confused. But I digress.

I do still love my Alexa’s. I hope number 4 is under the Christmas tree.