As smartphone makers established in the United States look to expand to emerging markets abroad, one key area that they need to address is language, and one of the best places for them to address it is the virtual assistants with which users interact when they need to complete voice-enabled tasks on their smartphones. While researchers continually compare Siri, Google Now, and Cortana on many different grounds, research conducted by Globalme for VentureBeat took a unique angle on a comparison. They tested Siri, Google Now, and Cortana in five languages to find out which had the best “multilingual chops.” Globalme’s researchers asked each assistant 18 questions in English, Italian, French, German, and Mandarin to get a sense of which was most accurate at interacting with users in each of those languages.
Each question had a possible score of 1.0, 0.5, or 0.0. Answers that scored 1.0 gave testers an easy correct response. Answers that scored 0.5 weren’t as straightforward and came back with a list of search results, within which researchers found the correct answer. Answers that scored 0.0 were those in which the assistant didn’t understand the question, gave irrelevant search results, or stated that it could not complete the task. English had the highest accuracy rates across all of the voice assistants. Siri logged the top score, with 14.5 out of 18. Google Now and Cortana weren’t far behind, with scores of 11.5 and 12.5 respectively. The researchers noted that they were especially impressed with Cortana.
In Italian, Siri left Cortana and Google Now in the dust. Siri’s score in Italian was identical to its score in England at 14.5. Google Now scored just 8.5, and Cortana scored only 6 out of 18. The researchers note that they were running the developer release of Cortana in beta for Italian, French, German, and Mandarin. They add, “We’re convinced that that with some proper localization (or maybe some practice with Duolingo), Cortana could rise to the top.”
The results of the tests in French and German were similar to those of the Italian tests. Siri dropped slightly in German and French while Google Now and Cortana improved marginally in both languages. Globalme says that the tests in Mandarin were a way to “set apart the boys from the men,” and noted that while Siri scored 13.5, only one point below its English score, Cortana scored a low 5 and Google Now dropped to 1.5. Only three out of the 18 questions posed in Mandarin scored as high as a 0.5, indicating that the ability to use Google for voice-enabled search is “questionable at best.”
Google Now is offered in more than 50 languages and dialects, a number that far exceeds both Siri and Cortana. The researchers report that many of these languages, like Mandarin, are only set up for voice search. They don’t yet have the capability to function as a full-fledged voice assistant in each of those languages. Globalme shared the specific test cases and results on its website, sharing the questions asked and the scores that each assistant earned for the English-language queries.
Cult of Mac’s Luke Dormehl notes that the demonstration of Siri’s multilingual abilities bodes well for Apple’s expansion into new markets like China. Mandarin has 995 million speakers worldwide — approximately three time the number of English speakers — and is by far the world’s most widely spoken language. Tim Cook has repeatedly spoken about how China will become Apple’s primary market in the near future, and Siri will likely prove useful in helping with adoption. As Apple expands to developing markets with lower rates of literacy, Dormehl says that he envisions virtual assistants that enable people to speak to their smartphones offering a higher level of functionality than they do in the United States, where they’re essentially just a substitute for text-based queries.
Many researchers and tech publications aside from Globalme and VentureBeat have recently compared the virtual assistants, testing their speed, their accuracy, the scope of the questions that they can successfully answer, how helpful their answers are, and how much personality the developers have programmed in to each assistant. And as Business Insider noted recently, the frontier of virtual assistants is starting to feel like a real race now that Microsoft has joined Apple and Google.
Siri, which Apple introduced in 2011 with the iPhone 4s, and Google Now, which was unveiled in 2012, aren’t truly the veterans in the space, as Microsoft has been experimenting with voice recognition for over a decade. Cortana, which seemed a latecomer to the scene, has been shown as an impressive alternative in some trials. Microsoft’s assistant is also regarded as an up-and-coming, key part of the company’s competitive strategy. As Search Engine Land’s Greg Sterling recently reported, Microsoft has denied that Cortana is a “me too” product, or built as a direct response to Siri or Google Now.
The technology behind Cortana has been evolving within Microsoft for years, primarily in the areas of speech recognition and natural language processing. Sterling notes that while Google has some of the same “deep technology” assets as Microsoft, Apple does not. Siri is relatively “thin,” and relies on third parties for speech recognition. Sterling reports that Apple risks falling behind as the digital assistant race gains momentum.
No comments:
Post a Comment