Amazon uses dead grandma of child in morbid demo of Alexa audio deepfake

amazon echo dot gen 4
enlarge The 4th generation Amazon Echo Dot smart speaker.


Amazon is figuring out how its Alexa voice assistant can deepfake anyone’s voice, dead or alive, with just a short recording. The company demonstrated the feature on its re:Mars conference in Las Vegas on Wednesday, using the emotional trauma of the ongoing pandemic and grief to sell interest.

Amazon’s re:Mars focuses on artificial intelligence, machine learning, robotics and other emerging technologies, with technical experts and industry leaders on stage. During the keynote on the second day, Rohit Prasad, senior vice president and chief scientist of Alexa AI at Amazon, showed a feature being developed for Alexa.

In the demo, a kid asks Alexa, “Can Grandma be done reading to me?” Wizard of OzAlexa replies, “Okay,” in her typical effeminate, robotic voice. But then the child’s grandmother’s voice comes out of the loudspeaker to read L. Frank Baum’s story.

You can watch the demo below:

Amazon re: MARS 2022 – Day 2 – Keynote.

Prasad only said Amazon is “working” on the Alexa capability and didn’t specify what work remains and when/if it will be available.

He did, however, provide meticulous technical details.

“This required an invention where we had to learn to produce a high-quality voice in less than a minute of recording versus hours of recording in a studio,” he said. “The way we managed to do it is by formulating the problem as a voice conversion task and not a speech generation task.”

Prasad briefly discussed how the feature works.
enlarge Prasad briefly discussed how the feature works.

Of course, deepfaking has built up a controversial reputation. Still, some effort has been made to use the technology as a tool rather than a tool for creepiness.

Audio deepfakes specific, as noted by The edgehave been used in the media to make amends when, say, a podcaster messes up a line or when a project’s star dies suddenly, as happened with the Anthony Bourdain documentary road runner

There are even cases where people use AI to create chatbots who work to communicate as if they were a lost loved one, the publication noted.

Alexa wouldn’t even be the first consumer product to use deepfake audio to stand in for a family member who can’t be there in person. The Takara Tomy smart speaker, as indicated by Gizmodouses AI to read bedtime stories for kids with a parent’s voice. Parents reportedly upload their voice, so to speak, by reading a script for about 15 minutes. Although this is notably different from Amazon’s demo, as the owner of the product decides to provide their vocals, rather than the product using the voice of someone who probably can’t give permission.

In addition to concerns that deepfakes are being used for scams, rip offand others nefarious activitythere are already some disturbing things about how Amazon is designing the feature, which doesn’t even have a release date yet.

Before showing the demo, Prasad talked about Alexa giving users a “companionship.”

“In this role of companionship, human qualities of empathy and affection are key to building trust,” said the director. “These traits have become even more important in these times of the ongoing pandemic, when so many of us have lost someone we love. While AI cannot take away that pain of loss, it can certainly make their memories lasting.”

Prasad added that the feature “enables lasting personal relationships”.

It is true that countless people are seriously seeking human “empathy and affect” in response to emotional stress caused by the COVID-19 pandemic. However, Amazon’s AI voice assistant is not the place to meet those human needs. Alexa also cannot enable “lasting personal relationships” with people who are no longer with us.

It’s not hard to believe that there are good intentions behind this evolving feature and that hearing the voice of someone you miss can be a great comfort. We might even find ourselves having fun with a feature like this, theoretically. Getting Alexa to make a friend sound like they said something stupid is harmless. And as we discussed above, there are other companies that are leveraging deepfake technology in a way similar to what Amazon showed.

But to frame an evolving Alexa capability as a way to rekindle a bond with deceased relatives is a giant, unrealistic, problematic leap. Meanwhile, it feels pointless to pull hearts by bringing in pandemic-related sadness and loneliness. There are some places Amazon doesn’t belong, and grief counseling is one of them.

Leave a Comment

Your email address will not be published.