Microsoft patents AI that changes what people say in live video in real time

What you could do with it

• A viewer could watch a live foreign-language broadcast and hear and see the speaker in their own language through real-time translation ¹.
• A broadcaster could use the system to scrub filler words like 'um' and 'ah,' reduce stutters, and fix mispronounced or misspoken words from a live stream ¹.
• A sports or awards-show producer could let viewers filter out offensive, insensitive, or inappropriate words while automatically syncing the speaker's face and mouth to match ¹.

How it would work

The patent, first flagged by the tracking site Patentlyze, describes a "near real-time artificial intelligence enhancement of video or augmented reality presentations of live events" ¹. An AI agent would watch a live broadcast and identify words it was told to change, then alter or remove them while editing the speaker's face and mouth so the new audio matches what you see .

The system would also edit the video and audio just before and after any removed or replaced word to create "smooth transitions" ¹. Those instructions could come from set parameters or from a viewer's personal profile, meaning two people watching the same live event could end up seeing two different versions .

What the patent says it could do

Microsoft's own patent sketch out several possible uses. One is real-time translation: a viewer could watch a foreign-language broadcast and hear and see the speaker in their own language ¹. Another is cleaning up speech — removing filler words like "um" or "ah," reducing stutters, or correcting mispronounced and misspoken words .

Some of the examples go much further. The patent mentions cutting "personal words or phrases" such as "colloquial terms" like the Southern American expression "y'all" or the idiom "bite the bullet" ¹. It also calls out the possibility of removing "mumbled words, offensive words, insensitive words, or inappropriate words" . When a word needs replacing, the patent notes the replacement could even be pulled from an "internet search" .

Concerns and safeguards

The idea of a live broadcast no longer being live raises obvious questions. If viewers can tune someone else's speech to their personal taste, do we lose shared reference points? Does erasing regional dialects or idioms flatten how people communicate online?

To address those worries, the patent does acknowledge the risk and says the original live video and audio could be recorded and stored as a "ground truth" — a backup copy of what actually happened, kept so people can check whether any editing took place ¹. The patent envisions the technology being applied to lectures, presentations, television productions including awards shows and game or cooking shows, and live sports .

As of now, Microsoft has not confirmed or denied any details beyond what is in the patent filing and did not respond to a request for comment ¹.

This story reports on a patent filing for information only. It is not advice or encouragement to use or implement any such technology.

Key facts

• Microsoft filed a patent for real-time AI video editing that changes speakers' words and lip movements.
• The system could translate live broadcasts, remove filler words, and even erase dialect expressions or idioms.
• The original broadcast could be stored as a "ground truth" record to preserve what actually happened.
• Potential uses include lectures, TV shows, awards ceremonies, and live sports.

Sources

1. Microsoft Patents an AI That Can Literally Put Words in People's Mouths in Real Time — futurism.com