Here’s how Android 10’s Live Caption actually works

By Kamran Zia -

Google IO 2019 Sundar Pichai stage live caption

Live Caption is one of the coolest Android features yet, using on-device machine learning to generate captions for local videos and web clips.

Google has published a blog post detailing exactly how this nifty feature works, and it actually consists of three on-device machine learning models, for starters.

There’s a recurrent neural network sequence transduction (RNN-T) model for speech recognition itself, but Google is also using a recurrent neural network for predicting punctuation.

The third on-device machine learning model is a convolutional neural network (CNN) for sound events, such as birds chirping, people clapping, and music. Google says this third machine learning model is derived from its work on the Live Transcribe accessibility app, which is able to transcribe speech and sound events.

Reducing the impact of Live Caption

The company says it’s taken a number of measures to reduce Live Caption’s battery consumption and performance demands. For one, the full automatic speech recognition (ASR) engine only runs when speech is actually detected, as opposed to constantly running in the background.

“For example, when music is detected and speech is not present in the audio stream, the [MUSIC] label will appear on screen, and the ASR model will be unloaded. The ASR model is only loaded back into memory when speech is present in the audio stream again,” Google explains in its blog post.

Editor's Pick

Pixel 4 Dual Exposure Controls, Live HDR won’t come to Google Pixel 3, 3a

The Google Pixel 4 packs quite a few neat camera features, including an astrophotography mode, Dual Exposure Controls, and Live HDR+ previews. We already know that the former feature is coming to the Pixel 3 …

Google has also used techniques such as neural connection pruning (cutting down the size of the speech model), reducing power consumption by 50% and allowing Live Caption to run continuously.

Google explains that the speech recognition results are updated a few times each second as the caption is formed, but punctuation prediction is different. The search giant says it delivers punctuation prediction “on the tail of the text from the most recently recognized sentence” in order to reduce resource demands.

Live Caption is now available in the Google Pixel 4 series, and Google says it’ll be available “soon” on the Pixel 3 series and other devices. The company says it’s also working on support for other languages and better support for multi-speaker content.

More posts about Android 10

Comments

Post a Comment

Popular posts from this blog

Woman clawed by zoo jaguar while trying to take selfie, learns valuable lesson

By Kamran Zia -
Image
Slate A woman’s trip to the Wildlife World Zoo, Aquarium, and Safari Park in Arizona was cut short when she was clawed by a captive jaguar (via CBS News ). The woman had climbed over a concrete barrier to get closer to the animal to snap a selfie with her smartphone. The woman was admitted to the hospital with her arm in “pretty bad shape,” according to medical responders. However, she was released shortly after with a series of stitches. According to witnesses on the scene — one of which captured a video of the woman writhing in pain after the jaguar clawed at her arm — the incident began when the unidentified woman climbed over a federally-regulated concrete wall that helps to keep a safe distance between the animal cage and the spectators. Once over the wall, the woman turned her back to the cage to snap the selfie. At some point shortly thereafter, the jaguar reached its two front paws outside the cage and grabbed the woman’s arm. The woman then tried to wrench it out of the
Read more

The 12 best TV shows on Netflix you can binge right now

By Kamran Zia -
Image
Looking for the best shows on Netflix can be hard given the huge selection the streaming service offers. This is why we have created a list of our favorite ones and are ready to share it with you! Netflix gave many critically acclaimed TV shows from regular broadcast and cable networks a second life. Some of the best television shows ever made are now available for binge viewing on the service. In addition, they have a healthy portfolio of original shows and mini series for your viewing pleasure. Best shows on Netflix: Breaking Bad Dracula Stranger Things Mad Men The Office (US) Twin Peaks The Witcher The Walking Dead Parks and Recreation The Good Place Black Mirror Russian Doll Editor’s note : This list of the best shows on Netflix will be updated as TV series leave and others make their debut on Netflix. 1. Breaking Bad Of all of the older shows streaming on Netflix, this series may have benefited the most. Breaking Bad is about a high-school chemistry t
Read more

10 best adult apps for Android (NSFW)

By Kamran Zia -
Image
The adult entertainment industry has come a long way in the last 20 years. Before, all you had were VHS tapes with videos that looked like they were shot in a mafioso’s basement with magazines that you had to ask actual people to get from behind the counter to give you. The horror. These days you can stream and download all the pornography you want, usually for free and in HD, and no one’s the wiser. With tech making it easier than ever to Netflix and chill yourself, it’s only natural that you’ve stumbled upon this page looking for the best porn apps for Android. Please note, if you’re under the legal age in your country, please leave. This isn’t for you. Also, since most of these apps aren’t available in the Play Store, it’s highly recommended that you exercise caution. Let’s get started! Don’t forget to protect your phone against potential problems with the best antivirus apps! Incognito mode gets you almost unlimited adult content. Check out the best Android browsers now! Apt
Read more