Now that the Google Pixel 2 and Pixel 2 XL phones are official, the “Now Playing” feature has gotten a lot of attention. Music playing in the background is automatically detected and identified on your lock screen. It basically takes Shazam to the next level. Google claims that the feature can work offline and won't send anything to the cloud, which is a sensitive topic after the Google Home Mini's “spying” issue came to light. Most importantly, the “Now Playing” feature won't drain your battery as it utilizes a proven technology, AmbientSense.
The new phones have a Pixel Ambient Services application, also available on the Google Play Store, which was the first clue that the “Now Playing” feature could take advantage of it. However, the pre-installed APK in /system/priv-app, called AmbientSense, all but confirmed it. A research paper from the 2013 IEEE International Conference on Pervasive Computing and Communications Workshops first described AmbientSense as a “real-time ambient sound recognition system on a smartphone.”
AmbientSense can be used as an Android app and only needs access to the phone's microphone to function. The research paper describes two different implementations, autonomous and server mode. Autonomous mode is strictly based on the phone itself and is limited to identifying audio based on locally stored data. Server mode sends information to a server that identifies the audio and sends it back to the phone, but this can't occur offline. Google is apparently using Autonomous mode as they claim that it works offline and nothing is sent to the cloud.
Performance testing of AmbientSense goes back several years, with the paper describing work on a Samsung Galaxy S II and original Google Nexus One. Recognition performance, runtime, CPU load and recognition delay were tested in a set of 23 ambient sound classes, using both autonomous and server modes. The feature ran for up to 13.75 hours on the Galaxy S II and up to 12.87 hours on the Nexus One, both archaic smartphones by today's standards. The Nexus One launched in 2010 with a 1,400 mAh battery, so AmbientSense performance on modern devices will crush these older figures.
Can the “Now Playing” feature be ported to a non-Pixel 2 device? XDA thinks it's possible, starting with older Pixel and Nexus phones. First and most importantly, the phones must be rooted. Then Pixel Ambient Services (AmbientSense.apk), the audio matching database and various missing libraries must be loaded, along with some systemUI modifications to ambient display.
They already have an audio matching database, a 53MB storage library based on Google's LevelDB called “matcher.leveldb.” It's the same database used by AmbientSense in autonomous mode. Other issues with getting older ambient displays to cooperate and implementing missing libraries need to be worked out, but getting “Now Playing” working on older Android phones is just a matter of time.
An interesting update from Google came in today (10/16) in which they claim that “Now Playing” is actually not based on AmbientSense, which definitely creates some confusion. XDA is waiting for a response after sending Google some additional questions.
Source: XDA
