Welcome to the age of Soli. Google has shown us how the Soli Sensor will do a couple of neat tricks in the Pixel 4, and while waving your hand like a Jedi to change what song is playing is pretty self-explanatory (and honestly, not that exciting), using it in tandem with dedicated hardware to unlock your phone with your face is something we haven't seen before.
This isn't the face unlocking you're used to seeing.
Facial unlocking, however, is something we've seen before. First debuting on Android 4.0 Ice Cream Sandwich with the Galaxy Nexus, using your phone's selfie camera — or, in newer iterations, infrared cameras— to unlock your phone is an incredible experience. It's also fairly fast, easy to use, and until Apple developed Face ID on the iPhone X, extremely insecure. Face unlock that only uses a camera to capture data on a two-dimensional plane is easy to fool with a photo. That made it more of a convenience feature, one even Google said shouldn't be used if you value security.
I've previously written about the tech that drives Face ID; those same words apply to Google's implementation on the Pixel 4. That's not surprising as it's well-established technology that can map and recreate a 3D image of basically anything. The only amount of detail captured is only limited by how long you want to wait for the results; the more data captured, and the more time allowed to process it, the better and more thorough the results can be. But most people don't want to wait that long to unlock their phones, which is why companies need to compromise between speed and security.
A quick refresher on how Face ID works: using a special LED that "washes" your face in infrared and near-infrared light, along with a special component that projects a matrix of contrasting IR "dots" on your face, cameras collect everything in front of it and build a 3D map. This map is then converted into a special token based on the shapes present (usually a face, or part of it) and the token is used to check against the data stored inside a secure element in the phone. If the two signatures match, the phone unlocks.
That means the token can do more than unlock your phone. Apple has integrated Face ID to replace the fingerprint sensor, and Google says the facial mapping from the Pixel 4 will also be able to authenticate you as needed, too. So far, Face ID and Google's facial unlocking as seen on the Pixel 4 seem to be very much the same animal. A quick look at the hardware, though, and we see Google has integrated a lot more into that bezel — and it's all courtesy of the Soli Sensor.
How Soli works
Saying that the Soli Sensor is tightly integrated with facial unlocking and gesture recognition isn't just speculation. Google makes it very clear that the two aren't separate the Soli chip is a major component of the whole system. From the blog post:
Unlocking your phone should be easy, fast, and secure. Your device should be able to recognize you—and only you—without any fuss. Face unlock may be a familiar feature for smartphones, but we're engineering it differently.Other phones require you to lift the device all the way up, pose in a certain way, wait for it to unlock, and then swipe to get to the homescreen. Pixel 4 does all of that in a much more streamlined way. As you reach for Pixel 4, Soli proactively turns on the face unlock sensors, recognizing that you may want to unlock your phone. If the face unlock sensors and algorithms recognize you, the phone will open as you pick it up, all in one motion. Better yet, face unlock works in almost any orientation—even if you're holding it upside down—and you can use it for secure payments and app authentication too.
This works because of what Soli is — a small and power-efficient RADAR transceiver that has a very fine resolution. That means it can detect a thing that's moving and notice very fine details in real-time versus being able to detect a thing that's very fast with less detail.
I've seen Soli in action and it can easily and reliably detect the slightest tapping of a finger, or the movement of fingers acting as if they're turning a miniature dial or knob. While these were controlled demos under the very best conditions, noticing a moving phone approaching the face should be easy to pull off. You don't need a fine resolution to do that and a simple proximity sensor does something similar.
Another big difference between Apple and Google's implementation of 3D facial recognition is that Google is using two face unlock IR cameras. These cameras are what reads the projected IR matrix and turns the contours and curves into a secure token. Apple only uses one of these cameras in Face ID. There could be several reasons for this: Apple is using better parts, Google is building a stereo image, or maybe a second sensor is needed so that the phone can work when turned 180-degrees on its head. Likely, all three of these things are true. It also solves the issue of unlocking the phone while it's sitting on a table, something Face ID still struggles with.
Having two face unlock cameras collecting information will make the system faster and more reliable.
Google calls these "face unlock IR cameras" but they are also used for any motion recognition the Pixel 4 will support. To cover everything happening in front of the phone, having a camera on each side makes perfect sense. You'll never be able to be perfectly aligned in the front dead center of a camera on your phone when you wave your hand to perform an action. A single infrared camera is able to pick up most of your motions, no matter how far away from center they may be, but adding a second can do a lot to make the system more reliable. And unless it's reliable, Google's Motion Sense is nothing more than a gimmick.
The same applies to facial unlocking, especially any "smart" algorithms Google has working to get better at recognizing facial features. Two cameras can make a stereoscopic view of a facial map and collect a lot more data, or they can work with each other to double-check for accuracy, making facial unlocking faster and less able to be fooled. Having two cameras also means that when the system starts "looking" for your face to see if it should prepare to unlock, it can do so much earlier because it doesn't have to wait for your face to be closer to the center.
Finally, because the tech has matured enough to find its way into a small battery-powered device like a phone, there's a good chance these "face unlock IR cameras" aren't as expensive as the ones Apple sourced for the iPhone X. A less expensive sensor that can't work quite as fast or has a lower resolution is perfectly fine if you're adding a second one to the mix.
Motion Sense
Google says that the new Motion Sense feature of the Pixel 4 will be able to "skip songs, snooze alarms, and silence phone calls, just by waving your hand", Admittedly, that's pretty cool but recognizing a waving hand isn't something we haven't seen before.
Besides the LG G8, we can go back to the Moto X to see hand gestures in action. Using a set of sensors placed in the front of the phone, a simple wave of the have was recognized even when the screen was off. Google's ATAP (Advanced Technology and Projects) team was born with Motorola and are responsible for Motion Sense so they have experience here.
Motion Sense won't be using the hardware in the Pixel 4 to its full extent because there is no need to draw a three-dimensional map of your hand each time to want to use it. But it will be using the Soli chip and some smart algorithms. That means what we've seen so far from Google is probably only the beginning.
With actual fine-detail RADAR to sense very intricate movement, the only limiting factor of Motion Sense will be digesting the information it collects. Once algorithms are able to accurately identify other gestures I expect Google to build them into Android on some level and hopefully open an API so third-party developers can get data from the Soli Sensor.
How accurate Motion Sense will be and how often it sees a "false positive" where a fluttering curtain or your cat triggers a gesture remains to be seen. Like any facial recognition, expect bumps in the road.
But will it suck?
Probably not. But it all depends on what you're expecting.
Current face unlock solutions on Android usually rely on the front-facing camera, which only captures only a 2D image. It's fast but also easy to fool, and it's why Google doesn't let device makers use it as a way to pay for things or do anything other than unlock your phone. Other solutions, like Samsung's now-defunct iris scanner or Huawei's spatial-mapping solution on the Mate 20 Pro, are similar to Face ID and Google's upcoming solution because they're considerably more secure.
They're not just matching your face to a known image — they're creating and reading a spatial map.
Everything that's new goes through a phase where it sucks.
What that means is the "image" does a lot more than say, "Hey, this is Jerry. Go ahead and unlock his phone," when it sees my face or a photo of my face. Modulated acquisition of spatial distortion maps — a technical term that means "checking out all the bumps, curves, and lines of a thing and saving all the data about what you see" — takes time. Comparing the data collected to a known good sample also takes time. It's also an order of magnitude more secure than looking at and parsing the photo of a face.
Like Face ID in its beginnings, there will be bumps in the road. Google is trying to stave off one of them by collecting facial scans of people on the street because this sort of tech has a tough time with people of color. That's something an AI can learn to work around, and hopefully Google scanning all those faces helps make it happen reasonably quickly. Other issues will be ambient lighting, detection of cosmetic changes like plucked eyebrows of shaving your mustache, and last but not least, glasses. All of these problems are "easily" sorted out, but then the fixes have to be done in a way that doesn't make the system less secure.
This tech isn't that difficult, and like Face ID on the iPhone, it will soon become second nature.
My prediction is that Reddit will explode with how bad face unlock on the Pixel 4 is about a week after the phone gets into people's hands, then slowly simmer down as these problems get addressed in software patches. Once algorithms are tuned and sensors are dialed in, other manufacturers can look int incorporating the tech into their own Android models.
At first it may be bad. All tech is. But secure facial mapping has a solid background behind it and is used every day by countless other companies for many other purposes. Tuning it to work on a power-limited handheld device that reads your face won't be an insurmountable problem.
