At the beginning of every month, Google releases the monthly Android Security Bulletin and starts to send updates to Pixel phones. It's great that the company is transparent about what is going on and how things are being fixed even if you're not the type of person who likes to read the source code.
There is a lot of work that goes into these patches before they are made public, and there is even more work involved before it comes to other phones — if it comes at all. Let's take a look at how the sausage is made and try to have a better understanding of why the timeline for security patches is a little blurry.
First you fix Android
Android is a complicated beast. Over 5 million lines of code, it exists to help companies that make mobile products get up and running with a complete application platform including access to Google Play and other services. It's not something that can be used as-is; these companies spend a lot of time trying to get Android tailored to merge into the other software they may be using to create a nice homogenized operating system.
Google has some rules about how this should be done should a company want to include its services, but manufacturers have a long leash on how the final product is built.
This code is where a security patch comes to life. Someone, be it a security researcher or just an average Joe, finds a flaw in a phone that could be used to lessen the device's security layer. If that flaw isn't something an OEM created, the Android team is tasked to find out what's happening, why it's happening, and how to fix it in the least disruptive way.
Often, the flaw isn't something Google can fix. Like us, Google doesn't have access to firmware from companies that make hardware like Qualcomm or LG. If the flaw needs to be addressed at the hardware level there is a good chance the company that supplies some of the components used will need to make changes first. If this is the case, those changes are forwarded to Google so that it can see what needs to be done to accommodate them in Android's code.
These changes take time, especially if a hardware vendor is involved. There is patching and testing and more patching and more testing for each and every flaw addressed in a patch. Once Google is confident that they have a valid fix for a security flaw, every company who makes Android phones is given early access (at least 30 days before the patch is made public by Google) so they can get to work.
This is where most of the work is done. Google may write and maintain Android itself, but the bulk of devices that use it are not made by Google. The ones that are — Pixel phones — are also included here. Google hardware is a customer of Android the same way Samsung or Motorola is.
All of these companies get to work on a couple of things as soon as they have new code from Google. The first — and possibly most important — part is determining what part of the patch is not needed. And there are plenty of things in every patch a single company can freely ignore.
For example, if NVIDIA had to make changes that are pushed back into Android, no Samsung phones will need that part of the patch. A more extreme example would be the changes that BlackBerry or Samsung made that already address the problem in a different way. Finding out what's needed and what isn't can be time-consuming, especially when a company makes large changes to certain portions of the operating system. Google investigated accusations that OEMs were sending security patches that did not address some things they should have, and this is what it found.
Once that's done, the rest of the patch needs to be merged into a vendor's custom Android code, then built and tested. The "built and tested" part can become a big headache if the patch can't just be applied because it touches files that custom code is using or depends on. We see that a lot, too. Whenever Bluetooth or Wi-Fi is patched, whether it be the hardware or the software behind them, it will touch code that has been altered by a large OEM that makes a fancier operating system than "stock" Android. There are a lot of parts of Android that an OEM can touch.
Once the engineers at Samsung or another vendor get an operating system that boots up and runs, it needs to be tested. And tested some more. The testing may include getting network engineers from various carriers involved, as well as having Google and/or the manufacturer of any component back into the mix. It has to be right. A patch sent out to thousands and thousands of phones could potentially cripple a carrier's network, eat up every user's data cap, or even cause the phone itself to stop working. Anything of the sort is unacceptable and has to be found before it leaves the building.
The company that made your phone, Google, and maybe your carrier work together to get a mass over-the-air update ready. If you've ever seen the URL that is used to download a patch, you'll notice it has "Google" in the web address. That's because the engine inside your phone that can fetch and process an OTA update is looking in a very specific place for a patch. It needs to know that the patch is 100% correct and signed by the right digital signature. It will check this again once the patch is fully downloaded.
Your carrier may have some rules about when and who can download a patch once it's live if their name is on the phone. Companies like Samsung or LG make custom versions of their most popular models for each carrier, which has plenty of input into how things are done. It should since its name is on the box. This can be frustrating, but it makes sense. If everyone in Pittsburgh (for example) who has a Samsung Galaxy S8 phone tries to fetch an 800MB patch at the same time, the network is going to crumble in spots. Your carrier will do anything it needs to do in order to keep the network alive.
Google also places a sort of hold on OTA rollouts. A specific number of users will receive a patch, and after a set amount of time, Google determines if those users had a good experience or a bad one. If all goes well, a larger number of users will get the patch in a second wave. This repeats several times before the floodgates are opened. Users who do not wish to wait for this final testing can manually download a patch through their device settings.
When it's your turn and you gave your phone the green light to grab that file, it's downloaded and then your phone takes control.
In your hands
A patch is downloaded to your phone and verified as being the right stuff. Older versions of Android have a dedicated cache, which is a section of your storage that has been divided off for things like an update file to live; things that are only temporarily on the phone. Phones that use Android's seamless update feature (which should be most phones running Android Nougat when sold) "slip" the downloaded files into what are called slots. In either case, you need to have enough space for the OTA file to be extracted and worked on.
The OTA updater software in your phone is a part of Android. A script in the downloaded file tells it how to go about finding the files that need altered and it copies those files either into your device cache or into the designated slot. It then compares the original files on your phone with the files that have been downloaded. Some may be a simple swap — take file X from the phone and delete it, then replace it with file X from the OTA download. Others aren't the full file and only contain small specific changes. The updater and installer software in your phone know what to do here.
Many files in Android, especially the applications and software libraries, are really a lot of files compressed into a special archive. You can take an APK file and change it to a .zip file and open it with Windows. Sometimes these archives need to be opened and portions of them need to be swapped with new versions downloaded for the security patch. That's why you need that working space in your cache partition — that's where these files are extracted.
Once every file in the OTA update has been processed and changes made to copies of system files, it's time to run the system with them. This happens when the phone asks you to reboot after it processes the OTA you received because there are often files that need to be patched but are in use while the phone is running. You may see a screen showing that there is work going on during the reboot or you may just see the Android logo. In either case, files are being checked, moved into place, and checked again. The old files are kept in the cache just in case there is a problem and you can't boot with the new files.
All that's left is for you to make sure everything is still just how you like it, and you have a newer date for the Security Patch version in the settings of your phone. Now you're ready for the next update!
How much easier would it be for manufacturers to update if they ran "stock" ish skins and additional features where downloaded as apps?
No kidding, tho I suppose that would put a crimp in that whole "different-but-together" business. Still considering how much modifying Samsung does, I've been wondering if they aren't a big part of the reason android P is being locked down from third-party theming. I do know that a number of Substratum themes specifically targeted Samsung builds, and I have no problem believing that Samsung customer service was fielding lots of questions about phones acting erratically that they eventually discovered were fundamentally theming-related issues.
I think about this a lot. Probably too much. Anyhoo, here's the only way I can ever see the Android update mess being fixed: Google removes all system apps from AOSP. This means dialer, sms, browser, everything. AOSP is then just an application framework. Google allows OEMs to fork AOSP and changes the compatibility testing suite so that it only considers how Google services work. Tell OEMs exactly which services are required for Google Apps to be provided and give exact standards that must be met surrounding them. That means that Google Play Services is also changed so that it becomes a full compatibility layer to ensure that every app in Google Play will work. Google will never do this. Ever. And if they did, Samsung and LG would never, ever be able to meet the compatibility testing standards. So my whole idea sucks.
Yep. Will probably never happen. :)
True Jerry, that will probably never happen, but I think Google is making some serious strides in changing the upgradability of Android. The Google Play Services system was a beginning shot, and we now have Project Treble. What follows next will bring close than ever to your dream.
Wouldn't one solution be - less choice? OEMs would still need to be motivated to do these updates. And I think the only Android program that calls out updates is Android Enterprise Recommended. https://forums.androidcentral.com/general-news-discussion/893412-enterpr... BTW Jerry - loved that although you mentioned carriers, it's clear in your words that OEMs need to put in the work before we even get to a point where a carrier could "block" an update. With less choice, it's less work. Take Essential for example. They have essentially one device. (oh puns!) It could appear easier for that team to scale to have resources to review, develop, and test these updates. But look at Samsung and their 50 smartphone SKUs in 2018. Not to mention the 120 smartphone SKUs that they would probably still be supporting from the last two years.
What a mess. If you care about your security, it sounds like it's really best to stick with Pixels to get timely updates as tempting as some of the other smartphone hardware is.
It's the nature of Android's openness.
The problem with Pixels is their availability, and of course they are not necessarily the best on everything.
Mr. Jerry Hildenbrand - Thank you. Another excellent article. There's way more that goes into Android Security Updates than most people, even most "fans" of Android think. Question for you - your own personal preference of course - but if Jerry LLC made a JerryPhone based on Android and say the Android Security Bulletin for July 2018 had no changes applicable for JerryPhone, would you increment JerryPhone's security patch level to July 2018 or leave it at June 2018?
He would definitely leave that supercool JerryPhone showing the true June 2018 security patch level. What could maybe be good though is to (using the same security update channel) deliver a calming message stating that there is no need for security patch for this particular device this month. This would make the phone owner/user to relax and know that he has not been ignored, and would be a committed statement to Google that there was no need for patching for that month.
I'd have a place where customers can opt-in and receive timely developer notes via email or text and not worry about a date. Keep every phone up to date. Don't bother people who aren't concerned about it. People who are concerned can have a history and a road map of all planned software changes.
Love reading your articles Jerry, informative as always.
Nice insight into how Android security updates work.
Get the best of Android Central in in your inbox, every day!
Thank you for signing up to Android Central. You will receive a verification email shortly.
There was a problem. Please refresh the page and try again.