In 2017, what do benchmark numbers even mean??

Update, March 2017: This post has been updated with information on the latest phones and benchmark techniques.

When it's time for Samsung to show us a new phone, talk about the hardware inevitably brings up the subject of benchmarks. The Galaxy S8, Samsung's showcase phone for 2017, is no different. And as expected, just by existing these numbers got plenty of people talking about them.

Some conversation about benchmarks is just idle chatter. "Oh, cool! The Snapdragon calculates the "stuff" in a benchmark application about as well as the Exynos" is fun to talk about. it's an interesting conversation that leads to talk about how a new phone can meet or exceed our expectations because it is using state of the art hardware to do cool things. That's why most of us are here, to talk about things that run Android and how we can use them to enrich our lives.

But some folks get serious about benchmark numbers and consider them an important part of a buying decision. We all should encourage this because it's always great to have people excited about something, but we should also talk about what benchmark numbers really mean in the grand scheme of things. One way to try and put things into perspective is to compare the benchmarks of a pair of new processors that Android manufacturers will be buying to Apple's latest.

This doesn't mean that an iPhone using the A10 is automatically a better experience than a Galaxy S8 will be. There's so much more that determines which is better, and the majority of it is user preference. you like what you like and I like what I like. Numbers in a tweet won't change that and the numbers don't mean what you might think they mean.

How those numbers come to be

Benchmarks on mobile phones aren't really benchmarking any hardware, at least not the way we think they are. They don't have access to the hardware itself because they are using the operating system's application layer. They have a laundry list of things they have the phone do through the APIs exposed by the operating system, then they calculate how well it did them. There is an intermediary layer of a sort to go through to get to the "brains" behind the operating system, which is the part that controls the hardware directly. So a benchmark app is benchmarking the hardware through some software.

You might have heard iOS people talking about Metal or Android people talking about an NDK. These are ways applications can interface with the hardware, through that intermediate layer without having to go through the full software stack. Notice that the benchmark numbers for Samsung's Exynos 8895 and Qualcomm's Snapdragon 835 are very close to each other. They both use the same software and the performance differences between them are minimized because of it.

Apple's "intermediary layer" is better. Let's throw that out there right where we all can see it. Apple builds its own processor with a focus on doing certain things really well then builds software that takes advantage of it. Google has to build software that can be adapted to work with anything. It's done an amazing job and the software that powers an Android phone is a beautiful thing that's incredibly complicated. Something like a benchmarking app using Apple's interface to the hardware automatically has an advantage over Android, no matter who built it, because the interface itself is more streamlined and "faster" with iOS. That happens because the hardware and the software were designed to work with each other and nothing else.

You're benchmarking the phone as a whole, not just the processor. When it comes to crunching numbers on each CPU core the iPhone 7 Plus does it a lot better.

Let's look at those cores in Apple's A10 processor. That thing is undeniably the best consumer ARM chip ever designed when it comes to raw performance per core. That's because the hardware was designed to do just that and the software was designed to use it. We've talked about ARM architecture before, and the A10 is a great example of how you can scale ARM to do just about anything you want. So are the Qualcomm 835 and the Exynos 8895, they just were designed with different criteria in mind.

The difference in benchmark numbers isn't an accident

We compare them because they all are inside a phone, but Apple is thirsty to build one ARM processor that can power an iPhone, an iPad, and a MacBook. Qualcomm and Samsung build processors to sell to other companies for small mobile devices. Qualcomm and Samsung could build a processor that excels in the same areas as the A10 and would work great for a Windows laptop. Qualcomm is actually interested in doing it and the Snapdragon 835 is the company's first step towards that goal.

A couple years from now and we'll see a Snapdragon chip that can be worked hard enough to run a full-blown laptop and still be efficient enough to be used in a mobile device with a tiny battery. We'll also see more chips that aren't as powerful, are even more efficient when it comes to battery use and are a lot cheaper. These will be the CPUs that companies who make phones will buy.

When you take a tool designed to only do certain things in a certain order and see how "fast" they can be done, the A10 will always win. It should always win, and we should want it to always win. A CPU designed for a 13-inch MacBook needs to perform single core calculations faster than an Exynos 8895. The A10 isn't that CPU, but it is a step in that direction. And Apple is a tech company that we should want to do really cool things to drive tech forward just like we want Samsung or Google or Microsoft to do.

The little snip of a benchmark scoreboard that has no context you see above shows the most important thing: These numbers have little bearing on how great something like a phone is to use. The user experience has little to do with the hardware because the hardware has been good enough for a while now. The innards of a Galaxy S5 or Nexus 7 or Note 4 are more than enough to do the things we expect a phone to do as long as the software is up to snuff. You don't have to take my word on that, just stumble over to XDA where people who don't want or can't afford to buy something newer have built custom software for each. We're not asking a phone to do anything complicated enough to need more processing power than these devices can deliver.

I'm convinced even mobile VR would be fine if companies cared enough to support Vulkan correctly on their older processors. We'll never know because the companies involved exist to make new things and sell them to us and that's where they focus their time and money. New chips aren't just designed to be new. They all offer small incremental increases in performance, security and efficiency and those small increases add up over time. Right now Moore's Law isn't focusing on quadrupling performance in every generation, it's focusing on using better manufacturing techniques to provide more energy efficient chips and performance gains are just natural evolution.

What we really see from these benchmarks

What we can take away from these benchmark scores is that the way a CPU core calculates things and works with GPU cores isn't broken. Numbers can be crunched a little faster with newer hardware that was designed to be more energy efficient. The way a CPU core can crunch a number is no longer the bottleneck, so these small differences and increases won't be noticed when you're not running a benchmark application. Newer hardware might be better than last year's, and one processor might be better than another. The performance increases are real, but they don't translate into a noticeable difference when you are using them and won't unless you skip a few generations. Going from a Qualcomm S4 Pro to a Qualcomm 835 brings a bump in performance that you will notice right away. Going from a Qualcomm 821 to a Qualcomm 835 doesn't.

The Galaxy S8 will bring a user experience that is better than last year's Galaxy S7. Many of us here will consider it a better experience than Apple offers with the iPhone 7 while many will feel the opposite. None of this is because of a benchmark score.