In 2017, what do benchmark numbers even mean??

By Jerry Hildenbrand
last updated

Update, March 2017: This post has been updated with information on the latest phones and benchmark techniques.

When it's time for Samsung to show us a new phone, talk about the hardware inevitably brings up the subject of benchmarks. The Galaxy S8, Samsung's showcase phone for 2017, is no different. And as expected, just by existing these numbers got plenty of people talking about them.

The numbers are in, but what do they mean?

Some conversation about benchmarks is just idle chatter. "Oh, cool! The Snapdragon calculates the "stuff" in a benchmark application about as well as the Exynos" is fun to talk about. it's an interesting conversation that leads to talk about how a new phone can meet or exceed our expectations because it is using state of the art hardware to do cool things. That's why most of us are here, to talk about things that run Android and how we can use them to enrich our lives.

But some folks get serious about benchmark numbers and consider them an important part of a buying decision. We all should encourage this because it's always great to have people excited about something, but we should also talk about what benchmark numbers really mean in the grand scheme of things. One way to try and put things into perspective is to compare the benchmarks of a pair of new processors that Android manufacturers will be buying to Apple's latest.

See more

This doesn't mean that an iPhone using the A10 is automatically a better experience than a Galaxy S8 will be. There's so much more that determines which is better, and the majority of it is user preference. you like what you like and I like what I like. Numbers in a tweet won't change that and the numbers don't mean what you might think they mean.

How those numbers come to be

Benchmarks on mobile phones aren't really benchmarking any hardware, at least not the way we think they are. They don't have access to the hardware itself because they are using the operating system's application layer. They have a laundry list of things they have the phone do through the APIs exposed by the operating system, then they calculate how well it did them. There is an intermediary layer of a sort to go through to get to the "brains" behind the operating system, which is the part that controls the hardware directly. So a benchmark app is benchmarking the hardware through some software.

You might have heard iOS people talking about Metal or Android people talking about an NDK. These are ways applications can interface with the hardware, through that intermediate layer without having to go through the full software stack. Notice that the benchmark numbers for Samsung's Exynos 8895 and Qualcomm's Snapdragon 835 are very close to each other. They both use the same software and the performance differences between them are minimized because of it.

If you buy a phone because you like to run benchmarks, you should probably buy an iPhone.

Apple's "intermediary layer" is better. Let's throw that out there right where we all can see it. Apple builds its own processor with a focus on doing certain things really well then builds software that takes advantage of it. Google has to build software that can be adapted to work with anything. It's done an amazing job and the software that powers an Android phone is a beautiful thing that's incredibly complicated. Something like a benchmarking app using Apple's interface to the hardware automatically has an advantage over Android, no matter who built it, because the interface itself is more streamlined and "faster" with iOS. That happens because the hardware and the software were designed to work with each other and nothing else.

You're benchmarking the phone as a whole, not just the processor. When it comes to crunching numbers on each CPU core the iPhone 7 Plus does it a lot better.

Let's look at those cores in Apple's A10 processor. That thing is undeniably the best consumer ARM chip ever designed when it comes to raw performance per core. That's because the hardware was designed to do just that and the software was designed to use it. We've talked about ARM architecture before, and the A10 is a great example of how you can scale ARM to do just about anything you want. So are the Qualcomm 835 and the Exynos 8895, they just were designed with different criteria in mind.

The difference in benchmark numbers isn't an accident

We compare them because they all are inside a phone, but Apple is thirsty to build one ARM processor that can power an iPhone, an iPad, and a MacBook. Qualcomm and Samsung build processors to sell to other companies for small mobile devices. Qualcomm and Samsung could build a processor that excels in the same areas as the A10 and would work great for a Windows laptop. Qualcomm is actually interested in doing it and the Snapdragon 835 is the company's first step towards that goal.

A couple years from now and we'll see a Snapdragon chip that can be worked hard enough to run a full-blown laptop and still be efficient enough to be used in a mobile device with a tiny battery. We'll also see more chips that aren't as powerful, are even more efficient when it comes to battery use and are a lot cheaper. These will be the CPUs that companies who make phones will buy.

When you take a tool designed to only do certain things in a certain order and see how "fast" they can be done, the A10 will always win. It should always win, and we should want it to always win. A CPU designed for a 13-inch MacBook needs to perform single core calculations faster than an Exynos 8895. The A10 isn't that CPU, but it is a step in that direction. And Apple is a tech company that we should want to do really cool things to drive tech forward just like we want Samsung or Google or Microsoft to do.

Qualcomm or Samsung could build an ARM processor that is as powerful as the A10, but they have no reason to do it.

The little snip of a benchmark scoreboard that has no context you see above shows the most important thing: These numbers have little bearing on how great something like a phone is to use. The user experience has little to do with the hardware because the hardware has been good enough for a while now. The innards of a Galaxy S5 or Nexus 7 or Note 4 are more than enough to do the things we expect a phone to do as long as the software is up to snuff. You don't have to take my word on that, just stumble over to XDA where people who don't want or can't afford to buy something newer have built custom software for each. We're not asking a phone to do anything complicated enough to need more processing power than these devices can deliver.

I'm convinced even mobile VR would be fine if companies cared enough to support Vulkan correctly on their older processors. We'll never know because the companies involved exist to make new things and sell them to us and that's where they focus their time and money. New chips aren't just designed to be new. They all offer small incremental increases in performance, security and efficiency and those small increases add up over time. Right now Moore's Law isn't focusing on quadrupling performance in every generation, it's focusing on using better manufacturing techniques to provide more energy efficient chips and performance gains are just natural evolution.

What we really see from these benchmarks

What we can take away from these benchmark scores is that the way a CPU core calculates things and works with GPU cores isn't broken. Numbers can be crunched a little faster with newer hardware that was designed to be more energy efficient. The way a CPU core can crunch a number is no longer the bottleneck, so these small differences and increases won't be noticed when you're not running a benchmark application. Newer hardware might be better than last year's, and one processor might be better than another. The performance increases are real, but they don't translate into a noticeable difference when you are using them and won't unless you skip a few generations. Going from a Qualcomm S4 Pro to a Qualcomm 835 brings a bump in performance that you will notice right away. Going from a Qualcomm 821 to a Qualcomm 835 doesn't.

The Galaxy S8 will bring a user experience that is better than last year's Galaxy S7. Many of us here will consider it a better experience than Apple offers with the iPhone 7 while many will feel the opposite. None of this is because of a benchmark score.

Jerry Hildenbrand
Jerry Hildenbrand
Senior Editor — Google Ecosystem

Jerry is an amateur woodworker and struggling shade tree mechanic. There's nothing he can't take apart, but many things he can't reassemble. You'll find him writing and speaking his loud opinion on Android Central and occasionally on Twitter.

42 Comments
  • You have taken your punishment with aplomb.
  • I understood it all.... except that stuff after "I pulled a dumbass stunt"
  • I understood it, up until the word "Recently..."
    But I read it cause Jerry wrote it!
  • I understood it all and I liked yours dumbass stunt.
  • Lol. Nice article, have a good rest of the weekend.
  • Stop making so much sense Jerry 😉
  • Exactly! He should start making dollars instead. Right?
  • Jerry, will you be purchasing a Galaxy S8 or an LG G6 or will you be sticking to your trusty Pixel and Pixel XL?
  • He's a tech reviewer so he'll probably use both as his daily driver at some point.
  • My phone use is as follows, YouTube videos, Chromecasting music and videos, browsing the internet, Facebook, Twitter, Google+, Instagram, taking pictures and recoding video, emails and other clerical stuff. Phones like the Nexus 6, S6, Moto F4 Plus, Google Pixel and the Moto X (I own all these phones) easily gets the job done for me. I'm not too concerned about benchmarks these days. I think we're passed that stage at this point.
  • Moto F4 Plus eh sounds menacing.
  • My old Note 10.1 from 2012 still does most of what I use an Android tablet for with little lag, Web browsing has gotten slow but still works great for notes, sketching, and video consumption. I have an S7 for my daily driver, and I still have an old Note 3, S7 is quicker, but the Note 3 will still run any app or game I want without issue. Benchmarks mean little to me as well at this point, mobile just doesn't need much processing power.
  • I run benchmarks when I make changes to the kernel or change ROMs. I don't think they are useful in determining whether I would enjoy one phone over another - or even one ROM over another.
  • This couldn't have been more beautifully put. Dude is a wizard. 👍
  • You know what really grinds my gears? When people keep using battery benchmark scores to compare battery performance instead of using real-world examples.
  • You have to have some basis to get an idea of longevity, unless you personally purchase each handset and then conduct your own battery testing. I like to check out GSM Arena and CNET, while giving a passing look at Android Authority's seemingly less consistent testing, and just a passing glance at Phonearena's basically useless battery tests. I then compare their results as an aid to assess the probable battery life of different handsets, both stand alone and in comparison to one another. Like I said, unless you go buy every new handset, you have to form a basis, and get an idea, somewhere.
  • True. but I get tired when people just use that and not use any real-world examples to back up their claims.
  • The only usage for the battery is "real world usage. Any test that tries to emulate that is not really reliable. When you live with it, and use it,
    you find out if it claims are accurate. I don't judge a phone on test, I judge battery usage, when someone uses it on the street. I hope real world usage, doesn't go away from reviews of phones, it's the most accurate gauge of the phone.
  • The problem with this is, it is virtually impossible to replicate tests unless it is in a laboratory setting.
  • You rely on lab settings too much. Using the phone for at least 2 weeks, shows operational problems/features/peculiarities of it. Apple has the quickest hardware, but butt OS. That can't be replicated by lab test.
  • Well, in order to get replicable results, they need to be tested in an environment where settings remain constant and the given values don't change. I guess what I'm saying is, I've had BlackBerrys for the last 10 years. I am currently using a Samsung Galaxy S3 because my Note 4 is down with a broken screen. I have found that my BlackBerrys have had consistently stronger cell signals than either of my Samsungs have. My brother currently uses a Moto Z, and his signal strength is worse than mine. Now, real-world "testing" shows my BlackBerrys had stronger service than my Android devices, but how was the weather? Do I hold my Android devices different than how I held my BlackBerry devices. I'm sure there are dozens - if not hundreds - of variables that can be eliminated through bench-top laboratory testing. I fully understand that lab testing shows one processor "out-performs" the other, however marginal the results, but in real life, it may appear the other way around. At least with a lab test, there are no variables. Plus, if one is an Android "fan", odds are, said person cares not for how an iPhone performs. I'm one of those people.
  • Hope xda editors pick up on this article, it says a lot, in easy to understand "for us all" people. This may be enough to keep the apple trolls away.
  • A great article. But still, the iPhone has a buttery smoothness that still trumps the newest and purest Android experience.
  • The Pixel is arguably the closest, though.
  • Interesting, my Axon 7 is buttery smooth, it never hangs period.
  • I've think most phones nowadays are smooth enough, especially if you have bought one within the last year or so. If you went on benchmarks alone you would think my Lenovo P2 would lag all the time, yet it never has any problems running the apps I use.
  • Most phones are smooth. With better chips every year, they're all good. And ive seen tons of iPhone lags and being slow over time. Alot of them.
  • I have an iPhone 7 plus and you are correct. It stutters every now and then just like any other phone, in fact, more often than not, it will hang on an app causing me to reboot. Before anyone says "it's not the phone, it's the app", let me tell you Apple vets all apps and updates before it's released. So if that's the case, then why does it happen?
  • Interesting. I've personally never had an iPhone but whenever I've used one, it's always so slick! Android is almost there though.
  • Wrong on the iPhone, i still own a couple of gens, even the iPad Air that's a couple of years old has never lag! Stop making assumptions if you don't own them
  • And for obvious reasons as set forth quite well in the article.
  • Well said Jerry, but you still didn't tell me if it's a single core or multi core benchmark😉.
  • The benchmark he showed is a single-core.
  • Just like 3DMARK, benchmarks are useless tbh unless you like to see pointless numbers. Always test it in real world scenarios, multi apps loading, switching between them, gaming for 30min on intensive games, see who throttles fast and who is more steady, ram management etc...
  • Thank you Jerry for possibly the best short explanation I have read on what benchmarks really are and why they don't matter(very much).
  • Consider your penance served Jerry. 😉 By the way, is there any truth to the theory that multicore performance is more important on Android than on iOS?
  • Nope. Single core performance is more important. The entire user interface is a single thread that calls other single threaded processes. Plenty of apps and games are spread across multiple cores, but getting to them and opening them is all done with one core.
  • Jerry speaks the absolute truth. Storage speed is also important. When you have fast I/O, everything from bootup to app launch speeds and even taking burst photos become faster because your NAND controller is able to process all the data much faster. Storage controller tech like UFS and NVMe often take a backseat on the spec sheet behind processors, but they are very critical to overall system performance, maybe even more so now as processors have become extremely powerful.
  • "Apple's intermediary layer is better. Let's throw that out there right where we all can see it. Apple builds its own processor and its own software so that they work really well together. Google has to build software that can be adapted to work with anything."
    - This is nonsense. If this were an actual factor, Macs would be much faster than PCs. In fact, similarly spec'ed Windows, Mac, and Linux machines routinely perform relatively closely. The core problem here is Google doesn't have a standard hardware spec for Android in the same way PCs have x86. Doing so would allow Android to be optimized for all devices from AOSP instead of at the OEM level.
  • Speaking of nonsense ... (makes me sound like a dick doesn't it? That works both ways) It never fails that someone equates thermally constrained and battery-powered appliances with PCs. By your logic we should expect the same performance from a Galaxy S7 that we see from the new Chromebook since they both use Linux on the same CPU.
  • Jerry: Is there a comparison between Snapdragon 400 and Snapdragon Wear 2100 SOC?
    I'm very curious to know if the 25%efficiency gains claimed by Qualcomm is true or not...
    Cheers
  • While I appreciate the attempt, your article COMPLETELY misses the point about benchmarks, and why they matter. It's about relatively. Nobody gives a crap about the numbers until you compare them to another set of numbers. It's all about one phone versus another, versus another and so on. How well can this phone run this task compared to this phone, etc. People want to know how fast a particular phone is RELATIVE to another. And as long as they benchmarking software is the same, the benchmarks DO matter.