Have you heard the one about the guy who got excited about the benchmark performance of the untested processor in an unconfirmed environment that is headed for an unannounced phone that couldn't possibly be ready for production and official testing just yet? It's a real knee slapper, you'll love it.
It's that time of year, when we start getting that steady trickle of information regarding phones we'll hear about in an official capacity over the next month or two. For obvious reasons, the Samsung Galaxy S7 is high on the list of things people want to hear more about. That leads to a whole lot of unconfirmed reports, blurrycam leaks, and wringing of hands about what it all means. Above all that we have benchmarks, the most useless information we could get about unreleased hardware with brand new processors.
Truth is most smartphone benchmarks are pointless, but early unfinished and unofficial benchmarks are doubly so. Here's a quick look at why.
We are nowhere near seeing finished products yet
Any benchmarks we're going to see pop up over the next couple of weeks are coming from two places. We're either seeing the Samsung-made Snapdragon 820 developer boards or unfinished prototypes of the Samsung Galaxy S7. Neither of these is going to give you anywhere near the results you'll get running the benchmarking apps on your retail unit when it comes time to purchase the phone. These pre-release benchmark numbers have never matched the results off of release units, and they never will.
Plain and simple, these results tell you nothing.
Developer boards frequently don't have to account for being on battery power, or needing to be aware of things like temperature. These kits are built for developers to quickly optimize their software for the new processing architectures or underlying software tools Qualcomm is providing this year for things like better image handling in cameras or better audio support. Benchmarks on developer board don't even really give you a broad idea of what these chips are capable of, because often times the OS that runs on them isn't particularly well optimized to begin with. Plain and simple, these results tell you nothing.
Pre-production units are also not indicative of the real-world experience. These versions of phones are frequently running unfinished software, or are missing bloaty apps like security software that slow down phones just by running in the background. Prototype benchmarks often are closer to reality than dev kits, but still far enough away from the real world experience that they aren't worth taking seriously.
In short, until we see numbers on retail units, just say no.
Benchmarks frequently are manipulated anyway
Over the last couple of years we've seen report after report from benchmark app makers of manufacturers getting a little shady with the way their phones interact with these performance rating apps. We've seen special code that increases processor clock speed only when a benchmark is running, leading to numbers that are nowhere near the real world experience for these phones. Samsung is no angel here — and same goes for others — it's happened on more than one occasion and for whatever reason many users don't seem to care.
Why get excited about benchmarks when there are repeat cases of manufacturers writing code to boost their numbers in a way that doesn't reflect real-world usage on these phones? We all know the whole reason most users run benchmarks is to compete with the other phones out there as though their phones are in some kind of weird digital strongman competition, but what's the point when the information isn't real?
All that really matters is how the phone — whether it's the Galaxy S7 or something else — performs when it is in your hand doing the things you want it to do, and if benchmarking apps can't accurately measure this because the manufacturers are fudging the results then we need better ways to measure performance in the real world.
The software you install as you use the phone matters
Seeing benchmarks that someone else posts from their Galaxy S7, even when it's a retail version and even if there's no software to fudge results, still isn't going to accurately tell you what the experience is going to be like once you've moved in to your own Galaxy S7. A phone fresh out of the box with nothing but the stock apps is going to perform differently once you've loaded all of your apps and started using them. App cache, running processes, and even accessing data from external storage will all have a bearing on the overall performance of your phone.
Benchmarks show you what things will look like when trying to push the phone to its limits.
This doesn't mean benchmarks under similar-enough conditions are totally worthless, it just means you still aren't really seeing the whole picture. And you won't, as there really isn't a way for benchmarks to give you a good idea of how your phone is going to behave once it is in your hand and being used by you. What you get from benchmarks, when they are from retail devices without performance-altering software, is a blurry look at the ceiling of potential performance ability. You can kind of see what things will look like when trying to push the phone to its limits, which is only really going to happen with resource-intensive games or having fun with the camera settings. You don't really get a feel for much of anything else.
None of this is to say you shouldn't get excited about the Samsung Galaxy S7. We saw a lot of impressive things from Samsung over the last year, and if the rumors of three phones heading to the world with this branding all at once turn out to be true there's going to be even more reason to pay attention to how these phones perform so you can best determine which experience is the one for you. What it does mean is benchmarks aren't going to offer you the information you want in any reliable way, and the best thing to do right now is wait for these phones to start making their way to people who can show you how these phones behave in the real world.