Galaxy Note 7 benchmarks and the nerds who love to hate them

Despite going through a massive recall at the moment, the Galaxy Note 7 is still one of the top phones on the market. And there's something deep in the heart of every smartphone nerd that twinkles every time they own a phone that someone with any measure of authority calls "the best" and defends that statement in a way they agree with.

Note 7 fans had that feeling taken away from them recently by the smart folks at XDA-Developers. Through a combination of "real world" tests, benchmarks, and system monitoring tools the XDA writers told a tale of stuttering performance that seemed to demonstrate this shiny new phone was, in some ways, not deserving the space at the top of the heap so many tech reviewers has placed it.

It's a fun read, especially if you only kind of understand what you're reading. Before you consider flinging your Note 7 back into the store it came from with that homemade catapult, there's a few things you should know about benchmarks and "real world" tests.

Benchmark apps are pointless

If you use a benchmark app to tell you how great or terrible your phone performs, you're not getting anything anywhere near a complete picture. Benchmark apps used to be great for troubleshooting, existing so you could see if there had been some kind of performance loss over time or damage to your equipment. Today, many manufacturers implement special code that forces that hardware to perform above the typical thresholds when a benchmark app is being used, which irreparably alters the results. You're measuring a version of the hardware your phone otherwise never lets you have access to, so you gain nothing but an attempt at the highest score when you run benchmark apps today.

Those performance altering software changes lead to a much bigger issue. When you are comparing a Galaxy Note 7 to a Moto Z Droid Edition and a Nexus 6P, you're comparing three different experiences that will never be doing the same thing at the same time. Samsung and Lenovo both have background tasks that can't be replicated on the Nexus 6P. Features that can't be disabled to get a 1:1 compare of the software performance. If you see a Note 7 performing anywhere near as smoothly as a Nexus 6P, consider how many more things that Note 7 is doing. Better yet, take a look at the immeasurably more thorough Anandtech review of the Note 7 performance as it compares to all other high performing phones, and see how it regularly outpaces the Nexus 6P.

There's value in testing for things like dropped frames, and reporting on those dropped frames in context is an important thing to do when your goal is to educate and inform potential buyers. It's hard to say that's what happened with the presentation from XDA, given the lack of context or proper comparison. Does the Note 7 drop more or less frames than the Galaxy S7 or S7 Edge? Could this be an issue exclusive to the Snapdragon variant of this model? Is this happening because Samsung's new Grace UI was rushed out and could be fixed in a future update? None of these questions are answered, because the goal wasn't to inform.

Cherry picking in the "real world"

Not long after XDA published their findings, a post from The Verge's Vlad Savov attempted to apply some reality to the initial Internet explosion. In this Vladitorial, it was pointed out that some of the findings on XDA weren't really findings. Specifically, claiming that a 200ms difference in launching apps was an example of "embarrassing performance" is silly and not representative of how people actually use smartphones. XDA fired back with more data, trying to fortify their position. The counter argument, that a 200ms delay would add up over time to a phone in such a way that a Note 7 user would grab a Nexus 6P and be amazed at how much faster it "feels" is enforced by GIFs demonstrating those performance differences.

What's fascinating about this demonstration is the cherry picking. Samsung's software is far from perfect; in fact, after uninstalling 13 apps from my shiny new Verizon Note 7 and finding eight more that I can't disable I kind of want to smash mine with a hammer right now. Sitting that phone next to my freshly Nougat-filled Nexus 6P, there are absolutely aspects of this Note 7 that are demonstrably faster. The camera, for example, routinely launches a full second faster. It wouldn't be difficult to cherry pick half a dozen examples, GIF them up, and write something that looks like evidence of how superior the Note 7 is to the Nexus 6P.

The thing is, that's not how real world testing works. The point of real world testing, as the name suggests, is to offer performance examples of how the whole phone functions as though an "average" user is going to use the phone. Using that data to compare one experience to another is tricky, instead of offering a 1:1 compare of software as it functions you're offering a 1:1 compare of the experience, what using the phones feels like. Showing how a share menu loads, especially when those phones are clearly not set up the same way with the same apps, doesn't really fit that description. Unless, of course, you're pushing a different narrative.

Better tools and actual analysis

The real flaw in all of this is a mistaken assumption that Samsung's Note 7 and Google's Nexus 6P are built for the same purpose. Plain and simple, they aren't. Google's Nexus experience has evolved from clean Developer Kit to a demonstration of the Android Open Source Project with a handful of extras. Samsung's phones haven't started with AOSP in a long time, and there's no reason for the company to consider another path right now.

Samsung phones are Android-based, not Android. As a result, Samsung phones are optimized in whatever way they deem most important. Right now those optimizations are for delivering unique Samsung features, like Samsung's camera, Samsung Pay and the unique S Pen functions. Android, by which I mean the OS, doesn't place priority on those things. In recent releases there's been a focus on things like battery consumption when you aren't using the phone, security at all times, and a consistent 60FPS user interface. It's difficult to argue that any of these things aren't important, but neither Samsung's Android nor Google's Android places a priority on all of these things.

Really, what this comes down to is what you as the user place priority. If you want the most secure phone with a slick, unburdened interface, you probably want the thing Google and HTC are going to be announcing soon. If you care about a great camera and software built to offer you a ton of feature you may or may not use, Samsung's Galaxy Note 7 is tough to beat right now. Trying to claim the Note 7 is somehow underperforming because it doesn't behave like a phone it wasn't built to behave like is ridiculous, no matter how you tightly you try to wrap that narrative in benchmarks.