Galaxy Note 7 benchmarks and the nerds who love to hate them
Despite going through a massive recall at the moment, the Galaxy Note 7 is still one of the top phones on the market. And there's something deep in the heart of every smartphone nerd that twinkles every time they own a phone that someone with any measure of authority calls "the best" and defends that statement in a way they agree with.
Note 7 fans had that feeling taken away from them recently by the smart folks at XDA-Developers. Through a combination of "real world" tests, benchmarks, and system monitoring tools the XDA writers told a tale of stuttering performance that seemed to demonstrate this shiny new phone was, in some ways, not deserving the space at the top of the heap so many tech reviewers has placed it.
It's a fun read, especially if you only kind of understand what you're reading. Before you consider flinging your Note 7 back into the store it came from with that homemade catapult, there's a few things you should know about benchmarks and "real world" tests.
Benchmark apps are pointless
If you use a benchmark app to tell you how great or terrible your phone performs, you're not getting anything anywhere near a complete picture. Benchmark apps used to be great for troubleshooting, existing so you could see if there had been some kind of performance loss over time or damage to your equipment. Today, many manufacturers implement special code that forces that hardware to perform above the typical thresholds when a benchmark app is being used, which irreparably alters the results. You're measuring a version of the hardware your phone otherwise never lets you have access to, so you gain nothing but an attempt at the highest score when you run benchmark apps today.
Those performance altering software changes lead to a much bigger issue. When you are comparing a Galaxy Note 7 to a Moto Z Droid Edition and a Nexus 6P, you're comparing three different experiences that will never be doing the same thing at the same time. Samsung and Lenovo both have background tasks that can't be replicated on the Nexus 6P. Features that can't be disabled to get a 1:1 compare of the software performance. If you see a Note 7 performing anywhere near as smoothly as a Nexus 6P, consider how many more things that Note 7 is doing. Better yet, take a look at the immeasurably more thorough Anandtech review of the Note 7 performance as it compares to all other high performing phones, and see how it regularly outpaces the Nexus 6P.
There's value in testing for things like dropped frames, and reporting on those dropped frames in context is an important thing to do when your goal is to educate and inform potential buyers. It's hard to say that's what happened with the presentation from XDA, given the lack of context or proper comparison. Does the Note 7 drop more or less frames than the Galaxy S7 or S7 Edge? Could this be an issue exclusive to the Snapdragon variant of this model? Is this happening because Samsung's new Grace UI was rushed out and could be fixed in a future update? None of these questions are answered, because the goal wasn't to inform.
Cherry picking in the "real world"
Not long after XDA published their findings, a post from The Verge's Vlad Savov attempted to apply some reality to the initial Internet explosion. In this Vladitorial, it was pointed out that some of the findings on XDA weren't really findings. Specifically, claiming that a 200ms difference in launching apps was an example of "embarrassing performance" is silly and not representative of how people actually use smartphones. XDA fired back with more data, trying to fortify their position. The counter argument, that a 200ms delay would add up over time to a phone in such a way that a Note 7 user would grab a Nexus 6P and be amazed at how much faster it "feels" is enforced by GIFs demonstrating those performance differences.
What's fascinating about this demonstration is the cherry picking. Samsung's software is far from perfect; in fact, after uninstalling 13 apps from my shiny new Verizon Note 7 and finding eight more that I can't disable I kind of want to smash mine with a hammer right now. Sitting that phone next to my freshly Nougat-filled Nexus 6P, there are absolutely aspects of this Note 7 that are demonstrably faster. The camera, for example, routinely launches a full second faster. It wouldn't be difficult to cherry pick half a dozen examples, GIF them up, and write something that looks like evidence of how superior the Note 7 is to the Nexus 6P.
The thing is, that's not how real world testing works. The point of real world testing, as the name suggests, is to offer performance examples of how the whole phone functions as though an "average" user is going to use the phone. Using that data to compare one experience to another is tricky, instead of offering a 1:1 compare of software as it functions you're offering a 1:1 compare of the experience, what using the phones feels like. Showing how a share menu loads, especially when those phones are clearly not set up the same way with the same apps, doesn't really fit that description. Unless, of course, you're pushing a different narrative.
Better tools and actual analysis
The real flaw in all of this is a mistaken assumption that Samsung's Note 7 and Google's Nexus 6P are built for the same purpose. Plain and simple, they aren't. Google's Nexus experience has evolved from clean Developer Kit to a demonstration of the Android Open Source Project with a handful of extras. Samsung's phones haven't started with AOSP in a long time, and there's no reason for the company to consider another path right now.
Samsung phones are Android-based, not Android. As a result, Samsung phones are optimized in whatever way they deem most important. Right now those optimizations are for delivering unique Samsung features, like Samsung's camera, Samsung Pay and the unique S Pen functions. Android, by which I mean the OS, doesn't place priority on those things. In recent releases there's been a focus on things like battery consumption when you aren't using the phone, security at all times, and a consistent 60FPS user interface. It's difficult to argue that any of these things aren't important, but neither Samsung's Android nor Google's Android places a priority on all of these things.
Really, what this comes down to is what you as the user place priority. If you want the most secure phone with a slick, unburdened interface, you probably want the thing Google and HTC are going to be announcing soon. If you care about a great camera and software built to offer you a ton of feature you may or may not use, Samsung's Galaxy Note 7 is tough to beat right now. Trying to claim the Note 7 is somehow underperforming because it doesn't behave like a phone it wasn't built to behave like is ridiculous, no matter how you tightly you try to wrap that narrative in benchmarks.
Get the Android Central Newsletter
Instant access to breaking news, the hottest reviews, great deals and helpful tips.
If a 850$ phone has to struggle to provide comparable performance to a 400$ OP3, I find it really disturbing...
-laggy performance. phone should not have lag. they did not optimize the software at all.
-screen is way to soft. scratchs like a plastic display
-bad design of a screen. screen protectors dont fit right, small drops break the phone multiple reports on that. XDA shown proof the phone is subpar. They shown its laggy and then AC does damage control for Samsung. There is no reason the phone should be so laggy. First year Android devs can bake faster roms with more features and still use crappy touchwiz. The N7 is a flop. Out of the box its the laggyest phone you can buy. And now we know for a fact they used the cheapest li-ion/li-po charge protection ic's money could not buy and 3 phones are causing fires a day. Samsung lost me when they locked down harder then Apple. Now they are reselling the older phone with qa few gimmicks like Apple but unlike Apple they are cutting down on quality to the point the devices are extremely unsafe. I wont ever use another Samsung phone for the pure fact i do not want to burn to death when im sleeping. Good job Samsung cant wait to see Apple use this against you
the note was never made to be a bare bones light software smart phone,
it is and always was firstly a productivity and utility device , hence the s pen, the 2k screen, wireless charging, and the many convenience functions the note has that the one plus has not.
The note has 10 sensors for example, and just like the hamburger with the lot hardware, there is a lot of software, in a lot of these stupid tests you don't see a lot of the cool features integrated into the note software..
Sadly some of these dudes sooking about the price can't fathom just how powerful the note series of devices are , and being powerful does now mean opening an app a tenth of a second faster .
So, we can assume that the new Nexus/Pixel will show faster than iPhone 6s / plus and probably iPhone 7.
Last Android I saw beat iPhone in this particular Phonebuff testing, was HTC m8 Note 7 is shown ridiculously slow there - minute late vs iPhone and around 30s slower than s7 exynos https://youtu.be/9AY5tRg1Wo8 https://youtu.be/P3kTDeHUk4Q Just realized that N6 and Note 4 (sd805) both did better than iPhone 6.
Speaking of OP3 - since Moto Z still has a very clean ROM I expected similar results. Somehow Lenovo managed to get the same score as Moto XPE.
For example I Watched a fellow doing the finger print test on n7 and it looked slow , I watched a similar test by another bloke , and the note was so bloody fast, I doubt I have seen anything so fast, I mean it was bloody lightning.
And I'm a serious phone geek, I watch all these phone tests.
And you often see the same inconsistency in these tests despite what numbers say.
As for performance for a 900 dollar Device , from what I have seen the n7 is every bit as fast as my G5 for the most part and faster in some areas, yet it has the same hardware with more features to put load on it ,
yes I would be happy with it .
enerally see the same results on comparisons that are performed practically and similarly. I've done plenty as well, more between the 6s and android than anything else lately, and the results I've gotten have been spot on
Long live the note 4, it shall echo throughout eternity
The S series flagship= mass sales. Also,a Note series device should never have less battery than anything in the S series, excluding the very niche active model, and the glass should not be prone to easily scratching. GG4>GG5.
2. There can still be value in cherry picked examples of poor performance if a user can relate those particular examples to their own daily workflows and they amount to more of an impact to the user than anything else. Yes I do get the point that there are clearly other features or actions that are omitted in their article that work faster or more efficiently than other devices but to turn the argument around - dismissing one viewpoint on performance by saying it cherry picked it's metrics by picking other examples where results would sway things the other way could also be perceived as cherry picking depending upon the perspective of the reader who may have different opinions on what measures are important to them.
3. As for the share menu argument, it doesn't matter how different phones are set up. If a user shares frequently and accesses the menu many times a day for various reasons then the issue is still valid. Would it be a wash if the most frequent workflow for the user involved using the faster camera app? Surely, but if said user has the propensity to detect lag and isolate it from the entire experience then it won't matter to them how much faster it was for them to get to the point at which they initiated the share menu, the lag would be noticed and an opinion drawn. That is the type of user of XDA that the article targeted. Separating how one gets to the point of sharing or what happens after they make a selection from the menu can be relevant for that type of user. Again target audience - how data is presented is something writers choose based upon their own perceptions and opinions on what is important to the target audience. It's up to the readers to take everything in and evaluate for themselves. I get the point of this and Vlad's article, to point out that there are areas where the device outshines in performance and that there are differences in software design that make comparisons difficult, but I have a hard time dismissing the comparisons completely if I find value in both the additional software features Samsung provides and the areas within which perceived lag are going to affect my daily workflows. I'm not sure that I want to forgive Samsung for menu lag or app switching delay just because they provide me with a richer or different experience in the software features side - not when it comes at a much higher price point than ever before.
https://www.youtube.com/watch?v=v4O8_JbaoMU The 2nd one has been good so far, but it's only a little over a month old, so only time will tell how good it is 6 months from now.
The N7 has been good. Would I say it's better than the S7E...nope and that's just because I'm not a Spen user. I only got the phone, because I wanted
black and I missed out on the Injustice edition S7E. Do I care that the phone cost $100 more than the S7E again nope. I knew the phone was coming out, saved my money for it, paid full price for it and went home. Got what I wanted no contract no regrets. Now I get to take it back and get a new one + a $25 phone credit.
Performance is a subjective thing, and Speed is only one metic of performance. Camera quality is performance. Being waterproof is performance. Having the the best display is performance. Many hate it, but i think touch wiz has come into its own with some some great features and add to the experience of the phone as well, not to mention the numerous other features. An iris scanner is performance etc... Of course a phone that doesn't have those top notch features might be slightly faster, but i wouldn't trade slightly more speed for loosing all those great features.
Samsung also got a raw deal on this recal. This is apparently a battery manufacturer's problem, out of the control of Samsung. Only so much testing can be done before product is released to the masses to see what issues may arise over time. I do believe the phone is grossly overpriced but if people pay that much for the phone the manufacture is going to sell it for that price (obviously the case with the iPhone). That's only common sense. Stop buying the phone and the prices will drop.
Don't buy the gdamn phone... hahah wow