Should we hate the player? Or hate the game? Both, probably.

With today's shocking revelation that it's possible to cheat Android benchmark apps, it's worth a quick reminder that we've always been able to manipulate Android benchmarks. And, frankly, it shouldn't come as any surprise that manufacturers do everything they can to score as high as possible.

I quite fondly recall sitting in a Hell's Kitchen bar in 2011 2010, just hours after the launch event for the original Samsung Galaxy S. I had a brand new Motorola Droid X in my pocket — announced just a week prior — and (now former) Samsung engineers asking to see how it benchmarked. Not knowing better at the time, I happily obliged. 

Here's the thing: We have only ourselves to blame for this nonsense.

Android sites — ourselves included for some time — have long made a big deal about benchmarks. And here's why: They're easy to do. Fire up an app and a video camera, and watch the fun begin. "It benchmarks 9,000!!!" is a joke we still make today, long after Android Central stopped worrying about benchmarks.

Anyhoo. Or own Jerry Hildenbrand back in May 2011 walked us through just how easy it is to manipulate benchmark apps through software — never mind what Samsung's doing now, optimizing for specific apps. Can you really blame Samsung, though? Fans and bloggers make a big deal about benchmarks. Why wouldn't the manufacturers do everything they can to score as high as possible?

Where it gets shady, of course, is that Samsung is controlling the CPU for specific apps, and not exactly in a transparent manner.

"They should have a checkbox in the (operating system level) settings that says 'Run in full throttle,' " said Marcus Adoflsson, founder and CEO of Mobile Nations.

I'm inclined to agree. But these aren't Samsung's apps. It'd behoove the app developers to perhaps point out just how the hardware is interacting with the software. Using more cores than with other apps? Tell me.

But then again, if someone's going to game the system we helped create, we shouldn't act so surprised.

 

Reader comments

Cheating at Android benchmarks isn't new

86 Comments

"It benchmarks 9000!"

That quote warms the cockles of my heart, lol.

On topic: I'm not sure why, benchmarks still even matter, to be honest.

Posted via Android Central App

They don't. Neither do most of the other specs that we get but to some individuals they are all they go by.

I mean look at Tantai, he degrades the X just because of them without ever having touched the device.

At least when I hate on the Nexus 4 I have very tangible reasons, storage being the biggest. If the N4 would have went to 32-64GB, the rest of the OEMs might have followed. But instead it was another year of 8-16-32. I also need 32 at a minimum (yes I know I am odd in that respect) so I had my very real reasons

One of my biggest hates on Apple is the lack of NFC a few years ago that would have thrown payment systems into the forefront. Everyone would have scrambled to get it together to accommodate iCrud (yeah I hate Apple for a ton of other reasons but that was the last one. I stopped really paying too much attention after that)

Cores and clock speed do not matter all that much but somehow it is a huge deal. Architecture of those cores is another story but that is starting to hit a flat line as well.

If you quote a benchmark as your reason for buying a phone, your doing it wrong...

I need more coffee

If I may politely disagree, I actually happen to think that it makes sense to benchmark Android phones in particular. The diversity of hardware and software is enormous; today you can buy phones with 800MHz single-core processors and 8-core 2.3GHz processors. With market-distorting subsidies in place, most of which are not understood by the average person buying a phone, the cost difference between the two is remarkably small, and frankly, you could easily end up spending $200 and get stuck in a 2-year contract for a phone that is a completely obsolete piece of crap.

Thus, benchmarks, when run under fair and reasonable conditions, provide a good way to compare what kind of raw performance you can expect to get from the phone. I have no problem with Samsung or anyone else permitting locking the processors at the highest possible settings - provided that's how everything else is run as well. What Samsung did here was cheating, without a shred of doubt.

It's marketing boys. And marketing is has nothing to do with anything, but misleading the uninformed. I applaud AC for largely ignoring benchmarks and specs and focusing on the experience of using the devices. And writing pieces like this.

In another industry (that I used to work in) the same things have been done for decades. I used to work at HI-FI audio/video shops. And in the audio world, "the spec", the be-all-end-all spec, that most consumers focus on is watts (or watts per channel).

I see cheap, very cheap, stereo/audio gear boasting 200 watts, 400 watts, etc etc etc. It's been that way forever. And as even as you climb up the quality ladder a number of rungs, sadly, that spec gimmick remains the same. Because the manufacturers know that to the uninformed (read: most) consumers, more watts equal more sales and more profits.

Until you climb into the really expensive categories of equipment... and then the marketing gimmicks change (and the really creative snake oil marketing begins, and that's another story).

It may not be new, but that doesn't make it okay.

Why should we be concerned with the numbers anyway? Is it really that imperative that your device score higher on a test than another device?

I think it's about time we stop obsessing over specifications and benchmarks and start pushing OEMs to truly optimize perfectly capable "mid-range" (and top-of-the-line) hardware to provide a wonderful Android experience. Doing this would also lower the cost of our phones. Think about that.

This. THIS. The specs and benchmarks race needs to come to a screeching halt. I think we've reached a point, where Android is lean enough to be optimized for what is now considered mid-range specs. The Moto X proves this.

Posted via Android Central App

Yes. The moto x performs just as well as my HTC one and in some cases better. Some serious software optimizations.

Posted via Android Central App

Exactly. The MOTO X proves that two cores are enough and shedding central processor loads to specialized auxiliary processors is far more effective than adding cores that you have to turn off most of the time to keep the processor from melting.

Its not software optimizations. Its offloading to special processors. I'd point you to a Cnet article entitled "Top Motorola engineer defends Moto X specs (Q&A)" but posting a link is a hanging offense around here.

Got it. I remember you guys mentioning that, in your review. I just wasn't sure, if
most of the performance enhancements came from optimizing the software, offloading specific tasks to specific cores, or both. But, I appreciate the reply, Phil!

P. S: From some of my own posts (without links) being fried by the spam filter, I know it's not perfect, lol.

Posted via Android Central App

the spam filter won't let you type numbers either as it triggers the spam filters I have found, like entering a monetary unit of numbers, or trying to type four dot two dot two for jellybean

Just a mild correction: the Galaxy S and Droid X were released in 2010, not 2011. I vividly recall getting my X in July 2010.

Ooooooo, you're right. Can't believe it was that long ago. Fixing that (and clarifying Marcus' quote at his request.)

We greatly regret the error. :p

My roommate's mom once asked him how to get more "sheeps" in her computer. It took him like 20 minutes to realize she was asking about RAM. :D

If you follow the first link in Phil's post, read it carefully, and think about it for a minute, you come to two conclusions.

1) The power saving algorithms that Samsung (and most others) use impose a HUGE performance penalty on the devices. Turning the power saving features off yields a 20% performance boost. 4 cores are too power hungry to use. You can only afford to use them briefly.

2)Most apps can't benefit from 4 cores. Having another core standing ready means that the background mail check can be handled without slowing down your video game, your web browsing or Gmail or Calendar. None of these will be any faster with 4 cores.

The take away is:
A) By suppressing the use of 4 cores, except for fleetingly brief instants of time, (or any time a benchmark is running) Samsung has been proving Qualcomm's point: 2 Cores are better than 4.

B) Bringing a sleeping core up to full speed is no where near as instantaneous as we have been lead to believe. Its 20 percent slower at a minimum.

That's a very fascinating comment, brother.

What I take away from points "A" and "B," of your post is that quad-core devices take a "triple-hit" to battery life and performance:

1) Due to the device having to expend more energy, just to maintain four cores;

2) Due to the device having to suppress two of the cores, just to function "normally;"

3) Due to the device having to "ramp-up" an underpowered "sleeper-core," possibly along with the two suppressed cores, when under higher-demanding tasks and processes.

Maybe, the Moto X proves that higher-clocked dual-core processors, coupled with an optimized version of Android, are more effective and efficient than quad-core or even Samsung's "octa-core" processor.

EDIT: I just saw your comment, about the Moto X not utilizing optimized software.

Posted via Android Central App

Didn't mean to say some optimization wasn't done, just that more performance was achieved by offloading. At least that's what the Cnet article says.

While I can appreciate the thought process here (I would *never* deride someone for trying to work something out and thinking for themselves) I do feel the need to interject a little here:

Having a quad-core device (Note 2), it's not that only two cores are ever used. If I play Galaxy On Fire 2 for an hour, I can guarantee that all four cores are being used (and that the phone get's hot as the sidewalk in a Houston summer, too).

A really great app for monitoring this sort of thing is called "System Panel" in the Play Store. Basic version is free, and gives you basic task manager features plus greater visualization of what the hardware is doing. Tapping on the "cpu usage" data at the top will open a detail view showing the load of each core.

While I'm sure that there is overhead in the cores "spinning up", watching it happen with System Panel it looks to be pretty quick.

I think the "performance boost" here is that, since the cores are being kept at 100% (which is possible with root and an app like CPUBooster by setting it to "Performance" mode) there's no commands processed at the slower speed while a core ramps up.

Again, I don't argue that there is probably no *real* need for quad-core in our mobile devices, I don't think the take-away here should be that having quad-core make the device slower or eat more battery at idle than a dual core. Obviously, a quad-core is going to eat more battery when all four cores are running, but I'm not sold on the idea of that being true at idle.

I think the real question of "which is more efficient" depends greatly on exactly what you're doing with the device at any given moment, which is where the bigLITTLE SoC concept came from.

This is only tangentially related to what Ron Amadeo uncovered in the Galaxy Note for several reasons.

1) The point wasn't that it was possible to manipulate benchmarks, the point was that Samsung themselves are manipulating the benchmark scores. It's a bit like saying "hey, we caught this kid lying about his quarter mile time" and replying with "well, we already knew it was possible to lie so this isn't news and doesn't matter". The point is someone lied, not that it's possible to lie, there is a very big difference.

2) In the example you give in the article you overclock the SOC. This means many things would in fact perform faster, so an increased benchmark score accurately represents a performance improvement, or at least accurately represents an increase in what the benchmark is measuring across everything done with that device. On the Galaxy Note 3 IT ONLY OVERCLOCKS WHEN SPECIFIC BENCHMARKING APPS ARE RUN. That's completely different than changing the clock speed in general.

3) As far as I know there is no other proven case of another manufacture doing this (we know Nexus devices don't, it's possible other do BUT IT HAS NEVER BEEN PROVEN before the the public eye and that doesn't make it any less bad). This is the reason why when Samsungs cheat is disabled its benchmark score fits in more nicely with other similarly spec'd hardware. That's not to say it's impossible that anyone else is doing this. But this is big news because for the first time we have verifiable proof that a large (the largest), respected manufacture is doing this.

The fact that you place a low value on benchmark results doesn't make Samsung's intentional misinformation any less serious.

I think it is wrong of Android Central, Jerry Hildenbrand, and Phil Nickinson to try and pretend this isn't a story and I don't know why you're doing it. This is coming from a fan by the way, I usually trust your content which is why this is so disconcerting.

Well said.

Writing it off as business as usual is just a form of enabling that Android Central should not treat lightly.

It was pointed out on Slashdot that anyone who does business in the Far east learns there significant moral differences in the views of Koreans, Chinese and Japanese. Making excuses for the worst offenders doesn't solve the problem.

We all know that if AC voices any vigorous objections, they get cut off from advance evaluation phones, test units, and sneak peeks. Still, I'd prefer they call a Spade a F'ing shovel and let the chips fall where they may.

Who said it wasn't a story? I just said don't be surprised that Samsung (and other manufacturers) tried to win at a game we all helped create — and included a story of engineers playing the game right in front us. See anyone else admitting that yet?

I can't blame the manufacturers for playing it that way. 

Pretty sure I also said a little more transparency — either on the manufacturers' part, or the benchmark apps — would be a good thing. (Edit: That is to say, the benchmark apps should give us all the info they can. It's not their fault if the manufacturer circumvents things, like Samsung's done in this case. More info is better.)

I'm not surprised that a for-profit corporation would try and cheat its benchmark results, anymore than I would be surprised that a neighborhood pedaphile would molest somebody's children.

Doesn't mean it's excusable in any possible way.

I don't see why you can't blame the manufacturers for this. That might be how they choose to play the game, but it isn't an honest way to play the game, therefore I view it as morally wrong and it is quite easy to blame them for it. They could play the game fairly by not applying special "optimizations" when running the benchmarks. They should treat the benchmark app like any other app.

Posted via Android Central App

"I can't blame the manufacturers for playing it that way. "

This is the worst attitude to take. It is the steroids debate all over again. By accepting cheating.. to are condoning cheating. Simple as that.

No-one is accepting it, it just isn't a big deal in this case. Also, the steroid thing you mentioned doesn't compare. Two totally different circumstances.

Posted via Android Central App

I agree with demontooth: it's not like Samsung is advertising benchmarks in their commercials. Anyone that knows anything about benchmarks has to know that they don't affect real-world, day-to-day performance. So, Samsung "lying" or "cheating," to inflate their benchmark scores, is harmless, and you're overreacting, to say the least.

EDIT: Ok, maybe it's wrong of me, to say that you're overreacting. But, in my opinion, this doesn't really have any impact on the sales of Samsung's products or the mobile industry. I can't remember the last time someone purchased a product, based on its benchmark scores.

Posted via Android Central App

I've impressed many a lady with phone benchmarks. I call my benchmark program "the panty dropper". I've been the star of many dinner parties by whipping out the panty dropper and just watching people be amazed.

Posted via Android Central App

High five, brother.

I used my panty dropper benchmark app in a group of unannounced transsexuals and I almost lost my eye from all the "springing" action.

"With great power, comes great responsibility." - Peter Rabbit (or whatever the Super hero's name is that got bit by the radioactive cockroach)

Posted via Android Central App

While I can appreciate your outrage, what Phil/Jerry refer to in terms of "boosting" the CPU speed to get better benchmark scores is exactly what Samsung is doing.

These ARM-based CPU's save power by "scaling" their clock speed, meaning that when they're not being asked to do much, they run at 300-800Mhz, instead of their full 2.3Ghz potential.

The old way of "faking" benchmark scores involved altering the scaling "profile" (on a rooted device) to put the device into a "performance" profile. This effectively prevented the CPU's from trying to scale-down and conserve power and just caused them to stay spooled up to their full possible processing speed all the time.

What Samsung has done is put something into their OS code that enables this "performance" profile whenever it detects certain benchmarking apps are running.

It's not *exactly* giving fake numbers, since the CPU/GPU really are capable of those numbers, but it is artificially inflating them, since you won't see that same performance in the real world. I know, whether or not that constitutes "fake" numbers is very much a gray area.

I guess being a PC tech for many years, and seeing this go on in the PC hardware world over and over again, it really doesn't surprise or faze me much.

Its like saying Elvis is Still dead.

Why would they take this code out of their build, when they suffered no real black eye by leaving it in.

Benchmark writers just need to do as Ars Technia did, and disguise their benchmarks.

my god I'm old. I remember reading this back in the day. and think 3 years later we are still whining about. thanks for the repost.

Posted via Android Central App

"Benchmarks exist to measure the performance of a phone during normal usage, and a device should never treat a benchmark app differently than a normal app."

I don't agree with this statement made by ars. Look at the benchmark tests for pc graphic cards such as 3DMark. It doesn't test for how energy efficient a graphic card run during normal usage. Benchmarks exist to test for the maximum performance at maximum clock speed of the gpu and memory. It's become and been accepted as the industry standard to optimize the drivers to pump out the maximum possible benchmark scores. AMD and nVidia always optimize their drivers for the maximum possible benchmark scores and frame rates.

By the way, when gpu cores run at their maximum speed per the specification on any programs, it's not overclocking. Overclocking means forcing a cpu or gpu to run over the specified (advertized) maximum clock speed. The cpu runs at 2.3ghz, which is the specified clock speed. I don't see anywhere Samsung overclocking the cpu, do you?

Agreed. Ars complaining about Samsung forcing the N3 to use its hardware to the fullest is like Car & Driver complaining about Chevrolet forcing a Corvette to use all cylinders during a road test. This is complete & utter nonsense. Ars, Engadget and their ilk are simply trying to score page hits with this trollercoaster. I used to respect Ars, but not any more.

That's not a valid comparison. The Corvette will perform exactly the same for me on the street as it does for Car & Driver at the track. Samsung phones only run benchmark apps at that speed, no other apps benefit.

On that topic, car manufacturers have been caught in the past giving car magazines "ringers" that have different components and tuning than production cars. In 1964 Pontiac went so far as giving Car & Driver a 389 GTO to test that actually contained a specially prepared 421 that was not available in the GTO. The end result of that was a claim that the Pontiac GTO was faster than a Ferarri GTO. Which was a lie. And guess what? People complained about it. Car & Driver still feels the effects of that today as nobody has forgotten. Samsung's performance claims are also lies, and people have every right to complain about them.

Let me clarify my analogy. Forcing the N3 utilize all of its potential for a benchmark test that *determines maximum possible performance* is EXACTLY like having a Corvette run on all cylinders, disabling "valet" mode, launch control, etc. for balls-to-the-wall performance. The purpose of a benchmarking is to determine absolute maximum performance. You cannot make that determination without enabling every possible performance-enhancing option, which includes running all available features (including cores) at full speed for the duration of the benchmarking test.

Now if the Samsung kernel deliberately prevents non-benchmarking apps from using all cores at full speed (when they otherwise could), that is a different matter. Also, if Samsung overclocks beyond the standard production maximum in order to boost benchmark scores, that is a different matter as well.

Edit: I realize Samsung could be providing modded "ringer" N3s to bloggers knowing they will get benchmarked, but that is beside the point of my original comment.

'Why wouldn't the manufacturers do everything they can to score as high as possible?'

Some would consider that fraud.

Posted via Android Central App

It is a fraud, but it is not actually fraud.

It would be actual fraud if they used it solely as a selling tool but as someone said above, Samsung doesnt sell their phones using the benchmarks. They just fudged the numbers and let the tech sites do the rest.

The only purpose to "fudge numbers" is to sell more phones. It may be indirectly, but that is certainly the intent. There is no other purpose.

It's not fraud. The hardware is completely capable of performing just as it does in these "inflated" tests. Samsung simply puts the hardware (and the OS?) in the optimal state while these tests run. A state which the normal user won't ever see in normal operation, nor would they want to...battery life would be trash.

Posted via Android Central App

If it doesn't perform as claimed for me, it's fraud. It doesn't matter if it's practical or not.

Now, Samsung doesn't actually make that claim, so it's not technically fraud. But it is certainly deceptive and unethical.

This isn't quite the same as before though is it? Before they actually raised the gpu freq above what games could use.... only for a benchmark app... That doesn't happen on the note 3 AFAIK..

This is just locked all cores online and highest freq in certain apps right?

Posted via Android Central App

Love the Casablanca clip! All this deception makes me "feel like my insides were kicked out". *pursing lips*

Posted from my "CrackDroid" Nexus 4 via the Android Central App

Well I'm praise android central for not falling into the benchmark fiasco, I personally feel actually using the phone yourself is the best way to know if it's the powerful phone that gets you all wet. For some it may be Samsung for others like myself it's HTC and for others it's the mighty nexis (I like that word lol). Perfect example my dad loves Samsung he finds out to be the android powerhouse I find my HTC phones as the powerhouse, different phones for different people who use it differently an it director in an ivy league university vs a college student working at a warehouse lol.I mean how many times do Rom's and even mods work flawlessly for one person but after chaotic for another with the same phone, build, versions? It's all in the hands of the beholder!

Posted via Android Central App

Benchmarks lack when they don't simulate the real use of the hardware. Usually these benchmark apps throw a lot of arithmetic operations and count the result, eventually considering specific results involving the hardware. But, the effective benchmarks for us, as users, are the ones that have a lot of pieces, each one destinated to simulate ordinary actions of the user; then you can point where one is better while other not and vice-versa.

But I think benchmark still is an enthusiastic move. Lots of things must be considered so we can tell what is the best hardware around...

Posted via Android Central App

"Can you really blame Samsung, though?"

Can I blame Samsung for cheating even if cheating helps them? Yes! I can blame then and choose not to support them.

This is crazy to make a point of it. The question is what is the point of the benchmark. In most cases it is to show raw processing performance. More power to them for making sure that the phone will show is real potential when pushed to it's limits. One of the few ways to do that is to make sure the CPU runs at full throttle. I actually think the guys making the benchmarks are the ones that should be getting yelled at. If their app doesn't push the CPU to full clock and keep it there during the test it isn't valid to show how fast the device really is.

This isn't anything new when talking about raw processing power. As others have stated this has been common practice for years on pretty much every other major number crunching platform.

Ask any PC Gamer if they leave Intel Speedstep or AMD's Cool N' Quite on. They will tell you hell no if raw speed is what they are after. There is a performance hit for dynamic clock speeds and such.

The problem is people are misplacing the blame here. Samsung is doing what the benchmark app should already do. The Benchmark developers should get there act together and make sure they push the phone to it's limits.

But the PC gamer can actually use the full raw power. Samsung customers cannot, except to run benchmarks. See the difference?

This is one reason why I've never been a fan for Samsung. They want to be Apple so bad they are actually turning into the same thing that the evil fruit company represents. From adding as many total gimmick features as they can think of to inflated performance expectations and more, I think Sammy is trying to win the war in all the wrong way, and they have lost my respect a long time ago. I really can't stand Apple and Samsung is not much better. If they would just focus on making a genuinely good product the rest would take care of itself. Instead they want to kitchen sink all their products and its just overkill and reeks of inferiority complex. Sad part is that people are lapping it up, just like the isheep.

No thanks Apple or Sammy!

Posted via Android Central App on my white Nexus 4 with StraightTalk

Samsung has already proved they don't wanna be Apple by actually making something original which Apple is clearly incapable of doing.

Posted via Android Central App

What I don't get is what they were thinking. They were caught doing this before and recieved a lot of bad publicity because of it. The N3 peformed the benchmarks better than the G2 without this. They increased performance so much that it was immediately obvious somehting was fishy and there was never any doubt they would be caught.

All they gained out of this was more bad publicity. I really don't get how such a successful company can do such blatently stupid things.

Strange tone to article, very fanboy like imo , after the S4 I would think the phone makers would stop this practice. And for that Samsung should be ripped. The app developers should not be required to catch phones trying to game the system. I would prefer if Samsung would just add more control over kernel and let us crank up or down CPU/GPU performance. This way everything is out in open. No hidden optimizations that run with few specific apps.

Posted via Android Central App

Anand tech benchmarks always show the iPhones having better battery life than most android phones.

Yet even my first gen lte Bionic outlasted the iPhones by up to 6-8 hours in real life usage.

Benchmarks are the measure of the phones potential the os has to work with. Doesn't mean, but it could. That is how I look at this it.

Posted via Android Central App via HTC One

Motorola Droid Bionic Has Been Stolen and has been recovered! Yey!

Anand also only places iOS devices into benchmarks when he feels it shows to be competitive to android. So take all the Anand comparison charts without iOS devices to mean "Apple got their asses handed to them by Android again".

What matters is all cases is the user experience, benchmarks can say nothing to this.

What I want to know was did Apple do the same thing with the iPhone 5? Because in benchmarks run by Anandtech it crushed by far even the Galaxy S 4 which was known to chest the benchmarks as well.

So if Apple is crushing current Android top phones with a dual core chip despite the top Android phones cheating... that would be pretty bad.

In my opinion benchmarks mean nothing, but that has partly to do with news sites giving manufacturers a pass for cheating the system.

Posted via Android Central App

Anandtech had an article that addressed this recently, and this is quote from them:

"With the exception of Apple and Motorola, literally every single OEM we’ve worked with ships (or has shipped) at least one device that runs this silly CPU optimization. "