Cheaters

adb shell cat /sys/module/pvrsrvkm/parameters/sgx_gpu_clk can never tell a lie

Samsung got caught cheating. They are manipulating benchmark results for the Galaxy S4 by allowing the GPU to run at a higher clock speed and locking the CPU to the max frequency, in a different way than other system processes are allowed to do. They give a few of Samsung's own apps limited time with these higher speeds, but clearly favor some popular benchmark apps. They got caught red-handed by Brian and Anand over at AnandTech — two of the best hardware geniuses in the business — and the Internet is in a tizzy, as the Internet is wont to do.

We've been reading all about it, including the responses of people who are in no way affiliated with any of the involved parties, and now that it's past noon and I've had a few cocktails with lunch I want to write a few words about it.

Join the fray in the Galaxy S4 forums: Samsung cheats at benchmarks

To make things better (or something), Samsung released a statement clarifying a few things.

Under ordinary conditions, the Galaxy S4 has been designed to allow a maximum GPU frequency of 533MHz. However, the maximum GPU frequency is lowered to 480MHz for certain gaming apps that may cause an overload, when they are used for a prolonged period of time in full-screen mode. Meanwhile, a maximum GPU frequency of 533MHz is applicable for running apps that are usually used in full-screen mode, such as the S Browser, Gallery, Camera, Video Player, and certain benchmarking apps, which also demand substantial performance.

The maximum GPU frequencies for the Galaxy S4 have been varied to provide optimal user experience for our customers, and were not intended to improve certain benchmark results.

We remain committed to providing our customers with the best possible user experience.

The problem is, that's not quite true. Anandtech has data that shows Samsung isn't being completely truthful — especially the part where it claims things "were not intended to improve certain benchmark results." It also completely glosses over the part in the code where "BenchmarkBooster" actually names a few popular benchmarking applications that are able to set a special "boost_mode" flag when they are launched.

Maybe this statement came from someone who wasn't familiar with the situation, and it's an honest mistake. That doesn't mean it's not false. And people are going to rightly jump all over Samsung for saying it.

But it doesn't really matter.

Benchmarks suck. I know benchmarks suck, because I have a Galaxy S4 and a Nexus 4 here beside me, and the scores in no way reflect the user experience. They are just numbers for people who like to look at numbers and tell other people that their numbers are bigger than your numbers, or something like that. When I want to know how well any computing device, not just an Android, handles the system and apps we like to install, I turn it on and use it. Something like "25,611" means nothing to me. The fact that your phone got "25,612" also means nothing to me. In fact, we only run benchmark apps when our inboxes won't shut up and stop asking us to run them, and even then only in a momentary lapse of reason. Because they suck.

They're also super easy to manipulate, as evidenced by Samsung's latest gaffe as well as shown here. If you want numbers, just tell me what number you want and we'll make it happen. You still will have no idea what they mean in the real world and how it matters when you want to play Words with Friends or stream porn from the Internet.

Kudos to Anandtech for digging to the bottom of this, and shame on Samsung for playing into the game and then giving a wishy-washy response about it when they got caught. That needs said, so Ill say it. But as for benchmarks themselves, and whatever numbers Samsung's Galaxy S4 can turn in, I don't care, and I don't think you should care either. 

 
There are 64 comments

Roy G Biv says:

new phone released? let's post all benchmarks we can

samsung caught cheating on benchmarks? benchmarks suck

Please show me the benchmarks I posted. 

Edit: I guess this counts.

dacp283 says:

+1 To both. I remember when the DroidX first came out was about the time Bench marks started really becoming overly used. Hell I was even buying in to it. I then remember a very short time later AC posting about the flaws in such things and how easily manipulated the apps were. That's when I quit caring. So yes I believe the AC team might have posted a few back in the day when it was "the standard all phones were judged by", but there as very little weight to the claims and it was clearly stated as such.

Posted via Android Central App

dacp283 says:

I was implying Phil and Alex by the way. Guess I should have replied to the later. :)

Posted via Android Central App

dacp283 says:

Holy Hell I need to get off the juice while I type. Jerry and Alex... Smh

Posted via Android Central App

Alex Dobie says:

Er... no. I think we've been pretty consistent on our position that benchmarks suck. This is just another reason why they suck.

TenshiNo says:

AC announced that "benchmarks suck and that they were not going to do them anymore" like 2 years ago. And they really haven't, save for a couple of times where they were flooded by a torrent of requests and did it under duress. And even those times they were like "these benchmarks don't really mean anything, but we're tired of everyone asking for them."

HNNNNNGHHH says:

What's the point with even running benchmark apps on a Galaxy S4? The damn thing's already as overpowered as it is, and completely bloated with several TouchWiz features most of us will never use.

Jeff Kibuule says:

The terrible thing is when that "overpowered" handset is being used to power gimmicky features that STILL slow down the system.

ekimpadd says:

I disagree, phones always have gimmicky features. The speed of the system isn't about how many features you implement but the way you implement them.

rdude94 says:

Well, isn't the point of benchmarking to see maximum performance of a device? All that Samsung is doing is making sure that the S4 really performs at its maximum performance. Nothing devious here. All they did was have a special code to recognize if a benchmarking app was running so that that the CPU and GPU would run at its highest. Correct me if I'm wrong.
Either way, just as the article states, benchmarking is irrelevant and only real performance should be a factor whether to enjoy the device or not.

Yeah, I have also always thought that certain benchmarks just showed the potential of the hardware in question.

yea but the point here is the benchmark apps are now running at a max performance, that only they can run at, other apps (besides a few samsung apps as the article reads) will never see this boost mode. so esentially if i drive a car and hit a max 100 speed then you put a limitor on it and i drive again and max at 80 why do i care that 100 is the max speed if the limitor is there now, does that make sense because i am not the best at metaphors haha

rdude94 says:

What other apps need that extra boost and are not getting it? I'm pretty sure games and heavy apps do get that boost as they drain the battery more.. If the apps run fine, what's the problem? I know that the s4 had lag problems but thanks to optimization in later updates, the device runs well.

rampage1979 says:

Even with the latest updates it still lags.

Posted via Android Central App

ChrisFricke says:

But that's exactly how cars are bought. Millions and millions of people buy cars every year that have awesome max "benchmarks" when all of the time in the real world it doesn't matter because of a limiter put in place (called the speed limit).

All Samsung needed to do was to state "these benchmarks are optimized". No drama. Then they, or anyone else, can post "non-optimized" benchmarks for comparison. No drama. It's not a big deal and by next week no one will care but it is a bit of a gaffe.

SuperJonCA says:

By limiter he means something like a governor in a car, not the speed limit.

icebike says:

Exactly this.

Most things are slowed down by lowering the CPU speed. (A perfectly valid way of preserving battery).

Some apps can use the full speed, Browser, gallery, etc., when in full screen mode.
But most can't, almost certainly non running in the background.

Its a battery saving feature that every modern chipset uses. Some reduce cores, some jump into a companion core, but ALL of the modern chipsets reduce CPU speed at some time or another.

The DISHONEST part was including an exception for a benchmark that was not available to any other random app, and then NOT TELLING PEOPLE that they did it this way for a reason.

Jayq330 says:

Exactly, people don't seem to get this. Note I know why my HTC one would last longer in games than my current GS4 while playing games & even still HTC one would perform just about the same benchmark scores without the cheating. I been thinking they been doing something ever since galaxy 2, there scores 2; their scores were way ahead of anyone else's... Also to promote their exynos GPU as supreme GPU! Wrote what a bunch of BS, maybe that's why there GPU could never work right in the first place in all the galaxy generations, always a speratic compass or something with the map. anyway they tried to screw the faithful with there Oct core CPU, when it was really a dual quad core with the other 4 being slave. Motorola at least said it from the start & explained clearly how their 8 core chip works & that only 2 are actuary used for app's. Unlike Samsung. I have the SG4 for the Samsung fans out there justifying their BS, but if I see they're lying to me for my money then I call that scamming me for my money. I'm going with the MotorolaX for my next phone, no BS included.

Benchmarking is common in the pc industry. Its the only way to compare the performance of one device to another. As you stated cars do the same thing. My 2010 Chevy Impala is essentially underclocked since it's speed limited at 112 mph yet the speedometer goes up to 140 mph.

yes it is. It's a common method to figuring out what configuration is the best. I've been building computers for 17 years and I'm in IT.

chrisharmful says:

Greatest thing about benchmarks is it's an excuse to put a picture of Lance Armstrong on a phone :-)

Posted via Android Central App

vansmack says:

More tipsy Jerry articles, please!

S-Doo-1965 says:

Tipsy Jerry is more reliable then the rest of the internet combined.

cptaszek says:

+1

Posted via Android Central App

Linebarrel86 says:

But...but...the benchmarks...

Thanks Jerry. Maybe we'll only have to suffer these fools a little longer until some other company is caught doing something stupid.

Shouldn't be too long.

greg1007 says:

Yeah, for basic users, benchmarks mean very little compared to software optimization, but to those who overclock and flash custom roms, raw power means alot, and Samsung definitely showed that off. I think Samsung did good by these people, albeit a little shady.
PS. AC has been consistent in their dislike of benchmark hype, which I disagree with but respect and understand

Posted via Android Central App

codiusprime says:

I do both of those things and I still think benchmarks are pointless. I used to care about random numbers then I just started caring about user experience.

Posted via Android Central App

mintvilla says:

I think people are taking this the wrong way, the people are getting annoyed by this, are the same people who believe in benchmarks.

This is just Samsung's way of having a laugh, and taking the micky outa people who stress about pointless benchmarks.

I say well played Samsung, Well played.

God I loathe benchmarks. Who cares about a number... it's about how the device performs.

icebike says:

What is needed is better benchmarks, not condemning the very concept of benchmarks.

If you can't quantify "how the device performs", or if you can't even agree on wording more precise than "how the device performs" then you have no basis for comparison. None at all. You might as well judge phones strictly by the degree to which they "feel good in the hand".

A benchmark that scripted several USER SELECT-ABLE things that they do all the time, such open a browser, check their email, peek at facebook, check some ball game scores, then exit and report end-to-end time would be a good start (other than being dependent on the internet speed).

But I don't think you can condemn the entire concept of a benchmark and then suggest a measure like "how the device performs" as a replacement.

jamie says:

Nice picture choice!

Posted via Android Central App

nimer55 says:

"They are just numbers for people who like to look at numbers and tell other people that their numbers are bigger than your numbers, or something like that. When I want to know how well any computing device, not just an Android, handles the system and apps we like to install, I turn it on and use it."

Huh, not all of us get review devices from manufacturers. When I decided to buy my Nexus 4, I hadn't even seen a dummy plastic one in person. What I do is read reviews; which often vary due to being fairly subjective, watch videos, and look at benchmarks. Now, I don't let benchmarks dictate my purchase, but they help me get more informed (well I actually mainly use Anandtechs summary/analysis of the benchmarks; I don't care for the benchmarks themselves.) With the Nexus 4 for instance, I couldn't tell you a single benchmark score, but I remember reading that the Nexus 4 had thermal throttling, and got different scores on benchmarks depending on when you ran it, letting me know the phone would perform worst when it got hot.

nimer55 says:

worse*

heathroi says:

maybe Samsung has a sense of humor after all. you want Numbers, hell we'll give you numbers

Stupid Samsung. This is why I dong buy stuff from them

Posted via Android Central App

lol you said "dong".

icebike says:

he heh, heh, heh-heh-heh!

the1m.polo says:

So you don't buy samsung phones because???? Smh.. u knew they would cheat on benchmarks....O-kaaay... lol.

Posted via Android Central App

One way or the other , we have a common denominator for the 12000 devices...the benchmarks ...not only for the s4...

Posted via Android Central App

fightcrazy says:

I didn't buy the SGS4 but if the benchmarks suck so bad then do this, flash a custom rom without the Samsung apps on it that control the benchmarks, the benchmarks on the custom rom is sometimes even higher than on the stock rom. So to me that shows that the SGS4 is even faster than what Samsung is even advertising. The only people that are bitching about this are the anti Samsung people. Same old shit every day, even coming from the writers. I buy a phone because I like it, I would much rather take a SGS4 with 32gb of int memory than any Nexus phone to date. I would rather have the SGS4 than any other phone besides the HTC One with 64gb of int memory. Until these other Manufacturers can build a phone that gives the customer what we ask for than 9 out of 10 times Samsung is the phone. Until Nexus starts adding int storage to their phones I will have to pass.

MissyHaney says:

How do we know that Samsung didn't fudge the S4 so it appeals to reviewers on AndroidCentral?
(or any other so-called Android experts?)

The best judge of a new Android deice is yourself. Play with it. Buy it. If you hate it,
return it. It's not rocket science.

Don't buy a new Android phone or tablet because it has better benchmark results than your
friend's Android device!

NoNexus says:

Simple answer, the S4 lagged so much out of the box (first version) that you couldnt hide it from the reviewers short of making it so it didn't lag (for example, getting it right the first time)

the1m.polo says:

Agreed

Posted via Android Central App

return_0 says:

You pretty much reiterated the point of the article.

Posted via Android Central App

JoeZone says:

What a burn on Samsung. I'm glad they got caught lying because the truth will set you free! I will show this to my Samsung friends who think that s4 is greatest thing ever.

from my lean mean jelly bean machine N7!

troyboytn says:

In response....your friends will turn around and throw it back in your face as soon as similar code is found in those "other" android OEMs, which I will guarantee is hiding away, waiting to be found any time now.....

the1m.polo says:

I usually buy what I think is the greatest thing ever too... not your friend but I am a Samsung user ... this means nothing ... should I throw my Note 2 against the wall cause samsung possibly attempted to appeal to hackers who benchmark phones and would probably over clock their phone anyway??? I bought a phone not a company man...smh

Posted via Android Central App

the1m.polo says:

I dont mind people thinking they have the greatest thing ever. .. as long as they can give me reasons why it's the greatest thing ever. ..

Posted via Android Central App

the1m.polo says:

Benchmark : sminchmark.. its all about optimization. ..and actually performance when using the device. ..if u like touchwiz, sense, vanilla, or whatever. .. as long as u like it... the hell with a benchmark. .. I bet this...the fastest strongest player in the NFL isn't mostly the MVP... the best shooter in the NBA isn't necessarily gonna lead his team to a championship. ...the most horse power..doesn't put u in victory lane. its always overall performance! !!

Posted via Android Central App

rovex says:

Benchmarks are rubbish, but even then, Samsung didnt cheat as such.

Benchmarks are about finding out how fast a machine can be, but games on a phone are about running at the lowest acceptable frame rate in order to save power. Unless the frame rate drops very low there is no need to throttle up the GPU. With that in mind if you run a benchmark normally, it will just give you a throttled result. Its simple power and heat management

All Samsung did is lock it to maximum (533) and give the best result possible. So games dont run at 533 you say? Dont need to and indeed they shouldnt.

My HTC One hardly ever maxes out playing games either, so its not unique to the S4. If I force max clock the score are higher, but not that much, and the phone gets really hot.

the1m.polo says:

Agreed I don't consider it cheating. .it is misleading. ..but again who cares...

Posted via Android Central App

rampage1979 says:

They had to cheat cause they new touch wiz would lag so tried to sell you on its specs the nexus 4 and the LG optimus G both run smoother than the s4 and with an older chip.

Posted via Android Central App

the1m.polo says:

LG is quickly moving up my fav OEM ladder. .. LG is smoother but... I will say touchwiz has far more features running.. eye scrolling. . Hand gesture. . So on and so on ... pick whats more important to u I guess

Posted via Android Central App

likwidkool says:

Reminds me of years ago when nvidia manipulated their drivers to dumb down the graphics on 3dmark to achieve better scores!! I have never trusted a benchmark since.

Posted via Android Central App

Paragon99 says:

Too funny Jerry....Samsung caught cheating, and a picture of Lance Armstrong. Good one! :)

Dan29466 says:

+1,000,000

Jerry wins the internet. Again.

jelias999 says:

Hey benchmarks mean nothing in the real world to me at all I got the sgs4 and I'll tell ya it's a damn good phone and it is very fast especially if you get the developer options to pop back up and you turn off that annoying animation speed. Either way I feel that it's not a big deal to me they clock it higher the processor is faster than anything else out currently. Essentially the same as the HTC one my sg4 runs beautiful during games and even fiery taxing applications I've never experienced lag on it. The clock speed on games makes sense bc this processor does suck down the juice heavily when your running alot of taxing apps. Either way the phone works really well and that's all that matters to me

Ant34 says:

It is the actually hardware of the phone producing the numbers so I don't see what the problem is.

sk8trix says:

I've been a big samsung fan for about two years now...kind mad too because even if benchmarks don't matter samsung should be cheating their devices are already good who cares about benckhmarks...but hopefully they learn their lesson after they got put on blast today that isnt cool fan or not i dont agree with cheating

verdana says:

They'll remove it on the S5 ;-)

Posted via Android Central App

I'd like to think of it as Samsung trolling the benchmark trolls.

Posted via Android Central App

adamschuetze says:

Benchmarks (and by extension, the hardware spec race) sucks until you're looking for reasons to disparage a handset because the hardware isn't cutting edge on release.

*cough* Z10 *cough*

http://m.androidcentral.com/specs-battle-blackberry-z10-versus-best-android

xlDeMoNiClx says:

People just like to hate on Samsung cause they're the big boy in the industry now.

Posted via Android Central App