How to cheat at benchmarks (and why we should downplay their importance)

By Jerry Hildenbrand
last updated

Benchmarks can be useful, whether we're talking mobile or any other computer hardware.  There is a problem though -- you have to depend on everyone being honest.  Let me explain.  Anytime you see a video or a screenshot of a benchmark score, what you're really seeing is a measure of how the software interacts with the hardware.  There are two big problems with that: No two Android phones run the same software, and there's always some joker who wants to cheat.  recently our pal John from Phone Arena noted some Quadrant scores of a new handset that are just too low compared to the other phones listed.  And good on John for pointing it out.

Sure, benchmarks can be fun, like see how little impact LG's UI has on the Optimus 2X compared to Stock Froyo, and they can be useful if you're running them yourself to gauge some changes you're making to your own software.  But we're over it. We're going to base our judgment of performance based on performance, and not numbers.

It's just too damn easy to cheat at all benchmarks, because they are just Android apps.  Look at all the other magic folks do with Android apps, and ask yourself -- are benchmarks so different that they can't be "hacked"?  The answer is no, and we're going to show you how to hack at the easiest (but definitely not the only) one to manipulate, Quadrant. 

Check it out after the jump.

Let's have a little look at exactly what we're about to do here, through the magic of video.

YouTube link for mobile viewing

You're going to need to be rooted for this example.  There are plenty of other ways to cheat on unrooted phones by editing the code and rebuilding the app itself, but this easy way needs root.  You're also going to need a few tools:

  • Have adb working, with root access in the shell.
  • Quadrant Standard from the Android Market.  We used Quadrant Advanced in our demo, because it's so easy to see the cheating.  You'll want to use Standard, because it's so easy to see the cheating.
  • System Panel (or any app that can kill all processes) from the Android Market.
  • SetCPU from the Android Market, or equivalent (if you have an overclocked kernel).
  • Two hands, a keyboard, and no conscience.

Install all your apps and reboot your phone.  Let it sit for 5 to 10minutes so all the syncing and scanning gets finished, then run Quadrant (see the "before" side in the image above) so you know the truth.

Hook your phone up to your computer, and enter the shell.  As root, run the following command:

mount -t tmpfs tmpfs /data/data/com.aurorasoftworks.quadrant.ui.standard

Enter it exactly as written.  What that's doing is simple -- the I/O portions of all phones are the biggest bottleneck, so we're moving the Quadrant data folder off the physical storage, and into the RAM.  RAM is good.  Reading and writing to it is fast.  Super fast.  This goes away at reboot, so it's easy to take things back to normal.

Next, we want to cancel out any CPU scaling effect.  Who needs to save battery, we want high scores!  Fire up SetCPU and crank the CPU clock speed as high as it will go, and choose "performance" as the CPU governor.  This will keep things running at top speed all the time.

Now use whichever app you decided on (I use System Panel to monitor things, and it includes the dreaded kill-all as an extra function) to kill all your background tasks.  If you can kill system processes, even better.  We don't care how well the phone works, only how high the numbers are.

Fire up Quadrant, and run your benchmark.  Run it two or three times to get it settled in, then start taking screenshots.  Realize just how easy it was to do this, then imagine what other cheats can be done by reverse engineering the benchmark apps and changing the math used to write the scores.  Add in a few other tricks like changing a line or two in the build.prop file to inflate the CPU scores, and you're golden.  Before someone says "Yeah, but I use Bechmark xyz9000 Pro and you can't cheat on that one."  Yes, you can.  Even I can, and there's plenty of people out there way more talented than I am.  Only believe benchmarks that use lots of meters and wires, with flashing lights, because anything else is just software that someone can hack.

Now the playing field is level.  Congratulations, you're a cheater.

Jerry Hildenbrand
Jerry Hildenbrand
Senior Editor — Google Ecosystem

Jerry is an amateur woodworker and struggling shade tree mechanic. There's nothing he can't take apart, but many things he can't reassemble. You'll find him writing and speaking his loud opinion on Android Central and occasionally on Twitter.

22 Comments
  • Nice post. Very informative. Another interesting point on the cheating front is to look at the impact of stagefright. On my droid 2 i score garbage with normal settings. If I enable stagefright in all 4 settings (media, scan, etc...) I can shoot benchmarks over 2.5k in quadrant. Honestly, I notice almost no change in the phone function.
  • Which is why I hate quadrant, most things you can do to improve the score, even without changing the calculation method, do little to nothing when it comes to effecting real world performance. Also, what I always noticed was that the Nexus 1, though obviously most hands on by Google coding and the JIT in Froyo being optimized around it, would outscore virtually every other processor on Froyo or not, without better real world performance in many cases. Lastly, the fact that many people look at these things as a whole, in many cases, without really having the ability to understand how the things break down individually for them, so they end up seeing a really great processor score on the N1, yet they are only gaming and texting on the phone so their quadrant score is even more meaningless than the average quadrant score.
  • I've always wondered this but how do you get that colored stacks on this app?
  • There are three versions of quadrant available, the priced, and perhaps the other paid version as well, provides the separate graphs. I tell you this fully hoping that you do not pay for either version, what a terrible benchmark that has misinformed too many buying decisions and been utilized in countless reviews in spite of all its obvious shortcomings.
  • from the video (around 5:29) he says he's using Quadrant ADVANCED, which breaks down the bar into different colors
  • Only a true looser would do this...
  • Only a true LOSER wouldn't be able to spell loser...
  • Jerry, it'd be awesome if you guys never use quadrant again. Please look at anand's review process and the benchmarks they employ over there, they definitely use the best benchmarks in the business. I hate to say it, but I go to them for 100% of my quantitative analysis, I still love you guys for your qualitative writing though.
  • So why not just state that from here on out this is the bench marking technique we are using for every single phone? At least if you make an "Android Central Benchmark process" we can decide if it is legit and if something we want to include in our decision then thats our choice. Since you have already stated that its easy enough to hack, it is up to the consumer to decide if it is legit. And besides, if you have the procedure steps displayed somewhere on the record, if people think that the numbers are fudged, someone can and will do their own test. Either proving or disproving the numbers in question.
  • Am I missing something? Jerry is clearly putting this out there so next time you see these numbers posted somewhere on the internet to take them with a grain of salt. It isn't that they are going to be fudging the numbers themselves. This is more of a consumer alert/PSA. Do they really need to say, "oh by the way, we don't do this."? There are other, and better benchmarks, that I would like to see substituted for, or supplemental to, Quadrant from Android Central. e.g. basemark, sunspider, rightware, neocore, linpack, quake 3, glbenchmark 2.0 In practice in anand's most recent review of the Thunderbold http://www.anandtech.com/show/4240/htc-thunderbolt-review-first-verizon-...
  • I personally like Smartbench 2011 as well. It also gives you incredible list of devices to compare against. XDA developers seem to agree with this.
  • Good on you guys! I've never been a fan of benchmarks being bantered about on enthusiast sites. You really need to understand benchmarks to interpret them properly, otherwise they can be totally misleading, and unfortunately most enthusiasts (myself included) don't have the proper knowledge and experience so we end up making bad decisions based on some number being higher, or lower, than another without understanding its significance. Real, everyday, in your hand, use will give the average user the best benchmark. Benchmarks are for those that know how to use them, and can put that information to proper use. Dave
  • Very sneaky on mounting the Quadrant data directory as tmpfs. You should put a limit on the amount of ram allocated just as a precaution (mount -o size=128m). In any case this is the fault of Quadrant for not checking that the IO portion of the test is on a physical drive. Edit: I see that your using Ubuntu. It that your distro of choice?
  • Benchmark results are fun to talk about with coworkers all with different Android phones but even top score and bottom score phones are crazy fast and we all love our phones!Drod,Droid2,DroidX,Droid Global,HTC Evo,Htc Incredible.
  • You forgot to mention all the Samsung phones. Unless that was intentional.
  • Hey hey, lay off of Tiger.
  • Why the Hero, Jerry? Out of all the lower end devices you choose to pull the trigger on our beloved Hero :P Very informative post.
  • but it's really only inflating the I/O operations. the cpu, memory, 2d/3d still suck.
  • exactly, if anything this shows to not trust results on Quadrant Standard as the Pro version pretty much gives it away.
  • I agree with others that we need a clear standard for measuring the speed of a phone, that everyone can reproduce, and get similar results. I'm thinking that one way to do it would be to ask everyone to run the benchmark AS SOON as the phone has finished booting. With the default settings on and with NO other programs loaded.
  • I don't consider Overclocking or killing background apps to get a better score as cheating.
  • Hello,
    I 've tried out running this command into the adb shell " mount -t tmpfs tmpfs /data/data/com.aurorasoftworks­.quadrant.ui.standard " But the result I get is " mount: operation not permitted " My phone is rooted. Is this the right command or is there anyone else with the same problem?
    Thank you