Benchmarks are a horrible way to gauge the performance of a smartphone, and here's why

There is a subset of smartphone users that like to benchmark the hardware inside their phones. Benchmarking on Android is difficult because many apps aren't running natively (Android apps mostly use an interpreter-only virtual machine called ART) even though they may have portions with direct access to the hardware. The entire system is easy to manipulate with minor changes. You might not be cheating when you share scores, but you can never be sure that others are being honest.

A number from a benchmark app isn't going to tell you how well your phone works for you.

Benchmarks also don't tell much about the actual user experience, especially with more modern hardware. The chips inside your flagship Android phone aren't exactly built for supercomputers, but they are overbuilt and very few tasks you'll ever do tax the processor or GPU of the latest chip from Qualcomm or Huawei. In other words, a phone that scores a lot lower can still give the same user experience as one that tops the charts.

That is, unless a chipmaker is cheating to inflate scores for its low-end offerings, which is precisely what MediaTek has been doing for a while.

AnandTech is a media outlet that does a lot of benchmarking. And I mean a lot — it benchmarks almost any product with a processor and noticed some strange goings-on with MediaTek devices. It turns out the company was distributing its own benchmark app whitelist in the vendor partition of Android devices, and it allowed the hardware to run well above its normal full-power rates.

Here's how this cheating works, in case it all sounds like gibberish to you. MediaTek keeps a list of popular benchmark apps. When a user runs one of these apps, MediaTek lets the processor run as fast as it can, allowing it to get much hotter than normal and allowing the memory controller to run as fast as possible before it is slowed down.

In essence, it's making the phone stop doing everything else and focus on the benchmark app until it almost gets to the hardware's breaking point before slowing things down. When things cool off enough, if the benchmark app is still running, it ramps things back up again. This gives higher scores but is not an actual representation of how the hardware would perform while you are using it.

Cheating isn't new

It's not just chip manufacturers that cheat; phone makers have been known to do the same thing. In fact, Samsung was sued for cheating at benchmarks (yes, really) for the Galaxy S4 and was forced to pay out $13.4 million because it got caught doing essentially the same thing as MediaTek.

HTC, Huawei, other Samsung phones, and Sony have all been caught modifying system code so that phones the companies sell would perform better at benchmarks, and only during benchmarking. But this time, things are a little different.

Usually, it's the phone makers doing the cheating. Not this time.

Why? Because this round of benchmark cheating isn't being done by phone makers; it's being done by the company that makes the actual chip: MediaTek. The configuration files that force a potentially unsafe high-performance mode during benchmark tests are in the actual board support package provided by Media Tek to phone makers. The phone maker can remove the offending code — which is what Oppo did after Anandtech's findings — but it's something that should be plug and play that a phone maker just drops into place.

Why it was done is easy to answer — it works, and MediaTek almost got away with it. It was discovered when benchmarking cheap phones with MediaTek chips against phones with higher-end processors and seeing benchmark scores that were too good to be true. MediaTek does make some newer high-end chips like the oddly named Dimensity series, but the company is generally better known for making budget chips that provide just enough power to run your apps.

We love budget phones, and so do a lot of other consumers — we don't buy them because of benchmark numbers.

Let me be clear by saying that there's absolutely nothing wrong with making a budget chip. Not everyone wants to pay $800 or more for a phone with Qualcomm's latest and greatest, and as long as the chip itself can run Android and run the apps you love, it's an all-around good thing. Even Apple has seen the demand for a sub-$500 phone, and we all can appreciate a good value when we see it.

But trying to pull the wool over our eyes is shameful. The next time you see a phone with a MediaTek chipset, remember that you might not see the performance that's being advertised.