We’ve all seen benchmarks, and used them as reference for how fast a device is. However, should we really give benchmarks as much weight as we once did? Probably not.
Some manufacturers have been caught out (some — Samsung — more than once) manipulating device performance specifically in response to known benchmarking apps being run on the device. It’s unfortunate we had to name Samsung for doing this, but as the biggest name in the Android space, and being almost synonymous with Android, it’s behaviour we had to point out.
Fast forward to this week, with the Galaxy Note 3 being released, and lo and behold, Samsung are busted again!
Ausdroid wants to be clear though — it’s not that other manufacturers don’t manipulate the figures. We know they do. It’s known that HTC have gamed benchmarks in the past, and Sony have been caught doing it as well.
How do they do it?
Simple. We’ll use Samsung as the example (because that’s the one we know) but the others would be much the same. Samsung’s devices detect that particular benchmark software is running, and instead of allowing CPU cores to sleep when they’re not needed (the default behaviour), when the benchmarking app is detected, all four cores are kept running, and other optimisations are put in place as well, to artificially boost performance which isn’t normally available in other apps.
It’s not really cheating, but it doesn’t represent real-world performance; unless you fiddle quite extensively with your phone’s settings, and many don’t even allow you to do this, you’re not going to be able to extract the same performance for real world use, like games and other apps.
They’re all doing it
The team at AnandTech have done a fabulous write-up of the state of benchmarking which we highly recommend you read, and we’ve reproduced a table they’ve made which shows just how prevalent manipulation of benchmarking is:
|Android devices that manipulate benchmarks|
|3DM||AnTuTu||AndEBench||Basemark X||Geekbench 3||GFXB 2.7||Vellamo|
|ASUS Padfone Infinity||Qualcomm Snapdragon 800||N||Y||N||N||N||N||Y|
|HTC One||Qualcomm Snapdragon 600||Y||Y||N||N||N||Y||Y|
|HTC One mini||Qualcomm Snapdragon 400||Y||Y||N||N||N||Y||Y|
|LG G2||Qualcomm Snapdragon 800||N||Y||N||N||N||N||Y|
|Moto RAZR i||Intel Atom Z2460||N||N||N||N||N||N||N|
|Moto X||Qualcomm Snapdragon S4 Pro||N||N||N||N||N||N||N|
|Nexus 4||Qualcomm APQ8064||N||N||N||N||N||N||N|
|Nexus 7||Qualcomm Snapdragon 600||N||N||N||N||N||N||N|
|Samsung Galaxy S 4||Qualcomm Snapdragon 600||N||Y||Y||N||N||N||Y|
|Samsung Galaxy Note 3||Qualcomm Snapdragon 800||Y||Y||Y||Y||Y||N||Y|
|Samsung Galaxy Tab 3 10.1||Intel Atom Z2560||N||Y||Y||N||N||N||N|
|Samsung Galaxy Note 10.1 (2014 Edition)||Samsung Exynos 5420||Y(1.4)||Y(1.4)||Y(1.4)||Y(1.4)||Y(1.4)||N||Y(1.9)|
|NVIDIA Shield||Tegra 4||N||N||N||N||N||N||N|
Table via AnandTech, link above.
But they used to be useful..
Benchmarks used to be the gauge of a devices brute speed and performance, and in the early days of Android, they were a somewhat useful differentiator of performance between devices, although the main argument against benchmarking is that it doesn’t mean much for real world performance — that, at least, has virtually always been the case.
Benchmarking has now turned the race to the top into a race to see who can best optimise their code to best perform under benchmarking conditions: what a disappointment. The reality is that the benchmarking scores can, within a reasonable margin, be manipulated to make a particular product look good, and thus others look worse.
Perhaps unsurprisingly, while some devices manipulate benchmarking scores, they don’t change the underlying understanding anyway. It’s no secret that Samsung’s hardware IS very powerful, and probably more powerful than a good portion of the competition. Manipulating benchmarking can’t make a crap device appear to be gold; it’s limited to making an already powerful device just look a bit better on paper.
But does it really matter?
Truthfully, no. The media (Ausdroid included) probably have a bit to answer for this situation coming up, obsessing over minute differentiation between devices in days gone by. While we don’t really focus on benchmarks anymore, it is this earlier focus which probably led to manufacturers wanting to game the system, to get a better picture of their products.
Nowadays, we’ve certainly moved away from the raw power aspect (as hardware performance is becoming fairly homogenised amongst the top-tier at least), and we now focus more on usability, design and experience. It’s all well and good to have insanely powerful hardware, but if it isn’t usable, there’s little point.
Fortunately, the Android manufacturers are alive to this move, and the focus on user experience (UX) is only increasing with each major release. We only hope that this becomes the real measure of a device’s performance, and the somewhat dated reliance on benchmarking scores goes the way of the dinosaurs.