Comments Locked

46 Comments

Back to Article

  • jkhoward - Tuesday, November 26, 2013 - link

    Way to go Futuremark!
  • ddriver - Wednesday, November 27, 2013 - link

    So, delisting devices which toggle their CPU to run at top frequency during the test, while it is completely OK to have intel processors overclock themselves hundreds of MHz? That doesn't seem objective and sharply contrasts with the alleged "plea for objectivity" the delisting is supposed to stand for...

    I think someone should check recent financial transactions from apple to futuremark, since this is obviously yet another PR stunt from apple to damage its main competitors.
  • Rogess - Wednesday, November 27, 2013 - link

    I don't get the first part of your comment. Do you mean the turbo boost clock speed of Intel-based CPU's? That wouldn't be cheating if the CPU behaves similarly in all application as the CPU is just using the thermal headroom availible.

    And do you have any references for linking Apple to the delisting of devices by Futuremark? I would be careful to claim something like that even though I'd support any company that tries to level the playing-field.
  • ddriver - Wednesday, November 27, 2013 - link

    The reason samsung lock the CPU governor at max clocks for the duration of the tests is because UNLIKE real world scenarios, those benchmarks are really really short, as AT own article showed they are only a few milliseconds each, which is way too short time for the governor to ramp up the clocks, and the benchmarks actually run at lower clock speed, which is not what an actual real life application would to faced with a computationally intensive load, which is sustained 99% of the time.

    By locking the CPU to maximum clocks, samsung ensure that the benchmark will measure the actual performance of the chip when faced with a heavy workload instead of its performance at idling lower clock state. The reality of the situation is exactly the opposite of your claim, as any real life computationally intensive will keep the CPU at its toes, running at maximum frequency, just like samsung have provisioned for the benchmark tests.

    My reference is obviousness, just look at the very image AT has deemed most fit to put on top of this article, it contains all the devices who sell the most and take most $$$ away from apple. Naturally, since futuremark is a Finish company, looking at nokia might be more logical, and nokia also have stuff to gain from tarnishing the name of their competitors, but nowhere nearly as much as apple.

    And this is most certainly not an act of leveling the field, it is a cheap act that only aims to make more money for greedy and exploitative corporations.

    But hey, if anything, it just shows how desperate apple is, for having to resort to such lowly stunts under fear of a healthy competition.
  • thunng8 - Wednesday, November 27, 2013 - link

    3dmark is not short
  • Black Obsidian - Wednesday, November 27, 2013 - link

    It's unfortunate that you can't understand the difference between "boosting CPU speed for benchmarks while providing ABSOLUTELY NO BENEFIT to actual user programs" and "boosting CPU speed for all programs equally."

    Rather than try to come up with an Intel-like solution that actually benefits users, Samsung has decided to be lazy and just cheat on specific benchmarks. I can't tell if you just don't understand what Samsung is actually doing, or if you're trying to lay some sweet, sweet astroturf. Either way, good luck with that.
  • lukarak - Wednesday, November 27, 2013 - link

    Actual real life applications also have short jumps in CPU demand, you are not doing 3 hour 100% utilization renderings on your phone, but rather various R2S scenarios.

    By cheating, they are giving a false impression of the performance of their devices, as the measured performance is unavailable to the user. Unlike Intel's Turbo Boost.
  • steven75 - Wednesday, November 27, 2013 - link

    "And this is most certainly not an act of leveling the field, it is a cheap act that only aims to make more money for greedy and exploitative corporations like Samsung.

    But hey, if anything, it just shows how desperate Samsung is, for having to resort to such lowly stunts as cheating on benchmarks under fear of a healthy competition."

    T;FTFY
  • Solandri - Wednesday, November 27, 2013 - link

    I agree this type of cheating on benchmarks is harmful (because benchmarks are supposed to be an objective measure of performance comparable between products). But this behavior is hardly a "lowly stunt." It's actually commonplace and the accepted norm pretty much everywhere else. I know because I usually don't abide by that norm.

    - I didn't study for the SAT or GRE, I took them cold - they're supposed to measure what I know and how I think, not what I studied for.

    - For the most part I didn't cram for tests at school. Either I knew the material or I didn't. If I knew I didn't know the material well enough, I would study it regardless of if there was a test.

    - Unless my manger overrides me, I don't specially prepare my work projects for a dog and pony show. What you see is exactly what I have, nothing prepared to hide the warts and flaws. I'll do a couple run-throughs to make sure it can actually perform the behavior we want to demonstrate, but I'm not gonna hide anything that doesn't work.

    - I usually don't clean my house when expecting company. Any cleaning I do is to make entertaining the guests easier, not to impress them with the cleanliness of my abode.

    - I'm not on my best behavior to try to impress a girl on a date. I act like I normally do.

    "Put your best foot forward" seems to be a common rule of human social behavior, so it's hardly surprising that some would try to mis-apply it to benchmarks.
  • KoolAidMan1 - Friday, November 29, 2013 - link

    I'm blown away at the level of false equivocation here, wow.
  • KPOM - Wednesday, November 27, 2013 - link

    I suppose Google/Motorola is "resorting to stunts" since they, also, don't cheat on their benchmarks?
  • Cptn_Slo - Monday, December 2, 2013 - link

    Samsung clock up their chips to appear higher in benchmarks without actually being faster in the real world. Intel actually clocks up in all applications.

    If understanding computing was a benchmark for your brain then it definitely did not clock up there.
  • Krysto - Wednesday, November 27, 2013 - link

    The problem is Turbo-Boost is rarely activated in normal situations or for a very long time, so you won't benefit a lot from that Turbo-Boost anyway, but it does give Intel the opportunity to "win benchmarks", and also say that their chips are efficient, because they are usually only using the base clock speed.

    So Intel is having its cake and eating it, too, by using Turbo-Boost mainly in benchmark situations to show their chips are "high-performance", because even if it does use it say in games, it can't do it for more than a few minutes at a time, making it pretty useless anyway, while for a normal user it will mainly use the base clock speed, and have decent battery life BECAUSE it's actually using the low-performance version of the chip (i.e. not Turbo Boost).

    Bottomline, Intel's chips are 99 percent of the time running in "power saving mode", and in benchmarks they run in "high-performance mode", which makes for very misleading benchmarks, because benchmarks are supposed to tell us how fast some chips are in GENERAL, not just for the 5 minutes the benchmark is running.
  • MrX8503 - Wednesday, November 27, 2013 - link

    There is no problem with turbo boost. The reason turbo boost rarely ramps up is because users' workload is mostly light weight. Turbo boost will activate in real world heavy workloads, not just benchmarks. This is in no way the same as smartphone vendors boosting their clocks for benchmarks, but never for real world usage apps. Stop trying to give cheating vendors a pass.
  • Tanclearas - Wednesday, November 27, 2013 - link

    You have no idea what you're talking about. The base clock for my i7 is 2.4GHz, but when running things like Handbrake it will sustain boost clocks (typically close to 3.2GHz) for extended periods (over 20 minutes). I have seen Turbo boost work in other programs as well, and I don't have any "benchmark" programs on my computer at the moment.
  • sundragon - Wednesday, November 27, 2013 - link

    The issue is that Turbo Boost runs on any program that requires it, not just benchmarks... Samsung and HTC don't boost for other programs, they artificially boost for specific benchmark programs.
    Stop kissing their butts and admit this is a disservice to everyone and out right cheating.
    Apple, fwiw, boots like Intel on all apps initially then drops to below the rated - If you Google Anadtech's benchmark you'd see it's uniform across all apps not just special special for benchmarks...
  • geniekid - Wednesday, November 27, 2013 - link

    Turbo-Boost looks at process behavior and tries to optimize CPU operation for it. If the benchmark happens to meet the performance profile to activate TB, there's no problem with that. I disagree with your assertion that benchmarks are supposed to tell us how fast chips are "in general". I don't even know what you would classify as "general" behavior since I use my desktop 99% of the time for gaming - others don't. Even assuming benchmarks are supposed to measure "general" performance, it would be up to the benchmark program to generate "general" load.

    In the case of HTC/Samsung, they're not simulating some kind of load they know will cause the CPU to ramp up. They are literally looking to see if the application is Futuremark. That's something the benchmark program just can't get around, which is why it makes perfect sense to delist these devices.

    Samsung/HTC, on the other hand, literally check to see if you are running Futuremark.
  • KPOM - Wednesday, November 27, 2013 - link

    Some applications, such as video encoders, will cause Turbo Boost to keep running for as long as possible. Typing up an e-mail or editing a basic presentation isn't going to tax a CPU, so Turbo Boost won't stay on for very long unless it really needs to.

    What the OEMs are doing is detecting the benchmark, and then actively sending signals to the processors to remain on boosted or even overclocked speeds, rather than letting the CPU toggle normally as it would when a "real" application is running.
  • Cptn_Slo - Monday, December 2, 2013 - link

    So you have no idea what turbo boost does.
  • KoolAidMan1 - Friday, November 29, 2013 - link

    Except that's not how normal CPU throttling works. The offending companies put flags to boost clock speed specifically for those benchmarks. If something else demands similar performance boosts, it doesn't do it. Intel and Apple don't cheat in this way, clock speed is regulated consistently.

    This is cheating, plain and simple. The baseless accusation at Apple in the end also makes your bias very very clear.
  • tipoo - Wednesday, November 27, 2013 - link

    Turbo boost isn't the same thing as benchmark boosting. Turbo boost works on all applications and benefits them. Benchmark boosting ONLY clocks up when certain apps are detected, it doesn't run like that in games or anything else stressing the system.
  • xype - Wednesday, November 27, 2013 - link

    Yes, obviously Apple _made_ the Android handset makers cheat at the benchmarks so they could culminate the effort in delisting their devices in the Futuremark benchmark listings. It all makes sense now!

    ddriver, I know you’re probably incapable of holding one thought long enough for this, but please find me a benchmark or specification that Apple deliberately mislead their customers with in in the past few years. Battery life, GPU performance, CPU performance. Show me where they cheated in order to look better.

    What these Android vendors did is a _deliberate_ effort to game the benchmarks. That’s not a mistake or an oversight. That’s not something that Apple had any part in. And that’s what’s upsetting people. Even as an Apple customer I can admit that a lot of Android devices are better in certain areas than what Apple is producing. And that’s exactly the reason why I find such attempts to game the benchmarks retarded, because it screams "We actually don’t think our product is good enough to stand on its own merits, we need to game the system!", and that’s pathetic, doubly so because it’s absolutely not needed.
  • steven75 - Wednesday, November 27, 2013 - link

    I've seen some stretches in my time, but this one takes the top of the list.

    Samsung and HTC's decision to cheat at specific benchmarks was Apple's fault all along!
  • dylan522p - Wednesday, November 27, 2013 - link

    Huh? You do realize that Qualomm and every other ARM vendor has boost as well, they just advertise the boost clock as the nominal because they are somewhat shady in that regard.
  • mark3785 - Thursday, November 28, 2013 - link

    I'm actually stunned that someone could be so delusional that he could conjure up an Apple/Futuremark plot to discredit Samsung and HTC. Kudos on the fertile imagination!
  • KoolAidMan1 - Friday, November 29, 2013 - link

    Except that's not how normal CPU throttling works. The offending companies put flags to boost clock speed specifically for those benchmarks. If something else demands similar performance boosts, it doesn't do it. Intel and Apple don't cheat in this way, clock speed is regulated consistently.

    This is cheating, plain and simple. The baseless accusation at Apple in the end also makes your bias very very clear.
  • Notmyusualid - Thursday, December 5, 2013 - link

    ^^ +1

    I'm more surprised some here are defending the cheating...
  • Drumsticks - Tuesday, November 26, 2013 - link

    Good job people :D
  • WinterCharm - Tuesday, November 26, 2013 - link

    Good. cheating on benchmarks only serves to harm consumers who are trying to make an educated decision on which devices to buy.
  • Tehk17 - Wednesday, November 27, 2013 - link

    Go check out the 3DMark reviews in the Play Store. Grab a bag of popcorn while you're at it. Pretty funny stuff in the review section.
  • OzedStarfish - Wednesday, November 27, 2013 - link

    So much butt hurt in those reviews, but at least Futurmark is on the case to explain the situation to those less than happy about their stance, and kudos to them for doing so.
  • Tehk17 - Wednesday, November 27, 2013 - link

    Agreed
  • djscrew - Wednesday, November 27, 2013 - link

    It's a shrewd business move and nothing more. Their product is valueless if gamed and knowledge of the game is public, as it has been made by the media.
  • Sunrise089 - Wednesday, November 27, 2013 - link

    Huge props to Futuremark.

    I know AT's public position is that 'name and shame' isn't working, and instead opting for a more measured response of behind the scenes pressure to device makers plus limited public criticism, but I'm pleased Futuremark is standing up and taking the lead on this.
  • FYoung - Wednesday, November 27, 2013 - link

    I don't know how feasible it would be, but I think a preferable approach would be to enhance the software so it could not be cheated, for example by adding code that would randomly change the file name of the benchmark executable each time it is downloaded or run.
  • thunng8 - Wednesday, November 27, 2013 - link

    Your suggestion would not work .. Just think about it.
  • hlovatt - Wednesday, November 27, 2013 - link

    Great news, hope everyone else including AnandTech follow suit.
  • Creig - Wednesday, November 27, 2013 - link

    Good going, Futuremark. It's nice to see them keeping everybody honest. I believe this is only the 2nd time they've called out companies for cheating on their benchmark suite. The first time was Nvidia with 3DMark03. THAT was some over-the-top, blatant cheating, though.

    http://www.kickassgear.com/downloads/3dmark03_audi...
  • hrrmph - Wednesday, November 27, 2013 - link

    Awesome.

    Now we need something to go after phone region locking, phone locking (in general), and movie region locking.
  • tushar001 - Sunday, December 1, 2013 - link

    I agree that listing benchmarking apps to provide 100% capacity in delisted device may seems fallacious to Anand, those are benchmarking apps which determine limit of CPU & GPU under rigorous test conditions, so to overpower them is dutiful to smartphones & tabs by provide their best! That's what exactly has HTC & Samsung has done. Considering expectation of benchmarking apps, its primary objective of gadget is expected to run at its maximum capacity to provide accurate readings, so why it is needed to delist a device which specifically runs CPU & GPU at it full clocked speed isn't that limit is tested by benchmarking concept??? Now here it goes fishy that those products have overthrown benchmark's expectation so may be the cause of delisting!!! ya, its hard to digest fact...
  • Ma Deuce - Sunday, December 1, 2013 - link

    "Considering expectation of benchmarking apps, its primary objective of gadget is expected to run at its maximum capacity to provide accurate readings"

    Unfortunately when Samsung and HTC game the system, instead of accurate readings we get false highs instead.

    " so why it is needed to delist a device which specifically runs CPU & GPU at it full clocked speed isn't that limit is tested by benchmarking concept???"

    When they cheat the benchmark they are effectively making it a less useful tool for for judging real world performance. I couldn't care less how fast Samsung can make their phone run. I want to know how fast it runs for me instead. The same with benchmarks, if a benchmark only shows me how fast Samsung can run apps on their phone, it's worthless to me.

    It's good to see Futuremark take a stand and it makes their product more credible for sure.
  • Ichinisan - Wednesday, December 4, 2013 - link

    The point is, cheating makes the benchmark run faster than any other application is allowed to run. What good is that score if your game or other performance-hungry application is never allowed to run at that speed/duration? They decrease thermal restrictions temporarily to get that score. Running the processor at that speed while playing a game would melt your device. The score is automatically invalid because you can't compare real-world performance between 2 different devices...so what's the point?

    Bravo, Futuremark. Cheaters will never stop if we don't call them out.
  • meacupla - Wednesday, December 4, 2013 - link

    wow, I can't believe this many people don't understand the difference between turbo boost and cheating on benchmarks.

    The difference is simple, people.
    Turbo boost: overclocking within thermal envelope
    Cheating: overclocking above thermal envelope

    I think we all know what happens when we run a stress testing program on an overclock higher than what the heatsink can handle, right?
  • Notmyusualid - Thursday, December 5, 2013 - link

    Full respect to Futuremark.
  • Notmyusualid - Thursday, December 5, 2013 - link

    And I should add, I'm rocking a HTC One.
  • gtjg66 - Saturday, December 7, 2013 - link

    I disagree with the the delisting of these devices, when you take a look at the records in 3dmark the highest benchmark scores are achieved with LN2 cooling on the PC. Unless I'm wrong that isn't real world performance in my book, unless pouring LN2 down the throat of a CPU/GPU all day is real world.

    If a CPU/GPU can run a benchmark without crashing then that seems to be acceptable with 3dmark scores, if your going for a world record in 3dmark scores I don't think those people are going to be too concerned if they destroyed the silicon to achieve it. If that happens to any of these devices then maybe there should be a disclaimer for running the benchmark, if the device can do that without meltdown then so be it. It's not like AMD/ Nvidia haven't tried this before in their drivers to get the highest scores possible, isn't that what 3dmark is about to find out just how far the silicon can be pushed to get the highest scores possible? If Futuremark want the devices to have a score that is real world then maybe the PC benchmark scores should be invalid if overclocked on LN2.

Log in

Don't have an account? Sign up now