Original Link: https://www.anandtech.com/show/2430



Introduction

One of the most controversial subjects when it comes to benchmarking graphics performance is undoubtedly Futuremark — specifically their "gaming" benchmarks, the 3DMark series. For 10 years now, we have seen graphics card reviews bicker and argue about the viability of using 3DMark. On the one hand, we have those who insist the 3DMark tools are nothing more than a synthetic graphics benchmark, encouraging heavy optimizations from the various GPU companies in order to come out on top. The other side of the equation consists of people looking for an easy way to categorize performance, plus a group of diehard benchmarkers who are in constant competition to come out on top of the ORB (Online Results Browser) charts. As with so many things in life, reality strikes more of a middle ground.

While there are occasions where the performance metrics generated by the Futuremark tools correlate well to certain real-world games, very few people are going to be interested in purchasing hardware based solely on 3DMark performance. On the other hand, there have been many occasions throughout the history of PC gaming where users have upgraded hardware purely to improve performance in the latest and greatest game. GL-Quake helped to sell thousands (millions even?) of 3dfx graphics cards, which in turn helped to kick-start our modern obsession with 3D gaming.




Take a look at the images in this article for a moment; certainly we're not the only people in the world who when first greeted by a new 3DMark have thought, "Daaaaamn! That is a sweet looking benchmark and it would make an awesome game. They should turn that concept into a real game rather than a 60 second benchmark scene." If you're with us on this one, the wait may be over... sort of.

It appears that Futuremark has been secretly hard at work on their first full retail game, and while we don't have any details on what sort of game it will be or when it will launch, they have announced the formation of Futuremark Games Studio. The plans sound ambitious, with the following statement: "For years, our fans have been asking us when we will start making games. Very soon they are going to get it - and then some!" If we're lucky, we may end up with not just one title but numerous cutting edge titles over the coming years.

That's the core of the announcement, but let's take a minute to discuss exactly why we think this is at all meaningful.



What's in a benchmark?

At their core, modern 3D games consist of a graphics engine designed to spit out polygons onto your screen, and the internal workings of these engines can vary wildly from game to game. To discount the 3DMark graphics engines as "synthetic" is missing the mark; they are no more synthetic than any other gaming engine. The problem of course is that while other gaming engines are actually used in — at the very least — one game, 3DMark exists as its own entity.

What do performance benchmarks using The Witcher, Unreal Tournament 3, Crysis, or any other title really tell us? They tell us how specific hardware happens to perform in that title; no more and no less. Sure, gaming engines are often used in multiple titles, but once the developers are done modding and tweaking things, there's no guarantee that performance of the engine in one title will be the same as what we find in another title. Consider Gears of War, Bioshock, and Unreal Tournament 3 as an example; generally speaking, performance is similar but rarely identical.






Things are further complicated when you consider that a benchmark of one specific area of a game/game engine doesn't tell you much about how hardware might perform in other areas. Take Oblivion as an example: performance in towns, dungeons, outdoor areas, underwater, and the gates can and does differ; if you want a truly comprehensive look at Oblivion performance, you would need to benchmark representative selections of all those environments. That's precisely what we did when Oblivion first launched, and we found that some while the GPU was clearly the bottleneck in outdoor environments, other areas demanded more of the CPU.

But is that really necessary? Most people are really only concerned with worst-case performance; you might be able to get 80 FPS or more inside a town, but if you drop to 15 FPS in outdoor environments it is unlikely you will be happy with the overall experience. We can substitute any other title in place of OblivionCrysis for example has some seriously taxing levels towards the end of the game, where frame rates might be less than one fourth of what you experience at the beginning of the game. The net result is that any benchmark is only a snapshot of performance.

When we look at performance in a variety of benchmarks, we do so in order that we might gain an overall impression of how hardware actually performs. History helps explain why we do this. Which is the faster architecture: Pentium 4 or Athlon XP? The answer: it depends on the application. Which GPU architecture is better: G70 or R500? Again, it depends what game you're playing. It makes our job easy when we get hardware that is clearly faster in virtually every situation, but that rarely happens. Driver and compiler optimizations can even help turn a mutt into a purebred — sort of like DNA tampering.



Enter the Futuremark Games Studio

Why is it that the ideal review needs to look at performance in a large variety of applications/games? It is precisely because it is difficult (if not impossible) to predict performance without such a broad selection of performance results. If we run performance benchmarks on a dozen applications and a component comes out ahead of its competition in all 12 tests, it's reasonably safe to state that this component is going to be faster in the majority of applications. This is currently the case with Intel's Core 2 pitted against AMD's Athlon X2 — and now Phenom — processors. More often, we encounter situations where some applications perform better on one architecture and the remainder are faster on the competition. Depending on the margins of victory, and even more importantly depending on how individual users plan to use their systems, which component is "better" is a matter of perspective.




If Futuremark Games Studio (FGS) can remain true to their roots and release games that include useful benchmarking tools, even better. It's not that difficult to include benchmarking tools with a game that will provide a very accurate overview of performance, but too few developers take the time to do so. Of course, there's a difference between benchmarking a good game and benchmarking junk. Before FGS can become relevant, they need to prove they can actually make games. We'll have to hold off on leveling a verdict in that area for a while. Regardless, getting more titles from more developers is never a bad thing, and if the games have good graphics and they use the graphics engines from the 3DMark utilities, ORB results take one large step away from being purely synthetic.

We're optimistic about what Futuremark Games Studio will be able to do in the gaming market, and with the resources of Futuremark behind them we will hopefully get to see creative new designs rather than cookie-cutter clones pushed out by corporate think-tanks. What we don't know (yet - we're trying to get more details and will update this article if/when they become available) is what sort of games they're planning to release, when they're planning to release them, and whether they'll be PC-exclusive or multi-platform. Given that the press release mentions "new IP", the door is wide open.

Log in

Don't have an account? Sign up now