Building a Better (Linux) GPU Benchmark
by Kristopher Kubicki on September 24, 2004 12:05 AM EST- Posted in
- Linux
Why Average Frames Per Second Can Be Misleading
Timedemos make up the core backbone of our video card benchmarks. Timedemos are simply pre-recorded scenes that are re-rendered as the program plays them back. The example that we use here is the timedemo "radar" from the game Wolfenstien: Enemy Territory. Depending on the conditions of the testbed, radar might render more or less frames than another machine, but both machines will always take the same amount of time to play back. The program generally reports the number of frames rendered and we divide by the number of seconds that the timedemo lasted. This calculates our average frames per second.Plotting more than just the average FPS is not a new concept. AMD, Intel, NVIDIA and ATI have all expressed their thoughts to us about plotting box-and-whisker or line graphs of our timedemos rather than just an average FPS. Of course, CPU manufacturers do not like timedemos at all, but that is an argument that we will leave for a different day. Let us claim during our timedemo that we record the instantaneous frames per second several times. "Instantaneous" may be a bit of a misnomer if you do not have an explicit math background, but for argument's sake, let's say that we are calculating how many frames we render in exactly one second and plotting those FPS over the course of a minute. The graph may look something like this.
This line graph demonstrates a 60-second clip from a timedemo with 45 average frames per second. When we look at an average FPS, we expect the timedemo's instantaneous frames per second to look something like this. In reality, the timedemo may look something more like this extreme example below.
Both graphs have an average ~45 frames per second. In fact, the lower graph actually averages to slightly higher. We make written notes of issues like this in our Windows GPU analyses, but a line graph puts it into a format that is less subjective.
Facebook
Twitter
RSS
23 Comments
View All Comments
- Saturday, October 24, 2009 - link
http://www.goph3r.com/mh">http://www.goph3r.com/mh(air jordan, air max, shox tn, rift, puma, dunk sb, adidas) nike jordan shoes 1-24 $32
lv, coach, chane bag $35
COOGI(jeans, tshirts, hoody, jacket) $30
christian audigier(jeans, tshirts, hoody) $13
edhardy(shoes, tshirts, jeans, caps, watche, handbag) $25
Armani(jeans, tshirts,) $24
AF(jeans, coat, hoody, sweater, tshirts)Abercrombie & Fitch $31
http://www.goph3r.com/mh">http://www.goph3r.com/mh
quanta - Friday, October 1, 2004 - link
If you are going to compare image quality, what is the basis of comparison? That means that a reference (software) renderer is needed. But even then, choosing right technique means ones need to draw extremely detailed scenes similar to CAD programs, but such renditions will be too slow. And of course, there's the issue of how to objectively define the 'right' way vs. 'wrong' way of optimization in ways that most can agree, and can be done by computers.In any case, we can't afford yet another GPU benchmark that is easily exploited (in bad ways, that is).
tygrus - Tuesday, September 28, 2004 - link
Nice start.I get frustrated by companies that delay support for <5% of the market because of small numbers when its the lack of support that is causing the low numbers. If ATI improved the drivers for linux then more people would use their existing ATI cards under linux and more importantly more people will buy ATI for their linux system instead of Nvidia. It's the issue of "what came first, the chicken or the egg", when their attitude should be more like "build it and they will come".
TrogdorJW - Tuesday, September 28, 2004 - link
Just a quick question here, as I see a potential performance issue with the "hacked" OpenGL/SDL framerate utility. You mention that the software will automatically capture screen shots periodically. That ought to increase the demand on the HDD and potentially the rest of the system for brief instants when the screen captures are done. Hopefully there will be multiple benchmarking runs done, one with the screen captures and one without. Although, I suppose we have to see what sort of difference the screen captures actually make in performance first. :)raylpc - Saturday, September 25, 2004 - link
I agree with fic. I'm going to buy a new AMD64 system and I will get Nvidia just because their linux support is way superior than ATi's, although ATi does make better cards.aaime - Saturday, September 25, 2004 - link
I hope you can add some 2D benchmarks as well, since, for example, the 2D performance of the ATI drivers really suck! A 2D perf. benchmark should involve also use of Render, text antialiasing and such to be representative.Anyway, thank you, it's nice to see some attention to the Linux world from you :-)
fic - Friday, September 24, 2004 - link
Probably the big reason that only 4% of owners of ATI cards run linux is that their support sucks. This is a self fulfilling thing. ATI doesn't support linux therefore if you are going to run linux you don't use ATI. I image that a huge percentage of linux users use Nvidia because they have decent drivers.Brule - Friday, September 24, 2004 - link
Great article. That's why they call it computer "science". Looking forward to the results on a personal linux-using level as well.javalino - Friday, September 24, 2004 - link
Well. I hope this type of benchmark didnt effect the continuos windows benchmarks. As the ATI said this week, 4% of owers of ati cards run linux, so, not much people want to see linux benchmarks, unless they are better than windows ones.Illissius - Friday, September 24, 2004 - link
Amazing. I've been looking for a tool like this for Linux for ages. Gimme now! Gimme now!