Quick Look: Comparing Vulkan & DX12 API Overhead on 3DMark
by Ryan Smith on March 25, 2017 9:00 AM EST- Posted in
- GPUs
- Radeon
- Futuremark
- GeForce
- 3DMark
- Vulkan
- DirectX 12
Earlier this week the crew over at Futuremark released a major update to their API Overhead testing tool, which is built into the larger 3DMark testing suite. The API Overhead tool, first rolled out in 2015, is a relatively straightforward test that throws increasingly large number of draw calls at a system to see how many calls a system can sustain. The primary purpose of the tool is to show off the vast improvement in draw call performance afforded by modern, low-level APIs that can efficiently spread their work over multiple threads, as opposed to classic APIs like DirectX 11 which are essentially single-threaded and have a high degree of overhead within that sole thread.
The latest iteration of the API Overhead test, now up to version 1.5, has added support for Vulkan, making it one of the first feature-level benchmarks to add support for the API. Khronos’s take on a low-level API – and a descendant of sorts of Mantle – Vulkan has been available now for just a bit over a year. However outside of a very successful outing with Doom, in the PC ream it has been flying somewhat under the radar, as few other games have (meaningfully) implemented support for the API thus far. By the end of 2017 we should be seeing some wider support for the API, but for the moment it’s still in the process of finding its footing among PC developers.
In any case, like OpenGL versus Direct3D 9/10/11 before it, there’s a lot of curiosity (and arguments) over which API is better. Now that Futuremark is supporting the API for their Overhead test, let’s take a quick look at how the two APIs compare here, and whether one API offers lower overhead than the other.
| CPU: | Intel Core i7-4960X @ 4.2GHz |
| Motherboard: | ASRock Fatal1ty X79 Professional |
| Power Supply: | Corsair AX1200i |
| Hard Disk: | Samsung SSD 840 EVO (750GB) |
| Memory: | G.Skill RipjawZ DDR3-1866 4 x 8GB (9-10-9-26) |
| Case: | NZXT Phantom 630 Windowed Edition |
| Monitor: | Asus PQ321 |
| Video Cards: | NVIDIA GeForce GTX 1080 Ti Founders Edition NVIDIA GeForce GTX 1060 Founders Edition AMD Radeon RX 480 8GB |
| Video Drivers: | NVIDIA Release 378.92 AMD Radeon Software Crimson 17.3.3 |
| OS: | Windows 10 Pro |
As a reminder for the API Overhead test, this is not a cross-system test or even a cross-GPU test. The purpose of the test is solely to measure overhead within a single setup. In practice it’s something of a combined GPU and driver test, as depending on where the bottleneck lies, the limiting factor can be CPU overhead from the driver or just outright hitting the limits of the GPU’s command processor.
The purpose of the test is to compare API performance on a single system. It should not be used to compare component performance across different systems. Specifically, this test should not be used to compare graphics cards, since the benefit of reducing API overhead is greatest in situations where the CPU is the limiting factor.
With that out of the way, let’s start somewhere in the middle of the pack with the GeForce GTX 1060 6GB.
Going into this I was not expecting Vulkan and DX12 overhead to be meaningfully different, so having run the GTX 1060 6GB first, it definitely caught me by surprise that Vulkan’s overhead was so much lower. The net result is that using the Vulkan API, the GTX 1060 can sustain 26.4M draw calls per second, 32% more than DirectX 12. And to be sure, this is consistent across multiple runs.
However before jumping to conclusions, let’s take a look at a couple of other cards.
AMD’s fastest GCN4 card and card most comparable to the GTX 1060 is the Radeon RX 480 8GB. Running the API Overhead test on this card produces something notably different from the GTX 1060. Rather than finding Vulkan well in the lead, we get not quite the inverse: DX12 is holding a small lead at 26M draw calls versus 24.9M draw calls.
The two obvious differences here are the GPU and driver – in other words, the two things that matter the most – and is why cross-GPU results are not directly comparable. However it does go to show that whatever is causing Vulkan to perform better on the GTX 1060 is not a consistent factor. Switching things up can easily put Vulkan performance on the back foot.
Finally, to give the test as much GPU power as possible, I’ve also gone ahead and run it on NVIDIA’s recently released GTX 1080 Ti. This gives us a datapoint where GPU bottlenecking has been reduced as much as possible, and also gives us another datapoint with NVIDIA’s driver set.
The results still put Vulkan in the lead, but not by anywhere near what we saw on the GTX 1060. 32.4M calls versus 31.3M calls is a much narrower 4% difference between the APIs. What this does hint at is that on NVIDIA cards, the Vulkan API path has an edge in overhead, but even within just the NVIDIA ecosystem it’s not a massive difference. As with the AMD RX 480, for the GTX 1080 Ti this is essentially a draw between the two APIs. Which to be fair, is what we’d expect to find.
Overall the latest 3DMark API Overhead benchmark proved both reassuring, and more interesting than I was expecting to find. At a high level, neither Vulkan nor DirectX 12 hold a consistent lead with regards to API overhead, which indicates that both AMD and NVIDIA have done a good job optimizing their drivers and runtimes for this API. And more to the point, no matter the API used, it’s still vastly more efficient than DirectX 11, to the point where the draw call throughput is significantly greater than anything a developer could hope to use in the real world.
However the GTX 1060 results present an interesting anomaly, with the Vulkan API path showcasing noticeably lower overhead. I do have to stress that this is absolutely academic – these low-level feature tests are designed to test one small aspect of a GPU/system, and game performance won’t be anything like this – but it is an unexpected find that hints that Vulkan and DX12 may not be so neck-and-neck at all times on NVIDIA cards. The question we’re left to ponder is whether this is product of NVIDIA’s drivers, or if there’s something at the API level that just maps a bit better to NVIDIA’s command processor…
Facebook
Twitter
RSS
47 Comments
View All Comments
SirPerro - Saturday, March 25, 2017 - link
What's the difference in features between both APIs? Is it much easier to use DX12 compared to Vulkan?I'm asking this because I don't see many reasons to use DX12, except for the answers to the questions above.
Kjella - Saturday, March 25, 2017 - link
Well both are low-level APIs, so both are complicated to use. I think most who use either will do it through some game engine, essentially shifting much of the work from Microsoft/nVidia/AMD/Intels to Unity/Unreal etc. but I'm guessing the biggest difference is platform support, as far as I know there's no Vulkan on XBone while there's no DX12 on anything but PC and XBone. I think that's far more likely to decide what you'll end up using.PseudoKnight - Saturday, March 25, 2017 - link
It's a little worse that that for DX12, as it's only available on Windows 10 (not 7, 8.1, OSX or Linux), which is just under 50% of Steam users. Currently only 42% of Steam gamers can do DX12.Freakie - Saturday, March 25, 2017 - link
Which is still the more users than DX11 Steam users. 75% of Steam users have a DX12 GPU which is rather significant, it's just that 34% of those people aren't on Windows 10 which won't last much longer in my opinion. As Microsoft gives Windows 10 more gaming features (as well as HDR support and better HiRes scaling, both coming at the end of this month with the Creator's Update) more and more people will get pulled to W10.TheJian - Saturday, March 25, 2017 - link
Umm, no...Many of us won't be switching regardless until 2020+ ;) I have never had a steam account (and most of the people I know refuse to use it also - prefer GOG etc). So while steam may show that, it's not exactly representative of the whole WORLD. DX12 is not a motivator for me in any way shape or form. Vulkan vs. DX12 game, Vulkan or you don't get my money. PERIOD. I'm guessing more devs will go Vulkan soon like star citizen guys. Win10 is dead to me and if enough stuff is vulkan by 2020, I have no need for MSFT again except for job related stuff, since I'm still on the IT treadmill, but that may change if I decide networking or linux is more attractive soon. Just a little more time and things coming to fruition and I'm out of MSFT. None of the features you mention mean much to me (4k not going to be my next monitor, etc).Freakie - Sunday, March 26, 2017 - link
I wasn't talking about non-Steam users? So not sure why you felt the need to say what you said.Also, why can't you enjoy Windows 10? Not only is it faster than Windows 7, but it has many more features. Don't want it talking to Microsoft? Your Windows 7 already does. Personally I don't want my computer talking to Microsoft, so I micromanage my firewall rules and make changes in the Registry as well as in Group Policy to prevent it. Not a single network communication goes to Microsoft without me allowing it, and right now all I allow is licensing communication for Windows and Office. Which is what I did for Windows 7 as well so it wasn't like Windows 10 was more effort for me. So Microsoft can't collect telemetry on me, I got a free upgrade to W10 so I gave them no money, and I get to enjoy all of the software features that I want, the more stable drivers, and my Operating System is setup to take advantage of all the technologies and improvements that are built into my GPU and CPU on the hardware level so I'm getting the double whammy of a better coded OS and an OS that can leverage my hardware to the fullest extent. The HiRes scaling doesn't matter to me at the moment, but the HDR does for my photo and video editing as well as my gameplay so those two things I mentioned are really just icing on the cake.
Barring just being ignorant and refusing to realize what Windows 7 is and unless there is a compatibility issue, then I can not think of a single reason to NOT be on Windows 10 if you have the sufficient technical skill. Perhaps you don't?
Arbie - Sunday, March 26, 2017 - link
TheJian, I read your comment as it claimed up front to negate Freakie's argument. But it doesn't begin to do that. Only a microscopic percentage of PC gamers will be buying graphics cards and not using Windows - hardly "many of us". This article is about the difference between Vulkan and DX12 - not between Linux or whatever and Windows. Save your rant for a different venue.benedict - Sunday, March 26, 2017 - link
Arbie, in your microscopic world everyone might be gaming on Windows. In the real world most people use their PCs for work first and then gaming second. TheJian's arguments are perfectly valid, as he points out there's plenty of people gaming on systems that are not W10 and for those we have Vulkan. I'm more than happy to see this test prove Vulkan is just as good as DX12.Freakie, the OS should serve me, not the other way round. If I have to spend time and effort to disable things that shouldn't be there in the first place, then the OS is not for me. And no, my W7 does not have any of the telemetry updates as it miraculously allows you to choose which updates to get, unlike the "upgrade" W10.
HollyDOL - Sunday, March 26, 2017 - link
quote: If I have to spend time and effort to disable things that shouldn't be there in the first place, then the OS is not for me.I wonder then what operating system are you going to use then since there isn't single one that wouldn't require getting rid of things each specific user considers useless...
kn00tcn - Wednesday, March 29, 2017 - link
benedict, not everyone in the real world plays games, but in the game world most people are windows (or console), there's nothing microscopic about this