ABOUT BENCH FORUMS PODCAST LOGIN REGISTER
PC Components
Smartphones & tablets
Systems
ENTERPRISE & IT
GUIDES

Supermicro SuperServer E302-9D Review: A Fanless 10G pfSense Powerhouse

 by Ganesh T S on July 28, 2020 3:00 PM EST
34 Comments | Add A Comment
34 Comments

pfSense Configuration for Benchmarking

A perusal of the FreeBSD firewall performance evaluation guidelines and the accompanying infrastructure helped us narrow down the scope of testing. As elaborated in the section covering the testing methodology, the DUT was configured in various states and the iPerf3 regular TCP benchmark and the pkt-gen sweep for different packet sizes were run for traffic passing through the firewall. A test of the L3 forwarding capabilities of the DUT was also performed using the ipgen benchmark while keeping in mind its stimulus-generating machine limited nature.

Supermicro E302-9D as pfSense Firewall - Benchmarked Modes
Mode DUT Commands / Rules
Router sysctl net.inet.ip.forwarding=1
pfctl -d
PF (No Filters) sysctl net.inet.ip.forwarding=1
pfctl -e
pfctl -F all
PF (Default Ruleset) sysctl net.inet.ip.forwarding=1
pfctl -e
(Additional firewall rules specified at end of sub-section)
PF (NAT Mode) sysctl net.inet.ip.forwarding=1
pfctl -e
pfctl -F all -f /home/username/nat.pf
PF (IPSec) sysctl net.inet.ip.forwarding=1
pfctl -e
(Additional firewall rules specified at end of sub-section)

The table above summarizes the different states of evaluation and the shell commands used to place the DUT in that mode.

The additional firewall rules for the PF (Default Ruleset) case (added using easyrule / firewall log view) are as below:
pass in quick on ixl2 inet from 172.16.0.0/24 to 172.16.1.0/24 flags S/SA keep state label "USER_RULE"
pass in quick on ixl2 inet from 172.16.0.0/24 to 172.16.10.0/24 flags S/SA keep state label "USER_RULE"
pass in quick on ixl3 inet from 172.16.1.0/24 to 172.16.0.0/24 flags S/SA keep state label "USER_RULE"
pass in quick on ixl3 inet from 172.16.1.0/24 to 172.16.11.0/24 flags S/SA keep state label "USER_RULE"
pass in quick on ixl0 inet from 172.16.10.0/24 to 172.16.0.0/24 flags S/SA keep state label "USER_RULE"
pass in quick on ixl0 inet from 172.16.10.0/24 to 172.16.11.0/24 flags S/SA keep state label "USER_RULE"
pass in quick on ixl1 inet from 172.16.11.0/24 to 172.16.1.0/24 flags S/SA keep state label "USER_RULE"
pass in quick on ixl1 inet from 172.16.11.0/24 to 172.16.10.0/24 flags S/SA keep state label "USER_RULE"
pass in quick on igb3 inet from 172.16.20.0/24 to 172.16.21.0/24 flags S/SA keep state label "USER_RULE"
pass in quick on igb2 inet from 172.16.21.0/24 to 172.16.20.0/24 flags S/SA keep state label "USER_RULE"

The contents of the /home/username/nat.pf file referenced in the PF (NATMode) row of the table are as below:
set limit states 100000000
nat on ixl0 from 172.16.0.0/16 to any -> ixl0
nat on ixl1 from 172.16.0.0/16 to any -> ixl1
nat on igb2 from 172.16.0.0/16 to any -> igb2
pass in quick all keep state
pass out quick all keep state

The IPsec evaluation doesn't follow the steps outlined for the other modes. Instead of using both the source and the sink, along with iPerf3 and pkt-gen programs running on either side, only the source and the DUT are used. A baseline iPerf3 run between the source and the DUT (with no IPsec communication) is used for comparison. The communication between the two sets of ports is configured for IPsec using the script template below (invoked from the shell as an argument to the setkey -f command). The previous security policies and associations are flushed prior to the invocation.
flush;
spdflush;
# Host to host ESP
# Security Associations
add 172.16.0.2 172.16.0.1 esp 0x10001 -E -A ;
add 172.16.0.1 172.16.0.2 esp 0x10002 -E -A ;
add 172.16.1.2 172.16.1.1 esp 0x10003 -E -A ;
add 172.16.1.1 172.16.1.2 esp 0x10004 -E -A ;
# Security Policies
spdadd 172.16.0.2 172.16.0.1 any -P in IPsec esp/tunnel/172.16.0.2-172.16.0.1/require;
spdadd 172.16.0.1 172.16.0.2 any -P out IPsec esp/tunnel/172.16.0.1-172.16.0.2/require;
spdadd 172.16.1.2 172.16.1.1 any -P in IPsec esp/tunnel/172.16.1.2-172.16.1.1/require;
spdadd 172.16.1.1 172.16.1.2 any -P out IPsec esp/tunnel/172.16.1.1-172.16.1.2/require;
The template above is for the DUT side, with the one on the source side being similar (the in and out are reversed in the security policies section).

The next section provides additional benchmark processing details along with the results for both iPerf3 and ipgen tests. That is followed by a discussion of pkt-gen benchmark results.

Packet Generation Options - A Quantitative Comparison Benchmarking with iPerf3 and ipgen
PRINT THIS ARTICLE
Comments Locked

34 Comments

View All Comments

  • eastcoast_pete - Tuesday, July 28, 2020 - link

    Thanks, interesting review! Might be (partially) my ignorance of the design process, but wouldn't it be better from a thermal perspective to use the case, especially the top part of the housing directly as heat sink? The current setup transfers the heat to the inside space of the unit and then relies on passive con
    vection or radiation to dispose of the heat. Not surprised that it gets really toasty in there.

  • DanNeely - Tuesday, July 28, 2020 - link

    From a thermal standpoint yes - if everything is assembled perfectly. With that design though, you'd need to screw attach the heat sink to the CPU via screws from below, and remove/reattach it from the CPU every time you open the case up. This setup allows the heatsink to be semi-permanently attached to the CPU like in a conventional install.

    You're also mistaken about it relying on passive heat transfer, the top of the case has some large thermal pads that will make contact with the tops of the heat sinks. (They're the white stuff on the inside of the lid in the first gallery photo; made slightly confusing by the lid being rotated 180 from the mobo.) Because of the larger contact area and lower peak heat concentration levels thermal pads are much less finicy about being pulled apart and slapped together than the TIM between a chip and the heatsink base.

  • Lindegren - Tuesday, July 28, 2020 - link

    Could be Solved by having the CPU on the opposite side og the board

  • close - Wednesday, July 29, 2020 - link

    Lower power designs do that quite often. The MoBo is flipped so it faces down, the CPU is on the back side of the MoBo (top side of the system) covered by a thick, finned panel to serve as passive radiator. They probably wanted to save on designing a MoBo with the CPU on the other side.

  • eastcoast_pete - Tuesday, July 28, 2020 - link

    Appreciate the comment on the rotated case; those thermal pads looked oddly out of place. But, as Lindegren's comment pointed out, having the CPU on the opposite site of this, after all, custom MB, one could have the main heat source (SoC/CPU) facing "up", and all others facing "down".
    For maybe irrational reasons, I just don't like VRMs, SSDs and similar getting so toasty in an always-on piece of networking equipment.

  • YB1064 - Wednesday, July 29, 2020 - link

    Crazy expensive price!

  • Valantar - Wednesday, July 29, 2020 - link

    I think you got tricked by the use of a shot of the motherboard with a standard server heatsink. Look at the teardown shots; this version of the motherboard is paired with a passive heat transfer block with heat pipes which connects directly to the top chassis. No convection involved inside of the chassis. Should be reasonably efficient, though of course the top of the chassis doesn't have that many or that large fins. A layer of heat pipes running across it on the inside would probably have helped.

  • herozeros - Tuesday, July 28, 2020 - link

    Neat review! I was hoping you could offer an opinion on why they elected to not include a SKU without quickassist? So many great router scenarios with some juicy 10G ports, but bottlenecks if you’re trafficing in resource intensive IPSec connections, no? Thanks!

  • herozeros - Tuesday, July 28, 2020 - link

    Me English are bad, should read “a SKU without Quickassist”

  • GreenReaper - Tuesday, July 28, 2020 - link

    The MSRP of the D-2123IT is $213. All D-2100 CPUs with QAT are >$500:
    https://www.servethehome.com/intel-xeon-d-2100-ser...
    https://ark.intel.com/content/www/us/en/ark/produc...
    And the cheapest of those has a lower all-core turbo, which might bite for consistency.

    It's also the only one with just four cores. Thanks to this it's the only one that hits a 60W TDP.
    Bear in mind internals are already pushing 90C, in what is presumably a reasonably cool location.

    The closest (at 235% the cost) is the 8-core D-2145NT (65W, 1.9Ghz base, 2.5Ghz all-core turbo).
    Sure, it *could* do more processing, but for most use-cases it won't be better and may be worse. To be sure it wasn't slower, you'd want to step up to D-2146NT; but now it's 80W (and 301% the cost). And the memory is *still* slower in that case (2133 vs 2400). Basically you're looking at rack-mount, or at the very least some kind of active cooling solution - or something that's not running on Intel.

    Power is a big deal here. I use a quad-core D-1521 as a CPU for a relatively large DB-driven site, and it hits ~40W of its 45W TDP. For that you get 2.7Ghz all-core, although it's theoretically 2.4-2.7Ghz. The D-1541 with twice the cores only gets ~60% of the performance, because it's _actually_ limited by power. So I don't doubt TDP scaling indicates a real difference in usage.

    A lower CPU price also gives SuperMicro significant latitude for profit - or for a big bulk discount.
Copyright © 2024. All rights reserved.
BENCH
TOPICS
FOLLOW
ABOUT

Log in

Don't have an account? Sign up now