Supermicro SuperServer E302-9D Review: A Fanless 10G pfSense Powerhouse

Benchmarking with iPerf3 and ipgen

The iPerf3 tool serves as a quick check to ensure that the network link is up and running close to expectations. As a simple passthrough device, we expect the Supermicro SuperServer E302-9D to achieve line-rates for 10G traffic across various interfaces. We do expect the rates to go down as more processing is added in the form of firewalling and NAT. Towards this, each tested mode is started off with an iPerf3 test. Following that, we perform the sweep of various packet sizes with the pkt-gen tool. In both cases, each 10G interface set is tested separately, followed by both sets simultaneously. After both sets of experiments, the L3 forwarding test using ipgen is performed from each of the three machines in the test setup. This section discusses only the iPerf3 and ipgen results. The former includes IPsec evaluation also.

iPerf3

Commands are executed on the source, sink, and DUT using the Conductor python package described in the testing methodology section. The setup steps on the DUT for each mode were described in the previous section. Only the source and sink [Run] phases are described here.

On the sink side, two servers are spawned out and terminated after 3 minutes. The spawn and timeout refer to keywords specified by the Conductor package.
spawn0: cpuset -l 1,2 iperf3 -s -B 172.16.10.2 -p 5201
spawn1: cpuset -l 3,4 iperf3 -s -B 172.16.11.2 -p 5201
timeout180: sleep 180
step3: killall iperf3

On the source side, the first link is evaluated for 30s, followed by the second link. In the third iteration, the tests are spawned off for both links simultaneously.
spawn1: cpuset -l 1,2 iperf3 -c 172.16.10.2 -B 172.16.0.2 -P 4 -O 5 -t 35 --logfile /tmp/.1c.0.txt
timeout45: sleep 45
spawn3: cpuset -l 1,2 iperf3 -c 172.16.11.2 -B 172.16.1.2 -P 4 -O 5 -t 35 --logfile /tmp/.1c.1.txt
timeout46: sleep 45
spawn5: cpuset -l 1,2 iperf3 -c 172.16.10.2 -B 172.16.0.2 -P 4 -O 5 -t 35 --logfile /tmp/.2c.0.txt
spawn6: cpuset -l 3,4 iperf3 -c 172.16.11.2 -B 172.16.1.2 -P 4 -O 5 -t 35 --logfile /tmp/.2c.1.txt

The table below presents the bandwidth numbers obtained in various modes. The interfaces specified in the headers refer to the ones in the DUT.

Line-rates are obtained for the plain router mode. Enabling the packet filtering lowers the performance, as expected - with more rules resulting in slightly lower performance. The NAT mode doesn't exhibit much performance loss compared to the plain PF mode, but, multiple streams on different interfaces needing NAT at the same time does bring the performance more compared to the PF (No Filters) mode.

IPsec Testing using iPerf3

IPsec testing also involves a similar set of scripts, except that only the ixl2 and ixl3 interfaces of the DUT are involved. The table below presents the iPerf3 bandwidth numbers for various tested combinations of encryption and authentication algorithms. The running of the iPerf3 server on the DUT itself may result in lower than actual performance - however, the comparison against the baseline case under similar conditions can still be made.

The above numbers are low compared to the line-rate, but closely match the results uploaded to the repository specified in the the AsiaBSDCon 2015 network performance evaluation paper for a much more powerful system. Given the 60W TDP nature of the SoC and the passively cooled configuration, coupled with the absence of QuickAssist in the SKU, the numbers are passable. It must also be noted that this is essentially an out-of-the-box benchmark number, and optimizations could extract more performance out of the system (an interesting endeavour for the homelab enthusiast).

L3 Forwarding Test with ipgen

The ipgen L3 forwarding test is executed on a single machine with two of its interfaces connected to the DUT. In the evaluation testbed, this condition is satisfied by the source, sink, and the conductor as well. The ipgen tool supports scripting of a sweep of packet and transmission bandwidth combinations. The script is provided to the tool using a command of the following form:
ipgen -T ${TxIntf},${TxGatewayIP},${TxSubnet} -R ${RxIntf},${RxGatewayIP},${RxSubnet} -S $ScriptToRun -L $LogFN
where the arguments refer to the transmitter interface, the IP of the gateway to which the interface connects, and its subnet specifications, along with a similar set for the receiver interface.

Twelve distinct runs were processed, once in each of the four tested modes for each of the machines connected to the DUT. As mentioned earlier, these numbers are likely limited by the capabilities of the source (like in the case of the Compulab fitlet-XA10-LAN), but the other two machines present some interesting results that corraborate with results observed in the iPerf3 and pkt-gen benchmarks. In general, increasing the number of rules seems to noticeably affect the performance. Enabling NAT, on the other hand, doesn't have such a discernible impact compared to other configurations with similar number of rules to process.