Recently, I fell into the soft router rabbit hole. Here’s how it happened:

1. I got Shanghai Telecom business broadband at home, along with premium network service. Upload 100Mbps, download 500Mbps. A good horse deserves a good saddle, so I planned to replace the K3 that had just been barely doing the job.
2. And so I stepped into the world of soft routers. At first, I planned to buy something around the price range of a 3850U CPU, so I eventually chose a J4150 industrial PC. I bought it at lightning speed and asked the seller to flash esxi and set up WAN passthrough. After bringing it home, I installed ROS and Debian: needless to say, ROS served as the main router (after all, it’s stable); Debian acted as a server, mainly doing ** tasks.
3. Things stayed peaceful and satisfying for a week, and then I started getting restless again. The reason was that after seeing the J4105 max out the 500M connection, CPU usage was also almost maxed out. As someone with obsessive tendencies, of course I couldn’t tolerate that, so I started looking for a more high-end (gui) solution.
4. My requirements were pretty high. First, the router had to be small; second, it had to perform well; ideally, it would be an industrial PC. I thoroughly filtered through the many industrial PCs on Taobao, and models like the 7500U and 8250U came into view. But with these oddly branded industrial PCs, first, quality couldn’t be guaranteed, and I still wanted stability; second, none of them looked very good (possibly the main reason)...
5. Then I looked into brand-name manufacturers and finally found the Zotac MI643 and the star of this article, the NUC10. The MI643 has dual network ports (tell me loudly, what can that be used for), but I heard they were from Realtek, which made me hesitate, and it was also expensive: at the time, the barebones version on Tmall was around 2850. The NUC10 Frost Canyon was priced around 2450 (on Xianyu); the single network port was frustrating, but otherwise it had no obvious flaws. After struggling for another day and browsing major soft router forums, I found that in fact NUC10 + a gigabit network card was also a feasible option.
6. Based on the principle that the NUC10 is popular and easier to troubleshoot if problems arise, I finally bought the Frost Canyon NUC10i5FNK. That same day, I bought a 2.5G network card on Pinduoduo, a Western Digital Blue 250G drive on JD, removed two memory sticks from an iMac, and officially stepped into the world of NUC.

First, a brief summary:

After a week of use, the NUC10i5FNK as a soft router has been very stable, aside from a bit of tinkering at the beginning.

Advantages:

1. It looks great, it’s compact, and its overall dimensions of 115*112*38 are truly unbeatable.
2. That’s it—looking good is enough. Just kidding~ It’s also powerful, and at a price of 2450, it’s not outrageously expensive compared with industrial PCs, especially considering the three-year warranty.

Disadvantages:

1. Of course, it only has one network port. If you want to use it as a soft router, you can only pair it with a gateway switch or add a USB network card. I’m using the USB network card solution. Relatively speaking, the card gets fairly hot, but there are no speed issues.

The adventure officially begins

When I got the NUC10 in hand, it really was tiny. Disassembly was very convenient. I inserted the newly arrived hard drive and memory, powered it on, plugged in the USB drive containing the previously prepared esxi 6.7 installation files, and began the installation—only to run into the first pitfall.

Pitfall 1: ESXI 6.7 does not natively support the NUC10 network card

Halfway through installation, ESXI reported that it could not find a network card, so the installation was terminated.

This issue was very easy to solve. A Google search for “NUC10 ESXI” led to the first result, which was a foolproof tutorial. In short, ESXI’s default NE1000 driver does not support the NUC10’s network card, but a developer has already adapted it; it just hasn’t been merged into the ESXI main branch, so you need to integrate it yourself.

The integrated driver can be found here. Since I use a Mac, I opened a Windows virtual machine to integrate it. Note that Mac PowerShell is not supported.

I wrote it back to the USB drive, and ESXI installation was successful at this point.

Pitfall 2: ESXI does not support 2.5Gbps USB network cards

In fact, before buying it, I had already checked the network card support situation for ESXI and knew that extra drivers would be needed. There were mainly two choices: the first was VMWare Flings, which seems to have a semi-official background; the second was a user-compiled RTL8152 series driver by Gomes, but it didn’t really explain clearly how to use it (at least, as an ESXI beginner, I took many detours).

The VMWare Flings driver supports gigabit USB network cards (such as ASIX88179 and RTL8152); for example, one from UGREEN is perfectly supported, but it does not support the 2.5Gpbs RTL8156 network card. The driver compiled by Gomes theoretically supports the RTL815x series. Since the one I bought from UGREEN was RTL8156, I could only choose the latter.

I ran into quite a few issues along the way, so let me just give the correct path for getting the RTL8156 network card driver working:

1. Install the vib; the driver is available at the link above. You can choose to integrate the vib directly when building ESXI, or after installing ESXI, transfer the vib file to ESXI via scp, install it from the command line, and reboot.

esxcli software vib install -v xx.vib # xx.vib is the filename

2. The most, most, most critical step: disable vmkusb. Newer versions of ESXI use vmkusb to replace deprecated modules such as usb, usbnet, and usb-storage, but the installed r8152 driver is still based on usbnet.

esxcli system module set -m=vmkusb -e=FALSE

3. After rebooting, check whether r8152 and usbnet were loaded successfully. According to the author, after vmkusb is disabled, usbnet and others should load automatically. But on my side they would not load automatically, so I could only manually use vmkload_mod. I tried commands such as esxcfg-module, but still couldn’t get it to work. If anyone knows the reason, please let me know~

vmkload_mod -l | egrep "r8152|usbnet" # Check the loading status of r8152 and usbnet
# If they were not loaded automatically, try executing the following two lines
vmkload_mod usbnet
vmkload_mod r8152

4. In theory, at this point you should be able to see the network card in the ESXI web interface, or you can also check it from the command line:

esxcli network nic list

5. Set up auto-start. Since the USB network card is not officially supported, after a reboot, the vSwitch association will be lost, so it needs to be re-associated after rebooting. This mainly references the script from VMWare Flings. You need to modify the /etc/rc.local.d/local.sh file and add the following content:

# Load modules (if you loaded them automatically, then these two lines are not needed)
vmkload_mod usbnet
vmkload_mod r8152

# My USB network card name is vmnic32. You can get this from point 4 above. Replace it with your own network card name as needed.
vusb0_status=$(esxcli network nic get -n vmnic32 | grep 'Link Status' | awk '{print $NF}')
count=0
while [[$count -lt 20 && "${vusb0_status}" != "Up"]]
do
    sleep 1
    count=$(( $count + 1 ))
    vusb0_status=$(esxcli network nic get -n vmnic32 | grep 'Link Status' | awk '{print $NF}')
done
# After waiting for the network card to come online, bind the vSwitch
if ["${vusb0_status}" = "Up"]; then
    esxcfg-vswitch -L vmnic32 vSwitch1
    #esxcfg-vswitch -M vmnic32 -p "Management Network" vSwitch1
    esxcfg-vswitch -M vmnic32 -p "VM Network1" vSwitch1
fi

At this point, ESXI could correctly recognize the network card, and setting passthrough for the NUC’s built-in network card was also no problem, though it felt like NUC performance probably didn’t really require it. And so, the ESXI adventure came to a temporary end...

The second adventure:

Since ESXI does not natively support USB network cards, could it make sense to consider Proxmox VE, which natively supports ESXI? My heart started stirring again, and on a dark and windy night, I used a USB drive containing the latest version of PVE and successfully managed to leave myself without internet.

I had done my research beforehand as well. There is less information about PVE USB network cards because, after all, it has native support, so I naively assumed my RTL8156 would also be supported. In fact, after installing PVE, I did indeed see two network cards (one built-in and one USB network card), and I was very excited. I tried using the USB network card as WAN for ROS dialing, but it never dialed successfully. Still unwilling to give up, I used the NUC built-in network card as WAN and the USB network card as LAN, and the dialing succeeded, but there was no internet access, and the negotiated speed of the USB network card was only 10Mbps...

Because I was a complete beginner with PVE, and although I had at least tinkered with a J4105 soft router on ESXI, I had absolutely no experience with PVE. In the end, I still found the clue in syslog: the USB network card was constantly connecting and disconnecting, so I guessed it was a driver problem.

Pitfall 3: PVE does not natively support the RTL8156 network card

I found the driver on GitHub, and it needed to be compiled manually. The compilation is actually very simple: just install the header files and dependencies, then compile:

# Install pve headers
wget http://download.proxmox.com/debian/pve/dists/buster/pvetest/binary-amd64/pve-headers-5.4.34-1-pve_5.4.34-2_amd64.deb
dpkg -i pve-headers-5.4.34-1-pve_5.4.34-2_amd64.deb
# Download the driver
wget https://github.com/wget/realtek-r8152-linux/archive/v2.13.20200712.tar.gz
tar zxvf v2.13.20200712.tar.gz
cd realtek-r8152-linux-2.13.20200712/
# Install build tools
apt install build-essential libelf-dev -y
# After compilation, you can get the r8152.ko module file
make
# I also looked up the original r8152 module and replaced it directly to save trouble. (Remember to back up the original file.)
cp r8152.ko /usr/lib/modules/5.4.34-1-pve/kernel/drivers/net/usb/r8152.ko

After rebooting, I found that PVE could correctly recognize the RTL8156 network card, and under ROS the USB network card could also be used as WAN for dialing. At this point, the PVE solution was also solved quite nicely. One more wordy note: under PVE, it’s best to choose the VirtIO model for the ROS network card. I found that with Intel E1000 selected, it couldn’t fully saturate the connection.

Finally

1. If you do not need 2.5Gbps, then just buy a gigabit network card, such as UGREEN’s RTL8152, which is quite good. I also tried the Dell DA300, and it can be perfectly recognized by PVE.
2. As an All in One device, the NUC10’s performance is completely sufficient. With the same tunnel maxing out 500M, CPU usage on the NUC10 is only 35%, which is more than enough. The single network port is a drawback of the NUC10, but a USB network card can completely make up for it. PVE has currently been running continuously for 6 days, and I haven’t found any problems yet.
3. If you have any questions about ESXI or PVE network card drivers (including requesting the compiled ko module for the PVE network card), feel free to leave a comment or send an email.