Ark Hardware Evaluation

In late 2023, CAIDA evaluated a bunch of x86 mini-PCs for field nodes for the Ark Internet measurement system. We currently use Raspberry Pis. They work great but have two disadvantages:

  1. Single source. When the Raspberry Pi Foundation has supply chain problems, we have nowhere else we can buy a compatible machine.
  2. Not x86. We also use 1U rackmount servers and have started using virtual machines and Docker containers as Ark field nodes. These invariably use an x86 processor. The Raspberry pi uses an Arm processor, which requires us to build, test and maintain software for two different machine architectures.

Since we began using Raspberry Pis a decade ago, folks have started building x86 "mini PCs" which, like the Raspberry Pi, are very compact. So, we figured we'd evaluate some of them and find out if they could meet our needs. We targetted a price point comparable to the Raspberry Pi, which once we buy the board, power supply, case and MicroSD card came in around $100 each. Were we willing to spend $250, there were many additional and promising options we could have considered.

In testing, we ran into two key issues:

Overall recommendation: BMAX B2 Pro

BMAX B2 Pro: https://www.amazon.com/dp/B0CL94GNNR zzz7-zz

Overall: Satisfactory. Preferred over B2S.
Price: $110
Best attributes: Good cooling, repairable
Worst attributes: Some fan noise. Larger than the Raspberry Pi

CPU: Intel J4105. 4 x 1.5ghz burst to 2.5ghz, 4M cache (2017)
RAM: 8 gigs
Cooling: CPU blower. Speed adjustable in bios. Audible. 38C idle -> 55C under load. Minor modification to plasic case (5 mins w/ dremel tool) to enlarge the exhaust port, drops temperature back to 48C.
Rack mounting: maybe. Very tight fit.
Disk: 256gigs m.2 2880 SATA SSD. Practical to write an image using a USB adapter on another computer (like we do for the Raspberry Pi) and the install it in the machine.
Disassembly: Trivial. All screws accessible. 5 minutes.
BIOS: auto power on, select UEFI boot drive.
Power: standard 3.0A @ 12V (36 watt) positive tip barrel adapter PS like is used with tons of SOHO routers and switches. Readily replaceable.

BMAX B2 S: https://www.amazon.com/dp/B08796NB62/ sea4-us

Overall: satisfactory, but B2 Pro is better for nearly the same price.
Price: $100
Best attributes: Good cooling, repairable
Worst attributes: Some fan noise. Larger than the Raspberry Pi

CPU: 2 x 1.1ghz turbo to 2.8ghz. Intel N4020 w/ 4meg cache (2019)
RAM: 6 gigs
Cooling: CPU blower. Speed adjustable in bios. Works well. 31C idle during 22C ambient. 42C under load. Recommend minor modification to plasic case (5 mins w/ dremel tool) to enlarge the exhaust port. Without enlaring the port and on the mild bios fan setting, Temperature increases from 38C to 63C under memtester load.
Rack mounting: maybe. Would be a tight fit in 1U.
Disk: 128G eMMC, accepts trim. Can be supplemented or replaced with an m.2 2280 SATA SSD drive (m.2 SATA, not nvme, not msata) making it repairable if we damage the eMMC drive with excessive writes. The SATA SSD is also much faster than the eMMC drive.
Disassembly: Trivial. All screws accessible. 5 minutes.
BIOS: auto power on, select UEFI boot drive, fan control, excellent.
Power: standard 2.5A @ 12V (30 watt) positive tip barrel adapter PS like is used with tons of SOHO routers and switches. Readily replaceable.
Notes: has a MicroSD slot too, but it connects via USB so doesn't support trim. Thus the MicroSD should not be used as a primary drive.

If we decide we like this one, we should evaluate the B2 Pro as well. For $129 it has 8 gigs of ram and a 256gb m.2 SATA SSD. If the hardware configuration is similar (which is likely) it may be a better choice.

Beelink thin-t T4 Pro: https://www.amazon.com/dp/B09CYXHY95/ sea5-us

Overall: satisfactory
Price: $90
Best attributes: Tiny! 11cm x 11cm x 2cm tall. Shorter than and almost as small as the Raspberry Pi.
Worst attributes: Not repairable

CPU: 2 x 1.1ghz turbo to 2.4ghz. Intel N3350 w/ 2meg cache (2016)
RAM: 4 gigs
Cooling: CPU blower. Works well. Good exhaust port. Not adjustable in BIOS. Runs about 2 degrees hotter than the Beelink, which isn't much. Temperature increases from 36C to 55C under memtester load.
Rack mounting: Would easily fit on a 1U rack shelf.
Disk: 64G eMMC, accepts trim. Not replaceable. If we kill the eMMC with excessive writes, this machine would no longer be usable for Ark. It would still be bootable from USB or LAN for other purposes. Realisitically: it'd be a paperweight.
Disassembly: Easy. Peel back bumpers to access screws. 5 minutes.
BIOS: auto power on, select UEFI boot drive, generally pretty good.
Power: standard 2.5A @ 12V (30 watt) positive tip barrel adapter PS like is used with tons of SOHO routers and switches. Readily replaceable.
Notes: Though not repairable, this one has a really impressive physical design. The folks who made this knew their craft.

Ouvislite AK3V: https://www.amazon.com/dp/B0CGXG94F1/ sea6-us

Overall: do not recommend.
Price: $100
Best attributes: Repairability. Replaceable drive. 2.5" SATA slot.
Worst attributes: doesn't cool properly (reliability concern), hard to disassemble

CPU: 2 x 1.1ghz turbo to 2.4ghz. Intel N3350 w/ 2meg cache (2016)
RAM: 8 gigs
Cooling: CPU blower. Physically does not move air through the case; only over the CPU. Did not activate during testing, even under load. Not adjustable in BIOS. Ran about 10 degrees hotter than the others (42C idle at 22C ambient). Temperature increases from 42C to 54C under memtester load. Grade: poor.
Rack mounting: Does not fit in 1U of rack space.
Disk: 128G msata. Better than eMMC or MicroSD. Replaceable. Also has a 2.5" SATA slot. Neither has any sort of cooling or heat sink - the fan (if it worked) only draws air past the CPU.
Disassembly: Moderate difficulty. Remove bumpers. Unscrew. Then unscrew more. Then peel pad. Then unscrew more. Hope you remember which screws go where. 20 minutes.
BIOS: auto power on, select UEFI boot drive. Option set is not as good as the Beelink or BMAX.
Power: standard 2.5A @ 12V (30 watt) positive tip barrel adapter PS like is used with tons of SOHO routers and switches. Readily replaceable.

Notes: the copper looking thing is just chromed plastic. No contribution to cooling.

Qotom Q401P: https://www.amazon.com/dp/B07GJFYX4J/ sea7-us

Overall: satisfactory but bulky, expensive and slow compared to the others evaluated.
Price: ~$150
Best attributes: Repairability. Swiss army knife of ports. Silent.
Worst attributes: Price. Older tech.

CPU: 2 x 1.4ghz fixed. Intel Celeron 2955U w/ 2meg cache (2013)
RAM: 8 gigs
Cooling: Passive with large outer heatsink case. Cools only the chipset, not the drive or memory. Runs a little warm but better than the Ouvislite. And it's actually designed for passive cooling. The CPU heat sink has a proper thermal pad linking it to the outer case, so it does transmit heat. Temperature increases to only 45C under memtester load.
Rack mounting: Does not fit in 1U of rack space. 2" tall.
Disk: 128G msata. Better than eMMC or MicroSD. Replaceable. Also has two more mSATA slots and a 2.5" sata cable and slot.
Disassembly: Moderate difficulty. Lots of screws and you have to fiddle with it to release it from the thermal pad. 15 minutes.
BIOS: auto power on, select UEFI boot drive. Console mirroring to serial port. Hardware watchdog. Most comprehensive set of BIOS options of the set.
Power: standard 3.0A @ 12V (36 watt) positive tip barrel adapter PS like is used with tons of SOHO routers and switches. Readily replaceable.

Notes: 4 EIA-232 serial ports. 2 RJ45 gige ports. 3 mSATA SSD drives. Socketed (replaceable) RAM. Nice expandability and repairability compared to the others. The 2955U is an older CPU than the N3350 or N3350. It runs hotter and slower.

CW-X86-P1: https://www.aliexpress.us/item/3256804822274979.html zzz5-zz

See also: Manufacturer and Amazon

Overall: Failed burn-in.
Price: $130
Best attributes: Passively cooled (no fan noise). Repairable.
Worst attributes: It beeps - must modify to remove speaker. No English documentation available. Order direct from China. Runs too hot unless CPU frequency limited.

CPU: Intel N3160. 4 x 1.6ghz turbo to 2.25ghz, 2M cache (2016)
RAM: 8 gigs. Socketed; replaceable.
Cooling: Passive. Cools only the CPU, not the drive or memory. Uses thermal paste instead of a thermal pad which is a mild hassle for full disassembly/reassembly but that shouldn't impact CAIDA's use. I'm a little concerned by how sloppily the paste was applied. They used way too much paste and it spilled over on to the chip surface. It seems to work but it's silver paste and I don't know how that impact electrical conduction or if it can short the surface mount components.
Limiting the CPU frequency reduced the heat, but after running hot for a week, it crashed anyway. Rebooted successfully after a power-cycle.

 kernel:[338081.346449] mce: [Hardware Error]: CPU 0: Machine Check: 0 Bank 2: 902000000001110a

Message from syslogd@zzz5-zz at Dec 11 10:33:33 ...
 kernel:[338081.351859] mce: [Hardware Error]: TSC 0

Message from syslogd@zzz5-zz at Dec 11 10:33:33 ...
 kernel:[338081.357018] mce: [Hardware Error]: PROCESSOR 0:406c4 TIME 1702319613 SOCKET 0 APIC 0 microcode 411

Message from syslogd@zzz5-zz at Dec 11 10:33:33 ...
 kernel:[338081.372774] mce: [Hardware Error]: CPU 1: Machine Check: 0 Bank 2: 902000000001110a

Message from syslogd@zzz5-zz at Dec 11 10:33:33 ...
 kernel:[338081.372783] mce: [Hardware Error]: TSC 0

Message from syslogd@zzz5-zz at Dec 11 10:33:33 ...
 kernel:[338081.372789] mce: [Hardware Error]: PROCESSOR 0:406c4 TIME 1702319613 SOCKET 0 APIC 2 microcode 411

Message from syslogd@zzz5-zz at Dec 11 10:33:33 ...
 kernel:[338081.346449] mce: [Hardware Error]: CPU 0: Machine Check: 0 Bank 2: 902000000001110a                                                                -
Message from syslogd@zzz5-zz at Dec 11 10:33:33 ...
 kernel:[338081.351859] mce: [Hardware Error]: TSC 0                           \
Message from syslogd@zzz5-zz at Dec 11 10:33:33 ...
 kernel:[338081.357018] mce: [Hardware Error]: PROCESSOR 0:406c4 TIME 1702319613 SOCKET 0 APIC 0 microcode 411                                                 |
Message from syslogd@zzz5-zz at Dec 11 10:33:33 ...
 kernel:[338081.372774] mce: [Hardware Error]: CPU 1: Machine Check: 0 Bank 2: 902000000001110a                                                                /
Message from syslogd@zzz5-zz at Dec 11 10:33:33 ...
 kernel:[338081.372783] mce: [Hardware Error]: TSC 0                           -
Message from syslogd@zzz5-zz at Dec 11 10:33:33 ...
 kernel:[338081.372789] mce: [Hardware Error]: PROCESSOR 0:406c4 TIME 1702319613 SOCKET 0 APIC 2 microcode 411                                                 
Rack mounting: Small enough to fit in 1U and has all ports on one side which is useful for rack mounting. However, do not recommend rack-mounting passively cooled equipment. Could theoretically have a rack mount case constructed for it.
Disk: 128G MSATA SSD. Has a standard SATA port too, but no space to put a SATA drive in it.
Disassembly: trivial. 4 easily accessed screws expose the board with the socketed MSATA drive and DDR RAM.
BIOS: auto power on, select UEFI boot drive.
Power: standard 3.0A @ 12V (36 watt) positive tip barrel adapter PS like is used with tons of SOHO routers and switches. Readily replaceable. Uses a snap-in module to make it a U.S. plug. Might be possible to order with the snap-in modules for all countries' plugs?

Notes: For reasons unknown, the manufacturer included a small speaker element on the board to implement the PC startup beep codes. This means that unlike the others, this machine will emit a noticeable BEEP sound when it boots. It's soldered to the board and cannot be disabled in the bios. Might be possible to remove by mashing it with needle nose pliers. Was easy to break off the board with needle nose pliers without doing damage. Squeeze it but don't push or pull. You'll hear a snap and it'll come free.

Box includes a SATA cable and a cable to connect a CPU fan. Thoughtful but not exactly useful.

hsx-1338np-12: https://www.aliexpress.us/item/3256805331501837.html zzz6-zz

See also: Amazon Amazon Amazon

Overall: satisfactory IF the CPU frequency is limited as described.
Price: $155
Best attributes: Newer, distinctly faster tech than the others reviewed. Passively cooled. Repairable. Power cable design should make it easy to buy international cables.
Worst attributes: Runs too hot unless CPU frequency limited to about 50%. It beeps!

CPU: Intel N6005, 4x2ghz turbo to 3.3ghz, 4M cache. 2021 processor. Noticeably faster than the others.
RAM: 8 gigs
Cooling: Passively cooled. Runs VERY hot: 32C (90F) idle -> 61C (142F) full load. 142F will cause premature failure of the device. If constrained with cpupower frequency-set -g powersave and cpupower frequency-set -u 1200Mhz, cools back down to a reasonable 41C (106F). Note that these are CPU temperatures not ambient temperatures.
Heat sink only cools the CPU, not the drive or memory. The two other temperature sensors (one ostensibly from the drive) lied about the temperature so I don't know how they were affected. The case bottom near them seemed pretty hot, so I'm betting they ran failure-level hot.
Rack mounting: No. Passively cooled and too tall for 1U. External ports on both sides of the unit.
Disk: 128G M.2 2280 M-Key NMVE. Faster than sata/msata but runs hotter. Space to mount a standard 2.5" sata drive but not recommended due to lack of cooling and vent obstruction.
Disassembly: Trivial. All screws accessible. 5 minutes.
BIOS: auto power on, select UEFI boot drive.
Power: standard 4.0A @ 12V (48 watt) positive tip barrel adapter PS like is used with tons of SOHO routers and switches. Readily replaceable. Note that this device is more power hungry than the others reviewed and dissipates more heat. Uses a power brick with a disconnectable cable, so the cable can be easily replaced with international ones.

Notes: For reasons unknown, the manufacturer included a small speaker element on the board to implement the PC startup beep codes. This means that unlike the others, this machine will emit a noticeable BEEP sound when it boots. It's soldered to the board and cannot be disabled in the bios. Can be removed by squeezing it hard with needle nose pliers and then wiggling it back and forth until it pops off the board.

Box is about the same physical size as the BMAX units. Has a serial port via an RJ45 connector with an unknown pinout (web reports NOT Cisco standard). Ethernet ports run up to 2.5G instead of just 1G. Fully backwards compatible. Noticeably faster than the other machines. Disk is about twice as fast.

Unclear availability: the N6005 version is no longer for sale at the link. The N5105 is -probably- comparable; it seems to be the same generation of chip but has a slightly lower clock speed. Would want to buy quantity 1 of the N5105 unit to be sure it really is the same before buying in bulk.

Raspberry Pi 4: https://www.amazon.com/dp/B07TC2BK1X/ hnl-us

Overall: satisfactory, system currently in use
Price: $100 (4 gig), $125 (8 gig).
$65 (4-gig), $90 (8 gig), $12 heat sink case, $10 power supply, $12 32 gig high endurance MicroSD card.
Best attributes: Known performance profile. Large troubleshooting community.
Worst attributes: Availability. Single source. Different CPU architecture than VMs and docker containers.

CPU: 4 x 1.5ghz ARM w/ 1 meg cache
Power: 3.5A @ 5V (18 watt) USB-C power adapter. Readily replaceable.
Functionally, the 2-core Intel processors above are faster than the 4-core ARM processor. The Intel processor has a significantly larger SRAM cache and uses more compact instructions, allowing it to spend more time running code and less time waiting for cache fills from DRAM.


Additional possibilities: