TrueNas Setup and Optimization

22nd November 2023

Running TrueNas for a year now as a backup destination. Cheap solution; used HP server (DL380 Gen9) w 12x 3.5″ disk bays, 24 Ram slots, 10Gb nics, iLo for remote mgt, still has support for a few years. 3.5″ HDD are substantially cheaper than 2.5″. Install an M.2 boot drive, set the internal raid controller to pass-thru. reliable.

Built same for ESXi storage.
HP Gen10 for ESXi hosts.

I expected VmWare to have many times the performance of the old system as it is running 10G network and the old system was 4x1G. Not so much. Starting with a completely unoptimized vdev Raidz1 7wide

TrueNas Optimizations:
-HP default iLo nic is set to 100mbps, change that to 1G
-on Dataset set Sync to disabled or write performance will be terrible. Make sure server is on a UPS.

Write Sync ON; the right column has low write speed

Write Sync OFF; now the right column has higher write speeds.

vSphere; turn on host cache. This is just-in-case you run out of RAM, so it shouldn’t cause your boot SSD to fail prematurely.

NFS with UDP enabled results in no performance gains

Storage Layout change from Raidz1 1x vdev 7wide TO Raidz1 2x vdev 4wide. No Gains.

Switch to iScsi on the 10g interface, didn’t realize NFS used the vMotion interface

Turning on Jumbo support on the network switch only

If you want to use Jumbo(9000mtu) frames you may need to dedicate a virtual switch to the iScsi connection and use a different virtual switch with 1500mtu for Client/Server connections.

Next Optimizations:
-move vMotion interface to 10g nic

Tips:

Get a Label maker and Label each disk with the last few digits of it’s serial number on the OUTSIDE. then when a disk goes bad you won’t be guessing as to which drive to pull. Especially important when running RaidZ1 as pulling the wrong disk will crash your zVol. Some hardware will allow you to flash the disk light but don’t count on that.

If you need to rebuild TrueNas, it labels each disk and you can see which Pool they previously belonged to when creating new vDevs.