Server Info

Truenas SCALE uses a suoeruser accound admin to log into the UI

Reserved IP addresses:

• 192.168.1.5 (Server) Password = "Dev" user from experiment server
• 192.168.1.11 (IPMI) Password = 8-character limited version of "services" password

BIOS Settings (SuperMicro X10DRH-C):

• Enable Bifurcation on PCIe slots 6 & 7 for NVME Drives:
  • Go to Chipset settings -> "IIOx Configuration" (where x = 0 for CPU 1 or 1 for CPU 2)
  • Select "IOUx Configuration" where x = the PCIe lane set desired:
    • IIO0/IOU0 = PCIe SLOTS 1 AND 2 (DEFAULT x8/x8 BIFURCATION)
    • IIO0/IOU1 = PCIE SLOT 3 AND DUAL INTEL NICs (DEFAULT x8/x8 BIFURCATION)
    • IIO0/IOU2 = ON-BOARD LSI SAS CONTROLLER (DEFAULT x8, NO BIFURCATION)
    • IIO1/IOU0 = PCIe SLOTS 6 AND 7 (DEFAULT x8/x8 BIFURCATION)
    • IIO1/IOU1 = PCIe SLOT 4 (FULL x16 SLOT, DEFAULT x16, NO BIFURCATION)
    • IIO1/IOU2 = PCIe SLOT 5 (DEFAULT x8, NO BIFURCATION)
  • Change Bifurcation settings to x4/x4/x4/x4 on IIO1/IOU0, to enable 2 sets of x4 lanes on both Slots 6 and 7. Each NVME drive uses x4 lanes, so Bifurcation must be enabled!
  • Future plan to install the GTX 1050 GPU: Install in Slot 4 and DO NOT enable Bifurcation!
    
    
• Set physical jumper JPS1 (bottom left corner of board next to SAS controller/SATA ports) to "DISABLE", this turns off the unused SAS controller, improves boot time and decreases power usage.

SYSTEM FANS:

System has no user fan controls, only generic fan profiles in IPMI. System will run fans as intended, until fan RPM (on ANY fan port!) runs too high or too low, including if a fan falls above/below critical tolerances, ALL FANS jump to 100% until system is cold booted (via IPMI or removing power COMPLETELY!)

After cold boot, system tests all fan headers, and only monitors the headers with fans on them. If a fan is unplugged or moved to an unpopulated header, this counts as out of tolerance! Populated headers are remembered across boots, until cold boot.

<B> Problem: </B> Factory fan settings are set for 1u chassis fans, with a low/high tolerance of 3000/35,000 RPM! Normal PC fans quickly fall out of tolerance!

<B> Fix: </B> Use ipmitool in OS shell (included with TN Scale) to set fan limits:

• sudo ipmitool sensor prints all MOBO sensors detected, including fan speeds:
• to change a fan header's thresholds, use two commands per header:
  • sudo ipmitool sensor thresh FANx upper 1500 1700 1900
  • sudo ipmitool sensor thresh FANx lower 200 300 500
• valid header names are FAN1, FAN2, FAN3, FAN4, FAN5, FAN6, FANA, and FANB
• Numbers after upper/lower are thresholds, always lowest value first:
  • non-critical limit (500/1500 in example): alarm will trip, but will reset once value comes within tolerances again. System will begin adjusting PWM to fix this error.
  • critical limit (300/1700 in example): A different alarm trip, also resets once cleared, but is considered more serious. system will temporarily bump PWM to 100% to try to "jump start" the fan to spin correctly, and clear error if issue is fixed.
  • non-recoverable limit (200/1900 in example): System assumes fan has died, so it throws a non-clearable alarm and ramps all fans to 100%! Only clearable with a cold boot.
• USE THE FAN'S TOLERANCES AS A BASELINE!! Example thresholds in above step are for a Noctua NF-P12 fan,currently on the CPU coolers. MAX RPM speed is 1300 RPM, so the smallest upper limit is set 100-200 RPM above. Same with lower thresholds: NF-P12 can run as low as ~600 RPM, so set as such.
• fan RPM settings persist through reboots, cold or otherwise. Likely reset by BIOS battery reset.
  
  

Current settings (n/a specifies no fan attached, set limits before attaching anything!:

FAN NAME	LOW N/R	LOW CRIT	LOW NON-CRIT	HIGH N/C	HIGH CRIT	HIGH N/R
FAN1	na	na	na	na	na	na
FAN2 (Noctua/CPU)	200.000	300.000	500.000	1500.000	1700.000	1900.000
FAN3	na	na	na	na	na	na
FAN4 (Case Fan)	300.000	500.000	700.000	3000.000	3200.000	3300.000
FAN5 (Noctua/CPU)	200.000	300.000	500.000	1500.000	1700.000	1900.000
FAN6 (Case Fan)	300.000	500.000	700.000	3000.000	3200.000	3300.000
FANA	na	na	na	na	na	na
FANB (Case Fan)	300.000	500.000	700.000	3000.000	3200.000	3300.000