Server Info
Truenas SCALE uses a suoeruser accound admin
to log into the UI
Reserved IP addresses:
- 192.168.1.5 (Server) Password = "Dev" user from experiment server
- 192.168.1.11 (IPMI) Password = 8-character limited version of "services" password
BIOS Settings (SuperMicro X10DRH-C):
- Enable Bifurcation on PCIe slots 6 & 7 for NVME Drives:
- Go to Chipset settings -> "IIO
x
Configuration" (wherex
=0
for CPU 1 or1
for CPU 2) - Select "IOUx Configuration" where
x
= the PCIe lane set desired:-
IIO0/IOU0
= PCIe SLOTS 1 AND 2 (DEFAULT x8/x8 BIFURCATION) -
IIO0/IOU1
= PCIE SLOT 3 AND DUAL INTEL NICs (DEFAULT x8/x8 BIFURCATION) -
IIO0/IOU2
= ON-BOARD LSI SAS CONTROLLER (DEFAULT x8, NO BIFURCATION) -
IIO1/IOU0
= PCIe SLOTS 6 AND 7 (DEFAULT x8/x8 BIFURCATION) -
IIO1/IOU1
= PCIe SLOT 4 (FULL x16 SLOT, DEFAULT x16, NO BIFURCATION) -
IIO1/IOU2
= PCIe SLOT 5 (DEFAULT x8, NO BIFURCATION)
-
- Change Bifurcation settings to x4/x4/x4/x4 on
IIO1/IOU0
, to enable 2 sets of x4 lanes on both Slots 6 and 7. Each NVME drive uses x4 lanes, so Bifurcation must be enabled! - Future plan to install the GTX 1050 GPU: Install in Slot 4 and DO NOT enable Bifurcation!
- Go to Chipset settings -> "IIO
- Set physical jumper
JPS1
(bottom left corner of board next to SAS controller/SATA ports) to "DISABLE", this turns off the unused SAS controller, improves boot time and decreases power usage.
SYSTEM FANS:
System has no user fan controls, only generic fan profiles in IPMI. System will run fans as intended, until fan RPM (on ANY fan port!) runs too high or too low, including if a fan falls above/below critical tolerances, ALL FANS jump to 100% until system is cold booted (via IPMI or removing power COMPLETELY!)
After cold boot, system tests all fan headers, and only monitors the headers with fans on them. If a fan is unplugged or moved to an unpopulated header, this counts as out of tolerance! Populated headers are remembered across boots, until cold boot.
<B> Problem:</B> Factory fan settings are set for 1u chassis fans, with a low/high tolerance of 3000/35,000 RPM! Normal PC fans quickly fall out of tolerance!
<B> Fix:</B> Use ipmitool
in OS shell (included with TN Scale) to set fan limits:
-
sudo ipmitool sensor
prints all MOBO sensors detected, including fan speeds: - to change a fan header's thresholds, use two commands per header:
-
sudo ipmitool sensor thresh FANx upper 1500 1700 1900
-
sudo ipmitool sensor thresh FANx lower 200 300 500
-
- valid header names are
FAN1
,FAN2
,FAN3
,FAN4
,FAN5
,FAN6
,FANA
, andFANB
- Numbers after
upper/lower
are thresholds, always lowest value first:-
non-critical limit (
500/1500
in example): alarm will trip, but will reset once value comes within tolerances again. System will begin adjusting PWM to fix this error. -
critical limit (
300/1700
in example): A different alarm trip, also resets once cleared, but is considered more serious. system will temporarily bump PWM to 100% to try to "jump start" the fan to spin correctly, and clear error if issue is fixed. -
non-recoverable limit (
200/1900
in example): System assumes fan has died, so it throws a non-clearable alarm and ramps all fans to 100%! Only clearable with a cold boot.
-
non-critical limit (
- USE THE FAN'S TOLERANCES AS A BASELINE!! Example thresholds in above step are for a Noctua NF-P12 fan,currently on the CPU coolers. MAX RPM speed is 1300 RPM, so the smallest upper limit is set 100-200 RPM above. Same with lower thresholds: NF-P12 can run as low as ~600 RPM, so set as such.
- fan RPM settings persist through reboots, cold or otherwise. Likely reset by BIOS battery reset.
Current
settings:
Current settings (n/a specifies no fan attached, set limits before attaching anything!:
FAN NAME
LOW N/R
LOW CRIT
LOW NON-CRIT
HIGH N/C
HIGH CRIT
HIGH N/R
FAN1
|na
na
|na
|na
na
|na
na
|
na | na | na | na | na
FAN2 (Noctua/CPU)|
600.000 | RPM | ok | 200.000
| 300.000
| 500.000
| 1500.000
| 1700.000
| 1900.000
FAN3
|na
na
|na
|na
na
|na
na
|
na | na | na | na | na
FAN4 (Case Fan)
| 1200.000 | RPM | ok | 300.000
| 500.000
| 700.000
| 3000.000
| 3200.000
| 3300.000
FAN5 (Noctua/CPU)|
600.000 | RPM | ok | 200.000
| 300.000
| 500.000
| 1500.000
| 1700.000
| 1900.000
FAN6 (Case Fan)
| 1300.000 | RPM | ok | 300.000
| 500.000
| 700.000
| 3000.000
| 3200.000
| 3300.000
FANA
|na
na
|na
|na
na
|na
na
|
na | na | na | na | na
FANB (Case Fan)
| 1300.000 | RPM | ok | 300.000
| 500.000
| 700.000
| 3000.000
| 3200.000
| 3300.000