Raid на серверах HP
Мне достались 2 сервера с разными контроллерами
lspci -s 09:00.0 -vv
09:00.0 RAID bus controller: Hewlett-Packard Company Smart Array G6 controllers (rev 01)
Subsystem: Hewlett-Packard Company Smart Array P410
lspci -s 06:00.0 -vv
06:00.0 RAID bus controller: Hewlett-Packard Company Smart Array Gen9 Controllers (rev 01)
Subsystem: Hewlett-Packard Company H240
P410
lspci | grep -i raid
09:00.0 RAID bus controller: Hewlett-Packard Company Smart Array G6 controllers (rev 01)
lspci -vv -s 09:00.0
09:00.0 RAID bus controller: Hewlett-Packard Company Smart Array G6 controllers (rev 01)
Subsystem: Hewlett-Packard Company Smart Array P410
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR- FastB2B- DisINTx+
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 256 bytes
Interrupt: pin A routed to IRQ 37
Region 0: Memory at fbc00000 (64-bit, non-prefetchable) [size=2M]
Region 2: Memory at fbbff000 (64-bit, non-prefetchable) [size=4K]
Region 4: I/O ports at d800 [size=256]
Expansion ROM at fbb00000 [disabled] [size=512K]
Capabilities: [40] Power Management version 3
Flags: PMEClk- DSI- D1- D2- AuxCurrent=375mA PME(D0+,D1-,D2-,D3hot-,D3cold-)
Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
Capabilities: [50] MSI: Enable- Count=1/1 Maskable- 64bit+
Address: 0000000000000000 Data: 0000
Capabilities: [70] Express (v2) Endpoint, MSI 00
DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s <1us, L1 <8us
ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 0.000W
DevCtl: CorrErr- NonFatalErr+ FatalErr+ UnsupReq-
RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
MaxPayload 256 bytes, MaxReadReq 4096 bytes
DevSta: CorrErr+ NonFatalErr- FatalErr- UnsupReq+ AuxPwr- TransPend-
LnkCap: Port #0, Speed 5GT/s, Width x8, ASPM not supported
ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk-
ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
LnkSta: Speed 2.5GT/s (downgraded), Width x8 (ok)
TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
DevCap2: Completion Timeout: Not Supported, TimeoutDis+ NROPrPrP- LTR-
10BitTagComp- 10BitTagReq- OBFF Not Supported, ExtFmt- EETLPPrefix-
EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
FRS- TPHComp- ExtTPHComp-
AtomicOpsCap: 32bit- 64bit- 128bitCAS-
DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR- OBFF Disabled,
AtomicOpsCtl: ReqEn-
LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
Compliance De-emphasis: -6dB
LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete- EqualizationPhase1-
EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest-
Retimer- 2Retimers- CrosslinkRes: unsupported
Capabilities: [ac] MSI-X: Enable+ Count=16 Masked-
Vector table: BAR=0 offset=001c2000
PBA: BAR=0 offset=001c4000
Capabilities: [100 v1] Advanced Error Reporting
UESta: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
UEMsk: DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt+ RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol-
UESvrt: DLP+ SDES+ TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
CESta: RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
CEMsk: RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
AERCap: First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
HeaderLog: 00000000 00000000 00000000 00000000
Kernel driver in use: hpsa
Kernel modules: hpsa
Утилиты
(я доставал файлы так как в пакете сломаны зависимости)
wget https://downloads.linux.hpe.com/SDR/repo/mcp/ubuntu/pool/non-free/hpacucli_9.40.1-1._amd64.deb
Файл:Hpacucli 9.40.1-1. amd64.deb
Сокращения, которые можно использовать в командах
chassisname
= ch
controller
= ctrl
logicaldrive
= ld
physicaldrive
= pd
drivewritecache
= dwc
Просмотр конфигурации
hpacucli ctrl all show config
Smart Array P410 in Slot 1 (sn: PACCR9SYRIHU )
unassigned
physicaldrive 2I:1:3 (port 2I:box 1:bay 3, SAS, 300 GB, OK)
SEP (Vendor ID PMCSIERA, Model SRC 8x6G) 250 (WWID: 5001438006116C2F)
Тут в примере - один диск который включен но никак не настроен
Более детально:
hpacucli ctrl all show config detail
Smart Array P410 in Slot 1
Bus Interface: PCI
Slot: 1
Serial Number: PACCR9SYRIHU
Cache Serial Number: PAAVP9SYYM4C
RAID 6 (ADG) Status: Disabled
Controller Status: OK
Hardware Revision: C
Firmware Version: 6.64
Wait for Cache Room: Disabled
Surface Analysis Inconsistency Notification: Disabled
Post Prompt Timeout: 0 secs
Cache Board Present: True
Cache Status: Not Configured
Drive Write Cache: Disabled
Total Cache Size: 512 MB
Total Cache Memory Available: 400 MB
No-Battery Write Cache: Disabled
Battery/Capacitor Count: 0
SATA NCQ Supported: True
unassigned
physicaldrive 2I:1:3
Port: 2I
Box: 1
Bay: 3
Status: OK
Drive Type: Unassigned Drive
Interface Type: SAS
Size: 300 GB
Rotational Speed: 10000
Firmware Revision: HPD7
Serial Number: 6XP5CLK80000B335CPL7
Model: HP EG0300FBLSE
PHY Count: 2
PHY Transfer Rate: 6.0Gbps, Unknown
SEP (Vendor ID PMCSIERA, Model SRC 8x6G) 250
Device Number: 250
Firmware Version: RevC
WWID: 5001438006116C2F
Vendor ID: PMCSIERA
Model: SRC 8x6G
Статус контроллера
hpacucli ctrl all show status
Smart Array P410 in Slot 1
Controller Status: OK
Cache Status: Not Configured
Управление кэшированием
hpacucli ctrl slot=1 modify dwc=disable
Error: This operation is not supported with the current configuration. Use the
"show" command on devices to show additional details about the
configuration.
Reason: Controller not configured
hpacucli ctrl slot=1 modify dwc=enable
Warning: Without the proper safety precautions, use of write cache on physical
drives could cause data loss in the event of power failure. To ensure
data is properly protected, use redundant power supplies and
Uninterruptible Power Supplies. Also, if you have multiple storage
enclosures, all data should be mirrored across them. Use of this
feature is not recommended unless these precautions are followed.
Continue? (y/n)
У меня не включился, думаю из-за
Battery/Capacitor Count: 0
Пересканировать устройства
Пересканировать устройства, добавленные с момента последнего сканирования
hpacucli rescandetects newly added devices since the last rescan
В моем случае ни к чему не привело - диск нашелся сам по себе
Просмотр информации по отдельному диску
hpacucli ctrl slot=1 pd 2I:1:3 show detail
Smart Array P410 in Slot 1
unassigned
physicaldrive 2I:1:3
Port: 2I
Box: 1
Bay: 3
Status: OK
Drive Type: Unassigned Drive
Interface Type: SAS
Size: 300 GB
Rotational Speed: 10000
Firmware Revision: HPD7
Serial Number: 6XP5CLK80000B335CPL7
Model: HP EG0300FBLSE
PHY Count: 2
PHY Transfer Rate: 6.0Gbps, Unknown
Во многих примерах приводят pd 2:3
(2-х значный адрес) вместо pd 2I:1:3
(3-х значный), возможно связано с версией контроллера или утилит.
Сам адрес диска, если он не известен можно посмотреть в hpacucli ctrl all show config detail
physicaldrive 2I:1:3
Port: 2I
Box: 1
Bay: 3
Статус дисков
Все диски
hpacucli ctrl slot=1 pd all show status
physicaldrive 2I:1:3 (port 2I:box 1:bay 3, 300 GB): OK
Или отдельный диск (а он тут всего один)
hpacucli ctrl slot=1 pd 2I:1:3 show status
physicaldrive 2I:1:3 (port 2I:box 1:bay 3, 300 GB): OK
Очистка
hpacucli ctrl slot=1 pd 2I:1:3 modify erase
Управление диодами на дисках
hpacucli ctrl slot=1 pd 2I:1:3 modify led=on
hpacucli ctrl slot=1 pd 2I:1:3 modify led=off
Показать информацию (детально)
hpacucli ctrl slot=1 ld all show detail
Error: The specified device does not have any logical drives.
Логических дисков пока нет
Создание дисков
логический диск (отдельный диск). Команда не возвращает вывод, результат смотреть отдельно.
hpacucli ctrl slot=1 create type=ld drives=2I:1:3 raid=0
Просмотреть что вышло?
hpacucli ctrl slot=1 ld all show detail
Smart Array P410 in Slot 1
array A
Logical Drive: 1
Size: 279.4 GB
Fault Tolerance: 0
Heads: 255
Sectors Per Track: 32
Cylinders: 65535
Strip Size: 256 KB
Full Stripe Size: 256 KB
Status: OK
Caching: Enabled
Unique Identifier: 600508B1001C8823B6E46FDAC58243EC
Disk Name: /dev/sdb
Mount Points: None
Logical Drive Label: A7044617PACCR9SYRIHUC7C6
Drive Type: Data
dmesg -T | tail -10
[Sun Sep 3 11:34:03 2023] loop7: detected capacity change from 0 to 8
[Sun Sep 3 16:36:18 2023] hpsa 0000:09:00.0: scsi 6:0:2:0: masked Direct-Access HP EG0300FBLSE PHYS DRV SSDSmartPathCap- En- Exp=0
[Mon Sep 4 08:48:25 2023] hpsa 0000:09:00.0: scsi 6:1:0:0: added Direct-Access HP LOGICAL VOLUME RAID-0 SSDSmartPathCap- En- Exp=1
[Mon Sep 4 08:48:25 2023] scsi 6:1:0:0: Direct-Access HP LOGICAL VOLUME 6.64 PQ: 0 ANSI: 5
[Mon Sep 4 08:48:25 2023] sd 6:1:0:0: [sdb] 585871964 512-byte logical blocks: (300 GB/279 GiB)
[Mon Sep 4 08:48:25 2023] sd 6:1:0:0: [sdb] Write Protect is off
[Mon Sep 4 08:48:25 2023] sd 6:1:0:0: [sdb] Mode Sense: 6b 00 00 08
[Mon Sep 4 08:48:25 2023] sd 6:1:0:0: Attached scsi generic sg2 type 0
[Mon Sep 4 08:48:25 2023] sd 6:1:0:0: [sdb] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
[Mon Sep 4 08:48:25 2023] sd 6:1:0:0: [sdb] Attached SCSI disk
Примеры из интерента которые не проверял
логический диск — чередование
hpacucli> ctrl slot=0 create type=ld drives=1:13,1:14 size=300 raid=1
логический диск — raid 5
hpacucli> ctrl slot=0 create type=ld drives=1:13,1:14,1:15,1:16,1:17 raid=5drives — конкретные диски, все диски (all drives) или неиспользуемые диски (unassigned drives)
size -размер логического диска в Мб
raid — тип raid 0, 1 , 1+0 and 5
Удаление
hpacucli> ctrl slot=0 ld 4 delete
Добавление дисков
hpacucli> ctrl slot=0 ld 4 add drives=2:3
Добавление дискового пространства (принудительно)
hpacucli ctrl slot=0 ld 4 modify size=500 forced
Добавление резервного диска spare
hpacucli> ctrl slot=0 array all add spares=1:5,1:7
smartctl
- В части
-d cciss,0
0 - это номер диска (а так как их в логическом устройстве всего один, то номер у него нулевой). Для более сложных рейдов будут доступны и другие значения.
smartctl -x /dev/sdb -d cciss,0 --all
smartctl -x /dev/sdb -d cciss,0 --all
smartctl 7.2 2020-12-30 r5155 [x86_64-linux-5.15.0-58-generic] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Vendor: HP
Product: EG0300FBLSE
Revision: HPD7
Compliance: SPC-3
User Capacity: 300,000,000,000 bytes [300 GB]
Logical block size: 512 bytes
Rotation Rate: 10000 rpm
Form Factor: 2.5 inches
Logical Unit id: 0x5000c5005fbbaa93
Serial number: 6XP5CLK80000B335CPL7
Device type: disk
Transport protocol: SAS (SPL-3)
Local Time is: Mon Sep 4 09:08:17 2023 UTC
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
Temperature Warning: Enabled
Read Cache is: Enabled
Writeback Cache is: Disabled
=== START OF READ SMART DATA SECTION ===
SMART Health Status: OK
Current Drive Temperature: 29 C
Drive Trip Temperature: 65 C
Manufactured in week 11 of year 2013
Specified cycle count over device lifetime: 10000
Accumulated start-stop cycles: 151
Specified load-unload count over device lifetime: 300000
Accumulated load-unload cycles: 3129
Elements in grown defect list: 146
Error counter log:
Errors Corrected by Total Correction Gigabytes Total
ECC rereads/ errors algorithm processed uncorrected
fast | delayed rewrites corrected invocations [10^9 bytes] errors
read: 0 0 0 0 0 467758.919 0
write: 0 0 0 2 0 44630.927 1
Non-medium error count: 23
No Self-tests have been logged
Background scan results log
Status: waiting until BMS interval timer expires
Accumulated power on time, hours:minutes 72101:55 [4326115 minutes]
Number of background scans performed: 2819, scan progress: 0.00%
Number of background medium scans performed: 2819
Protocol Specific port log page for SAS SSP
relative target port id = 1
generation code = 0
number of phys = 1
phy identifier = 0
attached device type: SAS or SATA device
attached reason: unknown
reason: power on
negotiated logical link rate: phy enabled; 6 Gbps
attached initiator port: ssp=1 stp=1 smp=1
attached target port: ssp=0 stp=0 smp=0
SAS address = 0x5000c5005fbbaa91
attached SAS address = 0x5001438006116c24
attached phy identifier = 6
Invalid DWORD count = 0
Running disparity error count = 0
Loss of DWORD synchronization = 0
Phy reset problem = 0
Phy event descriptors:
Invalid word count: 0
Running disparity error count: 0
Loss of dword synchronization count: 0
Phy reset problem count: 0
relative target port id = 2
generation code = 0
number of phys = 1
phy identifier = 1
attached device type: no device attached
attached reason: unknown
reason: unknown
negotiated logical link rate: phy enabled; 1.5 Gbps
attached initiator port: ssp=0 stp=0 smp=0
attached target port: ssp=0 stp=0 smp=0
SAS address = 0x5000c5005fbbaa92
attached SAS address = 0x0
attached phy identifier = 0
Invalid DWORD count = 0
Running disparity error count = 0
Loss of DWORD synchronization = 0
Phy reset problem = 0
Phy event descriptors:
Invalid word count: 0
Running disparity error count: 0
Loss of dword synchronization count: 0
Phy reset problem count: 0