web-dev-qa-db-ja.com

オフラインUncorrectableSectorとCurrentPendingSectorの電子メール-どうすればよいですか?

Centos 7の実行:

サーバーの1つでyum updateに続いてrebootを実行したところ、2つのメールが届きました。

最初:

OfflineUncorrectableSector
Device: /dev/sdb [SAT], 28 Offline uncorrectable sectors

2番目:

CurrentPendingSector
Device: /dev/sdb [SAT], 24 Currently unreadable (pending) sectors

これがsmartctl -x /dev/sdbです:

smartctl 6.5 2016-05-07 r4318 [x86_64-linux-3.10.0-957.1.3.el7.x86_64] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     HGST Ultrastar 7K6000
Device Model:     HGST HUS726060ALE610
Serial Number:    K1KLXSNN
LU WWN Device Id: 5 000cca 255f2e12a
Firmware Version: APGNT907
User Capacity:    6,001,175,126,016 bytes [6.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    7200 rpm
Form Factor:      3.5 inches
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ACS-2, ATA8-ACS T13/1699-D revision 4
SATA Version is:  SATA 3.1, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is:    Thu Dec 13 14:58:00 2018 CET
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
AAM feature is:   Unavailable
APM level is:     254 (maximum performance)
Rd look-ahead is: Enabled
Write cache is:   Enabled
ATA Security is:  Disabled, NOT FROZEN [SEC1]
Wt Cache Reorder: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x82) Offline data collection activity
                                        was completed without error.
                                        Auto Offline Data Collection: Enabled.
Self-test execution status:      (   0) The previous self-test routine completed
                                        without error or no self-test has ever
                                        been run.
Total time to complete Offline
data collection:                (  113) seconds.
Offline data collection
capabilities:                    (0x5b) SMART execute Offline immediate.
                                        Auto Offline data collection on/off support.
                                        Suspend Offline collection upon new
                                        command.
                                        Offline surface scan supported.
                                        Self-test supported.
                                        No Conveyance Self-test supported.
                                        Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine
recommended polling time:        (   2) minutes.
Extended self-test routine
recommended polling time:        ( 898) minutes.
SCT capabilities:              (0x003d) SCT Status supported.
                                        SCT Error Recovery Control supported.
                                        SCT Feature Control supported.
                                        SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAGS    VALUE WORST THRESH FAIL RAW_VALUE
  1 Raw_Read_Error_Rate     PO-R--   100   100   016    -    0
  2 Throughput_Performance  P-S---   062   062   054    -    2796
  3 Spin_Up_Time            POS---   100   100   024    -    0
  4 Start_Stop_Count        -O--C-   100   100   000    -    3
  5 Reallocated_Sector_Ct   PO--CK   100   100   005    -    25
  7 Seek_Error_Rate         PO-R--   100   100   067    -    0
  8 Seek_Time_Performance   P-S---   128   128   020    -    18
  9 Power_On_Hours          -O--C-   100   100   000    -    5790
 10 Spin_Retry_Count        PO--C-   100   100   060    -    0
 12 Power_Cycle_Count       -O--CK   100   100   000    -    3
192 Power-Off_Retract_Count -O--CK   099   099   000    -    1257
193 Load_Cycle_Count        -O--C-   099   099   000    -    1257
194 Temperature_Celsius     -O----   120   120   000    -    50 (Min/Max 25/61)
196 Reallocated_Event_Count -O--CK   100   100   000    -    25
197 Current_Pending_Sector  -O---K   100   100   000    -    24
198 Offline_Uncorrectable   ---R--   100   100   000    -    28
199 UDMA_CRC_Error_Count    -O-R--   200   200   000    -    0
                            ||||||_ K auto-keep
                            |||||__ C event count
                            ||||___ R error rate
                            |||____ S speed/performance
                            ||_____ O updated online
                            |______ P prefailure warning

General Purpose Log Directory Version 1
SMART           Log Directory Version 1 [multi-sector log support]
Address    Access  R/W   Size  Description
0x00       GPL,SL  R/O      1  Log Directory
0x01           SL  R/O      1  Summary SMART error log
0x02           SL  R/O      1  Comprehensive SMART error log
0x03       GPL     R/O      1  Ext. Comprehensive SMART error log
0x04       GPL,SL  R/O      8  Device Statistics log
0x06           SL  R/O      1  SMART self-test log
0x07       GPL     R/O      1  Extended self-test log
0x08       GPL     R/O      2  Power Conditions log
0x09           SL  R/W      1  Selective self-test log
0x10       GPL     R/O      1  SATA NCQ Queued Error log
0x11       GPL     R/O      1  SATA Phy Event Counters log
0x12       GPL     R/O      1  SATA NCQ NON-DATA log
0x15       GPL,SL  R/W      1  SATA Rebuild Assist log
0x21       GPL     R/O      1  Write stream error log
0x22       GPL     R/O      1  Read stream error log
0x24       GPL     R/O    256  Current Device Internal Status Data log
0x25       GPL     R/O    256  Saved Device Internal Status Data log
0x30       GPL,SL  R/O      9  IDENTIFY DEVICE data log
0x80-0x9f  GPL,SL  R/W     16  Host vendor specific log
0xe0       GPL,SL  R/W      1  SCT Command/Status
0xe1       GPL,SL  R/W      1  SCT Data Transfer

SMART Extended Comprehensive Error Log Version: 1 (1 sectors)
Device Error Count: 66 (device log contains only the most recent 4 errors)
        CR     = Command Register
        FEATR  = Features Register
        COUNT  = Count (was: Sector Count) Register
        LBA_48 = Upper bytes of LBA High/Mid/Low Registers ]  ATA-8
        LH     = LBA High (was: Cylinder High) Register    ]   LBA
        LM     = LBA Mid (was: Cylinder Low) Register      ] Register
        LL     = LBA Low (was: Sector Number) Register     ]
        DV     = Device (was: Device/Head) Register
        DC     = Device Control Register
        ER     = Error register
        ST     = Status register
Powered_Up_Time is measured from power on, and printed as
DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes,
SS=sec, and sss=millisec. It "wraps" after 49.710 days.

Error 66 [1] occurred at disk power-on lifetime: 5009 hours (208 days + 17 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER -- ST COUNT  LBA_48  LH LM LL DV DC
  -- -- -- == -- == == == -- -- -- -- --
  40 -- 53 00 38 00 00 14 e0 43 c8 40 00  Error: UNC 56 sectors at LBA = 0x14e043c8 = 350241736

  Commands leading to the command that caused the error were:
  CR FEATR COUNT  LBA_48  LH LM LL DV DC  Powered_Up_Time  Command/Feature_Name
  -- == -- == -- == == == -- -- -- -- --  ---------------  --------------------
  25 00 00 00 38 00 00 14 e0 43 c8 e0 08  7d+20:36:03.771  READ DMA EXT
  25 00 00 00 70 00 00 14 e2 10 00 e0 08  7d+20:36:00.941  READ DMA EXT
  25 00 00 04 00 00 00 14 e2 0c 00 e0 08  7d+20:36:00.938  READ DMA EXT
  25 00 00 04 00 00 00 14 e2 08 00 e0 08  7d+20:36:00.936  READ DMA EXT
  25 00 00 04 00 00 00 14 e2 04 00 e0 08  7d+20:36:00.934  READ DMA EXT

Error 65 [0] occurred at disk power-on lifetime: 5009 hours (208 days + 17 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER -- ST COUNT  LBA_48  LH LM LL DV DC
  -- -- -- == -- == == == -- -- -- -- --
  40 -- 53 00 38 00 00 14 e0 43 c8 40 00  Error: UNC 56 sectors at LBA = 0x14e043c8 = 350241736

  Commands leading to the command that caused the error were:
  CR FEATR COUNT  LBA_48  LH LM LL DV DC  Powered_Up_Time  Command/Feature_Name
  -- == -- == -- == == == -- -- -- -- --  ---------------  --------------------
  25 00 00 04 00 00 00 14 e0 40 00 e0 08  7d+20:36:00.421  READ DMA EXT
  25 00 00 04 00 00 00 14 e0 3c 00 e0 08  7d+20:35:57.616  READ DMA EXT
  25 00 00 04 00 00 00 14 e0 38 00 e0 08  7d+20:35:57.609  READ DMA EXT
  25 00 00 04 00 00 00 14 e0 34 00 e0 08  7d+20:35:57.601  READ DMA EXT
  25 00 00 04 00 00 00 14 e0 30 00 e0 08  7d+20:35:57.593  READ DMA EXT

Error 64 [3] occurred at disk power-on lifetime: 5009 hours (208 days + 17 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER -- ST COUNT  LBA_48  LH LM LL DV DC
  -- -- -- == -- == == == -- -- -- -- --
  40 -- 53 03 60 00 00 13 f9 4b 10 40 00  Error: UNC 864 sectors at LBA = 0x13f94b10 = 335104784

  Commands leading to the command that caused the error were:
  CR FEATR COUNT  LBA_48  LH LM LL DV DC  Powered_Up_Time  Command/Feature_Name
  -- == -- == -- == == == -- -- -- -- --  ---------------  --------------------
  25 00 00 03 60 00 00 13 f9 4b 10 e0 08  7d+20:35:00.738  READ DMA EXT
  25 00 00 00 d0 00 00 13 fa f2 70 e0 08  7d+20:34:57.902  READ DMA EXT
  25 00 00 04 00 00 00 13 fa ee 70 e0 08  7d+20:34:57.900  READ DMA EXT
  25 00 00 04 00 00 00 13 fa ea 70 e0 08  7d+20:34:57.898  READ DMA EXT
  25 00 00 04 00 00 00 13 fa e6 70 e0 08  7d+20:34:57.896  READ DMA EXT

Error 63 [2] occurred at disk power-on lifetime: 5009 hours (208 days + 17 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER -- ST COUNT  LBA_48  LH LM LL DV DC
  -- -- -- == -- == == == -- -- -- -- --
  40 -- 53 03 60 00 00 13 f9 4b 10 40 00  Error: UNC 864 sectors at LBA = 0x13f94b10 = 335104784

  Commands leading to the command that caused the error were:
  CR FEATR COUNT  LBA_48  LH LM LL DV DC  Powered_Up_Time  Command/Feature_Name
  -- == -- == -- == == == -- -- -- -- --  ---------------  --------------------
  25 00 00 04 00 00 00 13 f9 4a 70 e0 08  7d+20:34:57.413  READ DMA EXT
  25 00 00 04 00 00 00 13 f9 46 70 e0 08  7d+20:34:54.625  READ DMA EXT
  25 00 00 04 00 00 00 13 f9 42 70 e0 08  7d+20:34:54.622  READ DMA EXT
  25 00 00 04 00 00 00 13 f9 3e 70 e0 08  7d+20:34:54.619  READ DMA EXT
  25 00 00 04 00 00 00 13 f9 3a 70 e0 08  7d+20:34:54.614  READ DMA EXT

SMART Extended Self-test Log Version: 1 (1 sectors)
No self-tests have been logged.  [To run self-tests, use: smartctl -t]

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

SCT Status Version:                  3
SCT Version (vendor specific):       256 (0x0100)
SCT Support Level:                   1
Device State:                        Active (0)
Current Temperature:                    50 Celsius
Power Cycle Min/Max Temperature:     44/61 Celsius
Lifetime    Min/Max Temperature:     25/61 Celsius
Under/Over Temperature Limit Count:   0/0

SCT Temperature History Version:     2
Temperature Sampling Period:         1 minute
Temperature Logging Interval:        1 minute
Min/Max recommended Temperature:      0/60 Celsius
Min/Max Temperature Limit:           -40/70 Celsius
Temperature History Size (Index):    128 (100)

Index    Estimated Time   Temperature Celsius
 101    2018-12-13 12:51    50  *******************************
 ...    ..(126 skipped).    ..  *******************************
 100    2018-12-13 14:58    50  *******************************

SCT Error Recovery Control:
           Read: Disabled
          Write: Disabled

Device Statistics (GP Log 0x04)
Page  Offset Size        Value Flags Description
0x01  =====  =               =  ===  == General Statistics (rev 2) ==
0x01  0x008  4               3  ---  Lifetime Power-On Resets
0x01  0x018  6      6820807272  ---  Logical Sectors Written
0x01  0x020  6        20477840  ---  Number of Write Commands
0x01  0x028  6    444004265414  ---  Logical Sectors Read
0x01  0x030  6       448331816  ---  Number of Read Commands
0x01  0x038  6     20846500900  ---  Date and Time TimeStamp
0x03  =====  =               =  ===  == Rotating Media Statistics (rev 1) ==
0x03  0x008  4            5586  ---  Spindle Motor Power-on Hours
0x03  0x010  4            5586  ---  Head Flying Hours
0x03  0x018  4            1257  ---  Head Load Events
0x03  0x020  4              25  ---  Number of Reallocated Logical Sectors
0x03  0x028  4         1610568  ---  Read Recovery Attempts
0x03  0x030  4               0  ---  Number of Mechanical Start Failures
0x04  =====  =               =  ===  == General Errors Statistics (rev 1) ==
0x04  0x008  4              66  ---  Number of Reported Uncorrectable Errors
0x04  0x010  4              18  ---  Resets Between Cmd Acceptance and Completion
0x05  =====  =               =  ===  == Temperature Statistics (rev 1) ==
0x05  0x008  1              50  ---  Current Temperature
0x05  0x010  1              49  N--  Average Short Term Temperature
0x05  0x018  1              50  N--  Average Long Term Temperature
0x05  0x020  1              61  ---  Highest Temperature
0x05  0x028  1              25  ---  Lowest Temperature
0x05  0x030  1              58  N--  Highest Average Short Term Temperature
0x05  0x038  1              25  N--  Lowest Average Short Term Temperature
0x05  0x040  1              53  N--  Highest Average Long Term Temperature
0x05  0x048  1              25  N--  Lowest Average Long Term Temperature
0x05  0x050  4              34  ---  Time in Over-Temperature
0x05  0x058  1              60  ---  Specified Maximum Operating Temperature
0x05  0x060  4               0  ---  Time in Under-Temperature
0x05  0x068  1               0  ---  Specified Minimum Operating Temperature
0x06  =====  =               =  ===  == Transport Statistics (rev 1) ==
0x06  0x008  4              71  ---  Number of Hardware Resets
0x06  0x010  4              28  ---  Number of ASR Events
0x06  0x018  4               0  ---  Number of Interface CRC Errors
                                |||_ C monitored condition met
                                ||__ D supports DSN
                                |___ N normalized value

SATA Phy Event Counters (GP Log 0x11)
ID      Size     Value  Description
0x0001  2            0  Command failed due to ICRC error
0x0002  2            0  R_ERR response for data FIS
0x0003  2            0  R_ERR response for device-to-Host data FIS
0x0004  2            0  R_ERR response for Host-to-device data FIS
0x0005  2            0  R_ERR response for non-data FIS
0x0006  2            0  R_ERR response for device-to-Host non-data FIS
0x0007  2            0  R_ERR response for Host-to-device non-data FIS
0x0008  2            0  Device-to-Host non-data FIS retries
0x0009  2           70  Transition from drive PhyRdy to drive PhyNRdy
0x000a  2           71  Device-to-Host register FISes sent due to a COMRESET
0x000b  2            0  CRC errors within Host-to-device FIS
0x000d  2            0  Non-CRC errors within Host-to-device FIS

PASSEDと表示されているのは不思議ですが、ドライブはまだほとんど問題ないことを意味していると思いますが、25のReallocated_Sector_Ctは、すぐに失敗する可能性があることを示していますか?

私はこれまで(いくつかの専用ボックスを実行してから10年間)そのようなエラーが発生したことはなく、混乱したくありません。私は今どうすればいい?

これはHetznerストレージボックスであり、RAID6で最大48TBのスペースを提供するために10台の6GBドライブをセットアップしました。

彼らにメッセージを送るだけで最高のことであり、彼らは私のためにこれを世話してくれるでしょう、または私が最初にすべきことがあります...

私はすでにバックアップを持っており、Google CloudServicesまで毎晩実行されます。復元するのに時間がかかることがわかっているので、使用する必要はありません。

私はヘッツナーのサポートシステムにログインしましたが、ドライブに障害が発生した場合、彼らはこのアドバイスを提供します(これはチケットを送信する前ですが、まだ行っていません)。

Please give us the following information so that we can better understand your HDD/SSD defect:

  * Serial number(s) of the defective and/or intact HDD(s)/SSD(s)
  * Evidence of the defect (entire SMART log, less than one week old)

The following assistance is available:

  * [Instructions on establishing the serial numbers as well as information on defective HDDs/SSDs][1]
  * [Instructions on exchanging an HDD/SSD with software RAID][1]
  * [Instructions on creating a complete SMART log][1]

Please note that we can only exchange your defective drive for an empty drive. We do not carry out any data exchange or backups.

特に最後の行が私の注意を引いた。空のドライブを入れたらどうなりますか、それを再構築するために何をする必要がありますか、それとも自動的に起こりますか?

ありがとう!

1
Codemonkey

したがって、おそらくディスクに障害が発生しているようです。 RAID6では、2つのディスク障害が発生する可能性があるため、この1つのディスクが故障しても、データはそのままの状態であるはずです。 RAIDを使用することの全体的なポイントは、通常、アレイ内の残りの正常なディスクに書き込まれたデータを使用して、1つの障害が発生した(その後交換された)ディスクの内容を再構築できることです。

ディスクを交換する場合は、ディスクを初期化し、既存のアレイに追加して使用できるようにする必要があります。これは、アレイの構築方法によって異なります。ソフトウェアRAIDの場合は、正しいタイプの単一パーティションを作成し、mdadmを使用してアレイに追加する必要があります。その時点で、アレイは修復する必要があります/自分自身を再構築します。アレイ内の既存のディスクの1つのパーティションを見ると、複製する必要があるものがわかります。

このページには、実行する必要のある基本事項が表示されます。 https://www.thegeekdiary.com/replacing-a-failed-mirror-disk-in-a-software-raid-array-mdadm/

ハードウェアRAIDには通常、ベンダー固有の方法がありますが、個々のハードディスクを引き続き確認できるため、ソフトウェアRAIDを使用していると想定しています。

2
bodgit