Consistency Issues during/after disk rebuild

Please start any new threads on our new site at https://forums.sqlteam.com. We've got lots of great SQL Server experts to answer whatever question you can come up with.

All Forums

General SQL Server Forums

Data Corruption Issues

Consistency Issues during/after disk rebuild

Author

Topic

dscordle
Starting Member

3 Posts

Posted - 2008-04-09 : 15:38:57

We had one of the disks in our array fail yesterday morning. The rebuild process using the hot spare went into effect as one would expect. My question is: during this rebuild process, could additional errors be introduced into the database? I'm asking because the current production db has more errors than the restored database from last night.

Msg 8928, Level 16, State 1, Line 3
Object ID 210099789, index ID 255: Page (1:128895) could not be processed. See other errors for details.
Msg 8939, Level 16, State 106, Line 3
Table error: Object ID 210099789, index ID 255, page (1:128895). Test (m_freeCnt == freeCnt) failed. Values are 0 and 8094.
Msg 8939, Level 16, State 108, Line 3
Table error: Object ID 210099789, index ID 255, page (1:128895). Test (emptySlotCnt == 0) failed. Values are 1 and 0.
Msg 8928, Level 16, State 1, Line 3
Object ID 210099789, index ID 255: Page (1:469839) could not be processed. See other errors for details.
Msg 8939, Level 16, State 98, Line 3
Table error: Object ID 210099789, index ID 255, page (1:469839). Test (IS_ON (BUF_IOERR, bp->bstat) && bp->berrcode) failed. Values are 2057 and -1.
Msg 8928, Level 16, State 1, Line 3
Object ID 210099789, index ID 255: Page (1:469935) could not be processed. See other errors for details.
Msg 8939, Level 16, State 98, Line 3
Table error: Object ID 210099789, index ID 255, page (1:469935). Test (IS_ON (BUF_IOERR, bp->bstat) && bp->berrcode) failed. Values are 2057 and -1.
Msg 8928, Level 16, State 1, Line 3
Object ID 210099789, index ID 255: Page (1:491535) could not be processed. See other errors for details.
Msg 8939, Level 16, State 98, Line 3
Table error: Object ID 210099789, index ID 255, page (1:491535). Test (IS_ON (BUF_IOERR, bp->bstat) && bp->berrcode) failed. Values are 2057 and -1.
Msg 8928, Level 16, State 1, Line 3
Object ID 210099789, index ID 255: Page (1:680703) could not be processed. See other errors for details.
Msg 8939, Level 16, State 98, Line 3
Table error: Object ID 210099789, index ID 255, page (1:680703). Test (IS_ON (BUF_IOERR, bp->bstat) && bp->berrcode) failed. Values are 2057 and -1.
Msg 8928, Level 16, State 1, Line 3
Object ID 210099789, index ID 255: Page (1:762783) could not be processed. See other errors for details.
Msg 8939, Level 16, State 98, Line 3
Table error: Object ID 210099789, index ID 255, page (1:762783). Test (IS_ON (BUF_IOERR, bp->bstat) && bp->berrcode) failed. Values are 2057 and -1.
CHECKDB found 0 allocation errors and 13 consistency errors in table 'session_state' (object ID 210099789).
Msg 8928, Level 16, State 1, Line 3
Object ID 234028165, index ID 0: Page (1:14175) could not be processed. See other errors for details.
Msg 8944, Level 16, State 17, Line 3
Table error: Object ID 234028165, index ID 0, page (1:14175), row 0. Test (columnOffsets->offTbl [varColumnNumber] <= (nextRec - pRec)) failed. Values are 326 and 300.
Msg 8928, Level 16, State 1, Line 3
Object ID 234028165, index ID 0: Page (1:155583) could not be processed. See other errors for details.
Msg 8944, Level 16, State 12, Line 3
Table error: Object ID 234028165, index ID 0, page (1:155583), row 1. Test (ColumnOffsets <= (nextRec - pRec)) failed. Values are 3840 and 388.
Msg 8928, Level 16, State 1, Line 3
Object ID 234028165, index ID 0: Page (1:179199) could not be processed. See other errors for details.
Msg 8939, Level 16, State 98, Line 3
Table error: Object ID 234028165, index ID 0, page (1:179199). Test (IS_ON (BUF_IOERR, bp->bstat) && bp->berrcode) failed. Values are 2057 and -1.
Msg 8928, Level 16, State 1, Line 3
Object ID 234028165, index ID 0: Page (1:237663) could not be processed. See other errors for details.
Msg 8944, Level 16, State 12, Line 3
Table error: Object ID 234028165, index ID 0, page (1:237663), row 0. Test (ColumnOffsets <= (nextRec - pRec)) failed. Values are 91 and 26.
Msg 8928, Level 16, State 1, Line 3
Object ID 234028165, index ID 0: Page (1:1060304) could not be processed. See other errors for details.
Msg 8944, Level 16, State 12, Line 3
Table error: Object ID 234028165, index ID 0, page (1:1060304), row 1. Test (ColumnOffsets <= (nextRec - pRec)) failed. Values are 61455 and 397.
Msg 8928, Level 16, State 1, Line 3
Object ID 234028165, index ID 0: Page (1:1392799) could not be processed. See other errors for details.
Msg 8939, Level 16, State 98, Line 3
Table error: Object ID 234028165, index ID 0, page (1:1392799). Test (IS_ON (BUF_IOERR, bp->bstat) && bp->berrcode) failed. Values are 2057 and -1.
CHECKDB found 0 allocation errors and 12 consistency errors in table 'clientCampaignActivityAll' (object ID 234028165).

paulrandal
Yak with Vast SQL Skills

899 Posts

Posted - 2008-04-09 : 16:47:20

My guess is the disk has been failing for some time and introducing errors that have gone unnoticed, and have been backed up. How often do you run DBCC CHECKDB?

Paul Randal
Managing Director, SQLskills.com

dscordle
Starting Member

3 Posts

Posted - 2008-04-09 : 21:19:09

There is a weekly job that runs (Friday mornings), and it hasn't thrown any errors

paulrandal
Yak with Vast SQL Skills

899 Posts

Posted - 2008-04-10 : 10:25:24

So did the restored database have any errors? Your wording in the original mail suggested that it does.

Paul Randal
SQL Server MVP, Managing Director, SQLskills.com

dscordle
Starting Member

3 Posts

Posted - 2008-04-10 : 11:40:48

The restored db that was from a backup taken the evening of the failure/rebuild did. It was taken after the disk had completed rebuilding. In any case, we pulled restores from the night before and were able to recover the majority of the data.

Any idea how/why a disk failure in a raid array would cause unrecoverable consistency errors in 10 different databases? Is this more than likely a case of the errors being introduced prior to the actual failure?

One more question: Is there any way to recover the corrupted data from these databases? Either by 3rd party tools or companies? I've got a feeling the answer is no, but I told the powers that be that I would ask.

thanks for your help.

Subscribe to SQLTeam.com

SQLTeam.com Articles via RSS

SQLTeam.com Weblog via RSS

- Advertisement -

Resources