本文转自 惜纷飞 大师。

模拟基表事务未提交数据库crash,undo丢失恢复异常恢复，运维DBA反映Oracle数据库无法启动报错ORA-01157 ORA-01110，分析原因为Oracle数据库坏块导致

对于基表特别是bootstrap$中的表,如果进行了dml操作,但是没有提交,这个时候数据库crash,而且undo异常.对于类似的场景,都是很多数据库恢复的人都有畏惧,因为_corrupted_rollback_segments参数不能起作用.本实验dml修改seq$表,最终却引起了undo$和I_UNDO1异常(他们都在bootstrap$中)

1.模拟seq$表事务未提交

SQL> select * from v$version; 

BANNER

--------------------------------------------------------------------------------

Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - Production

PL/SQL Release 11.2.0.3.0 - Production

CORE    11.2.0.3.0      Production

TNS for Linux: Version 11.2.0.3.0 - Production

NLSRTL Version 11.2.0.3.0 - Production

SQL> select sysdate "www.oracleplus.com" from dual;

www.oracleplus

------------

31-AUG-13

SQL> CREATE SEQUENCE seq_oracleplus INCREMENT BY 1 START WITH 1 CACHE 3;

Sequence created.

SQL> select object_id from dba_objects where object_name='SEQ_oracleplus';

 OBJECT_ID

----------

     77282

SQL> SELECT MINVALUE,HIGHWATER,INCREMENT$, CYCLE#,ORDER$,CACHE from seq$ where obj#=77282;

  MINVALUE  HIGHWATER INCREMENT$     CYCLE#     ORDER$      CACHE

---------- ---------- ---------- ---------- ---------- ----------

         1          1          1          0          0          3

SQL> update seq$ set  CYCLE#=3 where obj#=77282;

1 row updated.

--另外一个会话abort数据库

SQL> shutdown abort;

ORACLE instance shut down.

--人工rm掉undotbs01.dbf文件

--启动数据库

SQL> startup

ORACLE instance started.

Total System Global Area  175775744 bytes

Fixed Size                  1343668 bytes

Variable Size             117444428 bytes

Database Buffers           50331648 bytes

Redo Buffers                6656000 bytes

Database mounted.

ORA-01157: cannot identify/lock data file 3 - see DBWR trace file

ORA-01110: data file 3: '/u01/oracle/oradata/ora11g/undotbs01.dbf'

--离线数据文件

SQL> alter database datafile 3 offline;

Database altered.

--尝试open数据库

SQL> alter database open;

alter database open

*

ERROR at line 1:

ORA-01092: ORACLE instance terminated. Disconnection forced

ORA-00704: bootstrap process failure

ORA-00604: error occurred at recursive SQL level 2

ORA-00376: file 3 cannot be read at this time

ORA-01110: data file 3: '/u01/oracle/oradata/ora11g/undotbs01.dbf'

Process ID: 17864

Session ID: 125 Serial number: 5

alert日志

SMON: enabling cache recovery

Mon Aug 26 17:08:49 2013

ARC3 started with pid=25, OS id=17902

ARC1: Archival started

ARC2: Archival started

ARC1: Becoming the 'no FAL' ARCH

ARC1: Becoming the 'no SRL' ARCH

ARC2: Becoming the heartbeat ARCH

Archived Log entry 13 added for thread 1 sequence 13 ID 0xfb920b77 dest 1:

Errors in file /u01/oracle/diag/rdbms/ora11g/ora11g/trace/ora11g_ora_17864.trc:

ORA-00704: bootstrap process failure

ORA-00604: error occurred at recursive SQL level 2

ORA-00376: file 3 cannot be read at this time

ORA-01110: data file 3: '/u01/oracle/oradata/ora11g/undotbs01.dbf'

Errors in file /u01/oracle/diag/rdbms/ora11g/ora11g/trace/ora11g_ora_17864.trc:

ORA-00704: bootstrap process failure

ORA-00604: error occurred at recursive SQL level 2

ORA-00376: file 3 cannot be read at this time

ORA-01110: data file 3: '/u01/oracle/oradata/ora11g/undotbs01.dbf'

Error 704 happened during db open, shutting down database

USER (ospid: 17864): terminating the instance due to error 704

Instance terminated by USER, pid = 17864

ORA-1092 signalled during: alter database open...

opiodr aborting process unknown ospid (17864) as a result of ORA-1092

做10046跟踪

SQL> conn / as sysdba

Connected to an idle instance.

SQL> startup nomount;

ORACLE instance started.

Total System Global Area  175775744 bytes

Fixed Size                  1343668 bytes

Variable Size             117444428 bytes

Database Buffers           50331648 bytes

Redo Buffers                6656000 bytes

SQL> alter database mount;

Database altered.

SQL> alter session set events  '10046 trace name context forever,level 12';

Session altered.

SQL> alter session set db_file_multiblocK_read_count=1;

Session altered.

SQL> alter database open;

alter database open

*

ERROR at line 1:

ORA-01092: ORACLE instance terminated. Disconnection forced

ORA-00704: bootstrap process failure

ORA-00604: error occurred at recursive SQL level 2

ORA-00376: file 3 cannot be read at this time

ORA-01110: data file 3: '/u01/oracle/oradata/ora11g/undotbs01.dbf'

Process ID: 17979

Session ID: 125 Serial number: 3

trace文件内容

PARSING IN CURSOR #3064524800 len=142 dep=3 uid=0 oct=3 lid=0 tim=1377508386082495 hv=361892850 ad='265c66d0' sqlid='7bd391hat42zk'

select /*+ rule */ name,file#,block#,status$,user#,undosqn,xactsqn,scnbas,scnwrp,DECODE(inst#,0,NULL,inst#),ts#,spare1 from undo$ where us#=:1

END OF STMT

PARSE #3064524800:c=2999,e=2162,p=0,cr=0,cu=0,mis=1,r=0,dep=3,og=3,plh=0,tim=1377508386082492

BINDS #3064524800:

 Bind#0

  oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00

  oacflg=08 fl2=0001 frm=00 csi=00 siz=24 off=0

  kxsbbbfp=b6a8ebd0  bln=22  avl=02  flg=05

  value=27

EXEC #3064524800:c=1999,e=2030,p=0,cr=0,cu=0,mis=1,r=0,dep=3,og=3,plh=906473769,tim=1377508386084833

WAIT #3064524800: nam='db file sequential read' ela= 65 file#=1 block#=321 blocks=1 obj#=34 tim=1377508386085083

WAIT #3064524800: nam='db file sequential read' ela= 21 file#=1 block#=225 blocks=1 obj#=15 tim=1377508386085208

FETCH #3064524800:c=0,e=375,p=2,cr=2,cu=0,mis=0,r=1,dep=3,og=3,plh=906473769,tim=1377508386085319

STAT #3064524800 id=1 cnt=1 pid=0 pos=1 obj=15 op='TABLE ACCESS BY INDEX ROWID UNDO$ (cr=2 pr=2 pw=0 time=362 us)'

STAT #3064524800 id=2 cnt=1 pid=1 pos=1 obj=34 op='INDEX UNIQUE SCAN I_UNDO1 (cr=1 pr=1 pw=0 time=223 us)'

CLOSE #3064524800:c=0,e=12,dep=3,type=0,tim=1377508386085657

WAIT #3064511164: nam='control file sequential read' ela= 33 file#=0 block#=1 blocks=1 obj#=15 tim=1377508386085816

WAIT #3064511164: nam='control file sequential read' ela= 18 file#=0 block#=15 blocks=1 obj#=15 tim=1377508386085889

WAIT #3064511164: nam='control file sequential read' ela= 18 file#=0 block#=17 blocks=1 obj#=15 tim=1377508386085994

WAIT #3064511164: nam='control file sequential read' ela= 18 file#=0 block#=23 blocks=1 obj#=15 tim=1377508386086058

WAIT #3064511164: nam='control file sequential read' ela= 17 file#=0 block#=1 blocks=1 obj#=15 tim=1377508386086135

WAIT #3064511164: nam='control file sequential read' ela= 15 file#=0 block#=15 blocks=1 obj#=15 tim=1377508386086196

WAIT #3064511164: nam='control file sequential read' ela= 15 file#=0 block#=17 blocks=1 obj#=15 tim=1377508386086255

WAIT #3064511164: nam='control file sequential read' ela= 18 file#=0 block#=32 blocks=1 obj#=15 tim=1377508386086317

DDE rules only execution for: ORA 1110

----- START Event Driven Actions Dump ----

---- END Event Driven Actions Dump ----

----- START DDE Actions Dump -----

Executing SYNC actions

----- START DDE Action: 'DB_STRUCTURE_INTEGRITY_CHECK' (Async) -----

Successfully dispatched

----- END DDE Action: 'DB_STRUCTURE_INTEGRITY_CHECK' (SUCCESS, 0 csec) -----

Executing ASYNC actions

----- END DDE Actions Dump (total 0 csec) -----

FETCH #3064511164:c=8998,e=9239,p=5,cr=6,cu=0,mis=0,r=0,dep=2,og=4,plh=2853959010,tim=1377508386088336

STAT #3064511164 id=1 cnt=0 pid=0 pos=1 obj=18 op='TABLE ACCESS BY INDEX ROWID OBJ$ (cr=0 pr=0 pw=0 time=82 us)'

STAT #3064511164 id=2 cnt=1 pid=1 pos=1 obj=37 op='INDEX RANGE SCAN I_OBJ2 (cr=3 pr=3 pw=0 time=1166 us)'

=====================

PARSE ERROR #3064512008:len=60 dep=1 uid=0 oct=3 lid=0 tim=1377508386088729 err=604

SELECT NULL FROM PROPS$ WHERE NAME='BOOTSTRAP_UPGRADE_ERROR'

ORA-00704: bootstrap process failure

ORA-00604: error occurred at recursive SQL level 2

ORA-00376: file 3 cannot be read at this time

ORA-01110: data file 3: '/u01/oracle/oradata/ora11g/undotbs01.dbf'

ORA-00704: bootstrap process failure

ORA-00604: error occurred at recursive SQL level 2

ORA-00376: file 3 cannot be read at this time

ORA-01110: data file 3: '/u01/oracle/oradata/ora11g/undotbs01.dbf'

_corrupted_rollback_segments无法打开数据库

SQL> show parameter _corrupted_rollback_segments;

NAME                                 TYPE        VALUE

------------------------------------ ----------- ------------------------------

_corrupted_rollback_segments         string      _SYSSMU22_1366820418$, _SYSSMU

                                                 23_2165404374$, _SYSSMU24_1842

                                                 376483$, _SYSSMU25_2373003110$

                                                 , _SYSSMU26_3635237253$, _SYSS

                                                 MU27_1730246918$, _SYSSMU28_31

                                                 59173804$, _SYSSMU29_141006113

                                                 4$, _SYSSMU30_2299069988$, _SY

                                                 SSMU31_290584066$

SQL> startup pfile='/tmp/pfile' mount

ORACLE instance started.

Total System Global Area  175775744 bytes

Fixed Size                  1343668 bytes

Variable Size             117444428 bytes

Database Buffers           50331648 bytes

Redo Buffers                6656000 bytes

Database mounted.

SQL> alter database open;

alter database open

*

ERROR at line 1:

ORA-01092: ORACLE instance terminated. Disconnection forced

ORA-00704: bootstrap process failure

ORA-00604: error occurred at recursive SQL level 2

ORA-00376: file 3 cannot be read at this time

ORA-01110: data file 3: '/u01/oracle/oradata/ora11g/undotbs01.dbf'

Process ID: 17979

Session ID: 125 Serial number: 3

dump block225

Block header dump:  0x004000e1

 Object id on Block? Y

 seg/obj: 0xf  csc: 0xb8c.3c0c297b  itc: 1  flg: O  typ: 1 - DATA

     fsl: 0  fnx: 0x0 ver: 0x01

 Itl           Xid                  Uba        Flag  Lck        Scn/Fsc

0x01   0x0000.001.00000034  0x00400221.0034.08  --U-    1  fsc 0x0000.3c0c297c

bdba: 0x004000e1

data_block_dump,data header at 0xb6b8a644

bbed 提交225事务

[oracle@oracleplus ~]$ bbed password=blockedit filename='/u01/oracle/oradata/ora11g/system01.dbf' blocksize=8192

BBED: Release 2.0.0.0.0 - Limited Production on Mon Aug 26 17:31:09 2013

Copyright (c) 1982, 2007, Oracle.  All rights reserved.

************* !!! For Oracle Internal Use only !!! ***************

BBED> set block 225

        BLOCK#          225

BBED> map

 File: /u01/oracle/oradata/ora11g/system01.dbf (0)

 Block: 225                                   Dba:0x00000000

------------------------------------------------------------

 KTB Data Block (Table/Cluster)

 struct kcbh, 20 bytes                      @0       

 struct ktbbh, 48 bytes                     @20      

 struct kdbh, 14 bytes                      @68      

 struct kdbt[1], 4 bytes                    @82      

 sb2 kdbr[32]                               @86      

 ub1 freespace[59]                          @150     

 ub1 rowdata[7979]                          @209     

 ub4 tailchk                                @8188    

BBED> p ktbbh

struct ktbbh, 48 bytes                      @20

   ub1 ktbbhtyp                             @20       0x01 (KDDBTDATA)

   union ktbbhsid, 4 bytes                  @24

      ub4 ktbbhsg1                          @24       0x0000000f

      ub4 ktbbhod1                          @24       0x0000000f

   struct ktbbhcsc, 8 bytes                 @28

      ub4 kscnbas                           @28       0x3c0c297b

      ub2 kscnwrp                           @32       0x0b8c

   b2 ktbbhict                              @36      -2047

   ub1 ktbbhflg                             @38       0x03 (KTBFONFL)

   ub1 ktbbhfsl                             @39       0x00

   ub4 ktbbhfnx                             @40       0x00000000

   struct ktbbhitl[0], 24 bytes             @44

      struct ktbitxid, 8 bytes              @44

         ub2 kxidusn                        @44       0x0000

         ub2 kxidslt                        @46       0x0001

         ub4 kxidsqn                        @48       0x00000034

      struct ktbituba, 8 bytes              @52

         ub4 kubadba                        @52       0x00400221

         ub2 kubaseq                        @56       0x0034

         ub1 kubarec                        @58       0x08

      ub2 ktbitflg                          @60       0x2001 (KTBFUPB)

      union _ktbitun, 2 bytes               @62

         b2 _ktbitfsc                       @62       0

         ub2 _ktbitwrp                      @62       0x0000

      ub4 ktbitbas                          @64       0x3c0c297c

BBED> set mode edit

        MODE            Edit

BBED> m /x 0180 offset 60

 File: /u01/oracle/oradata/ora11g/system01.dbf (0)

 Block: 225              Offsets:   60 to   91           Dba:0x00000000

------------------------------------------------------------------------

 01800000 7c290c3c 00012000 ffff5200 8d003017 30170000 20008e1f 4106f805 

 <32 bytes per line>

BBED> sum apply

Check value for File 0, Block 225:

current = 0x55db, required = 0x55db

BBED> verify

DBVERIFY - Verification starting

FILE = /u01/oracle/oradata/ora11g/system01.dbf

BLOCK = 225

Block Checking: DBA = 4194529, Block Type = KTB-managed data block

data header at 0xb6773244

kdbchk: row locked by non-existent transaction

        table=0   slot=20

        lockid=1   ktbbhitc=1

Block 225 failed with check code 6101

DBVERIFY - Verification complete

Total Blocks Examined         : 1

Total Blocks Processed (Data) : 1

Total Blocks Failing   (Data) : 1

Total Blocks Processed (Index): 0

Total Blocks Failing   (Index): 0

Total Blocks Empty            : 0

Total Blocks Marked Corrupt   : 0

Total Blocks Influx           : 0

BBED> p *kdbr[20]

rowdata[608]

------------

ub1 rowdata[608]                            @817      0x2c

BBED> dump

 File: /u01/oracle/oradata/ora11g/system01.dbf (0)

 Block: 225              Offsets:  817 to  848           Dba:0x00000000

------------------------------------------------------------------------

 2c011102 c115145f 53595353 4d553230 5f343035 34303430 31322402 c10202c1 

 <32 bytes per line>

BBED> m /x 002c

 File: /u01/oracle/oradata/ora11g/system01.dbf (0)

 Block: 225              Offsets:  817 to  848           Dba:0x00000000

------------------------------------------------------------------------

 002c1102 c115145f 53595353 4d553230 5f343035 34303430 31322402 c10202c1 

 <32 bytes per line>

BBED> sum apply

Check value for File 0, Block 225:

current = 0x79f6, required = 0x79f6

BBED> verify

DBVERIFY - Verification starting

FILE = /u01/oracle/oradata/ora11g/system01.dbf

BLOCK = 225

Block Checking: DBA = 4194529, Block Type = KTB-managed data block

data header at 0xb6773244

kdbchk: row locked by non-existent transaction

        table=0   slot=20

        lockid=44   ktbbhitc=1

Block 225 failed with check code 6101

DBVERIFY - Verification complete

Total Blocks Examined         : 1

Total Blocks Processed (Data) : 1

Total Blocks Failing   (Data) : 1

Total Blocks Processed (Index): 0

Total Blocks Failing   (Index): 0

Total Blocks Empty            : 0

Total Blocks Marked Corrupt   : 0

Total Blocks Influx           : 0

BBED> p *kdbr[20]

rowdata[608]

------------

ub1 rowdata[608]                            @817      0x00

BBED> d

 File: /u01/oracle/oradata/ora11g/system01.dbf (0)

 Block: 225              Offsets:  817 to  848           Dba:0x00000000

------------------------------------------------------------------------

 002c1102 c115145f 53595353 4d553230 5f343035 34303430 31322402 c10202c1 

 <32 bytes per line>

BBED> m /x 2c00

 File: /u01/oracle/oradata/ora11g/system01.dbf (0)

 Block: 225              Offsets:  817 to  848           Dba:0x00000000

------------------------------------------------------------------------

 2c001102 c115145f 53595353 4d553230 5f343035 34303430 31322402 c10202c1 

 <32 bytes per line>

BBED> sum apply

Check value for File 0, Block 225:

current = 0x55da, required = 0x55da

BBED> verify

DBVERIFY - Verification starting

FILE = /u01/oracle/oradata/ora11g/system01.dbf

BLOCK = 225

DBVERIFY - Verification complete

Total Blocks Examined         : 1

Total Blocks Processed (Data) : 1

Total Blocks Failing   (Data) : 0

Total Blocks Processed (Index): 0

Total Blocks Failing   (Index): 0

Total Blocks Empty            : 0

Total Blocks Marked Corrupt   : 0

Total Blocks Influx           : 0

dump 321 block

SQL> alter system dump datafile '/u01/oracle/oradata/ora11g/system01.dbf' block 321;

System altered.

SQL> select value from v$diag_info where name='Default Trace File';

VALUE

--------------------------------------------------------------------------------

/u01/oracle/diag/rdbms/ora11g/ora11g/trace/ora11g_ora_19704.trc

--trace文件

Block header dump:  0x00400141

 Object id on Block? Y

 seg/obj: 0x22  csc: 0xb8c.3c0c28ec  itc: 2  flg: -  typ: 2 - INDEX

     fsl: 0  fnx: 0x0 ver: 0x01

 Itl           Xid                  Uba         Flag  Lck        Scn/Fsc

0x01   0x0000.000.00000000  0x00000000.0000.00  ----    0  fsc 0x0000.00000000

0x02   0x0000.038.00000033  0x0040021e.0033.3d  --U-    1  fsc 0x0000.3c0c28ed

bbed提交321 block 事务

BBED> set block 321

        BLOCK#          321

BBED> map

 File: /u01/oracle/oradata/ora11g/system01.dbf (0)

 Block: 321                                   Dba:0x00000000

------------------------------------------------------------

 KTB Data Block (Index Leaf)

 struct kcbh, 20 bytes                      @0       

 struct ktbbh, 72 bytes                     @20      

 struct kdxle, 32 bytes                     @92      

 b2 kd_off[32]                              @124     

 ub1 freespace[7581]                        @188     

 ub1 rowdata[351]                           @7769    

 ub4 tailchk                                @8188    

BBED> p ktbbh

struct ktbbh, 72 bytes                      @20

   ub1 ktbbhtyp                             @20       0x02 (KDDBTINDEX)

   union ktbbhsid, 4 bytes                  @24

      ub4 ktbbhsg1                          @24       0x00000022

      ub4 ktbbhod1                          @24       0x00000022

   struct ktbbhcsc, 8 bytes                 @28

      ub4 kscnbas                           @28       0x3c0c28ec

      ub2 kscnwrp                           @32       0x0b8c

   b2 ktbbhict                              @36      -2046

   ub1 ktbbhflg                             @38       0x02 (NONE)

   ub1 ktbbhfsl                             @39       0x00

   ub4 ktbbhfnx                             @40       0x00000000

   struct ktbbhitl[0], 24 bytes             @44

      struct ktbitxid, 8 bytes              @44

         ub2 kxidusn                        @44       0x0000

         ub2 kxidslt                        @46       0x0000

         ub4 kxidsqn                        @48       0x00000000

      struct ktbituba, 8 bytes              @52

         ub4 kubadba                        @52       0x00000000

         ub2 kubaseq                        @56       0x0000

         ub1 kubarec                        @58       0x00

      ub2 ktbitflg                          @60       0x0000 (NONE)

      union _ktbitun, 2 bytes               @62

         b2 _ktbitfsc                       @62       0

         ub2 _ktbitwrp                      @62       0x0000

      ub4 ktbitbas                          @64       0x00000000

   struct ktbbhitl[1], 24 bytes             @68

      struct ktbitxid, 8 bytes              @68

         ub2 kxidusn                        @68       0x0000

         ub2 kxidslt                        @70       0x0038

         ub4 kxidsqn                        @72       0x00000033

      struct ktbituba, 8 bytes              @76

         ub4 kubadba                        @76       0x0040021e

         ub2 kubaseq                        @80       0x0033

         ub1 kubarec                        @82       0x3d

      ub2 ktbitflg                          @84       0x2001 (KTBFUPB)

      union _ktbitun, 2 bytes               @86

         b2 _ktbitfsc                       @86       0

         ub2 _ktbitwrp                      @86       0x0000

      ub4 ktbitbas                          @88       0x3c0c28ed

BBED> m /x 0180 offset 84

Warning: contents of previous BIFILE will be lost. Proceed? (Y/N) y

 File: /u01/oracle/oradata/ora11g/system01.dbf (0)

 Block: 321              Offsets:   84 to  595           Dba:0x00000000

------------------------------------------------------------------------

 01800000 ed280c3c 00008001 00000000 20006400 011e9d1d 00000000 00000000

 00000000 06000000 601f0000 561f4b1f 401f351f 2a1f1f1f 141f091f fe1ef31e

 e81edd1e d21ec71e bc1eb11e a61e9b1e 901e851e 7a1e6f1e 641e591e 4e1e431e

 381e2d1e 221e171e 0c1e011e 00000000 00000000 00000000 00000000 00000000

 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000

 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000

 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000

 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000

 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000

 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000

 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000

 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000

 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000

 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000

 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000

 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 

 <32 bytes per line>

BBED> sum apply

Check value for File 0, Block 321:

current = 0x990e, required = 0x990e

BBED> verify

DBVERIFY - Verification starting

FILE = /u01/oracle/oradata/ora11g/system01.dbf

BLOCK = 321

Block Checking: DBA = 4194625, Block Type = KTB-managed data block

**** actual rows locked by itl 2  = 1 != # in trans. header = 0

---- end index block validation

Block 321 failed with check code 6401

DBVERIFY - Verification complete

Total Blocks Examined         : 1

Total Blocks Processed (Data) : 0

Total Blocks Failing   (Data) : 0

Total Blocks Processed (Index): 1

Total Blocks Failing   (Index): 1

Total Blocks Empty            : 0

Total Blocks Marked Corrupt   : 0

Total Blocks Influx           : 0

BBED> p kdxle

struct kdxle, 32 bytes                      @92

   struct kdxlexco, 16 bytes                @92

      ub1 kdxcolev                          @92       0x00

      ub1 kdxcolok                          @93       0x00

      ub1 kdxcoopc                          @94       0x80

      ub1 kdxconco                          @95       0x01

      ub4 kdxcosdc                          @96       0x00000000

      sb2 kdxconro                          @100      32

      b2 kdxcofbo                           @102      100

      b2 kdxcofeo                           @104      7681

      b2 kdxcoavs                           @106      7581

   b2 kdxlespl                              @108      0

   sb2 kdxlende                             @110      0

   ub4 kdxlenxt                             @112      0x00000000

   ub4 kdxleprv                             @116      0x00000000

   ub1 kdxledsz                             @120      0x06

   ub1 kdxleunuse                           @121      0x00

BBED> m /x 00 offset 95

 File: /u01/oracle/oradata/ora11g/system01.dbf (0)

 Block: 321              Offsets:   95 to  606           Dba:0x00000000

------------------------------------------------------------------------

 00000000 00200064 00011e9d 1d000000 00000000 00000000 00060000 00601f00

 00561f4b 1f401f35 1f2a1f1f 1f141f09 1ffe1ef3 1ee81edd 1ed21ec7 1ebc1eb1

 1ea61e9b 1e901e85 1e7a1e6f 1e641e59 1e4e1e43 1e381e2d 1e221e17 1e0c1e01

 1e000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000

 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000

 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000

 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000

 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000

 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000

 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000

 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000

 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000

 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000

 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000

 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000

 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 

 <32 bytes per line>

BBED> sum apply

Check value for File 0, Block 321:

current = 0x380e, required = 0x380e

BBED> verify

DBVERIFY - Verification starting

FILE = /u01/oracle/oradata/ora11g/system01.dbf

BLOCK = 321

DBVERIFY - Verification complete

Total Blocks Examined         : 1

Total Blocks Processed (Data) : 0

Total Blocks Failing   (Data) : 0

Total Blocks Processed (Index): 1

Total Blocks Failing   (Index): 0

Total Blocks Empty            : 0

Total Blocks Marked Corrupt   : 0

Total Blocks Influx           : 0

bbed提交事务open依旧报错

SQL> alter database open;

alter database open

*

ERROR at line 1:

ORA-01092: ORACLE instance terminated. Disconnection forced

ORA-00704: bootstrap process failure

ORA-00604: error occurred at recursive SQL level 2

ORA-00376: file 3 cannot be read at this time

ORA-01110: data file 3: '/u01/oracle/oradata/ora11g/undotbs01.dbf'

Process ID: 20229

Session ID: 125 Serial number: 5

做10046的trace文件错误依旧

=====================

PARSING IN CURSOR #3065238148 len=142 dep=3 uid=0 oct=3 lid=0 tim=1377519352653391 hv=361892850 ad='269a8064' sqlid='7bd391hat42zk'

select /*+ rule */ name,file#,block#,status$,user#,undosqn,xactsqn,scnbas,scnwrp,DECODE(inst#,0,NULL,inst#),ts#,spare1 from undo$ where us#=:1

END OF STMT

PARSE #3065238148:c=2999,e=3152,p=0,cr=0,cu=0,mis=1,r=0,dep=3,og=3,plh=0,tim=1377519352653388

BINDS #3065238148:

 Bind#0

  oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00

  oacflg=08 fl2=0001 frm=00 csi=00 siz=24 off=0

  kxsbbbfp=b6b32268  bln=22  avl=02  flg=05

  value=27

EXEC #3065238148:c=2999,e=3182,p=0,cr=0,cu=0,mis=1,r=0,dep=3,og=3,plh=906473769,tim=1377519352657030

WAIT #3065238148: nam='db file sequential read' ela= 169 file#=1 block#=321 blocks=1 obj#=34 tim=1377519352657400

WAIT #3065238148: nam='db file sequential read' ela= 25 file#=1 block#=225 blocks=1 obj#=15 tim=1377519352657546

FETCH #3065238148:c=1000,e=578,p=2,cr=2,cu=0,mis=0,r=1,dep=3,og=3,plh=906473769,tim=1377519352657738

STAT #3065238148 id=1 cnt=1 pid=0 pos=1 obj=15 op='TABLE ACCESS BY INDEX ROWID UNDO$ (cr=2 pr=2 pw=0 time=538 us)'

STAT #3065238148 id=2 cnt=1 pid=1 pos=1 obj=34 op='INDEX UNIQUE SCAN I_UNDO1 (cr=1 pr=1 pw=0 time=338 us)'

CLOSE #3065238148:c=0,e=13,dep=3,type=0,tim=1377519352657942

WAIT #3065201888: nam='control file sequential read' ela= 111 file#=0 block#=1 blocks=1 obj#=15 tim=1377519352658170

WAIT #3065201888: nam='control file sequential read' ela= 19 file#=0 block#=16 blocks=1 obj#=15 tim=1377519352658269

WAIT #3065201888: nam='control file sequential read' ela= 18 file#=0 block#=18 blocks=1 obj#=15 tim=1377519352658329

WAIT #3065201888: nam='control file sequential read' ela= 19 file#=0 block#=24 blocks=1 obj#=15 tim=1377519352658391

WAIT #3065201888: nam='control file sequential read' ela= 17 file#=0 block#=1 blocks=1 obj#=15 tim=1377519352658469

WAIT #3065201888: nam='control file sequential read' ela= 16 file#=0 block#=16 blocks=1 obj#=15 tim=1377519352658528

WAIT #3065201888: nam='control file sequential read' ela= 28 file#=0 block#=18 blocks=1 obj#=15 tim=1377519352658787

WAIT #3065201888: nam='control file sequential read' ela= 20 file#=0 block#=32 blocks=1 obj#=15 tim=1377519352658883

DDE rules only execution for: ORA 1110

----- START Event Driven Actions Dump ----

---- END Event Driven Actions Dump ----

----- START DDE Actions Dump -----

Executing SYNC actions

----- START DDE Action: 'DB_STRUCTURE_INTEGRITY_CHECK' (Async) -----

Successfully dispatched

----- END DDE Action: 'DB_STRUCTURE_INTEGRITY_CHECK' (SUCCESS, 0 csec) -----

Executing ASYNC actions

----- END DDE Actions Dump (total 0 csec) -----

FETCH #3065201888:c=11998,e=12032,p=5,cr=6,cu=0,mis=0,r=0,dep=2,og=4,plh=2853959010,tim=1377519352661044

STAT #3065201888 id=1 cnt=0 pid=0 pos=1 obj=18 op='TABLE ACCESS BY INDEX ROWID OBJ$ (cr=0 pr=0 pw=0 time=19 us)'

STAT #3065201888 id=2 cnt=1 pid=1 pos=1 obj=37 op='INDEX RANGE SCAN I_OBJ2 (cr=3 pr=3 pw=0 time=1001 us)'

通过观察每次的10046文件,发现总是在查询select /*+ rule */ name,file#, block#,status$,user#,undosqn,xactsqn, scnbas,scnwrp,DECODE(inst#,0,NULL,inst#),ts#,spare1 from undo$ where us#=:1并且总是us#=27号回滚段上过不去,通过使用bbed或者dul修改数据文件,让数据库启动时候不再扫描27号回滚段(主要是修改undo$.status$=1,block位置可以通过10046确定,也可以通过odu来确定,亦或者找个相同版本的库查看)

继续open数据库

SQL> startup pfile='/tmp/pfile' mount

ORACLE instance started.

Total System Global Area  175775744 bytes

Fixed Size                  1343668 bytes

Variable Size             117444428 bytes

Database Buffers           50331648 bytes

Redo Buffers                6656000 bytes

Database mounted.

SQL> alter database open;

Database altered.

删除有问题undo,重建新undo

SQL> drop tablespace undotbs1;

drop tablespace undotbs1

*

ERROR at line 1:

ORA-01561: failed to remove all objects in the tablespace specified

SQL> !oerr ora 01561

01561, 00000, "failed to remove all objects in the tablespace specified"

// *Cause: Failed to remove all objects when dropping a tablespace

// *Action: Retry the drop tablespace until all objects are dropped

SQL> drop tablespace undotbs1 including contents;

drop tablespace undotbs1 including contents

*

ERROR at line 1:

ORA-01561: failed to remove all objects in the tablespace specified

SQL> drop tablespace undotbs1 including contents and datafiles;

drop tablespace undotbs1 including contents and datafiles

*

ERROR at line 1:

ORA-01561: failed to remove all objects in the tablespace specified

-------------------------------------------------

因为前面使用了bbed/dul之类的工具，修改了undo$表，

导致undo$中的标记27号回滚段与seg$中不一致

解决方法使得修改undo$.status$为3或者2

-------------------------------------------------

SQL> update undo$ set status$=3 where us#=27;

1 row updated.

SQL> commit;

Commit complete.

SQL> drop tablespace undotbs1 ;

drop tablespace undotbs1

*

ERROR at line 1:

ORA-01561: failed to remove all objects in the tablespace specified

SQL> shutdown abort;

ORACLE instance shut down.

SQL> startup pfile='/tmp/pfile'      这个地方可以用 _allow_resetlogs_corruption 这个参数打开

ORACLE instance started.

Total System Global Area  175775744 bytes

Fixed Size                  1343668 bytes

Variable Size             117444428 bytes

Database Buffers           50331648 bytes

Redo Buffers                6656000 bytes

Database mounted.

Database opened.

SQL> drop tablespace undotbs1 ;

Tablespace dropped.

SQL> create undo tablespace undotbs1 datafile '/u01/oracle/oradata/ora11g/undotbs1.dbf' size 10M;

Tablespace created.

undo丢失恢复异常恢复，运维DBA反映Oracle数据库无法启动报错ORA-01157 ORA-01110，分析原因为Oracle数据库坏块导致的更多相关文章

  1. 【转】运维DBA的4大纪律9项注意

    朋友们调侃说,运维是个把脑袋别在裤腰带上的活,更有人说,运维是个把脑袋别在他人裤腰带上的活,苦劳没人认,有锅就有得背! 测试的同学说,“吃瓜群众很难感知运维背后的付出,倒是出了事情更能体现我们的专业性 ...

  2. 运维DBA要不要学python

    运维DBA要不要学python 我个人认为是:要 现在python在运维数据库的工作中主要用在 1.编写一些运维脚本 2.编写运维管理平台 3.研究互联网大厂的运维脚本/工具并应有 特别是运维开源数据 ...

  3. docker启动报错iptables failed: -重建docker0网络恢复

    # docker启动报错 [root@localhost mysqlconf]# docker run -d -p 8080:8080 --link zookeeper:zookeeper -e du ...

  4. 一、linux-mysql 运维DBA介绍

    一.DBA数据库管理人员需要在整个架构中解决数据库的压力,前端业务通过扩展,加机器就可以很好的解决,但是存储.数据库就不是很好的可以进行扩展,数据也是分配不均的,所以,1)通过在数据库前面添加Memc ...

  5. LINUX运维实战案例之文件已删除但空间不释放问题的分析与解决办法

    1.错误现象 运维的监控系统发来通知,报告一台服务器空间满了,登陆服务器查看,根分区确实没有空间了,如下图所示: 这里首先说明一下服务器的一些删除策略,由于Linux没有回收站功能,我们的线上服务器所 ...

  6. 网站运维之 使用IIS日志分析器1.03.exe进行IIS服务器日志分析

    引言 对于网站运维是一个比较要细心有耐心的工作,当一个网站从开发到上线后,后期的维护也很关键,特别是对于引流的网站来说更是至关重要. 对于网站运维的内容大致可以分为: SEO流量监控方面:风险防控:访 ...

  7. 恢复云数据库MySQL的备份文件到自建数据库遇到的报错

    报错信息 : 恢复云数据库MySQL的备份文件到自建数据库,自建数据库版本5.6.36. 按照阿里云文档操作,启动数据库报错 -- :: [ERROR] /application/mysql/bin/ ...

  8. NBU恢复数据库数据文件报错RMAN-06091

    RMAN-06091: no channel allocated for maintenance (of an appropriate type) 一.错误信息 报错信息如下 Starting res ...

  9. 启动mysql错误解决方案，学会查看错误日志：mysql.sock丢失，mysqld_safe启动报错

    本人还是个菜鸟,下面是我的经验之谈,能解决一些问题,有不对的地方,敬请斧正. 我的是CentOS6.3+MySQL5.1.57. 重启了一次服务器后,使用> mysql -u root -p登陆 ...

随机推荐

  1. VUE 利用webpack 给生产环境和发布环境配置不同的接口地址

    第一步,分别设置不同的接口地址 首先,我们分别找到下面的文件: /config/dev.env.js /config/prod.env.js 其实,这两个文件就是针对生产环境和发布环境设置不同参数的文 ...

  2. agc002E - Candy Piles(博弈论)

    题意 题目链接 Sol Orz SovitPower #include<bits/stdc++.h> #define Pair pair<int, double> #defin ...

  3. 前端了解即可：OSS客户端如何使用，以实现资源分离

    前一阵,应为项目需要,才简单整理了一下oss客户端的使用,做静态资源分离,及在项目中引入路径.接下来讲解记录一下,阿里云对象存储(Object Storage Service,简称OSS)客户端的基础 ...

  4. iPhone手机怎么投屏到电脑上

    如今生活水平不断上升,人们更加追求高质量.高享受的生活,所以可以利用一切资源提高生活质量,享受更好的生活体验,比如说手机投屏电脑就可以提高我们的视觉体验,所以更多的人去尝试,那么iPhone手机怎么投 ...

  5. Tars 服务调服务

    1,创建一个 tars 服务工程 2,把需要调用的服务的 客户端接口文件 拷贝到当前服务 3,构建通信器 CommunicatorConfig 调用,如果是 springboot 开发客户端可以使用注 ...

  6. Android 实现锚点定位

    相信做前端的都做过页面锚点定位的功能,通过<a href="#head"> 去设置页面内锚点定位跳转. 本篇文章就使用tablayout.scrollview来实现an ...

  7. pyinstaller打包错误 UnicodeDecodeError: 'gbk' codec can't decode byte 0xab in position 160:

    注:我的博客原本在CSDN,现转到博客园,图片采用以前的图片,并没有盗图. 在将.py文件打包时,出现了下列错误 >>C:\Users\小呆\PycharmProjects\pycharm ...

  8. 章节七、2-Linked List

    package introduction9; import java.util.ArrayList; import java.util.LinkedList; import java.util.Lis ...

  9. session,cookie,sessionStorage,localStorage的区别

    浏览器的缓存机制提供了可以将用户数据存储在客户端上的方式,可以利用cookie,session等跟服务端进行数据交互. 一.cookie和session cookie和session都是用来跟踪浏览器 ...

  10. myeclipse10 blue版激活码

    http://blog.itpub.net/27042095/viewspace-1164998/