[20180612]删除bootstrap$记录无法启动.txt

[20180612]删除bootstrap$记录无法启动.txt

--//前几天看链接http://www.xifenfei.com/2018/05/willfully-delete-bootstrap.html.
--//按照介绍: 有人在数据库中注入了恶意脚本，导致数据库删除了bootstrap$中数据,关闭之后无法正常启动delete from bootstrap$;

--//我更多的思考如果我遇到这个问题如何解决:
1.如果有备份只要恢复到delete bootstrap$之前,但是这里有问题,因为删除后数据库还继续运行.不能继续应用日志,这样有恢复到
  删除bootstrap$后状态.

2.如果有备份很好解决,因为bootstrap$的相关块的信息不会变动,只要覆盖对应块就ok了.
  而且实际上只要oracle版本相同,OS平台一样,使用别的数据库的system表空间文件中对应的块替换应该一点问题都没有.

3.当然最笨的方法就是恢复删除的记录.因为执行删除记录多,手工恢复感觉还是比较麻烦.

--//我自己也测试看看,演示后2种恢复方法:千万不要再生产系统做这样的测试!!

1.环境:
SCOTT@book> @ ver1
PORT_STRING                    VERSION        BANNER
------------------------------ -------------- --------------------------------------------------------------------------------
x86_64/Linux 2.4.xx            11.2.0.4.0     Oracle Database 11g Enterprise Edition Release 11.2.0.4.0 - 64bit Production

--//最好备份数据库略.首先看看bootstrap$占用那些块.
SCOTT@book> select HEADER_FILE,HEADER_BLOCK,BLOCKS,EXTENTS from dba_segments where owner='SYS' and segment_name='BOOTSTRAP$';
HEADER_FILE HEADER_BLOCK     BLOCKS    EXTENTS
----------- ------------ ---------- ----------
          1          520          8          1

SCOTT@book> column PARTITION_NAME noprint
SCOTT@book> select * from dba_extents where owner='SYS' and segment_name='BOOTSTRAP$';
OWNER  SEGMENT_NAME         SEGMENT_TYPE       TABLESPACE_NAME                 EXTENT_ID    FILE_ID   BLOCK_ID      BYTES     BLOCKS RELATIVE_FNO
------ -------------------- ------------------ ------------------------------ ---------- ---------- ---------- ---------- ---------- ------------
SYS    BOOTSTRAP$           TABLE              SYSTEM                                  0          1        520      65536          8            1

--//在system数据文件头部也记录这个位置.通过bbed观察:
BBED> p dba 1,1 kcvfh.kcvfhrdb
ub4 kcvfhrdb                                @96       0x00400208

BBED> set dba 0x00400208
        DBA             0x00400208 (4194824 1,520)
--//dba= 0x00400208指向的位置就是1,520,也就是sys.BOOTSTRAP$的段头.

--//做一个sys.bootstrap$的备份:
SCOTT@book> create table bootstrap$bak as select * from sys.bootstrap$;
Table created.

SCOTT@book> select HEADER_FILE,HEADER_BLOCK,BLOCKS,EXTENTS from dba_segments where owner=OWNER and segment_name='BOOTSTRAP$BAK';
HEADER_FILE HEADER_BLOCK     BLOCKS    EXTENTS
----------- ------------ ---------- ----------
          4          858          8          1
--//开始破坏....
SYS@book> delete from sys.bootstrap$;
60 rows deleted.

SYS@book> commit ;
Commit complete.

--//实际上这个问题最严重的是如果你一直不重启,根本不知道这个问题的存在,知道下次重启才发现问题,
--//也就是可能备份的文件一直存在问题的.^_^.

2.重启数据库:
SYS@book> shutdown immediate ;
Database closed.
Database dismounted.
ORACLE instance shut down.
SYS@book> startup
ORACLE instance started.

Total System Global Area  634732544 bytes
Fixed Size                  2255792 bytes
Variable Size             197133392 bytes
Database Buffers          427819008 bytes
Redo Buffers                7524352 bytes
Database mounted.
ORA-03113: end-of-file on communication channel
Process ID: 54149
Session ID: 274 Serial number: 3

--//alert显示
SMON: enabling cache recovery
Exception [type: SIGSEGV, Address not mapped to object] [ADDR:0x0] [PC:0x977B258, lmebucp()+24] [flags: 0x0, count: 1]
Thu Jun 14 09:00:48 2018
ARC1 started with pid=46, OS id=54268
Errors in file /u01/app/oracle/diag/rdbms/book/book/trace/book_ora_54218.trc  (incident=2125010):
ORA-07445: exception encountered: core dump [lmebucp()+24] [SIGSEGV] [ADDR:0x0] [PC:0x977B258] [Address not mapped to object] []
Incident details in: /u01/app/oracle/diag/rdbms/book/book/incident/incdir_2125010/book_ora_54218_i2125010.trc
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
ARC1: Archival started
ARC0: STARTING ARCH PROCESSES COMPLETE
ARC0: Becoming the 'no FAL' ARCH
ARC0: Becoming the 'no SRL' ARCH
ARC1: Becoming the heartbeat ARCH
ARC0: Evaluating archive   log 3 thread 1 sequence 922
krss_find_arc: Selecting ARC1 to receive message as last resort
ARC1: Evaluating archive   log 3 thread 1 sequence 922
ARC0: Beginning to archive thread 1 sequence 922 (13280587483-13280607489) (book)
ARC0: Creating local archive destination LOG_ARCHIVE_DEST_1: '/u01/app/oracle/archivelog/book/1_922_896605872.dbf' (thread 1 sequence 922) (book)
ARC1: Unable to archive thread 1 sequence 922
      Log actively being archived by another process
ARC0: Closing local archive destination LOG_ARCHIVE_DEST_1: '/u01/app/oracle/archivelog/book/1_922_896605872.dbf' (book)
Committing creation of archivelog '/u01/app/oracle/archivelog/book/1_922_896605872.dbf'
Archived Log entry 16 added for thread 1 sequence 922 ID 0x4fb7d86e dest 1:
ARC0: Completed archiving thread 1 sequence 922 (13280587483-13280607489) (book)
Dumping diagnostic data in directory=[cdmp_20180614090051], requested by (instance=1, osid=54218), summary=[incident=2125010].
Incremental checkpoint up to RBA [0x39b.3.0], current log tail at RBA [0x39b.3.0]
Thu Jun 14 09:00:51 2018
PMON (ospid: 54174): terminating the instance due to error 397
Thu Jun 14 09:00:52 2018
System state dump requested by (instance=1, osid=54174 (PMON)), summary=[abnormal instance termination].
System State dumped to trace file /u01/app/oracle/diag/rdbms/book/book/trace/book_diag_54184_20180614090052.trc
Dumping diagnostic data in directory=[cdmp_20180614090052], requested by (instance=1, osid=54174 (PMON)), summary=[abnormal instance termination].
Instance terminated by PMON, pid = 54174
--//从以上跟踪文件很难发现是由于bootstrap$信息被删除.

3.使用10046跟踪分析.
SYS@book> startup mount
ORACLE instance started.

Total System Global Area  634732544 bytes
Fixed Size                  2255792 bytes
Variable Size             197133392 bytes
Database Buffers          427819008 bytes
Redo Buffers                7524352 bytes
Database mounted.
SYS@book> @ &r/10046on 12
old   1: alter session set events '10046 trace name context forever, level &1'
new   1: alter session set events '10046 trace name context forever, level 12'

Session altered.

SYS@book> alter database open ;
ERROR:
ORA-03113: end-of-file on communication channel
Process ID: 54336
Session ID: 274 Serial number: 3

SYS@book> @ &r/10046off
ERROR:
ORA-03114: not connected to ORACLE

--//通过跟踪文件发现如下错误:
=====================
PARSING IN CURSOR #140711922570672 len=188 dep=1 uid=0 oct=1 lid=0 tim=1528938422280523 hv=4006182593 ad='7d2995f0' sqlid='32r4f1brckzq1'
create table bootstrap$ (
END OF STMT
PARSE #140711922570672:c=2000,e=947,p=0,cr=0,cu=0,mis=1,r=0,dep=1,og=4,plh=0,tim=1528938422280521
EXEC #140711922570672:c=0,e=235,p=0,cr=0,cu=0,mis=0,r=0,dep=1,og=4,plh=0,tim=1528938422280848
CLOSE #140711922570672:c=0,e=8,dep=1,type=0,tim=1528938422280938
=====================
PARSING IN CURSOR #140711922570672 len=55 dep=1 uid=0 oct=3 lid=0 tim=1528938422281543 hv=2111436465 ad='7d297f30' sqlid='6apq2rjyxmxpj'
select line#, sql_text from bootstrap$ where obj# != :1
END OF STMT
PARSE #140711922570672:c=999,e=571,p=0,cr=0,cu=0,mis=1,r=0,dep=1,og=4,plh=0,tim=1528938422281541
BINDS #140711922570672:
 Bind#0
  oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00
  oacflg=08 fl2=0001 frm=00 csi=00 siz=24 off=0
  kxsbbbfp=7ffa0c291168  bln=22  avl=02  flg=05
  value=59
EXEC #140711922570672:c=1000,e=976,p=0,cr=0,cu=0,mis=1,r=0,dep=1,og=4,plh=867914364,tim=1528938422282657
WAIT #140711922570672: nam='db file sequential read' ela= 11 file#=1 block#=520 blocks=1 obj#=59 tim=1528938422282744
WAIT #140711922570672: nam='db file scattered read' ela= 24 file#=1 block#=521 blocks=3 obj#=59 tim=1528938422282974
FETCH #140711922570672:c=0,e=370,p=4,cr=5,cu=0,mis=0,r=0,dep=1,og=4,plh=867914364,tim=1528938422283068
STAT #140711922570672 id=1 cnt=0 pid=0 pos=1 obj=59 op='TABLE ACCESS FULL BOOTSTRAP$ (cr=5 pr=4 pw=0 time=371 us)'

*** 2018-06-14 09:07:02.283
Exception [type: SIGSEGV, Address not mapped to object] [ADDR:0x0] [PC:0x977B258, lmebucp()+24] [flags: 0x0, count: 1]
Incident 2127890 created, dump file: /u01/app/oracle/diag/rdbms/book/book/incident/incdir_2127890/book_ora_54445_i2127890.trc
ORA-07445: exception encountered: core dump [lmebucp()+24] [SIGSEGV] [ADDR:0x0] [PC:0x977B258] [Address not mapped to object] []

ssexhd: crashing the process...
Shadow_Core_Dump = partial
ksdbgcra: writing core file to directory '/u01/app/oracle/diag/rdbms/book/book/cdump'

--//可以发现在执行
select line#, sql_text from bootstrap$ where obj# != :1
--// :1 = 59时候就报错了.

--//通过bbed观察
BBED> x /rnnc *kdbr[0]
rowdata[6877]                               @8167
-------------
flag@8167: 0x3c (KDRHFL, KDRHFF, KDRHFD, KDRHFH)
lock@8168: 0x01
cols@8169:    0

BBED> x /rnnc *kdbr[1]
rowdata[6740]                               @8030
-------------
flag@8030: 0x3c (KDRHFL, KDRHFF, KDRHFD, KDRHFH)
lock@8031: 0x01
cols@8032:    0

BBED> dump /v
 File: /mnt/ramdisk/book/system01.dbf (1)
 Block: 521                               Offsets: 8030 to 8093                            Dba:0x00400209
-----------------------------------------------------------------------------------------------------------
 3c010301 80018081 43524541 54452052 4f4c4c42 41434b20 5345474d 454e5420 l <.......CREATE ROLLBACK SEGMENT
 53595354 454d2053 544f5241 47452028 2020494e 49544941 4c203131 324b204e l SYSTEM STORAGE (  INITIAL 112K N
<32 bytes per line>
--//可以发现相关记录被删除了.我手工修复2条记录看看:
BBED> set dba 1,521
        DBA             0x00400209 (4194825 1,521)

BBED> x /rnnc *kdbr[0]
rowdata[6877]                               @8167
-------------
flag@8167: 0x3c (KDRHFL, KDRHFF, KDRHFD, KDRHFH)
lock@8168: 0x01
cols@8169:    0

BBED> assign  dba 1,521 offset 8167 = 0x2c
Warning: contents of previous BIFILE will be lost. Proceed? (Y/N) y
ub1 rowdata[0]                              @8167     0x2c

BBED> x /rnnc *kdbr[0]
rowdata[6877]                               @8167
-------------
flag@8167: 0x2c (KDRHFL, KDRHFF, KDRHFH)
lock@8168: 0x01
cols@8169:    3

col    0[3] @8170: -1
col    1[3] @8174: -1
col    2[9] @8178: 8.0.0.0.0

BBED> assign  dba 1,521 offset 8030 = 0x2c
ub1 rowdata[0]                              @8030     0x2c

BBED> x /rnnc *kdbr[1]
rowdata[6740]                               @8030
-------------
flag@8030: 0x2c (KDRHFL, KDRHFF, KDRHFH)
lock@8031: 0x01
cols@8032:    3

col    0[1] @8033: 0
col    1[1] @8035: 0
col  2[129] @8037: CREATE ROLLBACK SEGMENT SYSTEM STORAGE (  INITIAL 112K NEXT 56K MINEXTENTS 1 MAXEXTENTS 32765 OBJNO 0 EXTENTS (FILE 1 BLOCK 128))

--//注意一个细节:cols@8169:    0的数量我被没有修改,也就是恢复删除记录仅仅修嘎flag从0x3c=>0x2c(多数情况是这样,除非出现行迁移或者分片之类情况).

4.恢复:
--//借助备份恢复,我测试环境一直保存一个冷备份,借助这个备份恢复看看.

$  dd if=/u01/backup/20170301B/system01.dbf of=/mnt/ramdisk/book/system01.dbf bs=8192 skip=520  seek=520 count=8 conv=notrunc
8+0 records in
8+0 records out
65536 bytes (66 kB) copied, 0.0163988 seconds, 4.0 MB/s

--//再次提醒使用dd命令,我现在每次使用都心存敬畏.就是自己做错了1次.
--//千万不要搞错输入输出的对象..
--//skip对应输入文件跳过的块数,seek对应输出文件跳过的块数,注意一定不要忘记加conv=notrunc参数,不然输出文件可能被截断.
--//这里数据块大小8192.
--//说明:/u01/app/oracle11g/oradata/test/system01.dbf这个是很久以前的冷备份.
--//再次通过bbed观察
BBED> set dba 1,521
        DBA             0x00400209 (4194825 1,521)

BBED> x /rnnc *kdbr[1]
rowdata[6740]                               @8030
-------------
flag@8030: 0x2c (KDRHFL, KDRHFF, KDRHFH)
lock@8031: 0x01
cols@8032:    3

col    0[1] @8033: 0
col    1[1] @8035: 0
col  2[129] @8037: CREATE ROLLBACK SEGMENT SYSTEM STORAGE (  INITIAL 112K NEXT 56K MINEXTENTS 1 MAXEXTENTS 32765 OBJNO 0 EXTENTS (FILE 1 BLOCK 128))
--//ok,现在已经修复.

--//再次重启数据库:
SYS@book> startup mount
ORACLE instance started.
Total System Global Area  634732544 bytes
Fixed Size                  2255792 bytes
Variable Size             197133392 bytes
Database Buffers          427819008 bytes
Redo Buffers                7524352 bytes
Database mounted.

SYS@book> alter database open ;
Database altered.

5.测试使用bbed的方法:
--//估计有点长另外写一篇文章.