TSM_ISSUE_123

dsmc 命令详解
http://www-ik.fzk.de/~apel/html/adsm_manual.html
TSM InfoCenter
http://www-01.ibm.com/support/knowledgecenter/SSGSG7/welcome
TSM Error Code
http://www-01.ibm.com/support/knowledgecenter/SSGSG7_6.3.0/com.ibm.itsm.msgs.client.doc/msgs_api_list_intro.html?lang=en

常见的数据库问题场景

场景1：
磁盘空间满，查看之后，发现该目录存放的是active log，存在大量的过期的日志。通过查看db2diag.log，
1）发现存在大量的归档日志失败，同时报TSM错误
2）归档日志成功，但是完成一个日志归档需要很长时间

场景2：

TSM 客户端升级或TSM server 修改，但是没有重启数据库

TSM 问题定位三把斧

）执行命令查看客户端与TSM server的连接，尝试上传一个小的文件
dsmc
dsmc i 'filename'
) 执行命令查看DB2 是否与TSM server连接正常
db2adutl query logs db db_name
) 检查client的错误日志 dsmerror.log

常见的获取信息的方法：

----查看环境变量获取错误日志信息

）通过如下命令查看环境变量DSM_LOG，指定了errorlog存放的路径，如果env没有输出，这说明使用的是默认路径/opt/tivoli/tsm/client/
env | grep -i dsm
[inst97@ha01 ~]$ env | grep -i dsm
DSMI_DIR=/opt/tivoli/tsm/client/api/bin64/
DSM_LOG=/home/inst97
DSM_DIR=/opt/tivoli/tsm/client/api/bin64/
DSM_CONFIG=/opt/tivoli/tsm/client/api/bin64/dsm.opt
DSMI_LOG=/home/inst97
DSMI_CONFIG=/opt/tivoli/tsm/client/api/bin64/dsm.opt

）错误日志文件名 dsmerror.log

----获取TSM Server hostname（IP）

）通过如下命令查看环境变量DSM（I）_CONFIG，获取定义SErver Name的dsm.opt文件，如果env没有输出，这说明使用的是默认路径/opt/tivoli/tsm/client/，api的在api/bin64 or api/bin, backup的配置在ba/bin
env | grep -i dsm
）查看文件dsm.opt，获取server name
）在DSM（I）_DIR路径下，查看dsm.sys文件，找到对应的server name的定义
）查看 TCPServeraddress 获取TSM server IP
TCPServeraddress xxxx.pok.ibm.com

----可以在TSM Client 使用命令查看到如下信息

query schedule
query mgmtclass

TSM 常见的错误
1. The TSM return code is -50
TSM server 不可用，需要engage OS team 和TSM Server support，均可以通过ops team 帮忙协调。

01/30/2015 12:00:13 ANS5216E Could not establish a TCP/IP connection with address '192.168.101.25:1500'. The TCP/IP error is 'Connection refused' (errno = 111).
01/30/2015 12:00:13 ANS9020E Could not establish a session with a TSM server or client agent.  The TSM return code is -50.
01/30/2015 12:00:13 ANS1017E Session rejected: TCP/IP connection failure

2. TSM RC=0x0000006A=106  DSM_RC_ACCESS_DENIED
   TSM Client 端某些文件权限存在问题，比如配置文件等，需要检测

【db2diag.log 中错误日志】
2015-02-02-22.32.21.766450+480 E29845E364          LEVEL: Error
PID     : 2182                 TID  : 140602221987648PROC : db2vend (db2logmgr.meth1 - 36 (D
INSTANCE: inst97               NODE : 000
FUNCTION: DB2 UDB, database utilities, sqluvint, probe:377
DATA #1 : TSM RC, PD_DB2_TYPE_TSM_RC, 4 bytes
TSM RC=0x0000006A=106 -- see TSM API Reference for meaning.

2015-02-02-22.32.21.766577+480 E30210E866          LEVEL: Error
PID     : 1102                 TID  : 139978105415424PROC : db2sysc 0
INSTANCE: inst97               NODE : 000
EDUID   : 36                   EDUNAME: db2logmgr (DB97) 0
FUNCTION: DB2 UDB, data protection services, sqlpInitVendorDevice, probe:1030
MESSAGE : ZRC=0x86100025=-2045771739=SQLP_MEDIA_VENDOR_DEV_ERR
          "A vendor device reported a media error."
DATA #1 : String, 29 bytes
Init failed!  Vendor rc info:
DATA #2 : Vendor RC, PD_DB2_TYPE_VENDOR_RC, 4 bytes
Vendor RC=0x0000000B=11 -- see DB2 API Guide for meaning.
DATA #3 : Hexdump, 48 bytes
0x00007F4EEEE78AD0 : 6A00 0000 3337 3720 3130 3600 0000 0000    j...377 106.....
0x00007F4EEEE78AE0 : 0000 0000 0000 0000 0000 0000 0000 0000    ................
0x00007F4EEEE78AF0 : 0000 0000 0000 0000 0000 0000 0000 0000    ................

2015-02-02-22.32.21.766715+480 I31077E429          LEVEL: Error
PID     : 1102                 TID  : 139978105415424PROC : db2sysc 0
INSTANCE: inst97               NODE : 000
EDUID   : 36                   EDUNAME: db2logmgr (DB97) 0
FUNCTION: DB2 UDB, data protection services, sqlpgArchiveLogVendor, probe:1820
RETCODE : ZRC=0x86100025=-2045771739=SQLP_MEDIA_VENDOR_DEV_ERR
          "A vendor device reported a media error."

2015-02-02-22.32.21.766813+480 E31507E461          LEVEL: Warning
PID     : 1102                 TID  : 139978105415424PROC : db2sysc 0
INSTANCE: inst97               NODE : 000
EDUID   : 36                   EDUNAME: db2logmgr (DB97) 0
FUNCTION: DB2 UDB, data protection services, sqlpgArchiveLogFile, probe:3150
MESSAGE : ADM1848W  Failed to archive log file "S0000000.LOG" to "TSM chain 0"
          from "/db2data/db97/inst97/NODE0000/SQL00001/SQLOGDIR/".

2015-02-02-22.32.21.767038+480 E31969E562          LEVEL: Error
PID     : 1102                 TID  : 139978105415424PROC : db2sysc 0
INSTANCE: inst97               NODE : 000
EDUID   : 36                   EDUNAME: db2logmgr (DB97) 0
FUNCTION: DB2 UDB, data protection services, sqlpgArchiveLogFile, probe:3160
MESSAGE : ZRC=0x86100025=-2045771739=SQLP_MEDIA_VENDOR_DEV_ERR
          "A vendor device reported a media error."
DATA #1 : <preformatted>
Failed to archive log file S0000000.LOG to TSM chain 0 from /db2data/db97/inst97/NODE0000/SQL00001/SQLOGDIR/.

3. TSM RC=0x0000000B=11 DSM_RC_ABORT_NO_REPOSIT_ SPACE 
   表明server端没有可用的存储空间

【db2diag.log 中错误日志】
2015-02-02-22.37.18.353566+480 E46638E364          LEVEL: Error
PID     : 2664                 TID  : 139694272812864PROC : db2vend (db2logmgr.meth1 - 22 (D
INSTANCE: inst97               NODE : 000
FUNCTION: DB2 UDB, database utilities, sqluvend, probe:1536
DATA #1 : TSM RC, PD_DB2_TYPE_TSM_RC, 4 bytes
TSM RC=0x0000000B=11 -- see TSM API Reference for meaning.

2015-02-02-22.37.18.354415+480 I47003E810          LEVEL: Error
PID     : 2638                 TID  : 139941229094656PROC : db2sysc 0
INSTANCE: inst97               NODE : 000
EDUID   : 22                   EDUNAME: db2logmgr (DB97) 0
FUNCTION: DB2 UDB, data protection services, sqlpCloseVendorDevice, probe:2800
MESSAGE : ZRC=0x86100025=-2045771739=SQLP_MEDIA_VENDOR_DEV_ERR
          "A vendor device reported a media error."
DATA #1 : Vendor RC, PD_DB2_TYPE_VENDOR_RC, 4 bytes
Vendor RC=0x00000011=17 -- see DB2 API Guide for meaning.
DATA #2 : Hexdump, 48 bytes
0x00007F4656E78AD0 : 0B00 0000 3135 3336 2031 3100 0000 0000    ....1536 11.....
0x00007F4656E78AE0 : 0000 0000 0000 0000 0000 0000 0000 0000    ................
0x00007F4656E78AF0 : 0000 0000 0000 0000 0000 0000 0000 0000    ................

2015-02-02-22.37.18.354561+480 I47814E429          LEVEL: Error
PID     : 2638                 TID  : 139941229094656PROC : db2sysc 0
INSTANCE: inst97               NODE : 000
EDUID   : 22                   EDUNAME: db2logmgr (DB97) 0
FUNCTION: DB2 UDB, data protection services, sqlpgArchiveLogVendor, probe:2870
MESSAGE : ZRC=0x86100025=-2045771739=SQLP_MEDIA_VENDOR_DEV_ERR
          "A vendor device reported a media error."

2015-02-02-22.37.18.354623+480 E48244E461          LEVEL: Warning
PID     : 2638                 TID  : 139941229094656PROC : db2sysc 0
INSTANCE: inst97               NODE : 000
EDUID   : 22                   EDUNAME: db2logmgr (DB97) 0
FUNCTION: DB2 UDB, data protection services, sqlpgArchiveLogFile, probe:3150
MESSAGE : ADM1848W  Failed to archive log file "S0000000.LOG" to "TSM chain 0"
          from "/db2data/db97/inst97/NODE0000/SQL00001/SQLOGDIR/".

4. 一些配置文件权限不够，导致dsmc 执行失败，需要engage OS team 帮忙修改。
0116 E DSM_RC_FILE_BEING_EXECUTED   File is in use; Write permission denied.

附录：正确的数据库归档日志
2015-02-02-22.45.02.803066+480 E50590E379          LEVEL: Info
PID     : 2638                 TID  : 139941216511744PROC : db2sysc 0
INSTANCE: inst97               NODE : 000
EDUID   : 39                   EDUNAME: db2logmgr (DB97) 0
FUNCTION: DB2 UDB, data protection services, sqlpgArchiveLogFile, probe:3108
DATA #1 : <preformatted>
Started archive for log file S0000000.LOG.

2015-02-02-22.45.05.216822+480 E50970E450          LEVEL: Info
PID     : 2638                 TID  : 139941216511744PROC : db2sysc 0
INSTANCE: inst97               NODE : 000
EDUID   : 39                   EDUNAME: db2logmgr (DB97) 0
FUNCTION: DB2 UDB, data protection services, sqlpgArchiveLogFile, probe:3180
DATA #1 : <preformatted>
Completed archive for log file S0000000.LOG to TSM chain 0 from /db2data/db97/inst97/NODE0000/SQL00001/SQLOGDIR/.

案例分享：
案例一：
IN5837476

the configuration file dsm.opt for API is linked to the one for BA.

There's one line that led tsm api issue

"subdir yes" is dedicately for for BA, shouldn't be in this file (but it is)

Besides, COMMMethod TCPip should be in this file (but it is not)

so the db2adutl got errors but dsmc is good.

we had OS team to create new configuration file for TSM API with correct contents, and recycled db2 instances that using this TSM client. issue fixed.

案例二：
0116 E DSM_RC_FILE_BEING_EXECUTED   File is in use; Write permission denied.
dsm.sys的 group 缺少实例用户组导致无法正常使用TSM

案例三：

TSM Client:

b01cxnp11081.gho.pok.ibm.com  1.18

TSM Case:

Server Info: xxx

Instance: opicmdb5

Detail: dsmc q b "/db/db2backup2/HVECODS2*", could not get any result.

Cause: the backup command is

dsmc q b "/db/db2backup2/HVECODS2*"  -inactive

dsmc incremental /db/db2backup2/HVECODS2* -se=pokxmet3_db2

if we need to check it, we should specify the SE

案例四：

TSM Case:

Server Info: xxx

Instance: opicmdb5

Detail:

Incremental backup of volume '/db/db2backup2/HVECODS2*'

Normal File-->    53,187,989,504 /db/db2backup2/HVECODS2.0.opicmdb5.NODE0000.CATN0000.20140310210103.001  ** Unsuccessful

**

ANS1228E Sending of object '/db/db2backup2/HVECODS2.0.opicmdb5.NODE0000.CATN0000.20140310210103.001' failed

ANS0326E This node has exceeded its maximum number of mount points.

Normal File-->    53,217,349,632 /db/db2backup2/HVECODS2.0.opicmdb5.NODE0000.CATN0000.20140311210104.001  ** Unsuccessful

**

ANS1228E Sending of object '/db/db2backup2/HVECODS2.0.opicmdb5.NODE0000.CATN0000.20140311210104.001' failed

ANS0326E This node has exceeded its maximum number of mount points.

Normal File-->    53,221,543,936 /db/db2backup2/HVECODS2.0.opicmdb5.NODE0000.CATN0000.20140312210104.001  ** Unsuccessful

**

ANS1228E Sending of object '/db/db2backup2/HVECODS2.0.opicmdb5.NODE0000.CATN0000.20140312210104.001' failed

ANS0326E This node has exceeded its maximum number of mount points.

ANS1802E Incremental backup of '/db/db2backup2/HVECODS2*' finished with 3 failure

Cause:

ANS0326E This node has exceeded its maximum number of mount points.

根本原因是多个dsmc同时执行上传命令导致

参考连接
TSM Doc 6.1 PDF下载
https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/Tivoli%20Storage%20Manager/page/PDF%20versions%20of%20the%20IBM%20Tivoli%20Storage%20Manager%20Version%206.1.x%20documentation

TSM Doc 5.5 PDF下载
https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/Tivoli%20Storage%20Manager/page/PDF%20versions%20of%20the%20IBM%20Tivoli%20Storage%20Manager%20Version%205.5.x%20documentation