unixadmin.free.fr Handy Unix Plumbing Tips and Tricks

Recommended Tivoli Storage Manager Server Post-Database Restore Processing Steps

Question

After a DSMSERV RESTORE DB is completed, are there any additional steps that should be taken before resuming normal operations of the Tivoli Storage Manager Server?

Cause

Restore of the Tivoli Storage Manager server database to an earlier point in time can cause Tivoli Storage Manager to encounter data on disk and tapes for which it has no record, or records in its database for which the data in question no longer exists (whether at all or in its original location), resulting in ANR1330E and ANR1331E errors and/or ANR0340E messages for 6.3+ level servers using node replication.

Answer

NOTE: It is important to save a copy of the volume history file (volhist.out, captured by running the BACKUP VOLHIST command) on a regular basis. The information in this file can be used to reduce the amount of volume auditing that is necessary for step 4 below.

After a Tivoli Storage Manager Server restore to a previous point in time, it is necessary to ensure that the storage pool volumes are consistent with the Tivoli Storage Manager database which has been restored The following steps should be performed prior to returning the Tivoli Storage Manager server to normal use:

1. Immediately after the Tivoli Storage Manager server database is restored, place the following options into the dsmserv.opt file:

NOMIGRRECL (Prevents migration and reclamation from kicking off when the server is started)
DISABLESCHED YES (prevents any schedules from starting when the server is started)
EXPINT 0 (prevents expiration from running when the server is started) If set to something other than 0, change the setting to 0 and record the original setting. In step 7 we will remove or reset this option back to the original setting.

2. Start the server in the foreground (DSMSERV)

3. When the Tivoli Storage Manager server prompt comes up, enter the command DISABLE SESSIONS so that clients will be unable to connect to the Tivoli Storage Manager Server. while these cleanup steps are being performed.

4. Review the saved volume history file, and locate sequential tape or FILE volumes marked as "STGDELETE" and "STGREUSE" in the list. You will need to audit only these types of volumes created after the date of the database backup that had been restored. If the volume history file is lost or unrecoverable, then all sequential volumes should be audited, for the sake of caution. The audit command is performed as follows:

AUDIT VOLUME FIX=YES

Note that this command will verify the data on the volume against the entries in the Tivoli Storage Manager Server database. Any items that are not found will be removed from the Tivoli Storage Manager Server.

5. It will also necessary to audit all volumes in random-access disk storage pool volumes with the same command above. These are volumes belonging to storage pools whose device class name is DISK.

6. Once all applicable volume audits have been completed, the data on the volumes should be synchronized with the Tivoli Storage Manager Database at the point in time of the last restored database backup.

7. Halt the server and remove the following two options added in step one: nomigrrecl, "disablescheds yes"; and set the expinterval back to its original setting (or comment it out if it had not previously been set).

8. Restart the server as normal. Connect to the running Tivoli Storage Manager server and Issue the command ENABLE SESSIONS to allow clients to connect to the server again.

9. For TSM Server 6.3 or higher, if using node replication between servers, resync replication. Reference the steps from the messages manual for message ANR0340E (Link below)

Windows 2008 – TSM Server upgrade fails due to DB2 upgrade problem

Problem

An active wmiprvse process prevents DB2 from upgrading.

Cause

Diagnosing the problem

Resolving the problem

ANR9999D_0521343731 and Could not determine media type , rc = 1

Problem

When the data is being backed up to a Tape Library/Drive, the following error is received during a write operation:
ANR9999D_0521343731 NtpOpen(pvrntp.c:1296) Thread<36>: Could not determine media type, rc = 1

Symptom

The label and checkin functions of the TSM Server work fine without any issues.

Cause

Wrong Drivers for the Library / Drives. Microsoft drivers were installed for the IBM LTO Drive.

Resolving the problem

Please verify that you are using the correct drivers for the device

ANR2969E Database restore terminated. DB2 sqlcode: -2033. DB2 sqlerrmc: 5802

Problem
Impossible to restore TSM database 6.1 on Windows 2008 64-bits (Ex: TSM Install directory : C:\TSM)
Problem for Tivoli Storage Manager to complete a database restoration with DB2 SQLCODE: -2033 and sqlerrmc 5802

Cause
With Tivoli Storage Manager 6.1 Fixpack 5 the TSM API display version 6.2.1

I note this when I run dsmsutil.exe utility before "dsmserv restore db"
Just for test, dsmapipw is normally use on Unix.

Resolving the problem

In most case the reinstallation of TSM client correct the problem.
In my case, I needed to compare version of all TSM API binary in Extraction directory with installation path.
right clic > properties > detail > and check the file version.
Ex: compare C:\tsm_image\TSM_BA_client\system64\tsmapi64.dll and C:\Windows\system32\tsmapi64.dll

In my case tsmapi64.dll, dsmntapi64.dll and tsmutil164.dll from C:\Windows\system32\ directory display 6.2.1 version, the new TSM client 6.1 installation wasn't overwrite file in system32 directory, but write 6.1.5.0 in registry ???
I must overwrite manually the wrong files with correct dll from C:\tsm_image\TSM_BA_client\system64, then reload TSM database recovery process ... that's works

Below a example of TSM 6.1 disaster recovery from scratch:

- remove TSM 6.1 => C:\TSM\_uninst\Uninstall Tivoli Storage Manager.exe

- remove TSM client 6.1 => Control panel > software > remove TSM client

- remove old Windows service => Ex: sc delete "TSM Scheduler"

- reboot Windows 2008 64-bits Operating system

- install TSM server 6.1.0.0 64-bits => CZ1N9ML.exe

- install TSM client 6.1.0.0 64-bits => 6.1.0.0-TIV-TSMBAC-WinX64.exe

- apply fixpack 6.1.5.0 => 6.1.5.0-TIV-TSMALL-WindowsX64.exe and 6.1.5.0-TIV-TSMBAC-WinX64.exe

- check Windows environment variable after new installation for DB2 and Global Secure ToolKit (GSK7_64)
Update the variable and remove old PATH that cannot be removed during uninstall.

- reboot and log back in with db2user1 account created during tsm install

- check and remove registry directory Server1 only:
HKEY_LOCAL_MACHINE\SOFTWARE\IBM\ADSM\CurrentVersion\Server\Server1

- run C:\TSM\server\dsmicfgx.exe and create a new TSM Server1 instance

- after, stop the Windows services for the fresh TSM instance (TSM Server1) and TSM db2 instance (DB2 - DB2TSM1 - SERVER1)

- open the db2 command line processor

- list db2 instance

- drop instance SERVER1

- create db2 instance for new TSM instance SERVER1

- set Windows environment variable DB2INSTANCE

- set db2 environment variable DB2_VENDOR_INI for DB2 instance Server1

- restart db2

- Ensure that the DSMI_CONFIG environment variable points to a valid TSM options file

- Enter the following command on one line

! A directory Server1 is created during the initial instance creation.
- remove all files and directory in the database, active log and archive log installation directory
Ex:
D:\tsm01\db01
E:\tsm01\db02
F:\tsm01\db03
G:\tsm01\db04
H:\tsm01\actlog
I:\tsm01\archlog

- restore the volhist.dat, dsmserv.opt, devconfig.dat need by "dsmserv restore db" files to C:\TSM\Server1

- create a text file C:\TSM\Server1\dbdir.txt with database paths like :
D:\tsm01\db01
E:\tsm01\db02
F:\tsm01\db03
G:\tsm01\db04

- from the C:\TSM\Server1 directory run :

- after database restore, try to start the Tivoli Storage Manager server in the foreground with dsmserv.exe :

Once this looks good, then halt it and start the Tivoli Storage Manager "TSM Server1" service

- If you have this below message, empty directory archive log and reload dsmserv.exe

Help

if you want to know what the DB2 error means :

For the SQLERRMC value check the include file "C:\TSM\api64\INCLUDE\dsmrc.h"

Good luck

System date/time change causes ANR0110E unexpected date sessions disabled

Problem

Operating system date/time changes cause the TSM server to disable sessions.

Resolving the problem

As a preventive measure, when the operating system date/time are changed on the TSM server it will automatically disable all TSM client sessions so that the TSM server does not perform undesired actions based on the new system date/time. For example, expiration processing uses the system date/time to determine when to purge files from server storage. If the system date/time was accidentally change, the server may expire data that was not meant to be purged and thus a loss of data could occur.

The TSM server will report the following when a system date/time change occurs:

ANR0110E An unexpected system date has been detected; the server is disabled. Use the ACCEPT DATE command to establish the current date as valid.

If you do not notice the above error and attempt a client backup the server will generate the following message:

ANR0420W Session 158 for node (PLATFORM) refused - server disabled for user access.

The TSM client would generate the following error to the dsmerror.log:

ANS1355E Session rejected: Server disabled

In order to correct this problem, perform the following commands on the TSM server:

1. Run the QUERY STATUS command and check for the availability parameter value, which should say "disabled'

2. Run the ACCEPT DATE command, which tells the TSM Server that you are aware of and accept the date/time change.

3. Run the ENABLE SESSIONS command, which re-enables the TSM Server again for normal operations.

Tivoli Storage Manager Server 1024 Database Connection Limit on AIX

Versions TSM 6.1 and 6.2

SOURCE: Technote 1428557

SCRIPT: dsmcvt2tcp

Abstract

The Tivoli Storage Manager server cannot establish more than 1024 simultaneous connections to its DB2 database when using local, IPC-based connections on AIX. Servers with large workloads might reach this limit and fail with a number of different symptoms. Internal errors in the server can also cause it to run out of connections even when the server is not running a large workload. A workaround is available for V6.1 and V6.2. A fix is available in V6.3.
Content

Summary

* The 1024 maximum connection limit, when using local database connections, is a permanent restriction on AIX systems. You can bypass the limitation by reconfiguring the server database to use TCP/IP to connect to the database instead of local connections.

Beginning with level 6.3.0, the server will automatically use TCP/IP connections. Both new servers and existing servers that are upgraded from earlier versions will be automatically reconfigured to use TCP/IP. No manual action is required when using level 6.3.0 or higher.
* You can convert to using TCP/IP connections using the dsmcvt2tcp script or by following the manual steps detailed below in section "Instructions for converting to TCP/IP connections." The script is installed with the server beginning in levels 6.1.4 and 6.2.1. You can also download it at: public.dhe.ibm.com.
* There are security implications to switching to TCP/IP connections. The dsmcvt2tcp script will set up firewall rules to reduce the impact of the change. Instructions for manually setting up the firewall rules are detailed below in section "Security implications of using TCP/IP connections."

Note: The Tivoli Storage Manager V6.3 server fixes the security concerns associated with the workaround. As a result, firewall rules are suggested only when using the workaround with V6.1 or V6.2 servers.
* A known problem on AIX (APAR IZ75548) can cause severe performance problems when enabling AIX IP Security with 10 GB network interfaces. A fix is available.
* An additional server problem has been discovered that might aggravate the problem by causing a rapid increase in the number of client sessions. A fix for the problem (APAR PK84259) is available in levels 6.1.3.4, 6.1.4, and 6.2.1.
* Tivoli Storage Manager server APAR IC76335 documents an error in which the server might fail to close unused connections to the database. This can cause the server to establish more than 1024 simultaneous connections to the database even when the server is not heavily loaded. Refer to http://www.ibm.com/support/docview.wss?uid=swg1IC76335 for more information about the APAR and the circumstances that can lead to the server failing to close connections to the database. APAR IC76335 is projected to be fixed in levels 6.1.5.1 and 6.1.6, and is available in levels 6.2.3 and 6.3.0.
* Some maintenance activities can cause the server to resume using local connections. If this problem occurs, it will be necessary to repeat the steps used to configure the server database to use TCP/IP. Section "Maintenance activities that affect TCP/IP connections" provides details on the activities that can cause this.
* If the server is running on a high availability cluster using solutions such as IBM PowerHA or Tivoli System Automation, then the steps that you take to configure the server database to use TCP/IP might need to be repeated on each node of the cluster.

Symptoms

Tivoli Storage Manager server operations might fail when the server is under heavy workload. Failure symptoms include some or all of the following server messages:

* ANR0171I dbieval.c(827): Error detected on 0:1026, database in evaluation mode.
* ANR9999D_0645605689 tbOpenX(tbtbl.c:4273) Thread: Failure participating on transaction.
* ANR9999D_1910305017 smExecuteSession(smexec.c:2379) Thread: Session NNNNN with client CLIENTNAME (PLATFORM) rejected - error creating the central logging vector.
* ANR0105E sspool.c(4451): Error setting search bounds for table "SS.Pools".
* ANR0102E admactlg.c(4212): Error 16 inserting row in table "Activity.Log".
* ANR2009E Activity log process has stopped - database error.
* ANR2006E Activity log process was not started, the default output stream cannot be opened

After issuing the messages, the server might hang or crash, although this does not happen in all cases.

Diagnosing the Problem

1. Locate the db2diag.log for the database instance.

The DIAGPATH database manager configuration parameter identifies the location of the log file. Issue this command to determine the value of the configuration parameter: db2 get dbm cfg | grep DIAGPATH.
2. Inspect the db2diag.log file. The db2diag.log file records errors that occur near the time of the failure. The specific errors that indicate that the 1024 connection limit has been reached are:

Resolving the Problem

Instructions for converting to TCP/IP connections

The dsmcvt2tcp script can be used to convert an existing Tivoli Storage Manager database instance to use TCP/IP connections. The script will be shipped with the AIX server beginning in levels 6.1.4 and 6.2.1.
It can also be downloaded at:ftp://public.dhe.ibm.com/storage/tivoli-storage-management/tools/aix/dsmcvt2tcp

Note: The dsmcvt2tcp script is not shipped with level 6.3.0 because the server will automatically use TCP/IP connections.

After you have downloaded the script, run it with the -h parameter for usage instructions.

Alternatively, follow these instructions to manually convert to TCP/IP connections. The following instructions assume that the Tivoli Storage Manager DB2 instance user is tsminst1. If the instance user has a different name, then you must replace any occurrences of "tsminst1" with the actual name of the instance user.

1. Find the service name in /etc/services that the DB2 instance uses. You can find this by running this command:

Tip: The service name that the tsminst1 instance uses is typically named "DB2_tsminst1".

If there is no entry for the database instance in the /etc/services directory, you must add one (while logged in as root). The DB2 service uses a range of four contiguous port numbers, and cannot use any number that is being used by another service. Also note that no port number can be greater than 65536. Search /etc/services for the proposed port numbers before using.

Note: The white space in the preceding example is a single tab character, not spaces.

2. Log in as the Tivoli Storage Manager instance user.

3. Run this command:

Take note of the 'Local database directory' value associated with the TSMDB1 database. This is typically set to the instance home directory, but might have a different value.

4. Run the following command:

Note the values of the settings for the AUTHENTICATION, TRUST_CLNTAUTH, SVCENAME, and NUM_POOLAGENTS parameters. You will need this information if you have to back out the change at a later time.

5. Shut down the Tivoli Storage Manager server.

6. Run the following command to ensure that the DB2 server is shut down:

7. Run the following commands:

Note: The value specified for SVCENAME should match the service name in /etc/services. Also note that NUM_POOLAGENTS is set to 0 to circumvent a crash in DB2 that has been observed after setting it up to use TCP/IP, but is not strictly needed for the conversion itself.

8. Run the following command:

9. Run the following command:

The last parameter should match the service name in /etc/services.

10. Run the following command:

12. Run the following command:

13. Run "db2start" to start the database manager. Issue the following command to verify that the configuration parameters that were set in step 7 were changed to the appropriate new values:

14. Run the following commands to verify that you can connect to the database:

If the connection is successful then Tivoli Storage Manager should start up normally.

Note: If Step 13 is slow, or if the server startup is abnormally slow, then your environment might have a DNS server set to ignore TCP/IP v6, which delays DB2 connections. This can be fixed by modifying /etc/netsvc.conf to only look up TCP/IP v4 addresses, assuming your environment does not use TCP/IP v6. For example, change /etc/netsvc.conf from:

to:

Security implications of using TCP/IP connections

Configuring the database manager to accept TCP/IP connections allows remote users to connect to the database. To prevent remote users from connecting, and to ward off potential security risks associated with remote users, consider configuring the AIX IP Security feature to only allow local users to connect to the database. Note, however, that in some cases, configuring IP Security can result in a severe performance degradation. Work-arounds are documented in AIX APAR IZ75548, and are also described below in "Performance implications of using TCP/IP connections."

Log in as root, and complete the following steps to set up IP Security to allow access to only local users. Note that these steps assume that the first port on which service for the instance was configured to listen is 60000. If your instance uses a different port, then substitute that port number for any occurrence of 60000.

1. Run the following command to enable IP Security for IP V4:

2. Run the following command to allow local users to connect to port 60000:

3. Run the following command to deny remote access to port 60000:

4. Run the following command to activate the two new filter rules:

Performance implications of using TCP/IP connections

Initial tests using TCP/IP database connections have shown a 10-20% increase in overall CPU utilization. Throughput should not be affected, provided your system is not already close to using 100% of the CPU.

Users of 10GB network interfaces might encounter severe performance problems, as documented in AIX APAR IZ75548. This APAR reports throughput as low as 300 Kb/sec when the 10GB adapter's 'large receive offload' feature is enabled along with the AIX IP Security feature. A fix is available for IZ75548. However, if the fix has not been installed, then the recommend work-around for this issue is to do one of the following:

* turn off large_receive,
* turn off chksum_offload. or
* turn off AIX IP Security.

Instructions for reverting to local connections

If you encounter problems using TCP/IP connections, complete the following steps to revert to local connections . You might also choose to revert to local connections if you reduce your server's workload to the point that you are no longer in danger of exceeding 1024 simultaneous database connections, and you want to reclaim the extra CPU utilization associated with using TCP/IP connections.

1. Shut down the Tivoli Storage Manager server,

2. Run the following command to ensure that the DB2 server is shut down:

3. Run the following command:

Take note of the 'Local database directory' value associated with the TSMDB1_L database. This is typically set to the instance home directory, but might have a different value.

4. Run the following command:

5. Run the following command:

6. Run the following command:

The last parameter (the directory name) should match the directory from the "db2 list database directory" command in Step 3.

7. Run the following commands:

8. Run "db2start" to start the database manager.

9. Run the following commands to verify that you can connect to the database:

If the connection is successful, Tivoli Storage Manager should start up normally.

After reverting to local connections, you may also delete any AIX IP Security filters that were put in place when setting up for TCP/IP connections. Log in as root, and complete the following steps to remove IP Security filters, or to disable IP Security entirely:

1. Run the following command to obtain a list of the current filters:

2. Each filter rule has a unique identifying number. For each rule that was defined when setting up for TCP/IP connections, run the following command to remove the rule, making sure to specify the rule's identifying number with the -n parameter.

3. Run the following command to activate IP Security with the new set of rules

or run the following command to deactivate IP Security filtering entirely:

4. (Optional) Run the following command to deactivate IP Security. Do not do this if you need IP Security to filter on any other ports.

Maintenance activities that affect TCP/IP connections

Maintenance activities that uncatalog and recatalog, or delete and recreate the server database will result in a database that is configured to use local connections. Examples of such activities include:

* Deleting the database instance using the db2idrop command. This will implicitly uncatalog the database, requiring that it be recataloged once the instance is recreated.
* Deleting the database using the DB2 DROP DATABASE command before restoring the database from a backup copy.
* Upgrading the server from version 6.1 to either 6.2.0 or 6.2.1. As part of the upgrade, existing database instances are dropped and recreated, and the server databases are recataloged. Version 6.2.2 correctly reconfigures the database to use TCP/IP connections; however, earlier 6.2 fix packs do not.

After performing one of these maintenance tasks, you can use the "db2 list database directory" command to determine whether the database is still configured to use TCP/IP connections.

* If the command lists a single database entry that has the same value specified for both "Database alias" and "Database name" then local connections are being used. Repeat the steps used to configure the server database to use TCP/IP connections.

* If the command lists two database entries that refer to one another, then the database is still configured to use TCP/IP connections and no further action is required.

IC56146: FSCSI_ERR10 LOGGED IN ERRPT LOG WHEN LIBRARY DEFINED WITH RESETDRIVES SET TO YES AND AIX DYNAMIC TRACKING IS USED.

Error description

When a Fiber attached library is defined to TSM with
resetdrives=yes, FSCSI_ERR10 errors are logged in the AIX errpt
log if the FC SCSI Protocol Device (fscsiX) of the HBA is
configured with AIX dynamic tracking.

The dynamic tracking is enabled by setting the "Dynamic Tracking
of FC Devices" option to "true" for the FC SCSI Protocol Device
(fscsiX) involved. It can also be enabled with the following
AIX command : chdev -l fscsiX -a dyntrk=yes

The FSCSI_ERR10 errors are logged because TSM issues ioctls
calls with SCSI_VERSION_0. AIX dynamic tracking requires
SCSI_VERSION_1 ioctl calls.

An example of a FSCSI_ERR10 error in the AIX errpt log follows :

LABEL: FSCSI_ERR10
IDENTIFIER: 1FB4FD21
Date/Time: Wed Mar 26 16:02:16 EST
Sequence Number: 50402
Machine Id: xxxxxxxxxxxx
Node Id: myaixbox
Class: S
Type: INFO
Resource Name: fscsi0
Description
CONFIGURATION MISMATCH
Probable Causes
APPLICATION PROGRAM
SOFTWARE DEVICE DRIVER
ADAPTER MICROCODE
STORAGE CONFIGURATION
Failure Causes
APPLICATION PROGRAM
SOFTWARE DEVICE DRIVER
ADAPTER MICROCODE
INCORRECT HARDWARE CONFIGURATION.
Recommended Actions
IDENTIFY OFFENDING SOFTWARE COMPONENT
VERIFY SYSTEM CONFIGURATION IS VALID
CORRECT CONFIGURATION
REFER TO PRODUCT DOCUMENTATION FOR ADDITIONAL INFORMATION
Detail Data
SENSE DATA
0000 0000 0000 00D3 0000 0016 0200 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0011 1C00 0000 0000
0000 0000 0300 0000 0400 0003 0000 0000 000B 5048 0000 0000 0011 0B00 0001 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000 0000
0864 4000
Duplicates
Number of duplicates
5
Time of first duplicate
Wed Mar 26 16:02:16 EST
Time of last duplicate
Wed Mar 26 16:02:16 EST

The FSCSI_ERR10 errors may be ignored.

TSM Versions Affected:
TSM 5.3, 5.4 and 5.5 servers on AIX only.

Additional Keywords:
Dyntrk dynamictracking

Local fix

To avoid FSCSI_ERR10 errors from being logged, either
turn off dynamic tracking on the fscsiX device, i.e,
chdev -l fscsiX -a dyntrk=no
--or--
update tsm library with resetdrives=no, i.e,
update library resetdrives=no

Problem summary

****************************************************************
* USERS AFFECTED: TSM customers using the TSM AIX server
* on AIX 5.2 and up systems
****************************************************************
* PROBLEM DESCRIPTION: See ERROR DESCRIPTION
****************************************************************
* RECOMMENDATION: Apply fixing level when available. This
* problem is currently projected to be fixed
*
* in level 5.5.1.
* Note that this is subject to change at the
*
* discretion of IBM.
****************************************************************

Problem conclusion

The problem has been fixed in the TSM server.