unixadmin.free.fr Handy Unix Plumbing Tips and Tricks

Potential problems when the Tivoli Storage Manager server processes files greater than the MAXFRAGMENTSIZE setting

Abstract

Various problems can occur while processing files larger than the MAXFRAGMENTSIZE option with a Tivoli Storage Manager V7.1.0 server. The default value for this option is 10G (10000M). The affected operations include Client Node Restore or Retrieve, Node Replication, Export, LAN-free Restore and Storage Pool Backup.

Content

Bypass the server and storage agent prerequisites during installation

Question

How do you bypass the Tivoli Storage Manager server and storage agent prerequisites during installation?

Answer

DSMSERV RESTORE DB FAILED

Environment:
TSM 6.3 - Windows 2008

1) make a copie of volhist.dat and check it content search impacted BACKUPFULL

2) creates a hex dump of TSM DB BACKUP and check value of Backup Series :

@ 0000020 => (0000 7f00) = 0x7f = 127

The Backup Series in DB BACKUP is different than Volhist file.

Workaround:
Edit volhist file and replace Backup Series value with value checked in hex dump of TSM DB BACKUP.
DSMSERV RESTORE DB work fine.

Check TSM FIXPACK

Migrating TSM Node Data from one Library to Another

Problem
Procedure to migrate any TSM data that resides in Storage Pools from one library to another library. This procedure can be used to move data from any devclass to any other devclass.

Cause
Data needs to be migrated to alternate library hardware.

Resolving the problem
1. Physically attach the new library to the server and define it in the TSM server.

2. Once it is defined to the O/S and TSM, create a new stgpool (NEWPOOL) using the new library. See the TSM Admin guide / quick start guide for procedure.
DEFINE STG NEWPOOL

3. disable sessions & events so that backups will not go to old stgpool.
DISABLE SESS
DISABLE EVENTS

It depends on how much data you have and what daily processing is going on at the time, but it is recommended to mark the OLDPOOL volumes 'read only' at this point.

4. Change the copygroup DESTination setting from your OLDPOOL to point to NEWPOOL. If your backups go to a Disk stgpool and then migrate to OLDPOOL, update the disk stgpool to point to NEWPOOL as well, without changing copygroup setting.

If your copygroups go directly to tape, then change those now as well. Validate and Activate the policy sets after making the changes to the copy destinations.

5. Enable sessions & events.

Now new backups will go to new library. Next we will migrate the data off of the old library..

6. Update your old stgpool NEXTstgpool setting to go to the new stgpool.
UPDATE STG OLDPOOL NEXT=NEWPOOL

7. Create the new copy storage pool, if you use them.
DEFINE STG NEWCOPY

8. Do this next step in increments as a large amount of data will be transferred during each step. During the migration you probably want to continue with every day production and you do not want to fill up your database. Migration can be controlled through your Hi / Lo settings, for example:
UPDATE STG OLDPOOL HI=80 LOW=70
UPDATE STG OLDPOOL HI=70 LOW=60 (next day, and so on.)

Look at Q STG OLDPOOL to verify your setting so migration can occur if you are having trouble at this point.

After each migration increment, Bring the new copy pool up to date (if used.)
BACKUP STG NEWPOOL NEWCOPY

This will run a backup from the NEWPOOL to the NEWCOPY pool. When this step is complete, do the next incremental migration and another backup stgpool and so on until the entire storage pool is migrated.

A way to move the data over to the new storage pool with more control is to use the MOVE DATA command. See the ADMIN REFERENCE for the syntax and options for this command.

Notes: Regardless of the method used, the only way to move copypool data to the new devclass is by backing it up again with 'backup stg' specifying a copy stgpool in the new devclass. The data in the copy stgpool in the old devclass will need to be removed by 'delete volume ##### discarddata=yes'. It is recommended to do this after the data has been copied to the new copy stgpool.

9. Once the BACKUP STG is done, get rid of the old pool
for each volume in "Q VOL STG=OLDCOPY" do
DELETE VOL

10. When done, delete the old copy stg pool.
DELETE STG OLDCOPY

Replacing a damaged primary storage pool volume

Problem

My primary storage pool volume is physically damaged and cannot be reused. Steps for recovering the data on the volume.

Resolving the problem

NOTE: The key factor is that a copy storage pool must exist for the primary storage pool, and the data on the volume(s) has been copied with the 'backup stgpool' command. If the data on the damaged volume(s) has not been copied to the copy storage pool, or if the data on the volume was damaged prior to copy, then it will not be recoverable. Any files that are unable to be restored would be eligible for backup from the client again.

1. The damaged volume must be marked as destroyed to prevent access:
UPDATE VOLUME ACCESS=DESTROYED

2. Check the volume out of the library before proceeding. When the restore commands are executed the destroyed volume is deleted from the TSM database, and cannot be removed within TSM if the volume has not been checked out. To checkout the volume:
CHECKOUT LIBVOLUME

3. This command will preview the restore and not move any physical data. This will produce a list of volumes needed for the restore:
RESTORE VOLUME PREVIEW=YES

4. Look in the Activity log for the list of volumes that must be returned from offsite storage and checked into the library.

5. Place the volumes from step #4 in the BULK I/O door and then check them into the library:
CHECKIN LIBVOLUME SEARCH=BULK CHECKLABEL=BARCODE STATUS=PRIVATE

If you do not have a bulk I/O door then place them in empty slots inside the library and change the SEARCH parameter:
CHECKIN LIBVOLUME SEARCH=YES CHECKLABEL=BARCODE STATUS=PRIVATE

6. The checked in volumes must be marked READONLY to prevent processes from using them. The important thing to remember is that all volumes listed in the preview need to be in a ACCESS=READONLY state so that the next command can run to completion:
UPDATE VOLUME ACCESS=READONLY WHERESTGPOOL=

7. The next command will start the volume restore process. The 'maximum number of processes' needs to be at least 2, one process to read from the copy pool volume, and one process to write the new primary storage pool volume. Ensure that sufficient scratch volumes are available.
RESTORE VOLUME MAXPROCESS=

8. Once the restore has been completed, the copy volumes will need to be sent back offsite. The first step is to change there access back to offsite. By updating all volumes from step #5 to have an ACCESS=OFFSITE, the server will not try to use them during reclamation.
UPDATE VOLUME * ACCESS=OFFSITE WHERESTGPOOL=

9. Check out the copy pool volumes so that they can be delivered back to the vault location:
CHECKOUT LIBVOLUME CHECKLABEL=NO

10. This command will need to be run on the damaged primary volume only if the TSM Server reports that files are still located on the volume. This could be because of previous problems with the backup storage pool command or files on that volume had not been copied to the copy storage pool as of yet. These files, as long as they are still present on the client, will be backed up again from the owning client during their next incremental:
DELETE VOLUME DISCARD=YES

Redefine Library and Drives on Windows TSM server

Problem
Drives not working after system, library or cabling reconfiguration.
Cause
There are times, especially when a system has been rebuilt, that it is necessary to remove & redefine the library & the drive(s). Sometimes there are specific errors associated with this as well (see Problem Abstract for example).
Resolving the problem

Common errors that may be encountered after a library or recabling configuration may be:
ANR8300E I/O error on library (OP=xx CC=xx KEY=xx ASC=xx ASCQ=xx SENSE=xx)
ANR8441E Initialization failed for SCSI library
ANR8301E I/O Error on library

NOTE: These steps apply to libraries of LIBTYPE=SCSI or (with the library path definition step removed) LIBTYPE=MANUAL. For libraries of LIBTYPE=ACSLS or LIBTYPE=EXTERNAL, refer to the TSM Server Administrator's Guide and the Storagetek (in the case of ACSLS) or External Media Management software vendor (in the case of EXTERNAL, e.g. Gresham) documentation for configuration information. This technote does not account for library client servers or storage agents in a library sharing environment. Additional steps will be needed to redefine drive paths for the library client servers and/or storage agents to the same drive definitions for which the library manager server's drive paths point to the same physical device.

This is specific to Microsoft Windows Operating Systems. These steps should only be used, after attempts to update the drive and paths using autodetect features have been exhausted.

1. Run the 'TSMDLST' command to produce a list of the configured OS devices. This command is located in the following directory of a standard install and should be run from a MS-DOS prompt:
C:\program files\tivoli\tsm\console\

This information is usually (although not always) also viewable from the TSM Management Console's "Device Information" window under TSM Device Driver for the machine on which the TSM server is running.

2. Write down the 'TSM Name' for the library and drives:
Library=lb#.#.#.#
Drives=mt#.#.#.#

3. Gather the output of the following for a reference:
QUERY LIBRARY
QUERY DRIVE F=D
QUERY PATH F=D

4. First the drive and library paths must be deleted.

Do this for all drives:
DELETE PATH SRCTYPE=SERVER DESTTYPE=DRIVE
LIBRARY=

Do the following for the library:
DELETE PATH SRCTYPE=SERVER DESTTYPE=LIBRARY

5. Now the device reference must be deleted.
Do the following for all drives:
DELETE DRIVE

Then for the library:
DELETE LIBRARY

6. Now the devices can be redefined to TSM. Use the query outputs from step 3 as a guide for the device names. First, redefine the library:
DEFINE LIBRARY LIBTYPE=
(NOTE: If this is a shared library, add SHARED=YES to the above definition)

7. Next, redefine the path to the library. Often, drive and drive path definitions cannot be created until the TSM server is able to communicate with the library in which the drives exist.

Redefine path to library using the library device value from step 2:
DEFINE PATH SRCTYPE=SERVER DESTTYPE=LIBRARY DEVICE=lb#.#.#.#

8. Redefine the drives:
DEFINE DRIVE

9. Redefine the paths to the drives. Use the information gathered from the 'tsmdlst' in step 2 for the device parameter. Use the query outputs from step 3 as a guide for the device names.

Redefine paths to all drives:
DEFINE PATH SRCTYPE=SERVER DESTTYPE=DRIVE LIBRARY= DEVICE=mt#.#.#.# AUTODETECT=YES

NOTE: For multivendor libraries (as in libraries whose drives are not from the same vendor, e.g. a Dell Powervault 136T with IBM LTO drives), the autodetect feature may not function in such a way to automatically detect the drive's element and serial numbers. In that event, refer to the device's worksheet (often found by going to www.ibm.com/support and searching on the make & model of the library itself) to obtain a list of valid drive element numbers. This information may also be available through the use of vendor-specific library querying utilities (the vendor equivalent of the IBMTape utility "ntutil.exe").

8. Check all of the Scratch volumes back in.
IMPORTANT - you MUST check in the Scratch volumes first or ALL volume will be marked as "Private."
CHECKIN LIBVOL SEARCH=YES CHECKLABEL=BARCODE STATUS=SCRATCH

9. Check in the Private volumes as well.
CHECKIN LIBVOL SEARCH=YES CHECKLABEL=BARCODE STATUS=PRIVATE

Redefining TSM Library and Drives for UNIX OS

Question
Frequently when hardware or firmware has changed it is necessary to remove the tape library and drive definitions from the (IBM Tivoli Storage Manager) TSM Server, then re-define them.
Cause
Sometimes there are specific errors, such as:
ANR0523W Transaction failed - error on output storage device
ANR8300E I/O error on library (OP=xx, CC=xx, KEY=xx, ASC=xx, ASCQ=xx, SENSE=xx)
ANR8301E I/O Error on library
ANR8355E I/O error reading label for volume NNNNNN on drive XXXXX
ANR8359E Media fault detected on volume NNNNNN in drive XXXXX
ANR8441E Initialization failed for SCSI library
ANR8779E Unable to open drive XXXXX, error number=ZZZ
ANR8944E Hardware or media error on drive
ANR8963E Unable to find path to match the serial number defined for drive

Frequently the TSM Server can automatically rediscover devices when using "SANDISCOVERY ON" or by using "UPDATE PATH" with "AUTODETECT=YES" to refresh the values.

However, there are times when that may not be successful. For example, if a tape drive, tape library, fibre/SCSI HBA, or SAN has experienced changes (such as hardware, firmware or device drivers) it may require rebuilding the TSM "special files" to re-establish connectivity to the library and drives. To rebuild the "special files," we must delete and re-define the hardware devices to the TSM Server (UPDATE does not rebuild).
Answer
Perform these tasks in this sequence to totally re-define the tape devices to TSM. These steps should be taken only if attempts to update the devices/paths using the autodetect features have failed:

1. Before deleting anything, gather the output from these commands, so you can use the same naming conventions when re-defining the tape devices:
QUERY STATUS (get SERVERNAME value for "")
QUERY DEVCLASS
QUERY LIBRARY FORMAT=DETAIL
QUERY DRIVE FORMAT=DETAIL
QUERY PATH FORMAT=DETAIL

2. Run the appropriate OS command to produce a list of the configured HW 'special file' device names.
AIX ==> lsdev -Cc tape (-or- 'cfgmgr')
lsdev -Cc adsmtape (for TSM devices)
lsdev -Cc library
Solaris ==> ls -l /dev/rmt/*st (-or- 'sysdef')
ls -l /dev/rmt/*smc
HP-UX ==> /usr/sbin/ioscan -funC tape
(-or 'ioscan -kfn')
Linux ==> ls -l /dev/IBM*
ls -l /dev/tsmscsi/*
(-or- 'more /etc/sysconfig/hwconf')

If the tape devices are not defined to the OS, please work with your OS or SCSI/SAN hardware support to configure them. Until the OS can use the drives (can write to them, for example using 'tar' or 'dd') the tape devices cannot be defined to TSM.

3. From the '/dev' directory, write down the OS-level device definitions for the library and drives:
AIX Linux Solaris HP-UX
TSM Drives mt# tsmscsi/mt# rmt/# rmt/tsmmt#
IBM Drives rmt# IBMtape# rmt/#st rmt/#m
TSM Library lb# tsmscsi/lb# rmt/#lb tsmchgr#
358x Library smc# IBMchanger# rmt/#smc rmt/#chng
3494 Library lmcp# 3494lib libmgrc# libmgrc#

4a. First the drives and drive paths must be deleted. From a TSM Server admin commandline, for all the drives:
DELETE PATH SRCTYPE=SERVER DESTTYPE=DRIVE LIBRARY=

4b. Then delete all the TSM drive definitions:
DELETE DRIVE

5a. Next, delete the path for the tape library:
DELETE PATH SRCTYPE=SERVER DESTTYPE=LIBR

5b. And finally delete the TSM library definition:
DELETE LIBRARY

If the OS cannot access the tape drives at this point, stop. Check hardware, device drivers, update firmware, swap cables; consider power-cycling the tape library then deleting and re-defining to the OS. There is no point attempting to get TSM to write to the devices if they are not recognized by the OS; work with OS and/or hardware vendors to resolve HW issues before proceeding.

6a. Now the tape library and library path can be re-defined. Use the TSM QUERY outputs from "Step 1" as a guide for the library name and LIBTYPE; no additional parameters are necessary in the syntax below. Redefine the library:
DEFINE LIBRARY LIBTYPE= SERIAL=AUTODETECT

Note: If this TSM Server is hosting a tape library for other systems, for example any "TSM Server Library Clients" or "TSM Storage Agents" then you also need "SHARED=YES" on the "DEFINE LIBRARY".

6b. Redefine the path to the library. For SCSI libraries, confirm the DEVICE value matches the latest OS-level info gathered from "Step 2". For 3494, ACSLS, and other types of libraries using software configuration files, use the previous values from "Step 1" to redefine the DEVICE or ACSID, and so on:
DEFINE PATH SRCTYPE=SERVER DESTTYPE=LIBRARY DEVICE=

7a. Redefine the drives and drive paths. Redefine all the drives using names from "Step 1" for example:
DEFINE DRIVE SERIAL=AUTODETECT ELEMENT=AUTODETECT

7b. Redefine paths to all drives, using the OS-level info gathered from "Step 2" for the DEVICE values. Keep in mind the OS-level DEVICE values may have changed since the they were previously defined.
DEFINE PATH SRCTYPE=SERVER DESTTYPE=DRIVE LIBRARY= DEVICE=

Note: If this TSM Server is hosting a tape library for other systems, for example any "TSM Server Library Clients" or "TSM Storage Agents" then in addition to the "TSM Server Library Manager" DRIVE PATH, you also need to define a new PATH for each drive for those systems, substituting the SERVERNAME (shown by "Q SERVER") for the value of "" and the local DEVICE value for the drive as seen by that other system.

8. Verify the library, drives, and paths are online:
QUERY LIBRARY FORMAT=DETAIL
QUERY DRIVE * FORMAT=DETAIL
QUERY PATH * * FORMAT=DETAIL

9. Since the library is "new" to TSM, the volumes must be checked in again to re-create the inventory (AUDIT LIBRARY does not CHECKIN). Use *this* sequence, first SCRATCH, then PRIVATE:
CHECKIN LIBVOL SEARCH=Y STATUS=SCR CHECKL=BARC
CHECKIN LIBVOL SEARCH=Y STATUS=PRIV CHECKL=BARC

NOTE: For ACSLS libraries, use "CHECKLABEL=NO" on the CHECKIN commands, because "CHECKLABEL=BARCODE" is not supported for an ACSLS Library.

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

If that doesn't resolve the issue, the tape drive problem seems beyond the control of the TSM Server (software). Review the output from the OS-level logs for additional hardware error information:
Remove OS Install
OS Diagnostics Devices OS Devices
------- ------------ --------- ------------
AIX errpt –a rmdev cfgmgr
Linux dmesg /dev/MAKEDEV
Solaris mbin/prtdiag rem_drv drvconfig
HP-UX dmesg rmsf insf -e

If you cannot reach HW support immediately, you could take the additional action of power-cycling in this order:
1. Tape library.
2. SAN switch (if any).
3. Consider updating to latest device drivers and/or firmware.
4. Halt TSM and reboot system with TSM Server.
5. Re-define the tape device to the OS (see commands above).
6. If tape device definitions have changed, DELETE & re-DEFINE to TSM.

And that is all we can do from a software perspective, if errors persist it points to an issue at a layer which TSM cannot repair.

Reclaiming volumes when out of scratch tapes

1. To start expiration issue the following command from a Tivoli Storage Manager command line:

EXPIre Inventory

2. Next, identify the RECLAMATION THRESHOLD is on your tape storage pool(s)

Query STGpool Format=Detail

3. If the threshold is set to 100 then adjust the reclamation threshold value down to a size that will cause reclamation to reclaim the volumes. *

UPDdate STGpool REClaim=

NOTE: Do not set the REClaim value lower that 50 as this will cause reclamation to run for an extended period of time.

* After expiration is finished you can see how many volumes would be reclaimed if the reclamation threshold were to be changed to a certain percent for your storage pool using the following command:

select count(*) from volumes where stgpool_name='XXXX' and upper(status)='FULL' and pct_utilized < ##

NOTE: Replace the XXXXX with the storage pool in UPPERCASE and the ## with the appropriate two digit numeric value.

Tivoli Storage Manager has the ability to run this at regular intervals during the day or at night. You will need to figure out what time would be the best for reclamation to run. After this you will need to check your system to see if it is set up to not run reclamation.

1. Check you DSMSERV.OPT file and make sure that the options NOMIGRRECL and EXPINTERVAL are not set. If they are set you will need to either remove them from the file or comment out the lines. Once this is done you will then need to stop and restart the Tivoli Storage Manager server so that it re-reads the file as these settings are retained in memory.

2. If the NOMIGRRECL and EXPINTERVAL were not in the file, then issue the following command

Query OPTions

Look for the ExpInterval in the first 'server option' column. The number to the right specifies the time, in hours, between automatic inventory expiration processes. You can specify from 0 to 336 (14 days). A value of 0 means that expiration must be manually started with the EXPIRE INVENTORY command. You can update this server option without stopping and restarting the server by using the SETOPT command.

SETOPT EXPINterval 24

In the above example, this command will cause Expiration to run every 24 hours.

3. If you then want to control when reclamation runs you can build some Administrative schedules to raise and lower the reclamation threshold.

DEFINE SCHEDULE ReclaimStart TYPE=ADMIN CMD="UPDATE STG REClaim= " ACTIVE=Yes DESC="Lower Reclamation Threshold" STARTT= DAY=ANY EXP=N

DEFINE SCHEDULE ReclaimStop TYPE=ADMIN CMD="UPDATE STG REClaim=100" ACTIVE=Yes DESC="Raise Reclamation Threshold" STARTT= DAY=ANY EXP=N

NOTE: In the second command set the time for 2 - 3 hours after the time you entered in the ReclaimStart command.

After accomplishing these three steps Tivoli Storage Manager will expire files and reclaim the volumes automatically everyday at the time you designated. Take into account that either of the two procedures stated above will need at least two scratch tapes available to move data off of the volumes it is reclaiming. If you do not have any scratch tapes available to initiate this in the beginning then you will need to perform the following steps.

1. Run the 'select' query from the first set of procedures after expiration is finished.

2. Manually move data from some of the Tape storage pool volumes to clear some space.

MOVe Data

This will move data from the volume to other volumes within the same storage pool.

3. Once this command is finished you then need to delete the volume from the storage pool with either one of the two following commands.

DEL V

DEL V DISCARD=Y

NOTE: Ensure that the move data command finished successfully (by reviewing the Tivoli Storage Manager server activity log) before running this command. If there is any data on the volume it will be deleted and lost from both the primary and copy storage pools.