unixadmin.free.fr just another IBM blog and technotes backup

16nov/16

emgr failure with noclobber ksh option

A customer never can apply a emergency fix with a lot of "file already exists" error.
The cause was noclobber ksh option was set in the ~/.kshrc of root user.

Checking space requirements ...
/usr/sbin/emgr[11177]: /tmp/emgrwork/10617068/.epkg.msg.buf.10617068: file already exists
/usr/ccs/lib/libc.a:
emgr: 0645-007 ATTENTION: /usr/sbin/fuser -xf returned an unexpected result.
/usr/sbin/emgr[11177]: /tmp/emgrwork/10617068/.global.warning.10617068: file already exists
emgr: 0645-007 ATTENTION: inc_global_warning() returned an unexpected result.
emgr: 0645-007 ATTENTION: isfopen() returned an unexpected result.
/usr/sbin/emgr[11177]: /tmp/emgrwork/10617068/.global.warning.10617068: file already exists
emgr: 0645-007 ATTENTION: inc_global_warning() returned an unexpected result.
/usr/sbin/emgr[11177]: /tmp/emgrwork/10617068/.epkg.msg.buf.10617068: file already exists
...
1 IV81303s1a INSTALL FAILURE

Workaround
check if noclobber ksh option is active.

set -o | grep noclobber

Remove noclobber option from ~/.kshrc or ~/.profile of root user and open a new terminal.

Disable noclobber:
set +o noclobber
Remplis sous: AIX Aucun commentaire
12oct/16Off

emgr secret flag for EFIXLOCKED

A customer have problem to update one AIX with EFIXLOCKED in some fileset. In normal situation fileset is locked by a emergency fix, and before apply update you must remove e-fix. With this customer no e-fix ... nothing to remove, and /usr/emgrdata was empty.

emgr command is a korn shell script that contains secret flag "-G" for unlock fileset with unlock_installp_pkg() function.

 G) # Secret flag. DO NOT USE THIS UNLESS YOU ARE SURE !
            emt || exit 1
            unlock_installp_pkg "$OPTARG" "0"; exit $?;; # secret flag

########################################################################
## Function: unlock_installp_pkg
## Parameters: <PKG> <EFIX NUM>
########################################################################

if no e-fix was installed, then use anything for second argument $2 like "unlock"

#  lslpp -qlc | grep EFIXLOCKED
/etc/objrepos:bos.rte.libc:7.1.4.0::COMMITTED:F:libc Library:EFIXLOCKED
/usr/lib/objrepos:bos.rte.libc:7.1.4.0::COMMITTED:F:libc Library:EFIXLOCKED

# emgr -G bos.rte.libc unlock    
Initializing emgr manual maintenance tools.
Explicit Lock: unlocking installp fileset bos.rte.libc.

Also -D flag display Debug mode with set -x.

20sept/16Off

AIX SCSI-2 Reservation on INFINIDAT

A customer encountered a problem with Disaster Recovery plan for AIX rootvg boot on SAN (reserve_policy=single_path) on Infinidat model F6130.

Problem: No boot disk on SMS menu.

Workaround: From the Infinidat Storage Unmap & Map rootvg LUN to the host.

Fix: Infinidat corrected Bug in Firmware 2.2.10.12 and add also a internal Script for SCSI-2 reservation.

Before the Fix Infinidat Storage managed only SCSI-3 reservation and AIX use SCSI-2 reservation.

Remplis sous: AIX Commentaires
12sept/16

Oracle RAC 11gR2 need multicast

After AIX / Oracle RAC migration to new Datacenter, the DBA encountered a problem to start Oracle RAC with network heartbeat error...
Root cause: Network Team has dropped multicast beetween Datacenter.

cat $GRID_HOME/log/grac2/cssd/ocssd.log
2016-09-11 21:15:55.296: [    CSSD][382113536]clssnmPollingThread: node 2, orac002 (1) at 90% heartbeat fatal, removal in 2.950 seconds, seedhbimpd 1
2016-09-11 21:15:56.126: [    CSSD][388421376]clssnmvDHBValidateNCopy: node 2, orac002, has a disk HB, but no network HB, DHB has rcfg 269544449, wrtcnt, 547793, LATS 2298

Workaround: Run tcpdump and grep multicast MAC on Oracle Interconnect Interface and send multicast address for Network Team. That's work better after.

root@orac002:/root #  tcpdump -en -i en0 |  grep "01:00:5e"

21:43:24.678611 76:82:b2:99:ac:0b > 01:00:5e:00:00:fb, ethertype IPv4 (0x0800), length 1202: 192.168.10.217.42424 > 224.0.0.251.42424: UDP, length 1160
21:43:24.678798 76:82:b2:99:ac:0b > 01:00:5e:00:01:00, ethertype IPv4 (0x0800), length 1202: 192.168.10.217.42424 > 230.0.1.0.42424: UDP, length 1160

Oracle Doc ID 1212703.1
Bug 9974223 : GRID INFRASTRUCTURE NEEDS MULTICAST COMMUNICATION ON 230.0.1.0 ADDRESSES WORKING

Remplis sous: AIX, ORACLE Aucun commentaire
27oct/15

NMON Visualizer

Pour ceux qui utilise l'outil de collecte de performance nmon développé par Nigel Griffiths et disponible pour IBM AIX/VIOS et Linux (Power, x86, x86_64, Mainframe & now ARM (Raspberry Pi)), comme vous le savez il faut se servir du fichier excel nmon analyser pour visualiser les fichiers de collecte nmon.

En complément de cet outil je vous conseille de tester NMONVisualizer, un projet IBM démarré par Hunter Presnall qui est un excelent outil pour comparer et analyser les fichiers nmon issue des collectes de performances de plusieurs systèmes ou VM AIX / Linux.

NMONVisualizer http://nmonvisualizer.github.io/nmonvisualizer/index.html

NMON
https://www.ibm.com/developerworks/aix/library/au-analyze_aix/

https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/Power+Systems/page/nmon

http://nmon.sourceforge.net

Nmon analyser
https://www.ibm.com/developerworks/community/wikis/home?lang=en#!/wiki/Power%20Systems/page/nmon_analyser

My thanks to you all for a job very well done. :)

13oct/15

How to add a hardware error in AIX errlog

Comment générer des erreurs hardware dans l'errlog d'AIX. Utile pour tester des logiciels de supervision ou la gestion des évènements sous PowerHA.
Voir le fichier /usr/include/sys/errids.h pour les LABELS errpt.

echo "EPOW_SUS\nEMULATE\n1\ntexte1\ntexte2" | /usr/lib/ras/ras_logger
echo "SCSI_ERR1\nEMULATE\n1\ntexte1\ntexte2" | /usr/lib/ras/ras_logger
# errpt
IDENTIFIER TIMESTAMP  T C RESOURCE_NAME  DESCRIPTION
0502F666   1013162315 P H EMULATE        ADAPTER ERROR
74533D1A   1013162115 U H EMULATE        LOSS OF ELECTRICAL POWER
Remplis sous: AIX, HACMP Aucun commentaire
25sept/15

AIX Crash always on d_map_list_tce()

Problem

AIX 6100-04-05-1015 Crash always on d_map_list_tce() with with EMC Symmetrix FCP VRAID storage disk.

EMC.Symmetrix.aix.rte:5.3.0.5
EMC.Symmetrix.fcp.rte:5.3.0.5
EMCpower.base:5.3.1.1
EMCpower.encryption:5.3.1.1
EMCpower.migration_enabler:5.3.1.1
EMCpower.mpx:5.3.1.1

(3)> stack
pvthread+001B00 STACK:
[041C85A0]d_map_list_tce+000A40 (0000000000000000, 0000000000000000,F1000A0380145AF0, 0000000000001020, 0000000000000001 [??])
[04255138]04255138 ()
[0425A668]efc_start+000508 (??)
[04226D58]efc_intr+0001B8 (??)
[0024280C]i_poll_soft+00012C (??)
[00242180]i_softmod+000480 ()
[0013FB44]flih_util+000250 ()
 
(3)> dr iar
iar   : 00000000041C85A0
041C85A0      tweqi    r3,0                r3=0
 
(3)> lke 041C85A0
    ADDRESS          FILE             FILESIZE FLAGS    MODULE NAME
 
  1 F1000000A05F2500 041B0000 00030000 00080252 /usr/lib/drivers/pci/pci_busdd
 
(3)> symptom
Instruction:
PIDS/5765G6200 LVLS/610 PCSS/SPI1 MS/700 FLDS/d_map_lis VALU/e8610110 FLDS/04255138 VALU/342e6b64

Local Fix
Stop the replication between storage and DR site

IBM APAR
Not yet tested with replication

Remplis sous: AIX Aucun commentaire
25août/15

Minimum NIM master levels for VIOS clients

The NIM master level for VIOS is also for me a good point of vue to know the AIX level vs VIOS ioslevel.

https://www-304.ibm.com/webapp/set2/sas/f/flrt/viostable.html

Minimum NIM master levels for VIOS clients

If using NIM to backup, install or update a VIOS partition, the NIM master must be greater than or equal to the levels shown below.

VIOS Release VIOS Level Minimum NIM master level
VIOS 2.2.3 VIOS 2.2.3.50 AIX 6100-09-05 7100-03-05
VIOS 2.2.3.4 AIX 6100-09-04 7100-03-04
VIOS 2.2.3.3 AIX 6100-09-03 7100-03-03
VIOS 2.2.3.2 AIX 6100-09-02 7100-03-02
VIOS 2.2.3.1 AIX 6100-09-01 7100-03-01
VIOS 2.2.3.0 AIX 6100-09 7100-03
VIOS 2.2.2 VIOS 2.2.2.6 AIX 6100-08-06 7100-02-06
VIOS 2.2.2.5 AIX 6100-08-05 7100-02-05
VIOS 2.2.2.4 AIX 6100-08-04 7100-02-04
VIOS 2.2.2.3 AIX 6100-08-03 7100-02-03
VIOS 2.2.2.2 AIX 6100-08-02 7100-02-02
VIOS 2.2.2.1 AIX 6100-08-01 7100-02-01
VIOS 2.2.2.0 AIX 6100-08 7100-02
VIOS 2.2.1 VIOS 2.2.1.9 AIX 6100-07-10 7100-01-10
VIOS 2.2.1.8 AIX 6100-07-09 7100-01-09
VIOS 2.2.1.7 AIX 6100-07-08 7100-01-07
VIOS 2.2.1.5 AIX 6100-07-05 7100-01-05
VIOS 2.2.1.4 AIX 6100-07-04 7100-01-04
VIOS 2.2.1.3 AIX 6100-07-02 7100-01-02
VIOS 2.2.1.1 AIX 6100-07-01 7100-01-01
VIOS 2.2.1.0 AIX 6100-07 7100-01
VIOS 2.2.0 VIOS 2.2.0.13 AIX 6100-06-05 7100-00-03
VIOS 2.2.0.12 AIX 6100-06-05 7100-00-03
VIOS 2.2.0.11 AIX 6100-06-03 7100-00-02
VIOS 2.2.0.10 AIX 6100-06-01 7100-00-01
VIOS 2.2.0.0 AIX 6100-06 7100-00
VIOS 2.1.3 VIOS 2.1.3.10 AIX 6100-05-02
VIOS 2.1.3.0 AIX 6100-05
VIOS 2.1.2 VIOS 2.1.2.13 AIX 6100-04-03
VIOS 2.1.2.12 AIX 6100-04-02
VIOS 2.1.2.11 AIX 6100-04-02
VIOS 2.1.2.10 AIX 6100-04-01
VIOS 2.1.2.0 AIX 6100-04
VIOS 2.1.1 VIOS 2.1.1.10 AIX 6100-03-01
VIOS 2.1.1.0 AIX 6100-03
VIOS 2.1.0 VIOS 2.1.0.10 AIX 6100-02-02
VIOS 2.1.0.1 AIX 6100-02-01
VIOS 2.1.0.0 AIX 6100-02
VIOS 1.5.2 VIOS 1.5.2.6 AIX 5300-08-08
VIOS 1.5.2.5 AIX 5300-08-05
VIOS 1.5.2.1 AIX 5300-08-01
VIOS 1.5.2.0 AIX 5300-08
17juil/15

Determining rlimit (ulimit) values for a running process

Question

How can I find out what limits are set for a currently running process?

Answer

The easiest way is to download the pdump.sh script and run it against the process. The pdump tool can be downloaded from here:
ftp://ftp.software.ibm.com/aix/tools/debug/

There is no installation needed, only to change the permissions of the file so it can be executed:
$ chmod +x pdump.sh

Then run it against the process-id (PID) of the process you wish to examine. The pdump.sh script will create an output file containing information regarding that process.

# ./pdump.sh 3408030
The output file will contain the name of the process, the PID and the current date. For example:

pdump.tier1slp.3408030.13May2015-11.18.13.out

This is an ASCII text file and can be inspected with "more" or "view".

Determining the limit values
Limits in a process are kept in the user area or "uarea" of the process memory. This section in the pdump output starts with the title "Resource limits:"

Resource limits:
fsblimit....00000000001FFFFF

rlimit[CPU]........... cur 7FFFFFFF max 7FFFFFFF
saved_rlimit[CPU]..... cur 7FFFFFFFFFFFFFFF max 7FFFFFFFFFFFFFFF
rlimit_flag[CPU]...... cur INF max INF

rlimit[FSIZE]......... cur 3FFFFE00 max 3FFFFE00
saved_rlimit[FSIZE]... cur 000000003FFFFE00 max 000000003FFFFE00
rlimit_flag[FSIZE].... cur SML max SML

rlimit[DATA].......... cur 08000000 max 7FFFFFFF
saved_rlimit[DATA].... cur 0000000008000000 max 7FFFFFFFFFFFFFFF
rlimit_flag[DATA]..... cur SML max INF

rlimit[STACK]......... cur 02000000 max 7FFFFFFF
saved_rlimit[STACK]... cur 0000000002000000 max 0000000100000000
rlimit_flag[STACK].... cur SML max MAX

rlimit[CORE].......... cur 3FFFFE00 max 7FFFFFFF
saved_rlimit[CORE].... cur 000000003FFFFE00 max 7FFFFFFFFFFFFFFF
rlimit_flag[CORE]..... cur SML max INF

rlimit[RSS]........... cur 02000000 max 7FFFFFFF
saved_rlimit[RSS]..... cur 0000000002000000 max 7FFFFFFFFFFFFFFF
rlimit_flag[RSS]...... cur SML max INF

rlimit[AS]............ cur 7FFFFFFF max 7FFFFFFF
saved_rlimit[AS]...... cur 0000000000000000 max 0000000000000000
rlimit_flag[AS]....... cur INF max INF

rlimit[NOFILE]........ cur 000007D0 max 7FFFFFFF
saved_rlimit[NOFILE].. cur 00000000000007D0 max 7FFFFFFFFFFFFFFF
rlimit_flag[NOFILE]... cur SML max INF

rlimit[THREADS]....... cur 7FFFFFFF max 7FFFFFFF
saved_rlimit[THREADS]. cur 0000000000000000 max 0000000000000000
rlimit_flag[THREADS].. cur INF max INF

rlimit[NPROC]......... cur 7FFFFFFF max 7FFFFFFF
saved_rlimit[NPROC]... cur 0000000000000000 max 0000000000000000
rlimit_flag[NPROC].... cur INF max INF

The resource limit for each ulimit value is represented here. As values could be either 32-bit or 64-bit, the include file /usr/include/sys/user.h tells us how to read them:

/*
* To maximize compatibility with old kernel code, a 32-bit
* representation of each resource limit is maintained in U_rlimit.
* Should the limit require a 64-bit representation, the U_rlimit
* value is set to RLIM_INFINITY, with actual 64-bit limit being
* stored in U_saved_rlimit. These flags indicate what
* the real situation is:
*
* RLFLAG_SML => limit correctly represented in 32-bit U_rlimit
* RLFLAG_INF => limit is infinite
* RLFLAG_MAX => limit is in 64_bit U_saved_rlimit.rlim_max
* RLFLAG_CUR => limit is in 64_bit U_saved_rlimit.rlim_cur

So using this and our pdump output, we can view the value of NOFILE for example:

rlimit[NOFILE]........ cur 000007D0 max 7FFFFFFF
saved_rlimit[NOFILE].. cur 00000000000007D0 max 7FFFFFFFFFFFFFFF
rlimit_flag[NOFILE]... cur SML max INF

The rlimit_flag for NOFILE is set to SML, so the value is a 32-bit integer, and is stored in the rlimit.cur variable.

0x7d0 = 2000 decimal, so the limit for that user, picked up by the process when it started, is 2000.

Source : IBM Technote

Remplis sous: AIX Aucun commentaire
10mai/15

Common EFS Errors and Solutions

Question

This document is a collection of errors encountered when using EFS and solutions to those issues.

Answer

1) Problem: Can't enable EFS on the system
# efsenable -a
/usr/lib/drivers/crypto/clickext: A file or directory in the path name does not exist.
Unable to load CLiC kernel extension. Please check your installation.

Solution:
Install CLiC filesets from AIX Expansion Pack CD

$ installp -l -d clic.rte
Fileset Name                Level                     I/U Q Content
====================================================================
clic.rte.includes           4.3.0.0                    I  N usr
#   CryptoLite for C Library Include File

clic.rte.kernext            4.3.0.0                    I  N usr,root
#   CryptoLite for C Kernel

clic.rte.lib                4.3.0.0                    I  N usr
#   CryptoLite for C Library

2) Problem: Can't enable EFS on the system

# efsenable -a
Unable to load CLiC kernel extension. Please check your installation.
(Please make sure latest version of clic.rte is installed.)

Double-check that you have installed the correct version of the CLIC filesets for your Technology Level of AIX.

For AIX 6100-01 use clic.rte.4.3.0.0.I on the Expansion Pack CD
For aix 6100-02 use clic.rte.4.5.0.0.I on the Expansion Pack CD

AIX 6100-03 has been updated to include clic.rte on the base media set to prevent boot issues on systems with EFS enabled. Use clic.rte.4.6.0.1.I

For AIX 6100-04 use clic.rte.4.7.0.0.I which is also included in the base OS media.

2) Problem: Can't view user's key:

$ efskeymgr -v
Problem initializing EFS framework.
Please check EFS is installed and enabled (see efsenable) on you system.
Error was: (EFS was not configured)

Solution:
Enable EFS on the system:
# efsenable -a
and give root's password when it asks for root's initial keystore.

3) Problem: Can't enable encryption inheritiance on a directory.
# efsmgr -E testdir
or
Can't enable encryption on a specific file
# efsmgr -e myfile

Problem initializing EFS framework.
Please check EFS is installed and enabled on you system.
Error was: (EFS was not configured)

Solution:
Make sure CLiC filesets are installed
Enable EFS on the system
Enable EFS and RBAC on the filesystem:

# chfs -a efs=yes /myfilesystem

4) Problem: Have enabled EFS on a filesystem but get error mounting:

# mount /efstest
The CLiC library (libclic.a) is not available. Install clic.rte and run 'efsenable -a'.

Solution:
Install CLiC filesets
Enable EFS on the system
Remount the filesystem

5) Problem: No encryption algorithms show up!
# efsenable -q
List of supported algorithms for keystores:
1
2
3

List of supported ciphers for files:
1
2
3
4
5
6

Solution:
Install CLiC filesets

# efsenable -q
List of supported algorithms for keystores:
1  RSA_1024
2  RSA_2048
3  RSA_4096

List of supported ciphers for files:
1  AES_128_CBC
2  AES_192_CBC
3  AES_256_CBC
4  AES_128_ECB
5  AES_192_ECB
6  AES_256_ECB

Source: IBM Technote

Remplis sous: AIX Aucun commentaire