unixadmin.free.fr IBM AIX Blog

PowerHA Enterprise Edition Support Cross Reference

Source: IBM Technote

The following shows the supported combinations available within PowerHA Enterprise Edition. The NPIV support was confirmed with PowerHA development, though no separate support flashes state it.

POWERHA Enterprise Edition Support Cross Reference

Addition EMC/SRDF software requirements:

• ·JAVA
  ·IBM’s Java 1.4 preferred
  ·EMC SYMCLI
  ·SYMCLI.SYMCLI.rte 7.0.0.0

The SYMCLI command line interfaces must be installed in the default location, /usr/symcli/bin

• ·EMC Storage Subsystem
  ·DMX-3 with microcode 5772.103.93 or higher
  ·DMX-4 with microcode 5773.134.94 or higher
  ·V-MAX with microcode 5874.188.156 or higher
  ·PowerPath
  ·5.3.0.0
  ·cluster.es.sr.rte
  ·cluster.es.sr.cmds
  ·cluster.msg.en_US.sr

Additional DS8700 Global Mirror software requirements

• • • ·cluster.es.genxd
      ·cluster.msg.en_US.genxd

  ·DS8700 Firmware Software Bundle V75.1.145.0 or later
  ·DSCLI version 6.5.1.203 or later
  ·JAVA
  ·Additional filesets from 6.1 SP3 specifically above what comes on 6.1 EE base media

Additional Hitachi software requirements

• • • ·cluster.es.tc
      ·cluster.msg.en_US.tc

  ·USPV Microcode 60-06-05/00, or later
  ·CCI Version 01-23-03/06 or later
  ·Additional filesets from 6.1 SP3 specifically above what comes on 6.1 EE base media

Additional HP software requirements
The following HP software and microcode were used as part of the testing:
1. HP StorageWorks XP Continuous Access Software (no version needed)
2. HP RAID Manager: 1.25.11
3. XP 24000 Microcode: 60-08-08-00/00
4. XP 12000 Microcode: 50-09-98-00/08
5. MPIO: MPIO ODM (5403) devices.fcp.disk.HP.xparray.mpio.rte 5.4.0.3

Server running AIX with Oracle RAC reboots itself

SOURCE: Technote T1011228

Problem(Abstract)

Server running AIX with Oracle RAC reboots itself with no warning
Symptom

AIX server shuts down and/or reboots.

A REBOOT_ID is logged in /var/adm/ras/errlog indicating "SYSTEM SHUTDOWN BY USER" although no shutdown or reboot command was issued by any user.

example error message...

Cause

Oracle Real Application Clusters (RAC) is known to reboot the operating system with no warning due to configuration of the oprocd daemon

Environment

AIX with Oracle RAC

Diagnosing the problem

Oracle Real Application Clusters (RAC) typically runs a process called oprocd.

The idea of OPROCD is quite straightforward. It’s goal is to provide I/O fencing. Basically oprocd works by setting a timer, then sleeping. If, when it wakes up again and gets scheduled onto cpu, it sees that a longer time has passed than the acceptable margin, oprocd will decide to reboot the node.

You can check for the oprocd process with the ps command...

These options to oprocd are saying -t 1000 (wake up every 1000 ms) and -m 500 (allow up to 500 ms margin of error on the time that oprocd wakes up before rebooting). In other words, if oprocd wakes up after > 1.5 secs it’s going to force a reboot.

Resolving the problem

The timeout and margin times are computed from the elements of diagwait and reboot time and it isn't recommended changing them via the init.cssd file, but rather through the command 'crsctl set css diagwait '.

There is a formula involved in the calculation of the times. For example, if the reboot time is 3 and you submit a diagwait setting of 13 you will get -t 1000 -m 10000.

You can see that the margin has changed to 10000 ms, that is 10 seconds in place of the default 0.5 seconds. This is a 20 fold increase allows oprocd more time to determine if the node needs to be rebooted.

IBM recommends the customer contact Oracle Support before modifying this value.

IBM and Oracle came to the agreement that a diagwait value of 13 is a suitable value if the best practices are used...

IBM recommends customers follow best practices, and if possible update to AIX 6.1 or AIX 7.1 with current Technology Levels which include the new non-pagable kernel as the preferred corrective action.

The Oracle master document can be found here...

Power Systems Memory Deduplication

Attention ! peinture fraîche
Après AMS et AME, IBM nous pond AMD comme Active Memory Deduplication, un complément de Active Memory Sharing.

Active Memory Deduplication is a virtualization technology that allows memory pages with identical contents to be deduplicated in physical memory. This is designed to free up physical memory positions so that more data can be held in memory at once.

Memory deduplication is intended to work in a shared memory environment. Therefore, it works together with Active Memory Sharing which is a technology to allow multiple partitions on a system to share a pool of physical memory, sometimes creating an over commitment of this physical memory. Active Memory Deduplication increases the performance of Active Memory Sharing since the savings can be used to either lower memory over commitment levels or to create room to increase logical partitions’ memory footprint.

Power Systems Memory Deduplication

How to convert AIX time_last_login

lastupdate.sh

How to Capture SAN Boot Debug for Virtual I/O Server and AIX on P6 Systems

Problem(Abstract)
How to capture boot debug of a SAN boot PowerVM Virtual I/O Server or AIX/NPIV client partition that is failing to boot.

Symptom
NPIV/AIX Client or VIOS fails to boot from SAN.

Environment
REQUIREMENTS

1. POWER6 System

2. A program where console terminal logging can be enabled will be needed. The following procedure uses PuTTY (a Windows ssh client program) as the means to open a console session to capture the boot debug data to a file. It's available for download at http://www.putty.org

Diagnosing the problem
Things to check PRIOR to gathering the debug

For a NEW NPIV/AIX Client Install

1. Ensure NPIV mapping is correct
2. Ensure SAN swtich is zoned correctly to the NPIV client's WWPN
3. Ensure resources (LUN) is assigned from the storage directly to the client's WWPN
4. Ensure Installation media meet minimum level required by the storage

For previously running LPAR

1. Check if boot device can be set in SMS
2. Check if rootvg is accessible in Service Mode.

Resolving the problem

1. To capture a boot debug to a file, open an ssh session via PuTTY to the HMC as follows

Under Category
Session
-> click on Logging
-> select "All session output" on the right
-> specify the filename in the "Log file name" box as shown in Figure 1

Terminal
-> click on Keyboard
-> select Control-H for the Backspace key

Click on Session
-> Type in the full domain to the HMC in the Host Name and Saved Sessions box
-> select SSH protocol (HMC must be configured to accept ssh connections)
-> Click on Open (See Figure 2). You will get a PuTTY Security Alert Window
-> Click No to connect just once

-> Login as hscroot and type 'vtmenu' to open a console session to the partition in question
-> Select the Managed System name
-> Select the partition in question =>You may or may not see activity at this point depending on the status of the partition.

2. Boot the partition to Open Firmware (0 >) prompt, run ioinfo utility, and select the FCINFO option as follows:

3. Select the desired path. In this example, we are selected the 2nd virtual Fibre Channel path:

FCINFO Main Menu
Select a FC Node from the following list:

4. Select a FC Device

5. Select the appropriate LUN. In this example option 1 happens to be the bootable device:

6. Select "Display Inquiry Data"

7. Select to "Boot from this Device"

SAN Switch Replacement in AIX Environments

Abstract: The purpose of this document is to describe the concepts and procedures used to replace SAN switches in an AIX Power environment. This includes direct attached or VIO attached storage, and with VIO both the VSCSI and NPIV cases. The article will first discuss dynamic tracking as this is important for making SAN changes. Then we'll look at SAN switch replacement in direct attached storage, VIO VSCSI attached storage, and VIO NPIV attached storage environments, including single SAN fabric and dual SAN fabric environments. We'll examine this from an MPIO environment, and consider how this applies to other multi-path code last.

SAN Switch Replacement in AIX Environments
Overview

The purpose of this document is to describe the concepts and procedures used to replace SAN switches in an AIX Power environment. This includes direct attached or VIO attached storage, and with VIO both the VSCSI and NPIV cases. The article will first discuss dynamic tracking as this is important for making SAN changes. Then we'll look at SAN switch replacement in direct attached storage, VIO VSCSI attached storage, and VIO NPIV attached storage environments, including single SAN fabric and dual SAN fabric environments. We'll examine this from an MPIO environment, and consider how this applies to other multi-path code last.

Dynamic tracking and LUN configuration

In environments with SAN switches, one will normally want to set certain attributes for the fscsi devices, specifically the dyntrk and fc_err_recov attributes. By default, these are set to no and delayed_fail respectively and assume the server is not attached to a SAN switch. This is important because the procedures to replace a switch are quite different depending on these settings.

Without dyntrk=yes, you will have to remove disk devices and reconfigure them. This means that any hdisk attribute settings you have changed will be undone, and you'll have to change them again. With dyntrk=yes, you do not have to remove the hdisk device definitions and you won't lose changes to the disk attributes. Disk attributes that are often changed include the reserve_policy for SCSI reserves, and the queue_depth for performance.

Here's how to look at these attributes:

You should set these as follows:

dyntrk=yes
fc_err_recov=fast_fail

via this command if no disks are in use:

or if the disks are in use:

and then reboot to make the changes go into effect. Thus, these changes are not dynamic for the LPAR. Preferably they attributes are set as recommended when the LPAR is installed and setup.

Note that at VIOSs (VIO Servers) if a LUN is mapped from the VIOS to a VIOC (VIO Client) as a VSCSI disk, then the disk is in use, even if the VIOC isn't using the disks. So in a typical dual VIOS environment, one would make this change to one VIOS, reboot it, then make the change to the other VIOS and reboot it.

Lacking these attribute settings, AIX includes information about the specific port on the switch, as part of the LUN configuration. Thus, to use a different port on the switch or another switch entirely, one will have to actually remove the disk definition (via a # rmdev -dl ), move the cables to the new switch and run cfgmgr. This also means stopping use of the disk and applications using it.

For more details on these settings see the documentation at :
http://www-1.ibm.com/support/docview.wss?uid=isg1520readmefb4520desr_lpp_bos

http://publib.boulder.ibm.com/infocenter/pseries/v5r3/index.jsp?topic=/com.ibm.aix.kernelext/doc/kernextc/fcp_overview.htm

http://publib.boulder.ibm.com/infocenter/systems/index.jsp?topic=/com.ibm.aix.prftungd/doc/prftungd/fast_fail_dynamic_interaction.htm

Can SAN switch replacements be done dynamically?

Provided the dyntrk and fc_err_recov are properly set, the answer is yes provided one ensures that there will always be at least one working path for each hdisk. Additionally, in some cases with only one path, provided we move the cable fast enough, then we can also do this dynamically; however this is discouraged. Fast enough is such that from the time we unplug a cable from a switch port, plug it into a port on the new switch, plus the time for the SAN fabric to recognize the new cabling, is less than 15 seconds so the IOs don't time out and fail. So when one path to the disk exists, SAN switch replacements are preferably done during maintenance windows. For example, if a cable isn't properly seated or a port is defective, then IOs can fail leading to problems.

Know your paths and the cables they use

As the previous paragraph makes clear, you want to make sure that a working path exists when dynamically migrating from one SAN switch to another. So you need to know what paths exist to your disks and the cables involved. To that end, it's important to understand that a path is uniquely described via the host port and the storage port used by the path, and that ports are uniquely identified via a WWPN (World Wide Port Name) which is 16 digits. How one determines this depends on the multi-path code used for the storage, and this article initially focuses on MPIO environments (which includes storage using SDDPCM as the multi-path code since SDDPCM uses MPIO under the covers). To list the paths for your disks with MPIO, use the lspath command as follows (here for hdisk2):

This shows hdisk2 has 4 paths, two from fcs0 (the parent device of fscsi0) two from fcs1, and going to two separate ports on the storage and identified via the WWPN of the storage port: 203900a0b848dda or 20180a0b8478dda. From this we can also conclude that we're only using one SAN fabric for this LUN (and probably for the LPAR as well - and this can be verified by looking at the paths for all LUNs and in which case they'd look similar) since both host ports connect to both storage ports. Thus, the cabling would look like:

In VIO VSCSI environments, you'd run the lspath commands on the VIOS (VIO Server) in the oem_setup_env shell, as we're concerned about paths from the VIOS to the storage. In a VIO NPIV environment, one would run the lspath commands on the VIOC plus one will need to know the vFC to real FC adapter mapping

Identifying adapter and port locations

It will be important to know which cables connect to which ports on the storage, host. and SAN switches. Since this document discusses SAN switch replacement, perhaps the easiest method is to obtain the host and storage WWPNs for the ports and then from the switch management interface, determine the ports to which they are connected. From the host side, you can list the port location code and WWPN via the following command:

Or for a description of the adapter and its location code:

These are a dual port adapters, and checking the Finding Parts Locations and Addresses manual for the specific system model (these manuals are available in the Power Hardware Information Center at http://publib.boulder.ibm.com/infocenter/powersys/v3r1m5/index.jsp) one can determine the specific slot location for the adapter. In this case the system model is a 9179-MHB and the P2-C2 in the location field indicates that the adapter is in slot 2 of the system unit. Then T1 refers to the top port and T2 to the bottom port. These adapters also have an identify light, so one can go into the diagnostics menu and get the light to flash to more easily locate the adapter.

Also note that it is possible that the fcs0 and fscsi0 don't refer to the same port, so you can't rely on the numbers for the devices. You can see the relationship via the location codes, e.g.:

This shows fcs0 is related to fscsi0 via the location codes.

Be aware if you are you using an active/passive disk subsystem

An active/passive disk subsystem is one in which one controller of a pair is used to handle all IOs to a LUN except in failure conditions. Examples of active/passive IBM storage include the DS3000, DS4000, DS5000, SVC and Storwize V7000. The reason this is important, is that it's preferable to not lose access to the primary controller for the LUNs during a switch replacement, or to at least be aware that LUNs will switch controllers if all paths to the primary controller are lost during the switch replacement. Usually half the LUNs have one storage controller as the preferred controller, with the other half of the LUNs using the other controller. So if only one cable is used per controller, this means that LUNs will fail over to the other controller if the server is doing IOs to the storage during the switch replacement.

It's also possible to use RDAC for DS4000 storage which requires that host adapterA is connected to storage controllerA, and host adapterB is connected to storage controllerB, without any cross connections. Please be aware that from AIX 6.1 and on, MPIO is strategic and preferred. One can choose the multi-path code used for the DS3/4/5000 via the manage_disk_drivers command available in AIX (note that there are also requirements from the storage side for MPIO).

Later when all paths are restored, one will normally want the storage administrator to switch the LUN back to the preferred controller. If all paths to one controller will be lost during a switch replacement, then it's recommended that the storage administrator move all IO handling to the controller that will be accessible prior to moving a cable.

Should you use ISLs to facilitate SAN switch replacement?

Connecting two switches via Inter Switch Links (ISLs) joins two switches into a fabric. Before going into ISLs, it's important to know that one should have dynamic tracking enabled prior to adding a switch to a fabric via an ISL as lacking that setting might cause IOs to be lost.

In the case where we have ISLs, we can move the cables in any order, and provided we do so quickly enough and we properly seat the cables, IOs will be slightly delayed. For the non-ISL environment, we have to use more care. First, we can't move the cables in any order. We have to move a server cable, then a storage cable; otherwise, the server will lose access to the storage. When we move cable E, any IOs using that cable will fail, and we'll have to rely on the multi-path code at the server to redirect the IO to use cable F. This delay will be longer as the IO must time out. If we move cable G first, and fc_err_recov=fast_fail and the switch supports fast fail, then the switch will inform the adapter driver that the port no longer has access to the storage and the multi-path code will immediately redirect the IOs to use cable H, so this will be less delay than if moving a server side cable.

Of course, if we stop the application and IOs, then we can move the cables without worrying about doing it quickly or regarding the order cables are moved. So we can see that ISLs facilitate switch replacement here, though the option of stopping IO entirely avoids some of the work required.

Why it's better to disable paths prior to cable movement

While we can use the path availability facilities to handle lost paths during a switch replacement, it's preferable to disable paths prior to moving cables for two reasons. First, in-flight IOs will be delayed if we don't disable the paths first. This delay might result in the application stalling while IOs time out and re-initiated down available paths, or with active/passive disk subsystems, while the storage moves IO processing from one controller to another. Secondly, and perhaps more importantly, bugs in the recovery portions of the code might have bugs which could result in IO failures. Given the matrix of multi-path code versions, storage firmware/microcode, and adapter firmware, it's difficult to test all possible combinations of code and failures. If you've tested path failure, observing failure detection, handling of IOs in-flight, and path recovery, then one can be assured that the code will work correctly.

How to disable and re-enable paths

The command to disable or enable paths for IO is the chpath command, e.g.:

VIO VSCSI environments

Here are two diagrams of a VIO Client (VIOC) using VSCSI to access SAN attached storage through a pair of VIO Servers (VIOSs) in a dual SAN fabric environment showing two cabling strategies:

It's important to realize there are 2 layers of multi-path code here. MPIO is always used at the VIOC for choosing a path to the VIOSs. The multi-path code at the VIOS depends on what the storage requires. From the VIOC, each LUN has two paths (to each VIOS). From the VIOS, there are potentially 8 paths to a LUN in example 1, and potentially 4 paths to a LUN for example 2. Besides having more paths, there is an availability difference between the two diagrams. Example 1 can continue running with the failure of a VIOS and a SAN fabric. Example 2 can also, provided the right pair of VIOS and fabric fail. Thus, typically you'll see cabling similar to the diagram on the left.

A difference in how one would disable paths here exists. For example 1, when replacing a switch, one preferably disables/enables paths at the VIOSs when replacing a switch. For example 2, one can simply disable the paths to the VIOS attached to the switch being replaced. And one can just disable all paths for a fibre channel port attached to the SAN switch being replaced, if the multi-path code provides this capability.

VIO NPIV environments

Here are two examples of a VIOC using NPIV thru two VIOSs in a dual SAN fabric to access SAN attached storage:

Here there is only one layer of multi-path code, and that is in the VIOC. In both examples, there are 8 potential paths per LUN. However example 3 has superior availability characteristics in that we can have a VIOS fail and a SAN fabric failure without losing access to the storage, while in example 4 we could lose access if a VIOS and the SAN fabric the other VIOS uses fails. So all path management commands are done from the VIOC. And in both cases one can just disable all paths for a fibre channel port attached to the SAN switch being replaced, if the multi-path code provides this capability.

Multi-path code other than MPIO

There are other multi-path code sets besides MPIO, and often MPIO isn't a choice as the storage vendor dictates what must be used for their storage, and MPIO is often not an option. Each multi-path code set has its own command for handling path management, and the concepts previously mentioned still apply.

For example, one can use SDDPCM (which is compliant with the MPIO architecture) and still use the MPIO commands; however, you may find using the pcmpath command to be easier to accomplish your objectives. SDD is another multi-path code set from IBM (though SDDPCM is strategic) and one can use the datapath command for path management. PowerPath is a common option for customers attaching EMC storage to Power, in which case one typically uses the powermt command for path management.

SOURCE: IBM TD105839

Navisphere configuration error after clone AIX

If you choose to clone IBM AIX operating system with alt_disk_install command to put a new system in production with EMC Clariion Storage, don't forget to reboot on the clone drive without SAN and make this change.

Modify ODM hostname and IP address (chdev command)
Modify /etc/hosts
Check /etc/resolv.conf

Reconfigure RSCT to avoid DLPAR mismatch

and finally ... remove Navisphere HostIdFile.txt file

When the clone partition reboot with new hardware (on same server) Navisphere start and recreate HostIdFile.txt with new IP address.

Easy.