File system verification and recovery

The fsck command checks and interactively repairs inconsistent file systems.
You should run this command before mounting any file system. You must be able
to read the device file on which the file system resides (for example, the /dev/hd0
device).
Normally, the file system is consistent, and the fsck command merely reports on
the number of files, used blocks, and free blocks in the file system. If the file
system is inconsistent, the fsck command displays information about the
inconsistencies found and prompts you for permission to repair them. If the file
system cannot be repaired, restore it from backup.
Mounting an inconsistent file system may result in a system crash. If you do not
specify a file system with the FileSystem parameter, the fsck command will
check all the file systems with the attribute check=TRUE in /etc/filesystems.
Note: By default, the /, /usr, /var, and /tmp file systems have the check
attribute set to false (check=false) in their /etc/filesystems stanzas. The
attribute is set to false for the following reasons:

The boot process explicitly runs the fsck command on the /, /usr, /var, and
/tmp file systems.

The /, /usr, /var, and /tmp file systems are mounted when the /etc/rc file is
run. The fsck command will not modify a mounted file system, and fsck
results on mounted file systems are unpredictable.
Chapter 7. LVM, file system, and disk problem determination 153
Fixing a bad superblock
If you receive one of the following errors from the fsck or mount commands, the
problem may be a corrupted superblock, as shown in the following example:
fsck: Not an AIX3 file system
fsck: Not an AIXV3 file system
fsck: Not an AIX4 file system
fsck: Not an AIXV4 file system
fsck: Not a recognized file system type
mount: invalid argument
The problem can be resolved by restoring the backup of the superblock over the
primary superblock using the following command (care should be taken to check
with the latest product documentation before running this command):
# dd count=1 bs=4k skip=31 seek=1 if=/dev/lv00 of=/dev/lv00
The following is an example of when the superblock is corrupted and copying the
backup helps solve the problem:
# mount /u/testfs
mount: 0506-324 Cannot mount /dev/lv02 on /u/testfs: A system call received a
parameter that is not valid.
# fsck /dev/lv02
Not a recognized filesystem type. (TERMINATED)
# dd count=1 bs=4k skip=31 seek=1 if=/dev/lv02 of=/dev/lv02
1+0 records in.
1+0 records out.
# fsck /dev/lv02
** Checking /dev/lv02 (/u/tes)
** Phase 0 - Check Log
log redo processing for /dev/lv02
** Phase 1 - Check Blocks and Sizes
** Phase 2 - Check Pathnames
** Phase 3 - Check Connectivity
** Phase 4 - Check Reference Counts
** Phase 5 - Check Inode Map
** Phase 6 - Check Block Map
8 files 2136 blocks 63400 free
Once the restoration process is complete, check the integrity of the file system by
issuing the fsck command:
# fsck /dev/lv00
154 IBM ^ Certification Study Guide - AIX 5L Problem Determination Tools and Techniques
In many cases, restoration of the backup of the superblock to the primary
superblock will recover the file system. If this does not resolve the problem,
recreate the file system and restore the data from a backup.
7.4.4 Sparse file allocation
Some applications, particularly databases, maintain data in sparse files. Files
that do not have disk blocks allocated for each logical block are called sparse
files. If the file offsets are greater than 4 MB, then a large disk block of 128 KB is
allocated. Applications using sparse files larger than 4 MB may require more disk
blocks in a file system enabled for large files than in a regular file system.
In the case of sparse files, the output of the ls command is not showing the
actual file size, but is reporting the number of bytes between the first and last
blocks allocated to the file, as shown in the following example:
# ls -l /tmp/sparsefile
-rw-r--r-- 1 root system 100000000 Jul 16 20:57 /tmp/sparsefile
The du command can be used to see the actual allocation, since it reports the
blocks actually allocated and in use by the file. Use du -rs to report the number
of allocated blocks on disk.
# du -rs /tmp/sparsefile
256 /tmp/sparsefile
Using the dd command in combination with your own backup script will solve this
problem.
7.4.5 Unmount problems
A file system cannot be unmounted if any references are still active within that file
system. The following error message will be displayed:
Device busy
or
A device is already mounted or cannot be unmounted
Note: The tar command does not preserve the sparse nature of any file that
is sparsely allocated. Any file that was originally sparse before the restoration
will have all space allocated within the file system for the size of the file. New
AIX 5L options for the backup and restore command are useful for sparse
files.
Chapter 7. LVM, file system, and disk problem determination 155
The following situations can leave open references to a mounted file system.

Files are open within a file system. These files must be closed before the file
system can be unmounted. The fuser command is often the best way to
determine what is still active in the file system. The fuser command will return
the process IDs for all processes that have open references within a specified
file system, as shown in the following example:
# umount /home
umount: 0506-349 Cannot unmount /dev/hd1: The requested resource is busy.
# fuser -x -c /home
/home: 11630
# ps -fp 11630
UID PID PPID C STIME TTY TIME CMD
guest 11630 14992 0 16:44:51 pts/1 0:00 -sh
# kill -1 11630
# umount /home
The process having an open reference can be killed by using the kill
command (sending a SIGHUP), and the unmount can be accomplished. A
stronger signal may be required, such as SIGKILL.

If the file system is still busy and still cannot be unmounted, this could be due
to a kernel extension that is loaded but exists within the source file system.
The fuser command will not show these kinds of references, since a user
process is not involved. However, the genkex command will report on all
loaded kernel extensions.

File systems are still mounted within the file system. Unmount these file
systems before the file system can be unmounted. If any file system is
mounted within a file system, this leaves open references in the source file
system at the mount point of the other file system. Use the mount command to
get a list of mounted file systems. Unmount all the file systems that are
mounted within the file system to be unmounted.
7.4.6 Removing file systems
When removing a JFS, the file system must be unmounted before it can be
removed. The command for removing file systems is rmfs.
In the case of a JFS, the rmfs command removes both the logical volume on
which the file system resides and the associated stanza in the /etc/filesystems
file. If the file system is not a JFS, the command removes only the associated
stanza in the /etc/filesystems file, as shown in the following example:
# lsvg -l testvg
testvg:
LV NAME TYPE LPs PPs PVs LV STATE MOUNT POINT
loglv00 jfslog 1 1 1 open/syncd N/A
156 IBM ^ Certification Study Guide - AIX 5L Problem Determination Tools and Techniques
lv02 jfs 2 2 1 open/syncd /u/testfs
# rmfs /u/testfs
rmfs: 0506-921 /u/testfs is currently mounted.
# umount /u/testfs
# rmfs /u/testfs
rmlv: Logical volume lv02 is removed.
# lsvg -l testvg
testvg:
LV NAME TYPE LPs PPs PVs LV STATE MOUNT POINT
loglv00 jfslog 1 1 1 closed/syncd N/A
This example shows how the file system testfs is removed. The first attempt fails
because the file system is still mounted. The associated logical volume lv02 is
also removed. The jfslog remains defined on the volume group.
7.4.7 Different output from du and df commands
Sometimes du and df commands are used to get a free block value. df is used to
report the total block count, and then the value returned by du -s
/filesystem_name is subtracted from that total to calculate the free block value.
However, this method of calculation yields a value that is greater than the free
block value reported by df. At AIX Version 4.1 and later, both df and du default to
512-byte units. Sample output from the du and df commands is below:
# du -s /tmp
152 /tmp
# df /tmp
Filesystem 512-blocks Free %Used Iused %Iused Mounted on
/dev/hd3 24576 23320 6% 33 1% /tmp
Here (total from df) - (used from du) + (false free block count): 24576 - 152 =
24424.
24424 is greater than 23320. The reason for this discrepancy involves the
implementation of du and df. du -s traverses the file tree, adding up the number
of blocks allocated to each directory, symlink, and file as reported by the stat()
system call. This is how du arrives at its total value. df looks at the file system
disk block allocation maps to arrive at its total and free values.
7.4.8 Enhanced journaled file system
The enhanced journaled file system (JFS2) contains several architectural
differences over the standard JFS, including:

Variable number of i-nodes for enhanced journaled file system
JFS2 allocates i-nodes as needed. Therefore, the number of i-nodes
available is limited by the size of the file system itself.
Chapter 7. LVM, file system, and disk problem determination 157

Specifying file system block size
File system block size is specified during the file system's creation with the
crfs and mkfs commands or by using the SMIT. The decision of file system
block size should be based on the projected size of files contained by the file
system.

Identifying file system block size
The file system block size value can be identified with the lsfs command or
the System Management Interface Tool (SMIT). For application programs,
the statfs subroutine can be used to identify the file system block size.

Compatibility and migration
The enhanced journaled file system (JFS2) is a new file system and is not
compatible with AIX Version 4.

Device driver limitations
A device driver must provide disk block addressability that is the same or
smaller than the file system block size.

Performance costs
Although file systems that use block sizes smaller than 4096 bytes as their
allocation unit might require substantially less disk space than those using the
default allocation unit of 4096 bytes, the use of smaller block sizes can incur
performance degradation.

Increased allocation activity
Because disk space is allocated in smaller units for a file system with a block
size other than 4096 bytes, allocation activity can occur more often when files
or directories are repeatedly extended in size. For example, a write operation
that extends the size of a zero-length file by 512 bytes results in the allocation
of one block to the file, assuming a block size of 512 bytes. If the file size is
extended further by another write of 512 bytes, an additional block must be
allocated to the file. Applying this example to a file system with 4096-byte
blocks, disk space allocation occurs only once, as part of the first write
operation. No additional allocation activity is performed as part of the second
write operation since the initial 4096-byte block allocation is large enough to
hold the data added by the second write operation.

Increased block allocation map size
More virtual memory and file system disk space might be required to hold
block allocation maps for file systems with a block size smaller than 4096
bytes. Blocks serve as the basic unit of disk space allocation, and the
allocation state of each block within a file system is recorded in the file system
block allocation map.
158 IBM ^ Certification Study Guide - AIX 5L Problem Determination Tools and Techniques

Understanding enhanced journaled file system size limitations
The maximum size for an enhanced journaled file system is architecturally
limited to 4 Petabytes. I-nodes are dynamically allocated by JFS2, so you do
not need to consider how many i-nodes you may need when creating a JFS2
file system. You need to consider the size of the file system log.

Enhanced journaled file system log size issues
In most instances, multiple journaled file systems use a common log
configured to be 4 MB in size. When file systems exceed 2 GB or when the
total amount of file system space using a single log exceeds 2 GB, the default
log size might not be sufficient. In either case, scale log sizes upward as the
file system size increases. The JFS log is limited to a maximum size of 256
MB.

JFS2 file space allocation
File space allocation is the method by which data is apportioned physical
storage space in the operating system. The kernel allocates disk space to a
file or directory in the form of logical blocks. A logical block refers to the
division of a file or directory contents into 512, 1024, 2048, or 4096 byte units.
When a JFS2 file system is created the logical block size is specified to be
one of 512, 1024, 2048, or 4096 bytes. Logical blocks are not tangible
entities; however, the data in a logical block consumes physical storage
space on the disk. Each file or directory consists of zero or more logical
blocks.

Full and partial logical blocks
A file or directory may contain full or partial logical blocks. A full logical block
contains 512, 1024, 2048, or 4096 bytes of data, depending on the file system
block size specified when the JFS2 file system was created. Partial logical
blocks occur when the last logical block of a file or directory contains less than
the file system block size of data.
For example, a JFS2 file system with a logical block size of 4096 with a file of
8192 bytes is two logical blocks. The first 4096 bytes reside in the first logical
block and the following 4096 bytes reside in the second logical block.
Likewise, a file of 4608 bytes consists of two logical blocks. However, the last
logical block is a partial logical block containing the last 512 bytes of the file's
data. Only the last logical block of a file can be a partial logical block.

JFS2 file space allocation
The default block size is 4096 bytes. You can specify smaller block sizes with
the mkfs command during a file system's creation. Allowable fragment sizes
are 512, 1024, 2048, and 4096 bytes. You can use only one block’s size in a
file system.
Chapter 7. LVM, file system, and disk problem determination 159
The kernel allocates disk space so that only the last file system block of data
receives a partial block allocation. As the partial block grows beyond the limits
of its current allocation, additional blocks are allocated.
Block reallocation also occurs if data is added to logical blocks that represent
file holes. A file hole is an "empty" logical block located prior to the last logical
block that stores data. (File holes do not occur within directories.) These
empty logical blocks are not allocated blocks. However, as data is added to
file holes, allocation occurs. Each logical block that was not previously
allocated disk space is allocated a file system block of space.

Increasing the file system size

In many instances, the size of a file system needs to be increased because the
demand for storage has increased. In AIX, this is a common procedure, and it is
possible to do by using the chfs command, as in the following example:
# chfs -a size=+300000 /u/testfs
Filesystem size changed to 458752
This example shows how the file system testfs is extended with 300000 512-byte
blocks. When the file system is extended, the logical volume holding the JFS is
also extended, with the number of logical partitions that is needed to fulfill the
space request. If the system does not have enough free space, the volume group
can either be extended with an additional physical volume, or the size specified
for the chfs command must be lowered so that it matches the number of free
LPs.

Extending the number of max physical partitions

When adding a new disk to a volume group, you may encounter an error due to
there being too few PP descriptors for the required number of PVs. This may
occur when the new disk has a much higher capacity than existing disks in the
volume group.
This situation is typical on older installations, due to the rapid growth of storage
technology. To overcome this, a change of the volume group LVM metadata is
required.
The chvg command is used for this operation using the -t flag and applying a
factor value, as shown in the following example:
# lsvg testvg
VOLUME GROUP: testvg VG IDENTIFIER: 000bc6fd5a177ed0
VG STATE: active PP SIZE: 16 megabyte(s)
VG PERMISSION: read/write TOTAL PPs: 542 (8672 megabytes)
MAX LVs: 256 FREE PPs: 42 (672 megabytes)
LVs: 1 USED PPs: 500 (8000 megabytes)
OPEN LVs: 0 QUORUM: 2
TOTAL PVs: 1 VG DESCRIPTORS: 2
STALE PVs: 0 STALE PPs: 0
ACTIVE PVs: 1 AUTO ON: yes
MAX PPs per PV: 1016 MAX PVs: 32
# chvg -t 2 testvg
0516-1193 chvg: WARNING, once this operation is completed, volume group testvg
cannot be imported into AIX 430 or lower versions. Continue (y/n) ?
y
0516-1164 chvg: Volume group testvg changed. With given characteristics testvg
can include upto 16 physical volumes with 2032 physical partitions
each.
144 IBM ^ Certification Study Guide - AIX 5L Problem Determination Tools and Techniques
This example shows that the volume group testvg with a current 9.1 GB disk has
a maximum number of 1016 PPs per physical volume. Adding a larger 18.2 GB
disk would not be possible; the maximum size of the disk is limited to 17 GB
unless the maximum number of PPs is increased. Using the chvg command to
increase the maximum number of PPs by a factor of 2 to 2032 PPs allows the
volume group to be extended with physical volumes of up to approximately 34
GB.
7.3 Disk replacement
AIX, like all operating systems, can be problematic when you have to change a
disk. AIX provides the ability to prepare the system for the change using the
LVM. You can then perform the disk replacement and then use the LVM to
restore the system back to how it was before the disk was changed. This process
manipulates not only the data on the disk itself, but is also a way of keeping the
Object Data Manager (ODM) intact.
The ODM within AIX is a database that holds device configuration details and
AIX configuration details. The function of the ODM is to store the information
between reboots, and also provide rapid access to system data, eliminating the
need for AIX commands to interrogate components for configuration information.
Since this database holds so much vital information regarding the configuration
of a machine, any changes made to the machine, such as the changing of a
defective disk, need to be done in such a way as to preserve the integrity of the
database.
7.3.1 Replacing a disk
The following scenario shows a system that has a hardware error on a physical
volume. However, since the system uses a mirrored environment, which has
multiple copies of the logical volume, it is possible to replace the disk while the
system is active. The disk hardware in this scenario are hot-swappable SCSI
disks, which permit the replacement of a disk in a production environment.
One important factor is detecting the disk error. Normally, mail is sent to the
system administrator (root account) from the Automatic Error Log Analysis
(diagela). Figure 7-1 on page 145 shows the information in such a diagnostics
mail.
Chapter 7. LVM, file system, and disk problem determination 145
Figure 7-1 Disk problem mail from Automatic Error Log Analysis (diagela)
Automatic Error Log Analysis (diagela) provides the capability to do error log
analysis whenever a permanent hardware error is logged. Whenever a
permanent hardware resource error is logged, the diagela program is invoked.
Automatic Error Log Analysis is enabled by default on all platforms.
The diagela message shows that the hdisk4 has a problem. Another way of
locating a problem is to check the state of the logical volume using the lsvg
command, as in the following example:
# lsvg -l mirrorvg
mirrorvg:
LV NAME TYPE LPs PPs PVs LV STATE MOUNT POINT
lvdb01 jfs 500 1000 2 open/syncd /u/db01
lvdb02 jfs 500 1000 2 open/stale /u/db02
loglv00 jfslog 1 1 1 open/syncd N/A
The logical volume lvdb02 in the volume group mirrorvg is marked with the status
stale, indicating that the copies in this LV are not synchronized. Look at the error
log using the error-reporting errpt command, as in the following example:
# errpt
EAA3D429 0713121400 U S LVDD PHYSICAL PARTITION MARKED STALE
F7DDA124 0713121400 U H LVDD PHYSICAL VOLUME DECLARED MISSING
41BF2110 0713121400 U H LVDD MIRROR WRITE CACHE WRITE FAILED
35BFC499 0713121400 P H hdisk4 DISK OPERATION ERROR
146 IBM ^ Certification Study Guide - AIX 5L Problem Determination Tools and Techniques
This error information displays the reason why the LV lvdb02 is marked stale.
The hdisk4 had an DISK OPERATION ERROR and the LVDD could not write the
mirror cache.
Based on the information in the example, hdisk4 needs to be replaced. Before
taking any action on the physical disk of the mirrored LV are recommended that
you do a file system backup in case anything should go wrong. Since the other
disk of the mirrored LV is still functional, all the data should be present. If the LV
contains a database, then the respective database tools for backup of the data
should be used.
Removing a bad disk
If the system is a high-availability (24x7) system, you might decide to keep the
system running while performing the disk replacement, provided that the
hardware supports an online disk exchange with hot-swappable disks. However,
the procedure should be agreed upon by the system administrator or customer
before continuing. Use the following steps to remove a disk:
1. To remove the physical partition copy of the mirrored logical volume from the
erroneous disk, use the rmlvcopy command as follows:
# rmlvcopy lvdb02 1 hdisk4
The logical volume lvdb02 is now left with only one copy, as shown in the
following:
# lslv -l lvdb02
lvdb02:/u/db02
PV COPIES IN BAND DISTRIBUTION
hdisk3 500:000:000 21% 109:108:108:108:067
2. Reduce the volume group by removing the disk you want to replace from its
volume group:
# reducevg -f mirrorvg hdisk4
# lsvg -l mirrorvg
mirrorvg:
LV NAME TYPE LPs PPs PVs LV STATE MOUNT POINT
lvdb01 jfs 500 1000 2 open/syncd /u/db01
lvdb02 jfs 500 500 1 open/syncd /u/db02
loglv00 jfslog 1 1 1 open/syncd N/A
3. Remove the disk as a device from the system and from the ODM database
with the rmdev command:
# rmdev -d -l hdisk4
hdisk4 deleted
This command is valid for any SCSI disk. If your system is using SSA, then an
additional step is required. Since SSA disks also define the device pdisk, the erroneous disk, use the rmlvcopy command as follows:
# rmlvcopy lvdb02 1 hdisk4
The logical volume lvdb02 is now left with only one copy, as shown in the
following:
# lslv -l lvdb02
lvdb02:/u/db02
PV COPIES IN BAND DISTRIBUTION
hdisk3 500:000:000 21% 109:108:108:108:067
2. Reduce the volume group by removing the disk you want to replace from its
volume group:
# reducevg -f mirrorvg hdisk4
# lsvg -l mirrorvg
mirrorvg:
LV NAME TYPE LPs PPs PVs LV STATE MOUNT POINT
lvdb01 jfs 500 1000 2 open/syncd /u/db01
lvdb02 jfs 500 500 1 open/syncd /u/db02
loglv00 jfslog 1 1 1 open/syncd N/A
3. Remove the disk as a device from the system and from the ODM database
with the rmdev command:
# rmdev -d -l hdisk4
hdisk4 deleted
This command is valid for any SCSI disk. If your system is using SSA, then an
additional step is required. Since SSA disks also define the device pdisk, the
Chapter 7. LVM, file system, and disk problem determination 147
corresponding pdisk device must be deleted as well. Use the SSA menus in
SMIT to display the mapping between hdisk and pdisk. These menus can
also be used to delete the pdisk device.
4. The disk can now be safely removed from your system.
Adding a new disk
Continuing the scenario from the previous section, this section describes how to
add a new disk into a running environment. After hdisk4 has been removed, the
system is now left with the following disks:
# lsdev -Cc disk
hdisk0 Available 30-58-00-8,0 16 Bit SCSI Disk Drive
hdisk1 Available 30-58-00-9,0 16 Bit SCSI Disk Drive
hdisk2 Available 10-60-00-8,0 16 Bit SCSI Disk Drive
hdisk3 Available 10-60-00-9,0 16 Bit SCSI Disk Drive
Use the following steps to add a new disk:
1. Plug in the new disk and run the configuration manager cfgmgr command.
The cfgmgr command configures devices controlled by the Configuration
Rules object class, which is part of the device configuration database. The
cfgmgr command will see the newly inserted SCSI disk and create the
corresponding device. Although the command requires no option, the -v flag
specifies verbose output, which helps in troubleshooting, as shown in the
following:
# cfgmgr -v
cfgmgr is running in phase 2
----------------
Time: 0 LEDS: 0x538
Invoking top level program -- "/etc/methods/cfgprobe -c
/etc/drivers/coreprobe.ext"
Time: 0 LEDS: 0x539
Return code = 0
*** no stdout ****
*** no stderr ****
----------------
Time: 0 LEDS: 0x538
Invoking top level program -- "/etc/methods/defsys"
Time: 0 LEDS: 0x539
Return code = 0
***** stdout *****
sys0
.....
.....
The result is a new hdisk4 added to the system:
# lsdev -Cc disk
148 IBM ^ Certification Study Guide - AIX 5L Problem Determination Tools and Techniques
hdisk0 Available 30-58-00-8,0 16 Bit SCSI Disk Drive
hdisk1 Available 30-58-00-9,0 16 Bit SCSI Disk Drive
hdisk2 Available 10-60-00-8,0 16 Bit SCSI Disk Drive
hdisk3 Available 10-60-00-9,0 16 Bit SCSI Disk Drive
hdisk4 Available 10-60-00-12,0 16 Bit SCSI Disk Drive
2. The new hdisk must now be assigned to the volume group mirrorvg by using
the LVM extendvg command:
# extendvg mirrorvg hdisk4
3. To re-establish the mirror copy of the LV, use the mklvcopy command.
# mklvcopy lvdb02 2 hdisk4
The number of copies of LV is now two, but the LV stat is still marked as stale
because the LV copies are not synchronized with each other:
# lsvg -l mirrorvg
mirrorvg:
LV NAME TYPE LPs PPs PVs LV STATE MOUNT POINT
lvdb01 jfs 500 1000 2 open/syncd /u/db01
lvdb02 jfs 500 1000 2 open/stale /u/db02
loglv00 jfslog 1 1 1 open/syncd N/A
4. To get a fully synchronized set of copies of the LV lvdb02, use the syncvg
command:
# syncvg -p hdisk4
The syncvg command can be used with logical volumes, physical volumes, or
volume groups. The synchronization process can be quite time consuming,
depending on the hardware characteristics and the amount of data.
After the synchronization is finished, verify the logical volume state using
either the lsvg or lslv command:
# lsvg -l mirrorvg
mirrorvg:
LV NAME TYPE LPs PPs PVs LV STATE MOUNT POINT
lvdb01 jfs 500 1000 2 open/syncd /u/db01
lvdb02 jfs 500 1000 2 open/syncd /u/db02
loglv00 jfslog 1 1 1 open/syncd N/A
The system is now back to normal.
7.3.2 Recovering an incorrectly removed disk
If a disk was incorrectly removed from the system, and the system has been
rebooted, the synclvodm command will need to be run to rebuild the logical
volume control block, as shown in the following examples.
Chapter 7. LVM, file system, and disk problem determination 149
In the examples, a disk has been incorrectly removed from the system and the
logical volume control block needs to be rebuilt.
The disks in the system before the physical volume was removed is shown in the
following command output:
# lsdev -Cc disk
hdisk0 Available 30-58-00-8,0 16 Bit SCSI Disk Drive
hdisk1 Available 30-58-00-9,0 16 Bit SCSI Disk Drive
hdisk2 Available 10-60-00-8,0 16 Bit SCSI Disk Drive
hdisk3 Available 10-60-00-9,0 16 Bit SCSI Disk Drive
The allocation of the physical volumes before the disk was removed are shown
as follows:
# lspv
hdisk0 000bc6fdc3dc07a7 rootvg
hdisk1 000bc6fdbff75ee2 volg01
hdisk2 000bc6fdbff92812 volg01
hdisk3 000bc6fdbff972f4 volg01
The logical volumes on the volume group:
# lsvg -l volg01
volg01:
LV NAME TYPE LPs PPs PVs LV STATE MOUNT POINT
logvol01 jfs 1000 1000 2 open/syncd /userfs01
loglv00 jfslog 1 1 1 open/syncd N/A
The logical volume distribution on the physical volumes is shown using the lslv
command:
# lslv -l logvol01
logvol01:/userfs01
PV COPIES IN BAND DISTRIBUTION
hdisk1 542:000:000 19% 109:108:108:108:109
hdisk3 458:000:000 23% 109:108:108:108:025
The system after a reboot has the following physical volumes:
# lspv
hdisk0 000bc6fdc3dc07a7 rootvg
hdisk1 000bc6fdbff75ee2 volg01
hdisk3 000bc6fdbff972f4 volg01
When trying to mount the file system on the logical volume, the error may look
similar to the following example:
# mount /userfs01
mount: 0506-324 Cannot mount /dev/logvol01 on /userfs01: There is an input or
output error.
150 IBM ^ Certification Study Guide - AIX 5L Problem Determination Tools and Techniques
To synchronize the logical volume, the following command should be run:
# synclvodm -v volg01
synclvodm: Physical volume data updated.
synclvodm: Logical volume logvol01 updated.
synclvodm: Warning, lv control block of loglv00 has been over written.
0516-622 synclvodm: Warning, cannot write lv control block data.
synclvodm: Logical volume loglv00 updated.
The system can now be repaired. If the file system data was spread across all
the disks, including the failed disk, it may need to be restored from the last
backup.
7.4 The AIX JFS
Similar to the LVM, most JFS problems can be traced to problems with the
underlying physical disk.
As with volume groups, various JFS features have been added at different levels
of AIX, which preclude those file systems being mounted if the volume group was
imported on an earlier version of AIX. Such features include large file enabled file
systems, file systems with non-default allocation group size, and JFS2.
7.4.1 Creating a JFS
In a journaled file system (JFS), files are stored in blocks of contiguous bytes.
The default block size, also referred to as fragmentation size in AIX, is 4096
bytes (4 KB). The JFS i-node contains an information structure of the file with an
array of eight pointers to data blocks. A file that is less then 32 KB is referenced
directly from the i-node.
A larger file uses a 4-KB block, referred to as an indirect block, for the addressing
of up to 1024 data blocks. Using an indirect block, a file size of 1024 x 4 KB = 4
MB is possible.
For files larger than 4 MB, a second block, the double indirect block, is used. The
double indirect block points to 512 indirect blocks, providing the possible
addressing of 512 x 1024 x 4 KB = 2 GB files. Figure 7-2 on page 151 illustrates
the addressing using double indirection.

LVM, file system, and disk

The following topics are discussed in this chapter:

Logical Volume Manager (LVM) problems

Replacement of physical volumes

JFS problems and their solutions

Paging space creation and removal, as well as recommendations about
paging space
To understand the problems that can happen on an AIX system with volume
groups, logical volumes, and file systems, it is important to have a detailed
knowledge about how the storage is managed by the Logical Volume Manager.
This chapter does not cover the fundamentals of the LVM; they are considered
prerequisite knowledge required to understand the issues addressed in this
chapter.
7
138 IBM ^ Certification Study Guide - AIX 5L Problem Determination Tools and Techniques
7.1 LVM data
The Logical Volume Manager (LVM) data structures that are required for the
LVM to operate are stored in a number of structures. This logical layout is
described in the following sections.
7.1.1 Physical volumes
Each disk is assigned a Physical Volume Identifier (PVID) when it is first
assigned to a volume group. The PVID is a combination of the serial number of
the machine creating the volume group and the time and date of the operation.
The PVID is stored on the physical disk itself and is also stored in the Object
Data Manager (ODM) of a machine when a volume group is created or imported.
You should not use the dd command to copy the contents of one physical volume
to another, since the PVID will also be copied; this will result in two disks having
the same PVID, which can confuse the system.
7.1.2 Volume groups
Each volume group has a Volume Group Descriptor Area (VGDA). There are
(commonly) multiple copies of the VGDA in a volume group. A copy of the VGDA
is stored on each disk in the volume group. The VGDA stores information about
the volume group, such as the logical volumes and the disks in the volume
group.
The VGDA is parsed by the importvg command when importing a volume group
into a system. It is also used by the varyonvg command in the quorum voting
process to decide whether a volume group should be varied on.
For a single disk volume group, there are two VGDAs on the disk. When a
second disk is added to make a two disk volume group, the original disk retains
two VGDAs and the new disk gets one VGDA.
Adding a third disk results in the extra VGDA from the first disk moving to the
third disk for a quorum of three with each disk having one vote. Adding this
additional disk adds a new VGDA per disk.
A volume group with quorum checking enabled (the default) must have at least
51 percent of the VGDAs in the volume group available before it can be varied
on. Once varied on, if the number of VGDAs falls below 51 percent, the volume
group will automatically be varied off.
Chapter 7. LVM, file system, and disk problem determination 139
In contrast, a volume group with quorum checking disabled must have 100
percent of the VGDAs available before it can be varied on. Once varied on, only
one VGDA needs to remain available to keep the volume group online.
A volume group also has a Volume Group Identifier (VGID), a soft serial number
for the volume group similar to the PVID for disks.
Each disk in a volume group also has a Volume Group Status Area (VGSA), a
127 byte structure used to track mirroring information for up to the maximum
1016 physical partitions on the disk.
7.1.3 Logical volumes
Each logical volume has a Logical Volume Control Block (LVCB) that is stored in
the first 512 bytes of the logical volume. The LVCB holds important details about
the logical volume, including its creation time, mirroring information, and mount
point (if it contains a journaled file system [JFS]).
Each logical volume has a Logical Volume Identifier (LVID) that is used to
represent the logical volume to the LVM libraries and low-level commands. The
LVID is made up of VGID.num, where num is the order in which it was created in
the volume group.
7.1.4 Object Data Manager (ODM)
The Object Data Manger is used by the LVM to store information about the
volume groups, physical volumes, and logical volumes on the system. The
information held in the ODM is placed there when the volume group is imported
or when each object in the volume group is created.
There exists an ODM object known as the vg-lock. Whenever an LVM
modification command is started, the LVM command will lock the vg-lock for the
volume group being modified. If for some reason the lock is inadvertently left
behind, the volume group can be unlocked by running the varyonvg -b
command, which can be run on a volume group that is already varied on.
7.2 LVM problem determination
The most common LVM problems are related to disk failures. Depending on the
extent of the failure, you may be able to recover the situation with little or no data
loss. However, a failed recovery attempt may leave the system in a worse
condition. This leaves restoring from backup as the only way to recover.
Therefore, always take frequent backups of your system.
140 IBM ^ Certification Study Guide - AIX 5L Problem Determination Tools and Techniques
7.2.1 Data relocation
When a problem occurs with a disk drive, data relocation may take place. There
are three types of data relocation, namely:

Internal to the disk

Hardware relocation ordered by LVM

Software relocation
Relocation typically occurs when the system fails to perform a read or write due
to physical problems with the disk platter. In some cases, the data I/O request
completes but with warnings. Depending on the type of recovered error, the LVM
may be wary of the success of the next request to that physical location, and it
orders a relocation to be on the safe side.
The lowest logical layer of relocation is the one that is internal to the disk. These
types of relocations are typically private to the disk and there is no notification to
the user that a relocation occurred.
The next level up in terms of relocation complexity is a hardware relocation
called for by the LVM device driver. This type of relocation will instruct the disk to
relocate the data on one physical partition to another portion (reserved) of the
disk. The disk takes the data in physical location A and copies it to a reserved
portion of the disk (location B). However, after this is complete, the LVM device
driver will continue to reference physical location A, with the understanding that
the disk itself will handle the true I/O to the real location B.
The top layer of data relocation is the soft relocation handled by the LVM device
driver. In this case, the LVM device driver maintains a bad block directory, and
whenever it receives a request to access logical location A, the LVM device
driver will look up the bad block table and translate it to actually send the request
to the disk drive at physical location B.
7.2.2 Backup data
The first step you should perform if you suspect a problem with LVM is to make a
backup of the affected volume group and save as much data as possible. This
may be required for data recovery. The integrity of the backup should be
compared with the last regular backup taken before the problem was detected.
7.2.3 ODM resynchronization
Problems with the LVM tend to occur when a physical disk problem causes the
ODM data to become out of sync with the VGDA, VGSA, and LVCB information
stored on disk.

Removing an HACMP Network

Note: Deleting all network interfaces associated with a network deletes the
network definition from HACMP.
To remove a network from the HACMP cluster definition:
1. Enter smit hacmp
2. In SMIT, select Extended Configuration > Extended Topology Configuration >
Configure HACMP Networks > Remove a Network from the HACMP Cluster and
press Enter.
SMIT displays the Select a Network to Remove panel.
3. Select the network to remove.
SMIT displays Are you sure?
4. Press Enter to remove the network. All of this network’s subnets and their interfaces are
removed from the HACMP configuration.
5. On the same node, synchronize the cluster configuration.
If the Cluster Manager is running on the local node, the synchronization triggers a dynamic
reconfiguration event. See Synchronizing the Cluster Configuration for more
information.
Converting an HACMP Network to use IP Aliasing
If you want to change the cluster network configuration to use IPAT via Aliases instead of the
previous IPAT via IP Replacement scheme for a specific network in the cluster, you should stop
the cluster services on all nodes to make the change. This change is not allowed during a
dynamic reconfiguration (DARE) of cluster resources.
Note: If you have an SP Switch network that has been configured in your
cluster in HACMP prior to version 5.1, and want to convert the SP
Switch to use the IP aliasing in HACMP, see the section Converting
an SP Switch Network to an Aliased Network.
To convert an HACMP network to use IP Aliasing:
1. Stop the cluster services on all cluster nodes.
2. Verify that no HACMP interfaces are defined with HWAT on that network.
3. Verify that the network is configured to support gratuitous ARP in HACMP, by checking
the Extended Configuration > Extended Topology Configuration > Configure an
HACMP Network Module > Show a Network Module SMIT panel for the Gratuitous
ARP setting for that network type.
Network Type Valid types are RS232, tmssa, tmscsi, diskhb.
Managing the Cluster Topology
Changing the Configuration of an HACMP Network
Administration Guide 399
13
4. To change the cluster network to use IPAT via IP Aliases instead of IPAT via IP
Replacement, see the steps in this chapter in the section Changing Network Attributes.
5. Verify and synchronize the cluster.
6. Restart the cluster services.
For more information on IPAT via IP Aliases see the relevant chapters in the Concepts and
Facilities Guide and in the Planning Guide.
Establishing Default and Static Routes on Aliased Networks
If you are setting up or converting to an IP aliased network and require establishing the default
route, and possibly, other static routes that have to be established on the IP aliased service
subnet, these routes will fail to be established automatically when the rc.net file runs at boot
time. This is because there is no address on that subnet in the Configuration Database.
To ensure that these routes are established at boot time, we recommend that you also configure
a persistent address on that subnet. After configuring the persistent address, HACMP
configures the routes.
If you do not configure persistent addresses, then you should use your own scripts that will
configure routes on aliased service subnets. For more information on the rc.net file see Chapter
1: Administering an HACMP Cluster and Chapter 9: Starting and Stopping Cluster
Services.
Converting an SP Switch Network to an Aliased Network
When you migrate your cluster to HACMP 5.4 from a version prior to HACMP 5.1, your
previously configured SP Switch network configuration remains valid. However, after
migration, HACMP by default treats your network as non-aliased, although in reality it
functions as an aliased network. Therefore, you may consider reconfiguring the existing SP
Switch network configuration.
To change the cluster configuration to use IPAT via IP Aliases instead of the standard IPAT via
IP Replacement scheme for the SP Switch network, stop the cluster services on all nodes and
make the following changes to the communication interface definitions. Such changes are not
allowed during a dynamic reconfiguration (DARE) of cluster resources.
To convert the SP Switch network to an aliased network, perform the following steps on all
cluster nodes:
1. Stop the cluster services.
2. Enter smit hacmp
3. In SMIT, select Extended Configuration > Extended Topology Configuration >
Configure HACMP Communication Interfaces/Devices > Remove Communication
Interfaces/Devices and press Enter.
4. Remove the non-service interfaces on the SP Switch network.
5. Remove the boot label that you had previously configured within HACMP for your SP
switch network.
Managing the Cluster Topology
Changing the Configuration of an HACMP Network
400 Administration Guide
13
6. Go back to the Extended HACMP Verification and Synchronization option and
synchronize the cluster. If the cluster network meets all the requirements of an aliased
network, the following message appears: “Setting attribute for network
to use IP Aliasing for IP address takeover”.
7. In SMIT, select the Extended Configuration > Extended Topology Configuration >
Configure HACMP Communication Interfaces/Devices and configure the base
interface address on the SP Switch network as a boot IP label/address in HACMP.
8. Put the service IP-based communication interface on a different subnet than the boot
interface to avoid errors during the verification process. If you have multiple service
addresses they should all be on a different subnet than the boot interface.
9. Verify that HWAT is disabled for all communication interfaces on this network.
10. Verify that the network is configured to support gratuitous ARP in HACMP, by checking
the Gratuitous ARP setting for that network type. See instructions in the section Changing
the Tuning Parameters to Custom Values.
11. In the Change/Show a Network in the HACMP Cluster SMIT panel, set the Enable IP
Address Takeover via IP Aliases field to Yes for this network.
12. Synchronize the cluster. HACMP verifies the configuration.
13. Restart the cluster services.
For more information on the SP Switch considerations, see Chapter 3: Planning Cluster
Network Connectivity in the Planning Guide.
For more information on IPAT via IP Aliases see Concepts and Facilities and Chapter 3 in the
Planning Guide.
Disabling IPAT via IP Aliases
If the network supports gratuitous ARP, you can configure the network in the HACMP cluster
to use IPAT via IP Aliases during fallover.
There are subtle differences between the operation of a network using IP aliasing and one that
does not. If you need to troubleshoot problems with external network equipment, clients, or
applications, you may want to disable IPAT via IP Aliases on the network and use IPAT via IP
Replacement instead.
To disable IPAT via IP Aliases facility for the entire network type:
1. In the SMIT Change/Show a Network in the HACMP Cluster panel, set the Enable IP
Address Takeover via IP Aliases field to No for this network.
2. Change the service IP labels to be on different subnets.
3. Press Enter to accept the changes.
4. Synchronize the cluster configuration.
Controlling Distribution Preferences for Service IP Label Aliases
To control the placement of the service IP label aliases on the cluster node physical network
interface cards, you can configure a distribution preference for the aliases of the service IP
labels that are placed under HACMP control.

Monitoring an HACMP Cluster

This chapter describes tools you can use to monitor an HACMP cluster.
You can use either ASCII SMIT or WebSMIT to configure and manage the cluster and view
interactive cluster status. Starting with HACMP 5.4, you can also use WebSMIT to navigate,
configure and view the status of the and graphical displays of the running cluster. For more
information about WebSMIT, see Chapter 2: Administering a Cluster Using WebSMIT.
Note: The default locations of log files are used in this chapter. If you
redirected any logs, check the appropriate location.
The main topics in this chapter include:
• Periodically Monitoring an HACMP Cluster
• Monitoring a Cluster with HAView
• Monitoring Clusters with Tivoli Distributed Monitoring
• Monitoring Clusters with clstat
• Monitoring Applications
• Monitoring Applications
• Displaying an Application-Centric Cluster View
• Using Resource Groups Information Commands
• Using HACMP Topology Information Commands
• Monitoring Cluster Services
• HACMP Log Files.
Periodically Monitoring an HACMP Cluster
By design, HACMP provides recovery for various failures that occur within a cluster. For
example, HACMP can compensate for a network interface failure by swapping in a standby
interface. As a result, it is possible that a component in the cluster has failed and that you are
unaware of the fact. The danger here is that, while HACMP can survive one or possibly several
failures, each failure that escapes your notice threatens a cluster’s ability to provide a highly
available environment, as the redundancy of cluster components is diminished.
To avoid this situation, you should customize your system by adding event notification to the
scripts designated to handle the various cluster events. You can specify a command that sends
you mail indicating that an event is about to happen (or that an event has just occurred), along
with information about the success or failure of the event. The mail notification system
enhances the standard event notification methods.
In addition, HACMP offers application monitoring capability that you can configure and
customize in order to monitor the health of specific applications and processes.
Monitoring an HACMP Cluster
Periodically Monitoring an HACMP Cluster
266 Administration Guide
10
Use the AIX 5L Error Notification facility to add an additional layer of high availability to an
HACMP environment. You can add notification for failures of resources for which HACMP
does not provide recovery by default. The combination of HACMP and the high availability
features built into the AIX 5L system keeps single points of failure to a minimum; the Error
Notification facility can further enhance the availability of your particular environment. See the
chapter on Configuring AIX 5L for HACMP in the Installation Guide for suggestions on
customizing error notification.
See Chapter 7: Planning for Cluster Events in the Planning Guide for detailed information on
predefined events and on customizing event handling. Also, be sure to consult your worksheets,
to document any changes you make to your system, and to periodically inspect the key cluster
components to make sure they are in full working order.
Automatic Cluster Configuration Monitoring
Verification automatically runs on one user-selectable HACMP cluster node once every 24
hours. By default, the first node in alphabetical order runs the verification at midnight. If
verification finds errors, it warns about recent configuration issues that might cause problems
at some point in the future. HACMP stores the results of the automatic monitoring on every
available cluster node in the /var/hacmp/log/clutils.log file.
If cluster verification detects some configuration errors, you are notified about the potential
problems:
• The exit status of verification is published across the cluster along with the information
about cluster verification process completion.
• Broadcast messages are sent across the cluster and displayed on stdout. These messages
inform you about detected configuration errors.
• A cluster_notify event runs on the cluster and is logged in hacmp.out (if cluster services
is running).
More detailed information is available on the node that completes cluster verification in
/var/hacmp/clverify/clverify.log file. If a failure occurs during processing, error messages and
warnings clearly indicate the node and reasons for the verification failure.
Tools for Monitoring an HACMP Cluster
HACMP supplies tools for monitoring a cluster. These are described in subsequent sections:
• The HAView utility extends Tivoli NetView services so you can monitor HACMP clusters
and cluster components from a single node. Using HAView, you can also view the full
cluster event history in the /usr/es/sbin/cluster/history/cluster.mmddyyyy file. The event
history (and other cluster status and configuration information) is accessible through Tivoli
NetView’s menu bar. For more information, see Monitoring a Cluster with HAView.
• Cluster Monitoring with Tivoli allows you to monitor clusters and cluster components
and perform cluster administration tasks through your Tivoli Framework console. For more
information, see Monitoring Clusters with Tivoli Distributed Monitoring.
• clstat (the /usr/es/sbin/cluster/clstat utility) reports the status of key cluster
components—the cluster itself, the nodes in the cluster, the network interfaces connected
to the nodes, the service labels, and the resource groups on each node.
Monitoring an HACMP Cluster
Periodically Monitoring an HACMP Cluster
Administration Guide 267
10
• WebSMIT displays cluster information using a slightly different layout and
organization. Cluster components are displayed along their status. Expanding the item
reveals additional information about it, including the network, interfaces and active
resource groups.
For more information, see Monitoring Clusters with clstat.
• Application Monitoring allows you to monitor specific applications and processes and
define action to take upon detection of process death or other application failures.
Application monitors can watch for the successful startup of the application, check that the
application runs successfully after the stabilization interval has passed, or monitor both the
startup and the long-running process. For more information, see Monitoring Applications.
• SMIT and WebSMIT give you information on the cluster.
You have the ability to see the cluster from an application-centric point of view.
• The HACMP Resource Group and Application Management panel in SMIT has an
option to Show Current Resource Group and Application State. The SMIT panel
Show All Resources by Node or Resource Group has an option linking you to the
Show Current Resource Group and Application State panel.
• Using the WebSMIT version lets you expand and collapse areas of the information.
Colors reflect the state of individual items (for example, green indicates online).
For more information, see Displaying an Application-Centric Cluster View.
The System Management (C-SPOC) >Manage HACMP Services > Show Cluster
Services SMIT panel shows the status of the HACMP daemons.
• The Application Availability Analysis tool measures uptime statistics for applications
with application servers defined to HACMP. For more information, see Measuring
Application Availability.
• The clRGinfo and cltopinfo commands display useful information on resource group
configuration and status and topology configuration, respectively. For more information,
see Using Resource Groups Information Commands.
• Log files allow you to track cluster events and history: The /usr/es/adm/cluster.log file
tracks cluster events; the /tmp/hacmp.out file records the output generated by
configuration scripts as they execute; the /usr/es/sbin/cluster/history/cluster.mmddyyyy
log file logs the daily cluster history; the /tmp/cspoc.log file logs the status of C-SPOC
commands executed on cluster nodes. You should also check the RSCT log files. For more
information, see HACMP Log Files.
In addition to these cluster monitoring tools, you can use the following:
• The Event Emulator provides an emulation of cluster events. For more information, see
the section on Emulating Events in the Concepts Guide.
• The Custom Remote Notification utility allows you to define a notification method
through the SMIT interface to issue a customized page in response to a cluster event. In
HACMP 5.3 and up, you can also send text messaging notification to any address including
a cell phone. For information and instructions on setting up pager notification, see the
section on Configuring a Custom Remote Notification Method in the Planning Guide.
Monitoring an HACMP Cluster
Monitoring a Cluster with HAView
268 Administration Guide
10
Monitoring a Cluster with HAView
HAView is a cluster monitoring utility that allows you to monitor HACMP clusters using
NetView for UNIX. Using Tivoli NetView, you can monitor clusters and cluster components
across a network from a single management station.
HAView creates and registers Tivoli NetView objects that represent clusters and cluster
components. It also creates submaps that present information about the state of all nodes,
networks, network interfaces, and resource groups associated with a particular cluster. This
cluster status and configuration information is accessible through Tivoli NetView’s menu bar.
HAView monitors cluster status using the Simple Network Management Protocol (SNMP). It
combines periodic polling and event notification through traps to retrieve cluster topology and
state changes from the HACMP management agent, the Cluster Manager.
You can view cluster event history using the HACMP Event Browser and node event history
using the Cluster Event Log. Both browsers can be accessed from the Tivoli NetView menu bar.
The /usr/es/sbin/cluster/history/cluster.mmddyyyy file contains more specific event history.
This information is helpful for diagnosing and troubleshooting fallover situations. For more
information about this log file, see Chapter 2: Using Cluster Log Files in the
Troubleshooting Guide.
HAView Installation Requirements
HAView has a client/server architecture. You must install both an HAView server image and
an HAView client image, on the same machine or on separate server and client machines. For
information about installation requirements, see Installation Guide.
HAView File Modification Considerations
Certain files need to be modified in order for HAViewto monitor your cluster properly. When
configuring HAView, you should check and edit the following files:
• haview_start
• clhost
• snmpd.conf or snmpdv3.conf
haview_start File
You must edit the haview_start file so that it includes the name of the node that has the
HAView server executable installed. This is how the HAView client knows where the HAView
server is located. Regardless of whether the HAView server and client are on the same node or
different nodes, you are required to specify the HAView server node in the haview_start file.
The haview_start file is loaded when the HAView client is installed and is stored in
/usr/haview. Initially, the haview_start file contains only the following line:
"${HAVIEW_CLIENT:-/usr/haview/haview_client}" $SERVER
You must add the following line to the file:
SERVER="${SERVER:-}"
Monitoring an HACMP Cluster
Monitoring a Cluster with HAView
Administration Guide 269
10
For example, if the HAView server is installed on mynode, the edited haview_start file appears
as follows:
SERVER="${SERVER:-mynode}"
"${HAVIEW_CLIENT:-/usr/haview/haview_client}" $SERVER
where mynode is the node that contains the HAView server executable.
Note: If you have configured a persistent node IP label on a node on a
network in your cluster, it maintains a persistent “node address” on the
node – this address can be used in the haview_start file.
clhosts File
HAView monitors a cluster’s state within a network topology based on cluster-specific
information in the /usr/es/sbin/cluster/etc/clhosts file. The clhosts file must be present on the
Tivoli NetView management node. Make sure this file contains the IP address or IP label of the
service and/or base interfaces of the nodes in each cluster that HAView is to monitor.
Make sure that the hostname and the service label of your Tivoli NetView nodes are exactly the
same. (If they are not the same, add an alias in the /etc/hosts file to resolve the name difference.)
WARNING: If an invalid IP address exists in the clhosts file, HAView will
fail to monitor the cluster. Make sure the IP addresses are valid,
and there are no extraneous characters in the clhosts file.
snmpd.conf File
The Tivoli NetView management node must also be configured in the list of trap destinations
in the snmpd.conf files on the cluster nodes of all clusters you want it to monitor. This makes
it possible for HAView to utilize traps in order to reflect cluster state changes in the submap in
a timely manner. Also, HAView can discover clusters not specified in the clhosts file on the
nodes in another cluster.
Note: The default version of the snmpd.conf file for AIX 5L v.5.2 and
AIX 5L v. 5.3 is snmpdv3.conf.
The format for configuring trap destinations is as follows:
trap 1.2.3 fe
For example, enter:
trap public 140.186.131.121 1.2.3 fe
Note the following:
• You can specify the name of the management node instead of the IP address.
• You can include multiple trap lines in the snmpd.conf file.
Note: HACMP now supports a SNMP Community Name other than
“public.” If the default SNMP Community Name has been changed in
/etc/snmpd.conf to something different from the default of “public”
HACMP will function correctly. The SNMP Community Name used
by HACMP will be the first name found that is not “private” or
“system” using the lssrc -ls snmpd command.
Monitoring an HACMP Cluster
Monitoring a Cluster with HAView
270 Administration Guide
10
Clinfo will also get the SNMP Community Name in the same manner.
Clinfo will still support the -c option for specifying SNMP
Community Name but its use is not required. The use of the -c option
is considered a security risk because doing a ps command could find
the SNMP Community Name. If it is important to keep the SNMP
Community Name protected, change permissions on
/tmp/hacmp.out, /etc/snmpd.conf, /smit.log and
/usr/tmp/snmpd.log to not be world-readable.
See the AIX documentation for full information on the snmpd.conf file. Version 3 has
some differences from Version 1.
Tivoli NetView Hostname Requirements for HAView
The following hostname requirements apply to using HAView in a Tivoli NetView
environment. If you change the hostname of a network interface, the Tivoli NetView database
daemons and the default map are affected.
Hostname Effect on the Tivoli NetView Daemon
The hostname required to start Tivoli NetView daemons must be associated with a valid
interface name or else Tivoli NetView fails to start.
Hostname Effect on the Tivoli NetView Default Map
If you change the hostname of the Tivoli NetView client, the new hostname does not match the
original hostname referenced in the Tivoli NetView default map database and Tivoli NetView
will not open the default map. Using the Tivoli NetView mapadmin command, you need to
update the default map (or an invalid map) to match the new hostname.
See the Tivoli NetView Administrator’s Guide for more information about updating or deleting
an invalid Tivoli NetView map.
Starting HAView
Once you have installed the HAView client and server, HAView is started and stopped when
you start or stop Tivoli NetView. However, before starting Tivoli NetView/HAView, check the
management node as follows:
• Make sure both client and server components of HAView are installed. See the installation
or migration chapters in the Installation Guide for more information.
• Make sure access control has been granted to remote nodes by running the xhost command
with the plus sign (+) or with specified nodes:
xhost + (to grant access to all computers)
or, to grant access to specific nodes only:
xhost
• Make sure the DISPLAY variable has been set to the monitoring node and to a label that
can be resolved by and contacted from remote nodes:
export DISPLAY=:0.0
Monitoring an HACMP Cluster
Monitoring a Cluster with HAView
Administration Guide 271
10
These actions allow you to access HACMP SMIT panels using the HAView Cluster
Administration option.
After ensuring these conditions are set, type the following to start Tivoli NetView:
/usr/OV/bin/nv6000
Refer to the Tivoli NetView User’s Guide for Beginners for further instructions about starting
Tivoli NetView.
When Tivoli NetView starts, HAView creates objects and symbols to represent a cluster and its
components. Through submaps, you can view detailed information about these components.
HAView places the Clusters symbol (shown below) on the Tivoli NetView map after Tivoli
NetView starts. The Clusters symbol is added to the Netview Root map and is placed alongside
the Tivoli NetView Collections symbol and other symbols:
HAView Clusters Symbol
Viewing Clusters and Components
To see which clusters HAView is currently monitoring, double-click the Clusters symbol. The
Clusters submap appears. You may see one or more symbols that represent specific clusters.
Each symbol is identified by a label indicating the cluster’s name. Double-click a cluster
symbol to display symbols for nodes, networks, and resource groups within that cluster.
Note that the cluster status symbol may remain unknown until the next polling cycle, even
though the status of its cluster components is known. See Customizing HAView Polling
Intervals for more information about the default intervals and how to change them using SMIT.
You can view component details at any time using the shortcut ctrl-o. See Obtaining
Component Details in HAView for information and instructions.
Read-Write and Read-Only NetView Maps
Normally, you have one master monitoring station for Tivoli NetView/HAView. This station
is supplied with new information as cluster events occur, and its map is updated so it always
reflects the current cluster status.
In normal cluster monitoring operations, you will probably not need to open multiple Tivoli
NetView stations on the same node. If you do, and you want the additional stations to be
updated with current cluster status information, you must be sure they use separate maps with
different map names. For more information on multiple maps and changingmap permissions,
see the Tivoli NetView Administrator’s Guide.

Configuring Volume Groups, Logical Volumes, and Filesystems as

You define volume groups, logical volumes, and filesystems in AIX 5L and then configure
them as resources for HACMP. Plan and note them on the worksheets before configuring to
HACMP. For more information, see Chapter 5 in the Installation Guide and Chapter 11:
Managing Shared LVM Components in this guide.
Server Name Enter an ASCII text string that identifies the server. You
will use this name to refer to the application server when
you define resources during node configuration. The server
name can include alphabetic and numeric characters and
underscores. Use no more than 64 characters.
Start Script Enter the pathname of the script (followed by arguments)
called by the cluster event scripts to start the application
server. (Maximum 256 characters.) This script must be in
the same location on each cluster node that might start the
server. The contents of the script, however, may differ.
Stop Script Enter the pathname of the script called by the cluster event
scripts to stop the server. (Maximum 256 characters.) This
script must be in the same location on each cluster node
that may start the server. The contents of the script,
however, may differ.
Configuring HACMP Cluster Topology and Resources (Extended)
Configuring HACMP Resources (Extended)
102 Administration Guide
4
Configuring Concurrent Volume Groups, Logical Volumes, and
Filesystems as Resources
Concurrent volume groups, logical volumes, and filesystems must be defined in AIX 5L and
then configured as resources for HACMP. They should be planned and noted on the worksheets
before configuring to HACMP. See Chapter 5: Planning Shared LVM Components in the
Planning Guide and Chapter 12: Managing Shared LVM Components in a Concurrent
Access Environment for information in this guide.
Configuring Multiple Application Monitors
HACMP can monitor specified applications using application monitors. These application
monitors can:
• Check if an application is running before HACMP starts it.
• Watch for the successful startup of the application.
• Check that the application runs successfully after the stabilization interval has passed.
• Monitor both the startup and the long-running process.
• Automatically take action to restart applications upon detecting process termination or
other application failures.
In HACMP 5.2 and up, you can configure multiple application monitors and associate them
with one or more application servers.
By supporting multiple monitors per application, HACMP can support more complex
configurations. For example, you can configure one monitor for each instance of an Oracle
parallel server in use. Or, you can configure a custom monitor to check the health of the
database along with a process termination monitor to instantly detect termination of the
database process.
Note: If a monitored application is under control of the system resource
controller, ensure that action:multi are -O and -Q. The -O
specifies that the subsystem is not restarted if it stops abnormally. The
-Q specifies that multiple instances of the subsystem are not allowed
to run at the same time. These values can be checked using the
following command:
lssrc -Ss Subsystem | cut -d : -f 10,11
If the values are not -O and -Q, change them using the chssys command.
Process and Custom Monitoring
You can select either of two application monitoring methods:
• Process application monitoring detects the termination of one or more processes of an
application, using RSCT Resource Monitoring and Control.
• Custom application monitoring checks the health of an application with a custom monitor
method at user-specified polling intervals.
Configuring HACMP Cluster Topology and Resources (Extended)
Configuring HACMP Resources (Extended)
Administration Guide 103
4
Process monitoring is easier to set up, as it uses the built-in monitoring capability provided by
RSCT and requires no custom scripts. However, process monitoring may not be an appropriate
option for all applications. Custom monitoring can monitor more subtle aspects of an
application’s performance and is more customizable, but it takes more planning, as you must
create the custom scripts.
Fallover and Notify Actions
In both process and custom monitoring methods, when the monitor detects a problem, HACMP
attempts to restart the application on the current node and continues the attempts until the
specified retry count has been exhausted.
When an application cannot be restarted within the retry count, HACMP takes one of two
actions, which you specified when configuring the application monitor:
• Choosing fallover causes the resource group containing the application to fall over to the
node with the next highest priority according to the nodelist. (See Note on the Fallover
Option and Resource Group Availability for more information.)
• Choosing notify causes HACMP to generate a server_down event, which informs the
cluster of the failure.
Monitor Modes
When you configure process monitor(s) and custom monitor(s) for the application server, you
can also specify the mode in which the application monitor is used:
• Startup Monitoring Mode. In this mode, the monitor checks the application server’s
successful startup within the specified stabilization interval and exits after the stabilization
period expires. The monitor in the startup mode may run more than once, but it always runs
during the time specified by the stabilization interval value in SMIT. If the monitor returns
within the stabilization interval, its zero return code indicates that the application had
successfully started. If the monitor returns a non-zero code within the stabilization interval,
this is interpreted as a failure of the application to start.
Use this mode for applications in parent resource groups. If you configure dependencies
between resource groups in the cluster, the applications in these resource groups are started
sequentially as well. To ensure that this process goes smoothly, we recommend configuring
several application monitors, and, especially, a monitor that checks the application startup
for the application that is included in the parent resource group. This ensures that the
application in the parent resource group starts successfully.
• Long-Running Mode. In this mode, the monitor periodically checks that the application is
running successfully. The checking begins after the stabilization interval expires and it is
assumed that the application server is started and the cluster has stabilized. The monitor in
the long-running mode runs at multiple intervals based on the monitoring interval value that
you specify in SMIT.
Configure a monitor in this mode for any application server. For example, applications
included in child and parent resource groups can use this mode of monitoring.
• Both. In this mode, the monitor checks for the successful startup of the application server
and periodically checks that the application is running successfully.
Configuring HACMP Cluster Topology and Resources (Extended)
Configuring HACMP Resources (Extended)
104 Administration Guide
4
Retry Count and Restart Interval
The restart behavior depends on two parameters, the retry count and the restart interval, that
you configure in SMIT.
• Retry count. The retry count specifies how many times HACMP should try restarting before
considering the application failed and taking subsequent fallover or notify action.
• Restart interval. The restart interval dictates the number of seconds that the restarted
application must remain stable before the retry count is reset to zero, thus completing the
monitor activity until the next failure occurs.
Note: Do not specify both of these parameters if you are creating an
application monitor that will only be used as in a startup monitoring
mode.
If the application successfully starts up before the retry count is exhausted, the restart interval
comes into play. By resetting the restart count, it prevents unnecessary fallover action that could
occur when applications fail several times over an extended time period. For example, a
monitored application with a restart count set to three (the default) could fail to restart twice,
and then successfully start and run cleanly for a week before failing again. This third failure
should be counted as a new failure with three new restart attempts before invoking the fallover
policy. The restart interval, set properly, would ensure the correct behavior: it would have reset
the count to zero when the application was successfully started and found in a stable state after
the earlier failure.
Be careful not to set the restart interval for a too short period of time. If the time period is too
short, the count could be reset to zero too soon, before the immediate next failure, and the
fallover or notify activity will never occur.
See the instructions for setting the retry count and restart intervals later in this chapter for
additional details.
Application Monitoring Prerequisites and Considerations
Keep the following in mind when planning and configuring application monitoring:
• Any application to be monitored must be defined to an application server in an existing
cluster resource group.
• If you have configured dependent resource groups, we recommend to configure multiple
monitors: for applications included in parent resource groups, and for applications in child
resource groups. For example, a monitor for a parent resource group can monitor the
successful startup of the application, and a monitor for a child resource group can monitor
the process for an application. For more information, see Monitor Modes.
• Multiple monitors can be configured for the same application server. Each monitor can be
assigned a unique name in SMIT.
• The monitors that you configure must conform to existing configuration rules. For more
information, see Configuring a Process Application Monitor and Configuring a Custom
Application Monitor.
Configuring HACMP Cluster Topology and Resources (Extended)
Configuring HACMP Resources (Extended)
Administration Guide 105
4
• We recommend that you first configure an application server, and then configure the
monitor(s) that you can associate with the application server. Before configuring an
application monitor, configure all your application servers. Then configure the monitors
and associate them with the servers. You can go back at any time and change the association
of monitors to servers.
• You can configure no more than 128 monitors per cluster. No limit exists on the number of
monitors per application server, as long as the total number of all monitors in the cluster is
less than 128.
• When multiple monitors are configured that use different fallover policies, each monitor
specifies a failure action of either “notify” or “fallover”. HACMP processes actions in the
order in which the monitors indicate an error. Forexample, if two monitors are configured
for an application server and one monitor uses the “notify” method and the other uses the
“fallover” method, the following occurs:
• If a monitor with “fallover” action indicates an error first, HACMP moves the
resource group to another node, and the remaining monitor(s) are shut down and
restarted on another node. HACMP takes no actions specified in any other monitor.
• If a monitor with “notify” action indicates an error first, HACMP runs the “notify”
method and shuts down that monitor, but any remaining monitors continue to
operate as before. You can manually restart the “notify” monitor on that node using
the Suspend/Resume Application Monitoring SMIT panel.
• If multiple monitors are used, HACMP does not use a particular order for the monitors
startup or shutdown. All monitors for an application server are started at the same time. If
two monitors are configured with different fallover policies, and they fail at precisely the
same time, HACMP does not guarantee it processes methods specified for one monitor
before methods for the other.
• The same monitor can be associated with multiple application servers using the
Application Monitor(s) field in the Change/Show an Application Server SMIT panel.
You can select a monitor from the picklist.
• If you remove an application monitor, HACMP removes it from the server definition for all
application servers that were using the monitor, and indicates which servers are no longer
using the monitor.
• If you remove an application server, HACMP removes that server from the definition of all
application monitors that were configured to monitor the application. HACMP also sends
a message about which monitor will no longer be used for the application. If you remove
the last application server in use for any particular monitor, that is, if the monitor will no
longer be used for any application, verification issues a warning that the monitor will no
longer be used.
Note on the Fallover Option and Resource Group Availability
Be aware that if you select the fallover option of application monitoring in the Customize
Resource Recovery SMIT panel—which could cause a resource group to migrate from its
original node—the possibility exists that while the highest priority node is up, the resource
group remains inactive. This situation occurs when an rg_move event moves a resource group
from its highest priority node to a lower priority node, and then you stop the cluster services on
the lower priority node with the option to take all the resources offline. Unless you bring the
resource group up manually, it remains in an inactive state.

Configuring an HACMP Cluster

This chapter describes how to configure an HACMP cluster using the SMIT Initialization and
Standard Configuration path.
Have your planning worksheets ready to help you through the configuration process. See the
Planning Guide for details if you have not completed this step.
The main sections in this chapter include:
• Overview
• Configuring a Two-Node Cluster, or Using Smart Assists
• Defining HACMP Cluster Topology (Standard)
• Configuring HACMP Resources (Standard)
• Configuring HACMP Resource Groups (Standard)
• Configuring Resources in Resource Groups (Standard)
• Verifying and Synchronizing the Standard Configuration
• Viewing the HACMP Configuration
• Additional Configuration Tasks
• Testing Your Configuration.
Overview
Using the options under the SMIT Initialization and Standard Configuration menu, you can
add the basic components of a cluster to the HACMP Configuration Database (ODM) in a few
steps. This HACMP configuration path significantly automates the discovery and selection of
configuration information and chooses default behaviors.
If you are setting up a basic two-node cluster, use the Two-Node Cluster Configuration
Assistant to simplify the process for configuring a two-node cluster. For more information, see
the section on Using the Two-Node Cluster Configuration Assistant in the chapter on Creating
a Basic HACMP Cluster in the Installation Guide.
You can also use the General Configuration Smart Assist to quickly set up your application.
You are not limited to a two-node cluster with this Assist.
You can use either ASCII SMIT or WebSMIT to configure the cluster. For more information
on WebSMIT, see Chapter 2: Administering a Cluster Using WebSMIT.
Configuring an HACMP Cluster (Standard)
Overview
54 Administration Guide
3
Prerequisite Tasks for Using the Standard Path
Before using the Standard Configuration path, HACMP must be installed on all the nodes, and
connectivity must exist between the node where you are performing the configuration and all
other nodes to be included in the cluster. That is, network interfaces must be both physically
and logically configured (to AIX 5L) so that you can successfully communicate from one node
to each of the other nodes. The HACMP discovery process runs on all server nodes, not just the
local node.
Once you have configured and powered on all disks, communication devices, serial networks
and also configured communication paths to other nodes in AIX 5L, HACMP automatically
collects information about the physical and logical configuration and displays it in
corresponding SMIT picklists, to aid you in the HACMP configuration process.
With the connectivity path established, HACMP can discover cluster information and you are
able to access all of the nodes to perform any necessary AIX 5L administrative tasks. That is,
you do not need to open additional windows or physically move to other nodes' consoles, and
manually log in to each node individually. To ease this process, SMIT fastpaths to the relevant
HACMP and/or AIX 5L SMIT screens on the remote nodes are available within the HACMP
SMIT screen paths.
HACMP uses all interfaces defined for the connectivity paths to populate the picklists. If you
do not want a particular interface to be used in the HACMP cluster, use the HACMP Extended
Configuration to delete it from the HACMP cluster (this way, it will not be shown up in picklists
and will not be available for selection).
By default, cluster heartbeats are sent through all discovered networks. The network is kept
highly available once it has an HACMP IP label assigned and you synchronize the
configuration.
Assumptions and Defaults for the Standard Path
HACMP makes some assumptions regarding the environment, such as assuming all network
interfaces on a physical network belong to the same HACMP network. Using these
assumptions, HACMP supplies or automatically configures intelligent and default parameters
to its configuration process in SMIT. This helps to minimize the number of steps it takes to
configure the cluster.
HACMP makes the following basic assumptions:
• Hostnames are used as node names. HACMP automatically configures and monitors all
network interfaces that can send a ping command to another network interface. The
network interfaces that can send a ping command to each other without going through a
router are placed on the same logical network. HACMP names each logical network.
• HACMP uses IP aliasing as the default mechanism for binding a service IP label/address to
a network interface. For more information, see the chapter on planning the cluster networks
in the Planning Guide.
Configuring an HACMP Cluster (Standard)
Overview
Administration Guide 55
3
Note: If you cannot use IP aliases because of hardware restrictions, such
as the limited number of subnets that are allocated for cluster
utilization, you will need to use IP replacement to bind your IP
labels to network interfaces. For instance, ATM network does not
support IP aliasing. IP replacement can be configured only under
the SMIT Extended Configuration path (where you disable IP
aliasing).
• IP Address Takeover via IP Aliases is configured for any logical network capable of taking
over a service IP label as an alias. Note, in the Extended Configuration path, you can also
configure IP Address Takeover that uses the IP replacement mechanism for binding IP
labels/addresses with network interfaces.
• You can configure the resource groups with any of the policies for startup, fallover, and
fallback (without specifying fallback timer policies).
• You can configure the application server start and stop scripts, but will need to use the
Extended Configuration path to configure multiple monitors to track the health of each
application server.
Also, since you can add, change, or remove serial (non-IP) networks and devices using the
Extended Configuration path, you must manually define which pair of end-points exist in the
point-to-point network, before adding, changing or removing serial networks and devices.
To manually configure any part of the cluster, or to add more details or customization to the
cluster configuration, use the SMIT HACMP Extended Configuration path. See Chapter 4:
Configuring HACMP Cluster Topology and Resources (Extended) for information on those
options.
Note: If you are using the Standard Configuration path and information that
is required for configuration resides on remote nodes, HACMP
automatically discovers the necessary cluster information for you.
Steps for Configuring a Cluster Using the Initialization and Standard
Configuration Path
Here are the steps to configure the typical cluster components:
What You Do Description
Step 1: Configure a basic
two-node cluster, or a
cluster with WebSphere,
DB2 or Oracle
(Optional) Use the Configuration Assistants panel to
configure any of the following:
• Basic two-node cluster
• Cluster with the WebSphere application
• Cluster with the DB2 UDB instances
• Cluster with an Oracle application server and or Oracle
database instances
• Cluster with another application.
Configuring an HACMP Cluster (Standard)
Overview
56 Administration Guide
3
Step 2: Configure the
cluster topology
Identify the cluster nodes and establish communication
paths between them using the Add Nodes to an HACMP
Cluster menu options. Here you name the cluster and
select the nodes (listed in /etc/hosts) either by their names
or their IP addresses. This gives HACMP the base
knowledge it needs to communicate with the nodes that are
participating in the cluster. Once each node is properly
identified and HACMP obtains information about working
communications paths, HACMP automatically runs a
discovery operation that identifies the basic components
within the cluster.
HACMP uses the discovered hostnames as the node names
and adds them to the HACMP Configuration Database
(HACMPnode ODM). HACMP also automatically adds
the networks and the associated interfaces that share
physical connectivity with two or more nodes in the cluster
to the HACMP Configuration Database (HACMPnetwork
and HACMPadapter ODMs).
Other resource information that HACMP discovers
includes shared disk PVIDs and volume groups.
Step 3: Configure the
cluster resources
Configure the resources to be made highly available. Use
the Configure Resources to Make Highly Available
menu to configure resources that are to be shared among
the nodes in the cluster.
You can configure these resources:
• IP address/IP label
• application server (a collection of start and stop scripts
for the application that HACMP uses)
• volume groups (shared and concurrent)
• logical volumes
• filesystems.
Step 4: Configure the
resource groups
Use the Configure HACMP Resource Groups menu to
create the resource groups you have planned for each set of
related resources. You can configure startup, fallover and
fallback policy for each resource group (without specifying
fallback timer policies).
Step 5: Put the resources to
be managed together into
their respective resource
groups
Use the Configure HACMP Resource Groups >
Change/Show Resources for a Resource Group
(standard) menu to assign resources to each resource
group.
What You Do Description
Configuring an HACMP Cluster (Standard)
Overview
Administration Guide 57
3
Step 6: Adjust log viewing
and management
(Optional) Adjust log viewing and management (settings
for the debug level and hacmp.out log file formatting
options per node).
Step 7: Verify and
synchronize the cluster
configuration
Use the Verify and Synchronize HACMP Configuration
menu to guarantee the desired configuration is feasible
given the physical connections and devices, and ensure that
all nodes in the cluster have the same view of the
configuration.
Step 8: Display the cluster
configuration
(Optional) Use the Display HACMP Configuration menu
to view the cluster topology and resources configuration.
Step 9: Make further
additions or adjustments to
the cluster configuration
(Optional) You may want to use some options available on
the Extended Configuration path. Such additions or
adjustments include, for example:
• Adding non-IP networks for heartbeating
• Configuring and changing the distribution preference
for service IP aliases
• Adding other resources to the cluster, such as SNA
communication interfaces and links or tape resources
• Configuring resource group runtime policies, including
Workload Manager
Adding resource group timers
• Configuring dependencies between resource groups
• Adding multiple application monitors for an application
server
• Configuring HACMP File Collections
• Configuring cluster security
• Customizing remote notifications (pager, SMS
messages, and email)
• Customizing cluster events
• Configuring site policies.
Step 10: Test the cluster
before it goes into the
production environment
(Recommended) Use the HACMP Cluster Test Tool to
test recovery procedures for the cluster.
What You Do Description
Configuring an HACMP Cluster (Standard)
Configuring a Two-Node Cluster, or Using Smart Assists
58 Administration Guide
3
Configuring a Two-Node Cluster, or Using Smart Assists
You can configure a basic two-node cluster with just a few configuration steps. For information,
see the section on Using the Two-Node Cluster Configuration Assistant in the chapter on
Creating a Basic HACMP Cluster in the Installation Guide.
If you are configuring a WebSphere, DB2 UDB or Oracle application, see the corresponding
HACMP Smart Assist guide.
To configure other applications, you can use the General Configuration Smart Assist.
Limitations and Prerequisites
The initial requirements for using Smart Assists are:
• The application must be installed on all cluster nodes where you want to run it.
• The Smart Assist must be installed on all cluster nodes that run the application.
Configuring Applications with the General Configuration Smart Assist
To configure your installed application (other than DB2, WebSphere, or Oracle):
1. On a local node, enter smitty hacmp
2. Select Initialization and Standard Configuration > Configuration Assistants > Make
Applications Highly Available > Add an Application to the HACMP Configuration
and press Enter.
If the cluster is not yet configured, you are directed to go to the Configure HACMP Nodes
and Cluster SMIT panel. Here you need to list the communication paths to all nodes in the
cluster. Then continue to the next step.
3. If the cluster is configured, SMIT displays a list of applications installed on this node.
Select Other Applications and press Enter.
4. Select General Application Smart Assist and press Enter.
5. Enter values for the following fields on the Add an Application to HACMP panel:
• Application Server Name
• Primary Node
• Takeover Nodes
• Application Server Start Script
• Application Server Stop Script
• Service IP Label.
6. Press Enter after you have filled in the values. The configuration will be synchronized and
verified automatically.
7. (Optional) Return to the panel Make Applications Highly Available to select Test the
HACMP Configuration and press Enter.
The Cluster Test Tool runs and displays results to the screen. If you get error messages,
make the necessary corrections.
Configuring an HACMP Cluster (Standard)
Defining HACMP Cluster Topology (Standard)
Administration Guide 59
3
Defining HACMP Cluster Topology (Standard)
Complete the following procedures to define the cluster topology. You only need to perform
these steps on one node. When you verify and synchronize the cluster topology, its definition
is copied to the other nodes.
To configure the cluster topology:
1. Enter smit hacmp
2. In SMIT, select Initialization and Standard Configuration > Configure an HACMP
Cluster and Nodes and press Enter.
3. Enter field values as follows:
4. Press Enter. Once communication paths are established, HACMP runs the discovery
operation and prints results to the SMIT panel.
5. Verify that the results are reasonable for your cluster.
6. Return to the top level HACMP SMIT panel to continue with the configuration.
Cluster Name Enter an ASCII text string that identifies the cluster. The
cluster name can include alphanumeric characters and
underscores, but cannot have a leading numeric. Use no
more than 32 characters. It can be different from the
hostname. Do not use reserved names. For a list of reserved
names see Chapter 7: Verifying and Synchronizing an
HACMP Cluster.
New nodes (via selected
communication paths
Enter (or add) one resolvable IP label (this may be the
hostname), IP address, or Fully Qualified Domain Name for
each new node in the cluster, separated by spaces. HACMP
uses this path to initiate communication with the node.
Example 1:
10.11.12.13 NodeC.ibm.com.
Example 2:
NodeANodeB
(where these are hostnames.)
The picklist displays the hostnames and/or addresses
included in /etc/hosts that are not already
HACMP-configured IP labels/addresses.
You can add node names or IP addresses in any order.
Currently configured
node(s)
If nodes are already configured, they are displayed here.
Configuring an HACMP Cluster (Standard)
Configuring HACMP Resources (Standard)
60 Administration Guide
3
Configuring HACMP Resources (Standard)
Using the Standard Configuration path, you can configure the types of resources that are most
often included in HACMP clusters. You must first define resources that may be used by
HACMP to the AIX 5L operating system on each node. Later you group together the associated
resources in resource groups. You can add them all at once or add them separately, as you
prefer.
This section shows how to configure the following components on all of the nodes defined to
the cluster using a single network interface:
• Application servers (a collection of start and stop scripts that HACMP uses for the
application).
• HACMP service IP labels/addresses. The service IP label/address is the IP label/address
over which services are provided and which is kept highly available by HACMP.
• Shared volume groups, logical volumes, and filesystems.
• Concurrent volume groups, logical volumes, and filesystems.
Configuring Application Servers
An HACMP application server is a cluster resource used to control an application that must be
kept highly available. It contains application start and stop scripts. Configuring an application
server does the following:
• Associates a meaningful name with the server application. For example, you could give the
application you are using with the HACMP software a name such as apserv. You then use
this name to refer to the application server when you define it as a resource. When you set
up the resource group that contains this resource, you define an application server as a
resource.
• Points the cluster event scripts to the scripts that they call to start and stop the server
application.
• Allows you to configure application monitoring for that application server. In HACMP 5.2
and up you can configure multiple application monitors for one application server. For
more information see Steps for Configuring Multiple Application Monitors in Chapter
4: Configuring HACMP Cluster Topology and Resources (Extended).
Note that this section does not discuss how to write the start and stop scripts. See the vendor
documentation for specific product information on starting and stopping a particular
application.
Ensure that the server scripts exist on all nodes that participate as possible owners of the
resource group where this application server resides.
To configure an application server on any cluster node:
1. Enter smit hacmp
2. In SMIT, select Initialization and Standard Configuration > Configure Resources to
Make Highly Available > Configure Application Servers > Add an Application Server
and press Enter.