How To Debug CCI Issues

Content

How To Debug CCI Issues – Version 1.3

Introduction

This short document describes some of the many ways CCI can break. It should be used as a list of the things to check with a customer whenever CCI problems are raised.

Check: This tag is used to highlight what to look for.

Installation and Environment Variables

UNIX

You would think it is easy to get this right – but we see many errors caused due to bad installation. Here is what should be done. Let us assume a non-default installation – as that is the easiest for the user to get wrong J

Let us assume we want to install CCI in /opt/HORCM

Copy the installation file to the hard disk somewhere. It is called RMHORC. Let’s copy it to /var/tmp
cd /opt
Change directory to where you want /HORCM created
cpio -idmu < /var/tmp/RMHORC
This will copy all the files in the RMHORC “package” to /opt/HORCM

Check: We have seen cases where users build their own installation “packages” for HORCM – and then copy the files from one Host to another. In UNIX particularly this is dangerous. CCI needs a “hidden” directory called .uds or it will not start.

In CCI 01-16-03 and below it was in /var/tmp
In CCI 01-17-03 and above it is in /yourdirectory/HORCM

This directory contains UNIX “pipes” when the instances are started. The “pipes” are deleted when the instance stops. Thus, you will see this:

root@SYD-E250-1:/opt/HORCM/.uds#:ls -al

total 4

drwxrwxrwx 2 root sys 512 Feb 22 15:26 .

dr-xr-xr-x 12 root sys 512 Feb 22 15:04 ..

root@SYD-E250-1:/opt/HORCM/.uds#:horcmstart.sh 4

starting HORCM inst 4

HORCM inst 4 starts successfully.

root@SYD-E250-1:/opt/HORCM/.uds#:ls -al

total 6

drwxrwxrwx 3 root sys 512 Feb 22 16:29 .

dr-xr-xr-x 12 root sys 512 Feb 22 15:04 ..

drwxrwxrwx 2 root other 512 Feb 22 16:29 .lcmcl04

srwxrwxrwx 1 root other 0 Feb 22 16:29 .lcmep04

Always check for this directory if you have a case where CCI does not start.

4 ln -s /opt/HORCM /HORCM

Check: You must create a link or the install in the next step will fail.

5 /HORCM/horcminstall.sh

Check: You must do this on UNIX to create links to the CCI commands.

6 raidqry -h

Here is what you see if the user has done everything right.

root@SYD-E250-1:/opt/HORCM/.uds#:raidqry -h

Model : RAID-Manager/Solaris

Ver&Rev: 01-19-03/04

Usage : raidqry [options] for HORC

-h Help/Usage

-I[#] Set to HORCMINST#

-IH[#] or -ITC[#] Set to HORC mode [and HORCMINST#]

-IM[#] or -ISI[#] Set to MRCF mode [and HORCMINST#]

-z Set to the interactive mode

-zx Set to the interactive mode and HORCM monitoring

-q Quit(Return to main())

-g Specify for getting all group name on local

-l Specify the local query

-r <group> Specify the remote query

-f Specify display for floatable host

Check: Always get the user to run this command and send you the output.

If user is using an old version of CCI, ask why. 01-19-03/04 or later is preferred for many reasons which will be discussed later.
In some special cases, like HPtM, a specific level of CCI may be stated in the ECN or Release Notes. In this case, it may be advisable to stick with that level. Also, the microcode ECNs for 9900V and USP always recommend a CCI level.
However, in my experience, CCI is always backwards compatible – and the developer has confirmed this – so one should always use the minimum level stated in ECNs.
Any command this user issues will be assumed to be TrueCopy (refer above - HORC). If the user is trying to perform ShadowImage operation, you now know why it is failing J
No instance has been set. Here is what you see if the instance variable has been set:

root@SYD-E250-1:/opt/HORCM/.uds#:export HORCMINST=4

root@SYD-E250-1:/opt/HORCM/.uds#:raidqry -h

Model : RAID-Manager/Solaris

Ver&Rev: 01-19-03/04

Usage : raidqry [options] for HORC[4]

-h Help/Usage

If the user is trying to control Instance 1, now you know why it is failing J

Windows

Installation is easier. Double click the EXE and follow the bouncing ball. I always recommend taking the default of C:\HORCM. The directory is only about 10 MB in size so we are not likely to fill the drive.

Check: The same rule applies as for UNIX. Always ask for raidqry output.

Environment Variables

Here is a ShadowImage example.

C:\HORCM\ETC>set horcminst=4

C:\HORCM\ETC>set horcc_mrcf=1

C:\HORCM\ETC>raidqry -h

Model : RAID-Manager/WindowsNT

Ver&Rev: 01-19-03/04

Usage : raidqry [options] for HOMRCF[4]

-h Help/Usage

-I[#] Set to HORCMINST#

-IH[#] or -ITC[#] Set to HORC mode [and HORCMINST#]

-IM[#] or -ISI[#] Set to MRCF mode [and HORCMINST#]

-z Set to the interactive mode

Check: horcminst is case insensitive on windows – it is case sensitive on UNIX

Change mode of operation to TrueCopy

C:\HORCM\ETC>set horcc_mrcf=

C:\HORCM\ETC>raidqry -h

Model : RAID-Manager/WindowsNT

Ver&Rev: 01-19-03/04

Usage : raidqry [options] for HORC[4]

Now try this on UNIX. Change mode of operation:

root@SYD-E250-1:/opt/HORCM/.uds#:export HORCC_MRCF=1

root@SYD-E250-1:/opt/HORCM/.uds#:raidqry -h

Model : RAID-Manager/Solaris

Ver&Rev: 01-19-03/04

Usage : raidqry [options] for HOMRCF[4]

root@SYD-E250-1:/opt/HORCM/.uds#:export HORCC_MRCF=

root@SYD-E250-1:/opt/HORCM/.uds#:raidqry -h

Model : RAID-Manager/Solaris

Ver&Rev: 01-19-03/04

Usage : raidqry [options] for HOMRCF[4]

Check: This does not work. You must do this.

root@SYD-E250-1:/opt/HORCM/.uds#:unset HORCC_MRCF

root@SYD-E250-1:/opt/HORCM/.uds#:raidqry -h

Model : RAID-Manager/Solaris

Ver&Rev: 01-19-03/04

Usage : raidqry [options] for HORC[4]

Sending logs to GSC

If you have to escalate the problem to GSC, we will need the complete set of HORCM logs and all the HORCM CONF files. In general, the preferred method of doing this is to run “getconfig”. These executables/scripts are available on the TUF Knowledge area here.

If for any reason you do not run these scripts, you must zip up all the LOG directories underneath the HORCM directory. Never pick and choose which log to upload – many of them have the same name – and GSC may need to refer to all of them!

In addition, the factory always asks for the output from these commands (Windows only):

inqraid -CLI -fgx $Phys

inqraid -CLI -fgvx $Vol

inqraid -CLI -fgx $LETALL

Finding Command Devices

You cannot create a HORCM CONF file or check it for accuracy without doing INQRAID commands for UNIX/Windows and RAIDSCAN commands for Windows.

UNIX

Check: Get the user to send you the result of this command:

root@SYD-E250-1:/opt/HORCM/.uds#:ls /dev/rdsk/* | inqraid -CLI -fxg

DEVICE_FILE PORT SERIAL LDEV CTG H/M/12 SSID R:Group PRODUCT_ID

...

c2t0d16s2 CL1-A-4 10111 0 - - - - OPEN-V-CM

c2t2d36s2 - - - - - - - -

c2t4d0s2 CL1-A-20 20169 43 - - - - OPEN-V-CM

c2t6d0s2 CL1-A-11 80025 31F - - - - OPEN-V-CM

c2t6d14s2 - - - - - - - -

c3t2d128s2 CL2-A-6 10262 2180 - - - - OPEN-V-CM

c3t3d4s2 CL2-A-7 3157 1 - - - - DF600F-CM

...

Here are 5 command devices – 2 are in bold. The first is a USP – the second is a 9570V. If the user wants to use the first one, they need to code:

/dev/rdsk/c2t6d0s2

in the HORCM CONF file.

Windows

Check: Get the user to send you the result of these commands:

C:\HORCM\ETC>raidscan -x findcmddev h0,20

cmddev of Ser# 10111 = \\.\PhysicalDrive2

cmddev of Ser# 10111 = \\.\PhysicalDrive5

cmddev of Ser# 41 = \\.\PhysicalDrive7

cmddev of Ser# 10262 = \\.\PhysicalDrive8

cmddev of Ser# 80025 = \\.\PhysicalDrive10

cmddev of Ser# 20169 = \\.\PhysicalDrive11

cmddev of Ser# 20169 = \\.\Volume{3c107ab6-7dbf-11db-a1ed-000e0c6abf1d}

Check: Do not use ANY of these names! If you find a user using this syntax, ask that it be changed J See INQRAID output below:

Harddisk numbers can change after a reboot.
GUID numbers can change in a MS Cluster environment after reboot
Do yourself a favour – do not use these names

C:\HORCM\ETC>inqraid $Phys -CLI

DEVICE_FILE PORT SERIAL LDEV CTG H/M/12 SSID R:Group PRODUCT_ID

Harddisk0 - - - - - - - 00JS-22MHB0

Harddisk1 - - - - - - - 00JS-22MHB0

Harddisk2 CL1-A 10111 515 - - - - OPEN-V-CM

Harddisk3 CL1-A 10111 1920 - s/s/ss 9997 5:03-02 OPEN-V

Harddisk4 CL1-A 10111 768 - s/s/ss 9993 5:06-02 OPEN-V

Harddisk5 CL1-A 10111 1856 - - - - OPEN-V-CM

Harddisk6 CL1-A 10111 2632 - P/s/ss 999A 5:06-02 OPEN-V

Harddisk7 CL1-A 41 0 - - - - DF600F-CM

Harddisk8 CL1-A 10262 16 - - - - OPEN-V-CM

Harddisk9 CL1-A 10262 8320 - s/s/ss 2000 5:01-05 OPEN-V

Harddisk10 CL1-A 80025 784 - - - - OPEN-V-CM

Harddisk11 CL1-A 20169 13 - - - - OPEN-V-CM

\\.\CMD syntax has been around since 01-17-03/05. There is no reason not to use it! If the user is running 01-17-03/05 or below, get them to use 01-19-03/04 or higher.

In this case, for Harddisk8/USP 10262, the correct syntax in the HORCM CONF file is:

\\.\CMD-10262-16 or even

\\.\CMD-10262-16-CL1-A-12 if you know this is HSD 12 - or

\\.\CMD-10262-16-CL1-A or, for slack people,

\\.\CMD-10262 J

\\.\CMD-10262-16 is my preferred coding technique – as this takes care of multipath environments as well.

Starting HORCM Instances

There are so many ways for this to fail that I could write a book on this topic J

So, always take the easy way out. Send the user a deck that is bound to work. If it does not, then you have very little to debug. Here is such a deck – HORCM4.CONF:

UNIX

HORCM_MON

#ip_address service poll(10ms) timeout(10ms)

10.129.2.53 11004 1000 3000

HORCM_CMD

#dev_name dev_name dev_name dev_name

/dev/rdsk/c2t6d0s2

HORCM_DEV

#dev_group dev_name port# TargetID LU# MU#

HORCM_INST

#dev_group ip_address service

There are only 3 things to check:

Is the IP address correct? Note: You can use “localhost” here, but this will not work for TC environments using 2 different CCI servers.

Is 11004 a “free” UDP port? Almost certainly it is.
Is the CMDDEV right? You can tell that from the commands we have already issued.

UNIX HORCM CONF files are kept in /etc

Windows

Here is HORCM8.CONF for Windows:

HORCM_MON

#ip_address service poll(10ms) timeout(10ms)

10.129.3.127 11008 1000 3000

HORCM_CMD

#dev_name dev_name dev_name dev_name

\\.\CMD-10262-16

HORCM_DEV

#dev_group dev_name port# TargetID LU# MU#

HORCM_INST

#dev_group ip_address service

Use the same logic as for UNIX. Windows HORCM CONF files are in C:\WINDOWS

Other recommendations:

HDvM uses HORCM CONF files called HORCM900.CONF to HORCM988.CONF for temporary HORCM CONF files. Do not use these numbers yourself.
I suggest that you use 0-799 for user created files and 800-899 for HDvM created permanent HORCM CONF files.
I also suggest a numbering convention of 1100x where x is the number in HORCMx.CONF. This means that you will need to “reserve” UDP ports 11000 to 11899 for HORCM CONF usage.

Updating the “Services” file

Many people code HORCM CONF files like this:

HORCM_MON

#ip_address service poll(10ms) timeout(10ms)

10.129.3.127 horcm8 1000 3000

In this case, the UDP port – horcm8 – must be defined in the “Services” file:

Windows C:\WINDOWS\system32\drivers\etc\services

UNIX /etc/services

Like this:

horcm0 11000/udp

horcm1 11001/udp

…

horcm8 11008/udp

horcm9 11009/udp

“blank line”

Check: Under Windows, if there is no blank line after horcm9 (in this example) that definition will be ignored! PS No blank lines at the end of the HORCM CONF file, please J

Check: If you have 2 CCI servers, using horcm8 and horcm9, for example, then both horcm8 and horcm9 have to be defined in both servers!

Reading the LOGS

Windows

Let’s start with Windows first this time J

In our example – we used Instance 8 – so you will find the log here:

C:\HORCM\log8\curlog\horcm_ml_acer510_log.txt

because this server is called ml_acer510.

Let us examine it in detail:

*****************************************************************************

- HORCM STARTUP LOG - Thu Feb 22 18:03:08 2007

*****************************************************************************

18:03:08-39210-05000- horcmgr started on Thu Feb 22 18:03:08 2007

18:03:08-39210-05000- _spawnvp() horcmd_08 using horcmgr [CWD=C:\HORCM\ETC]

18:03:08-3d090-07240- Fibre address conversion TBL has been set to '2'

P.P. : RAID Manager for WindowsNT

Model : RAID-Manager/WindowsNT

Ver&Rev: 01-19-03/04

Release: Production(GA)

HORCM(ml_acer510 7240) started by Administrator (0) on Thu Feb 22 18:03:08 2007

Lots of useful information here. See the data in bold.

18:03:08-3d090-07240- horcmd_08 started on Thu Feb 22 18:03:08 2007

18:03:08-3d090-07240- [horcmcfgrdf] access(conf_file) OK.

18:03:08-3d090-07240- [horcmcfgrdf] access(check) OK.

18:03:08-3d090-07240- [horcmcfgrdf] open(conf_file) OK.

18:03:08-3d090-07240- [horcmcfgetent] fseek(top) OK.

18:03:08-40b28-07240- converted CMDDEV filename \\.\CMD-10262-16 to \\.\PhysicalDrive8

Here is where \\.\CMD syntax is converted to a physical drive number.

18:03:08-40b28-07240- [horcmcfgetent] read(conf_file) OK.

18:03:08-40b28-07240- [horcmcfgrdf] close(conf_file) OK.

18:03:08-40b28-07240- [horcmcfgrdf] check(conf) OK.

18:03:08-40b28-07240- [horcmcfgrdf] horccmddev(0) OK.

18:03:08-40b28-07240- [horcread] cmddevopen() start

18:03:08-40b28-07240- [horcread] cmddevopen() finished

18:03:08-40b28-07240- ***** horcread *****

---ADDR--- -OFF- 0-1-2-3- 4-5-6-7- 8-9-A-B- C-D-E-F- ------CHAR------

[0x0012d8d4]0000: 48495441 43484920 4f50454e 2052454d HITACHI OPEN REM

[0x0012d8e4]0010: 4f544520 434f5059 20535953 54454d20 OTE COPY SYSTEM

[0x0012d8f4]0020: 44415441 20545950 45203031 00000000 DATA TYPE 01....

[0x0012d904]0030: f0f0f0f0 f0f0f0f1 f0f2f6f2 00040d09 000000010262....

Here is the USP serial number.

[0x0012d914]0040: 50090100 00040000 00040004 00040004 P...............

[0x0012d924]0050: ffffffff ffffffff 00060006 00060006 ................

[0x0012d934]0060: 00070007 00070007 000f0c00 00000000 ................

[0x0012d944]0070: 00000000 ef00e011 08030100 01004000 ..............@.

[0x0012d954]0080: 38000400 04400100 01000400 00ff0100 8....@..........

[0x0012d964]0090: 80000000 00000000 00000000 00000000 ................

[0x0012d974]00a0: 00000000 00000000 00000000 00000000 ................

[0x0012d984]00b0: 00800012 000e0002 00000000 00000000 ................

[0x0012d994]00c0: 00000000 00000000 00000000 00000000 ................

[0x0012d9a4]00d0: 00000000 00000000 00000000 00000000 ................

[0x0012d9b4]00e0: 00000000 00000000 00000000 00000000 ................

[0x0012d9c4]00f0: 00000000 00000000 00000000 00000000 ................

---ADDR--- -OFF- 0-1-2-3- 4-5-6-7- 8-9-A-B- C-D-E-F- ------CHAR------

[0x0012d9d4]0100: 0004ffff 00060007 ffffffff ffff000b ................

[0x0012d9e4]0110: ffff000d 000e000f 00100011 00120013 ................

[0x0012d9f4]0120: 00140015 00160017 9914ffff 001a001b ................

[0x0012da04]0130: 001c001d 001e001f 00200021 00220023 ......... .!.".#

[0x0012da14]0140: 20002001 00260027 00280029 002a002b . ..&.'.(.).*.+

[0x0012da24]0150: 002c002d ffff002f 00300031 00320033 .,.-.../.0.1.2.3

[0x0012da34]0160: 00340035 00360037 00380039 003a003b .4.5.6.7.8.9.:.;

[0x0012da44]0170: 003c003d 003e003f 00400041 00420043 .<.=.>.?.@.A.B.C

18:03:08-40b28-07240- [HORCREAD]: maxldev = 16384, unitnum = 256

18:03:08-40b28-07240- [HORCREAD]: maxhorc = 4, maxmrcf = 64, maxlun = 1024, maxctg = 256, maxjnlg = 256, mixport =1, slprflag = 1

18:03:08-40b28-07240- [HORCREAD]: Number of used instance(s) = 17, Number of attached instance(s) = UNKNOWN, Number of same instance(s) = 1

18:03:08-40b28-07240- [HORCREAD] execute-test read is done.:\\.\PhysicalDrive8

18:03:08-40b28-07240- [HORCMCFGRDF] SLPR is supported.

18:03:08-40b28-07240- ***** SLPR bitmap *****

---ADDR--- -OFF- 0-1-2-3- 4-5-6-7- 8-9-A-B- C-D-E-F- ------CHAR------

[0x0012db18]0000: 80000000 00000000 00000000 00000000 ................

[0x0012db28]0010: 00000000 00000000 00000000 00000000 ................

18:03:08-40b28-07240- [HORCRELOWNLBA] floatable LBA(e011) is released.ID:0:\\.\PhysicalDrive8