Brief Description and Timeline of the Performance Problem

Description. This is very important. Please remember, we do not know your server naming conventions nor which servers are connected to which ports. Please answer all these questions:

1. What are the customer's concerns?
2. What server(s) are affected (single or multiple)
3. What levels of z/OS are in use?
4. Which Ports/Array Groups/LDEVs are having performance issues?
5. What types of applications are affected?
6. Is replication used - TC, TCA, UR, COW, SI?

Timeline. Sometimes performance is good at certain times of the day - and bad at other times. We need to know the exact times at which it was good - as well as the exact times at which it was bad. Please answer all these questions:

1. Does the problem only occur at certain times or days of the week/month?
2. At what time(s) does the problem start?
3. At what time(s) does the problem go away?
4. Detail the exact timeline of all events before, during and after the event

Please be as specific as possible.

Data Collection from Performance Monitor

Supply the Performance Monitor data using the instructions How to Run the Export Tool for Enterprise Performance Data Collection

Time Difference between Storage and Server(s)

Supply the following:

• The log from the Export Tool. The log contains the time difference between the server collecting the Performance Monitor data and the SVP when the Export Tool is run on a customer PC/Server. The log also shows us what data was requested and any errors in the Export process.
• Time difference between the SVP and the time on the Host with the performance problem.

Ideally, the Export Tool is run on a customer PC/Server rather than on the SVP. If the CE runs the Export Tool on the SVP, the CE must supply the time difference as well as the log from the Export Tool.

In almost all cases, the "SVP time" is not the same as "server time". We have seen many cases where the SVP is set to be GMT - and the server is correctly set to the local time zone!

We suggest that you disable Performance Monitor data collection 1-2 hours after the USP dumps so that data is not lost in the event of problems running the Export Tool.

Detail Dump from Storage Array (CE Required)

Supply the following:

• Detail Dump of the subsystem. The dump must be taken within 24 hours of the problem.

You collect the dump in different ways, depending on the current situation:

• If the problem is still happening, turn on MODE 31 and after 5 minutes take a DETAILED DUMP of the USP system. Turn MODE 31 off when the dump has been taken.
• If the problem has gone away, turn on MODE 31 and take a DETAILED DUMP a few minutes later. This needs to happen as soon as possible after the incident. Turn MODE 31 off when the dump has been taken.
• If the problem can be recreated, turn on MODE 31, turn on Performance Monitor, turn on other available performance tools if applicable and take a DETAILED DUMP about 1 hour after the problem starts to happen again - and preferably whilst the problem is happening. Turn MODE 31 off when the dump has been taken.
• If Remote Copy is suspected, turn on MODE 31 on the RCU and take a DETAILED DUMP of the RCU a few minutes later. If Performance Monitor is available on the RCU, then capture its Performance data as well. Turn MODE 31 off when the dump has been taken.

Only the Hitachi hardware engineer can take a dump from a RAID subsystem. Please ask your Hitachi hardware engineer to do this for you.

Note:

If MODE 31 has been turned on, remember to turn it off after the DETAILED DUMP completes. MODE 31 can have a performance impact on some workload types and should not be left on unless specifically requested by GSC.

RMF or CMF Reports

Supply the following RMF reports, or the CMF equivalents:

1. Cache Utilization
2. Channel Activity
3. DASD Device Activity
4. IO queuing
5. ESS

Collect the data with the following guidelines:

• Supply reports in 15 minute or smaller intervals for a period of 1 hour
• Supply RMF reports in text format

We have tried numerous times to accept raw SMF data and frequently have had issues processing the data due to differences in operating system levels and/or maintenances levels.

Host and Subsystem Information

Please supply the following information:

1. Channel addresses on the host(s)
2. Port addresses on the subsystem
3. Directors/switches in between
4. Port address on the directors/switches
5. How the mainframe CU:LDEVs relate to the Subsystem CU:LDEVs

Batch Job Information from Affected Server(s) (if applicable)

If the problem is being experienced with batch job(s), supply the following for the server(s) in question:

• Joblog(s) covering the duration of the batch job(s)
  • They are generally not relevant for IMS or CICS performance problems
• Syslog covering the duration of the batch job(s)

External Storage Information (if applicable)

When External Storage is attached to the USP/USP-V/VSP, we need:

• Configuration diagram or description of the connectivity between the USP and external storage
• Detailed description of the usage/purpose of the external storage subsystem
• The time difference between the USP SVP clock and the external subsystem

If the external storage is a 9500V/AMS/WMS, we need:

• Simple Trace synchronised with USP/VSP dump
• Constitute files for RAID Group, System Parameters and Host Group
• 2 hours of DF Performance Statistics using a 1 minute interval and covering the time of the USP dump

If external storage is a 9900V, we need:

• MODE 31 DETAIL Auto Dump synchronised with the USP/VSP dump.
• Performance Monitor data

Remote Copy Information (if applicable)

If the performance problems involve Remote Copy, provide the data as specified below.

• Remote Copy Performance Data Collection