This is mostly common sense for and SAN, but it help to have it written somewhere;

Symptoms;

Databases down or slow.

Server response slow

Server down or showing SCSI errors

HBA errors and timeouts ;

Cluster disk resource Disk M:: is corrupt. Running ChkDsk /F to repair problems.

The file system structure on the disk is corrupt and unusable. Please run the chkdsk utility on the volume D:.

The device, \Device\Scsi\ql23001, did not respond within the timeout period.

An error was detected on device \Device\Harddisk1\DR1 during a paging operation.

Cluster resource Disk X: timed out. If the pending timeout is too short for this resource, then you should consider increasing the pending timeout value.

 

Recovery

1. Back-up and data that can be accessed
2. Shutdown all servers that attach to the SAN
3. Idle the switch
1. Log into SAN switch and back-up switch config, ensure config settings have been saved from memory
2. Save any log info to a test file (particularly any errors)
3. Save Zone configs (critical)
4. Ensure SAN switch traffic is minimal, if any (shutting down servers attached to switch should accomplish this
5. Shut down SAN switch
4. Log into SAN Disk Array
1. Ensure no active synch jobs, array rebuilds, etc are active (unless the jobs are hung)

                                                               i.      Note; Large SAN jobs can take hours to days to perform, are SAN memory intensive, but usually only slow down the SAN. These could be the cause. Find the chuclehead who started the jobs…

2. Save the log file and array configs
3. Restart the SAN (from the SAN management), never powercycle unless the SAN management is un-responsive (this can cause data loss, especially if the controller battery has failed)
4. If the array is not visible after restart, you may have to “import” the array (an array synch may occur after restart)
4. Restart the SAN switch
5. Restart the affected servers
1. Run checkdsk as neccessary

service performance issues

1. For IBM arrays, telnet to 6000
2. login as root (no pwd required)
3. use top to determine process using excessive cpu
4. check ibm (see link) to determine effect of killing process and process/service restart
5. If you are unsure and they system is having perf issues:
   1. unload the san as much as possible
   2. kill the process
   3. wait for system to stabilize
   4. Note: if process is critical, the storage software will restart it.
   5. Do not kill core linux processes (like init) it may be better to reboot
      1. to reboot>telnet (to std telnet port) >admin>system>reboot

IBM DS400 ref link; http://publib.boulder.ibm.com/infocenter/director/v5r2/index.jsp?topic=/eica7/eica7_trouble_unable_to_discover_ds400.htm