Where z/OSMF fits

I do not treat z/OSMF as “the web GUI for z/OS.” It is a management framework that exposes selected z/OS functions through a browser, workflows and REST APIs. The value is that the same platform can support human-guided administration and machine-driven automation without each team building a private interface to MVS commands, jobs and datasets.

For a sysprog, the most useful areas are Workflows, Software Management, jobs/dataset/file REST services, topology/sysplex services and the security/configuration model needed to expose those functions safely.

Core architecture

Browser / REST client
        │ HTTPS
        ▼
+-----------------------------+
| z/OSMF server (IZUSVR1)     |
| Liberty + z/OSMF services   |
+-----------------------------+
        │
        ├── SAF/RACF authorization
        ├── CEA and z/OS services
        ├── Jobs / data sets / files
        ├── Workflows
        └── Software Management

Angel process: IZUANG1 (where required by services)

The server runs on WebSphere Liberty technology. The z/OSMF server and angel address spaces are therefore real z/OS started tasks with datasets, USS files, ports, certificates, Java/runtime dependencies and SAF permissions that I troubleshoot like any other system service.

Configuration model

I separate base/nucleus configuration from service-specific enablement. The base setup establishes the server, security, networking and core prerequisites. Individual z/OSMF services can have additional authorization and configuration.

IZUPRMxx and the documented z/OSMF configuration process define the environment. I avoid hard-coding old Java paths, ports or prerequisite assumptions from an old build document because these change across z/OS releases and site standards.

Security model

Successful HTTPS authentication only gets a user through the front door. Each z/OSMF function still relies on SAF/RACF authorization to the service and to the underlying z/OS resource. A REST request that submits a job, for example, also has to obey the same security boundaries around job submission and datasets.

That layered model is why I troubleshoot authorization from both ends: z/OSMF service access and the underlying system resource.

Workflows

Workflows are where z/OSMF becomes especially interesting for systems programming. A workflow can turn a written implementation runbook into ordered steps with variables, instructions, JCL, REXX, shell actions and REST interactions. Some steps can be automated; others can deliberately require a human confirmation.

The useful design is not “automate every click.” I want the workflow to encode prerequisites, site variables, validation and backout points so another sysprog can execute the same process consistently.

• Inputs and site variables are defined once rather than edited across ten jobs.
• Steps have clear preconditions and completion criteria.
• Automation is used for deterministic work; approvals remain explicit.
• Failure messages tell the operator what evidence to collect or which step to roll back.

Software Management

Software Management provides inventory and management concepts around installed software and software instances. Portable software instances are part of the modern software-delivery story and can support movement/deployment of software content with metadata rather than treating every install as a loose collection of copied datasets.

For lifecycle work, this can help connect installed software information, deployment and workflows. It does not eliminate the need to understand SMP/E, product configuration or runtime activation—the sysprog still needs to know what is being deployed and how the product becomes active.

REST APIs

# Shape only — host/port/auth are site specific
GET https://host:port/zosmf/restjobs/jobs?owner=USERID
GET https://host:port/zosmf/resttopology/sysplexes

The REST services are what make z/OSMF important beyond the browser. Automation can submit and monitor jobs, work with datasets/files and query system information through documented HTTP interfaces. That is a cleaner integration point for modern tooling than screen-scraping ISPF or embedding console access into every script.

How I would use z/OSMF in lifecycle work

A product installation or upgrade already has the shape of a workflow: prerequisites, site inputs, installation jobs, configuration changes, validation and rollback. z/OSMF Workflows can encode that sequence. REST services can submit/monitor jobs and gather evidence. Software Management can add software-instance context.

That is also why I see a z/OSMF workflow skill as a natural extension for the lifecycle agent: the agent can produce a documented plan first, then eventually produce a controlled workflow definition for the steps that are safe to automate.

Troubleshooting sequence

1. Confirm IZUSVR1/IZUANG1 state and startup messages.
2. Check Liberty/server logs and USS filesystem availability.
3. Verify port/listener/TLS and certificate state.
4. Check the user identity and z/OSMF service authorization.
5. Check authorization to the underlying job/dataset/system resource.
6. Isolate whether the failing piece is the z/OSMF framework or one specific service.

That sequence keeps me from calling every HTTP 403 or failed workflow “a z/OSMF problem.” Often the framework is healthy and the underlying SAF or system resource is what is rejecting the operation.

IBM references

How I picture the machine

I find mainframe hardware easier to understand if I separate the physical box from the logical system that z/OS sees. IBM calls the physical machine the Central Processor Complex (CPC); most sysprogs simply say “the box.” Inside that CPC, PR/SM divides processor, memory and I/O resources into logical partitions. An LPAR is therefore not a small physical machine bolted inside the frame—it is a controlled logical view of shared hardware.

z/OS runs inside one of those LPARs. It sees logical processors, a defined amount of central storage and a set of I/O devices made visible through the I/O configuration. When I troubleshoot a hardware-looking problem, I keep asking which layer I am actually looking at: the z/OS image, the LPAR definition, the channel subsystem, the storage/network device, or the CPC itself.

Processors and capacity

A CPC can contain different processor types. General-purpose CPs execute normal z/OS work. Specialty engines exist so particular workloads can run on processors designed or licensed for those workloads—for example zIIPs for eligible work and ICF processors for Coupling Facility images. The important point operationally is that processor eligibility matters: adding idle specialty capacity does not automatically help work that can only run on general CPs.

PR/SM dispatches logical processors onto physical processors according to partition definitions, weights, caps and the overall CPC workload. That is why “LPAR CPU is high” is not a complete hardware diagnosis. I also want to know whether the LPAR is capped, whether shared processors are contended, whether logical processor configuration is sensible, and whether the workload is actually delayed for CPU.

D M=CPU
D M=STOR

D M=CPU gives the z/OS view of processors. D M=STOR gives the storage configuration visible to the image. For real capacity questions I correlate those displays with RMF and, when needed, HMC/partition data.

Memory: central storage is not virtual storage

Central storage is the real memory assigned to the LPAR. z/OS then builds virtual address spaces on top of it and pages less-active content as needed. I avoid comparing a process’s virtual size directly with CPC memory; the two are not equivalent. A 64-bit address space can reserve or address enormous ranges without all of that data being resident.

When memory pressure is suspected, I look for sustained paging, low frame headroom, auxiliary-storage pressure and the address spaces driving the working set. If the assigned LPAR storage itself changed, that becomes a PR/SM/HMC question rather than only a z/OS tuning question.

Channel subsystem and the I/O path

The channel subsystem is what lets processors and operating systems reach devices without the CPU micromanaging every I/O operation. The I/O definition describes channel paths, control units and devices, and z/OS works with the device numbers and paths made available to its LPAR.

The mental path I use is:

Application / subsystem
        ↓
z/OS access method / IOS
        ↓
Channel subsystem
        ↓
Channel path
        ↓
Control unit / storage or network adapter
        ↓
Device

If a job has high I/O delay, that does not immediately prove a bad disk. The delay can come from queueing, channel/path problems, control-unit response, serialization, a volume problem or the application’s own I/O pattern.

D IOS,CONFIG
D M
D U,,,
D M=DEV(device)

RMF device and channel reports are normally what turn those displays into a performance diagnosis.

LPARs, HMC and configuration ownership

PR/SM owns the partitioning boundary. The Hardware Management Console is where operators or platform teams define and control LPAR resources, profiles, activation and many hardware operations. z/OS can report what it has, but it cannot by itself tell me every physical scheduling decision occurring outside the image.

That boundary matters during incidents. A z/OS team may see CPU delay; the cause may be WLM policy inside z/OS, a defined-capacity cap on the LPAR, competition elsewhere on the CPC, or simply valid high workload. The fastest diagnosis usually comes from aligning both views instead of arguing whether the problem is “hardware” or “software.”

What I check when the problem looks physical

• Scope: one address space, one LPAR, several LPARs on the CPC, or multiple systems using the same external device?
• CPU: actual CPU delay, logical/physical utilization, caps and recent partition changes.
• Storage: assigned central storage, paging and whether memory was dynamically changed.
• I/O: device state, path state, RMF response components and whether affected work shares a control unit or storage group.
• Recent change: IODF activation, hardware maintenance, HMC profile changes, storage/network maintenance or processor reconfiguration.

The line I try not to cross is guessing at hardware from a single z/OS symptom. I want a chain of evidence from the workload down to the physical resource before I call the problem a CPC, channel or storage fault.

IBM reference points

Release-specific syntax and limits can change. Use the IBM documentation linked below for the z/OS and IBM Z level installed on the target system before making a configuration change.

• IBM Z hardware documentation
• IBM z/OS basic skills: mainframe hardware

Start with the problem Sysplex is solving

A Parallel Sysplex is not simply “several LPARs connected together.” The objective is to let multiple z/OS images cooperate closely enough that workloads can move, data can be shared and a single system failure does not have to become an application outage.

I break the technology into three layers. XCF handles membership, groups and signalling between systems. XES is the service applications use to exploit Coupling Facility structures. The Coupling Facility provides shared high-speed structures for locking, caching and lists. Couple data sets and policies hold the definitions that keep this coordinated across IPLs.

z/OS images
   │
   ├─ XCF membership / signalling
   ├─ XES requests for shared structures
   └─ shared DASD / subsystems
          │
          ▼
   Coupling Facility
   ├─ lock structures
   ├─ cache structures
   └─ list structures

XCF: membership and signalling

XCF is the base communication framework. Systems join the sysplex, applications form XCF groups, members join those groups, and XCF transports signalling between them. Some products expose their own higher-level functions, but underneath they still depend on healthy cross-system communication.

D XCF
D XCF,SYSPLEX,ALL
D XCF,GROUP
D XCF,PATHIN,DEVICE=ALL
D XCF,PATHOUT,DEVICE=ALL

When several unrelated sysplex-aware products degrade at the same time, I check XCF early. One failed signalling path may be tolerated because another path exists; the incident begins when redundancy or capacity is no longer enough.

Coupling Facility structures

A Coupling Facility runs CFCC, normally in an LPAR using ICF processors. Its memory is divided into structures requested by connected applications.

• Lock structures coordinate ownership and serialization across systems.
• Cache structures help data-sharing products coordinate cached data and invalidation.
• List structures hold shared lists, queues and other structured state; System Logger is a common example of a service that can use CF list structures.

The structure name alone is not enough to diagnose a problem. I also want to know who connects to it, where it is allocated, its size, entry/element usage, whether duplexing or rebuild is active, and what the owning product expects during failure.

D XCF,CF
D XCF,STR
D XCF,STR,STAT=ALLOC
D XCF,STR,STRNAME=<STRUCTURE>
D CF

Couple data sets and policies

Couple data sets are persistent control data used by sysplex services. Depending on the function, there are couple data sets for SYSPLEX, CFRM, LOGR and other policy types. I think of them as the durable configuration that all members agree to use—not as ordinary data sets that should be edited directly.

D XCF,COUPLE
D XCF,COUPLE,TYPE=SYSPLEX
D XCF,COUPLE,TYPE=CFRM
D XCF,COUPLE,TYPE=LOGR

The CFRM policy describes Coupling Facilities and structure attributes such as size, preference list and other allocation characteristics. The active policy lives in the CFRM couple-data-set environment.

Defining and changing CFRM safely

The exact policy content is site- and product-specific, but the workflow is consistent: format the CFRM couple data sets, define a policy, activate it, then verify the structures and connectors rather than assuming the command completed the change.

//CFRMCDS  EXEC PGM=IXCL1DSU
//SYSPRINT DD SYSOUT=*
//SYSIN    DD *
  DEFINEDS SYSPLEX(YOURPLEX) TYPE(CFRM)
            DSN(YOURHLQ.CFRM.PRIMARY) VOLSER(VOL001)
/*

//CFRMPOL  EXEC PGM=IXCMIAPU
//SYSPRINT DD SYSOUT=*
//SYSIN    DD *
  DATA TYPE(CFRM) REPORT(YES)
  DEFINE POLICY NAME(CFRMPOL1) REPLACE(YES)
    STRUCTURE NAME(APPL_LOCK)
      SIZE(4096) INITSIZE(2048) PREFLIST(CF01)
/*

SETXCF START,POLICY,TYPE=CFRM,POLNAME=CFRMPOL1
D XCF,POLICY,TYPE=CFRM
D XCF,STR

I never treat a structure resize or relocation as a generic infrastructure change. The owning subsystem’s documentation determines whether the structure can be rebuilt, reallocated or altered dynamically and what persistence/recovery semantics apply.

Failure and rebuild behaviour

Sysplex availability comes from planned redundancy and recovery behaviour. A CF or structure can fail, an XCF signalling path can disappear, or a system can stop responding. The technology provides mechanisms such as rebuild, alternate CF placement, duplexing and system-status detection, but those mechanisms only work well when the policies and alternate resources were designed beforehand.

During a structure incident I first determine whether the structure is full, unavailable, rebuilding or merely showing high utilization. Those are different conditions. A nearly full cache structure may need sizing analysis; a failed structure in active rebuild needs careful monitoring; a policy allocation failure points me back to CFRM definition, CF capacity or connectivity.

The sequence I use for problem isolation

1. Confirm sysplex membership and whether every system sees the same topology.
2. Check XCF signalling paths and groups if cross-system functions are slow.
3. Check CF availability and the specific affected structure.
4. Identify connectors and the owning subsystem.
5. Check CFRM/LOGR policy and couple-data-set state.
6. Correlate with hardware/channel, CPU and subsystem messages before taking SETXCF recovery action.

The main lesson is that Sysplex is shared infrastructure. If Db2, Logger, GRS and another sysplex consumer all become unhealthy at the same time, I do not start by restarting four products independently.

IBM reference points

Release-specific syntax and limits can change. Use the IBM documentation linked below for the z/OS and IBM Z level installed on the target system before making a configuration change.

• IBM Parallel Sysplex concepts
• IBM XCF overview
• IBM coupling facility planning

A data set is more than a file name

The quickest way I learned to stop getting confused by z/OS data sets was to separate four questions: what is the data set called, how is it organized, where is it physically allocated, and how is it found? The catalog answers the naming/location question; the VTOC describes allocations on a volume; the data-set organization and access method describe how records are stored and accessed.

That separation becomes important during failures. “Dataset not found” can mean a missing catalog entry, an unavailable volume, a wrong GDG generation, a typo, or a security/allocation failure that was summarized badly by the application.

PS, PDS and PDSE

A sequential data set (PS) is a linear sequence of records. A PDS stores members in one library but uses a directory plus member extents and needs compression after enough replacements/deletions. A PDSE keeps the library/member model but removes many classic PDS limitations and is the normal choice for modern program and source libraries unless a product explicitly requires PDS behaviour.

For load libraries I also care about whether the consuming product supports PDSE, whether sharing rules are correct and whether the library participates in LINKLIST, LPA, APF or LLA management.

DCB attributes and why mismatches hurt

• RECFM defines record format: fixed, variable, blocked variants and so on.
• LRECL defines logical record length.
• BLKSIZE controls physical blocking; BLKSIZE=0 normally lets the system choose an efficient value.
• DSORG identifies the organization.

I do not copy DCB values blindly from old JCL. If the output is being created by a program or utility, I first check whether that program supplies or derives the attributes. Bad DCB assumptions commonly surface as allocation/open errors or records being interpreted incorrectly.

Space, extents and DASD placement

SPACE describes primary and secondary allocation. The primary quantity is what the data set asks for initially; secondary quantities are used when it extends. Repeated extents are normal up to a point, but a data set that constantly runs out of extents usually needs a sizing or storage-management correction, not repeated manual extension.

SMS can select storage class, data class, management class and volumes according to policy. Non-SMS allocation may explicitly use UNIT/VOL. I treat volume selection as storage policy, not application trivia, because catalog/VTOC state and storage-group capacity are part of the reliability of the workload.

Catalog, VTOC and VVDS

The catalog maps a data-set name to the information needed to locate it. The master catalog also owns aliases that route high-level qualifiers to user catalogs. A VTOC exists on a DASD volume and records allocations on that volume. VSAM/SMS environments also use VVDS metadata.

When a catalog problem is suspected, I avoid “repairing” anything until I know which metadata layer is wrong. Re-cataloging a name when the on-volume components are not what I think they are can make a recoverable problem worse.

//LISTCAT  EXEC PGM=IDCAMS
//SYSPRINT DD SYSOUT=*
//SYSIN    DD *
  LISTCAT ENTRY(YOURHLQ.DATASET) ALL
/*

VSAM

VSAM is a family of access methods and data-set organizations rather than one data-set type. The common structures are KSDS, ESDS, RRDS and LDS. KSDS uses keys and indexes; ESDS is entry-sequenced; RRDS provides relative-record access; LDS is essentially byte-oriented space used by products such as zFS and other system components.

For a sysprog, the important part is recognizing when IDCAMS definitions, catalog components, data/index components, share options or control interval/control area behaviour are involved. I do not assume a generic sequential copy utility is appropriate for a VSAM object.

//REPRO    EXEC PGM=IDCAMS
//SYSPRINT DD SYSOUT=*
//INFILE   DD DISP=SHR,DSN=YOURHLQ.INPUT
//OUTFILE  DD DISP=SHR,DSN=YOURHLQ.OUTPUT
//SYSIN    DD *
  REPRO INFILE(INFILE) OUTFILE(OUTFILE)
/*

GDGs

A Generation Data Group gives me a stable base name plus ordered generations. Relative references such as (0), (-1) and (+1) are convenient in batch, but they are evaluated in the context of job execution and catalog state. I am careful with restart/re-run scenarios because an extra created generation can shift what “current” means.

The GDG base controls the generation limit and rollover behaviour. Changing those parameters is a data-retention decision, not merely a JCL cleanup.

Naming and system libraries

Data-set naming conventions matter because HLQs frequently drive catalog aliases, SMS rules, RACF profiles, automation and ownership. System libraries such as PARMLIB, PROCLIB, LINKLIB, LPALIB and product target libraries have additional operational meaning; moving or renaming them can affect IPL, program fetch and authorization.

For that reason, I try to understand a data set’s role before treating it as storage. Two equally sized PDSEs can have completely different risk if one is an application library and the other is APF-authorized and in LINKLIST.

My allocation / failure checklist

1. Confirm the exact data-set name and whether GDG/symbol resolution changes it.
2. Check LISTCAT and the owning catalog/alias.
3. Check volume/storage-group availability and free space.
4. Check DISP, organization and DCB compatibility.
5. Check RACF access.
6. Check ENQ/GRS contention if allocation/open hangs.
7. For VSAM, separate catalog metadata from VVDS/VTOC and physical components before repair.

That sequence keeps me from using IDCAMS ALTER/DELETE/RECATALOG as trial-and-error commands.

IBM reference points

Release-specific syntax and limits can change. Use the IBM documentation linked below for the z/OS and IBM Z level installed on the target system before making a configuration change.

• IBM: What is a data set?
• IBM: VSAM
• IBM: Access methods

The two questions I keep separate

Program fetch problems become much easier once I keep two questions separate: where will z/OS search for the module? and is the library/module allowed to run authorized? Search order is about finding code. APF is about authorization. Mixing those concepts leads to bad fixes such as APF-authorizing a library just because a module was not found.

Default search order

1. Already-loaded usable copy / job pack area
2. TASKLIB
3. STEPLIB, otherwise JOBLIB
4. LPA (dynamic / fixed / modified / pageable as applicable)
5. LINKLIST

The exact fetch path can be influenced by the environment and product, but this order is the useful working model for normal batch/STC program fetch. A private STEPLIB can deliberately override a module that also exists in LINKLIST, which is one reason wrong-level problems can affect only one started task.

LPA

The Link Pack Area holds shared executable modules so multiple address spaces can use the same common copy. Parts of LPA are established during IPL; dynamic LPA lets selected modules be added or removed without an IPL when supported and appropriate.

I treat LPA changes cautiously because they affect shared code. Before adding a module dynamically I confirm that the product actually expects LPA residency and that the module/library level matches the active configuration.

D PROG,LPA,MODNAME=<MODULE>
SETPROG LPA,ADD,MODNAME=<MODULE>,DSNAME=<DSN>

LINKLIST and LLA

LINKLIST is a logical concatenation of program libraries used for program fetch. LLA improves lookup efficiency for library members and is commonly associated with LINKLIST-managed libraries. Updating a data set on disk does not automatically mean every address space will begin executing the new member immediately; product restart, LLA refresh or other activation may be required.

D PROG,LNKLST
D LLA
F LLA,UPDATE=<XX>

When maintenance replaces load modules, I always ask what the activation mechanism is. “PTF applied” and “new code executing” are separate milestones.

APF

APF identifies libraries trusted to contain authorized programs. An authorized program can perform privileged operations that normal application code cannot, so APF is part of the z/OS security boundary.

That means I do not add a library to APF as a convenience. I verify vendor requirements, library integrity, dataset protection and whether every library in the relevant authorization chain meets the rules for the execution path.

D PROG,APF
SETPROG APF,ADD,DSNAME=<DSN>,SMS
SETPROG APF,DELETE,DSNAME=<DSN>,SMS

PROGxx and dynamic changes

PROGxx members can define APF lists, LNKLST sets and other program-management configuration. Dynamic SETPROG commands are useful, but I still make the persistent PARMLIB change when the configuration must survive IPL. Otherwise the system can look correct today and quietly revert at the next IPL.

SET PROG=<XX>
SETPROG LNKLST,TEST,NAME=<SET>,MODNAME=<MODULE>
D PROG,LNKLST

How I diagnose S806

1. Capture the exact module name from the failure message.
2. Check STEPLIB/JOBLIB/TASKLIB in the failing job or started task.
3. Check LINKLIST and LPA for the module.
4. Confirm the intended library is cataloged/available and the member actually exists.
5. Check whether the wrong procedure, symbol or environment selected an old library.

S806 is a search/fetch problem first. APF only enters the investigation if the eventual failure is about authorization.

How I diagnose a wrong-level module

A wrong-level problem is usually harder because the module is found. I compare all candidate copies: STEPLIB, LINKLIST and LPA. Then I check timestamps/service level, LLA state and whether a running address space must be recycled to pick up the replacement.

During upgrades I deliberately avoid leaving both old and new libraries ambiguously reachable. Side-by-side releases are useful only when the execution path for each release is explicit.

IBM reference points

Release-specific syntax and limits can change. Use the IBM documentation linked below for the z/OS and IBM Z level installed on the target system before making a configuration change.

• IBM: Search order for programs
• IBM DISPLAY command reference

The abend code is the start, not the answer

A system completion code such as S0C4 or SB37 tells me which broad failure class occurred. It does not tell me the root cause by itself. I want the message set, reason code, failing step/program, recent change and—when it is a program check—the PSW/register/module context.

The same code can have very different fixes. An 0C4 caused by a bad pointer after a software change is not approached the same way as an 0C4 caused by storage overlay. A B37 on a sequential output file is not the same as an extent problem on a heavily fragmented library.

What I collect first

Job name / job ID / failing step
System and exact timestamp
System completion code + reason code
User abend code if present
JESJCL / JESMSGLG / JESYSMSG
Program or load-module level
Relevant application/runtime messages
PSW / registers / module + offset for program checks
Dump or traceback identifiers
Recent software, JCL or data change

Program checks: 0C1, 0C4, 0C7

• 0C1 generally means an operation exception: the CPU tried to execute something that is not a valid instruction in that context. Wrong branch addresses, bad module levels or overwritten code are common investigation directions.
• 0C4 is a protection/addressing family. I look at the reason, PSW and registers to see what address was referenced and why that storage was not accessible.
• 0C7 is a data exception, commonly invalid packed/zoned numeric data being used as numeric input. I trace the failing instruction and data field rather than simply blaming “bad data.”

For application code, compiler/runtime traceback information often gets me to the failing statement faster than raw IPCS. For system/product code, a dump and vendor symbol information may be required.

Space abends: B37, D37 and E37

All three indicate a space/allocation failure family, but the exact code/reason matters. I check the data-set allocation, primary/secondary space, extent count, volume free space, SMS behaviour and whether the data set can extend to another eligible volume.

The right fix is not always “increase SPACE.” If a secondary quantity is missing, the volume is full, extents are exhausted or SMS cannot find eligible capacity, changing only the primary allocation may not solve anything.

Storage and resource failures

Storage-related abends need the same discipline: identify which address space and which storage area was exhausted. Private storage, common storage, 64-bit storage and auxiliary-storage pressure are different problems.

D VIRTSTOR,HVCOMMON
D VIRTSTOR,HVSHARE
D ASM
D A,L

Likewise, a job that appears “hung” before an abend may have been waiting on serialization. I check GRS when dataset/control-resource contention is possible.

D GRS,C
D GRS,RES=(SYSDSN,<DSN>)

Cancel, time and output-limit failures

Some completion codes are operational outcomes rather than code defects: operator cancellation, time limits, spool/output limits or subsystem termination. I verify why the system ended the step and who/what initiated it before sending the problem to an application team.

If a timeout is involved, I also check whether the job was actually consuming CPU or spending the elapsed time waiting on ENQ, I/O, WLM or an external dependency.

Security and data-set access

Not every failed OPEN or allocation is a storage problem. RACF denial messages, catalog errors and SMS allocation failures can all surface near the failing DD statement. I read the accompanying IEC/IGD/ICH messages instead of diagnosing from the final abend line alone.

Before I change anything

I try to preserve the failing evidence. Cancelling/restarting, increasing REGION, changing SPACE or granting broad access may make the next run succeed while destroying the explanation. A good escalation package contains the code/reason, messages, failing program/offset, input context and the exact difference from a known-good run.

IBM reference points

Release-specific syntax and limits can change. Use the IBM documentation linked below for the z/OS and IBM Z level installed on the target system before making a configuration change.

• IBM: Overview of an abend
• IBM z/OS MVS System Codes

Why I treat utility choice as part of the design

On z/OS there is usually more than one way to move, define or inspect data. I try not to choose a utility just because I remember its syntax. The right question is what object I am manipulating: a sequential data set, a library, VSAM/catalog metadata, whole volumes, couple data sets, SMF records or a UNIX file.

That matters because a utility that works perfectly for PS data can be the wrong tool for VSAM; a DFSMSdss dump is not the same thing as an IEBCOPY library copy; and IXCMIAPU is an administrative-policy tool, not a generic text editor.

IEFBR14: allocation and deletion through DD statements

//ALLOC    EXEC PGM=IEFBR14
//NEWDD    DD DSN=YOURHLQ.NEW.DATA,
//            DISP=(NEW,CATLG,DELETE),UNIT=SYSDA,
//            SPACE=(CYL,(5,2)),
//            DCB=(RECFM=FB,LRECL=80,BLKSIZE=0)

IEFBR14 itself does essentially nothing useful to the data. The allocation/deletion side effect comes from JCL allocation processing. I use it for simple dataset lifecycle tasks, not for copying content.

IEBGENER: sequential copy/generation

//COPY     EXEC PGM=IEBGENER
//SYSUT1   DD DISP=SHR,DSN=YOURHLQ.INPUT
//SYSUT2   DD DSN=YOURHLQ.OUTPUT,
//            DISP=(NEW,CATLG,DELETE),UNIT=SYSDA,
//            SPACE=(CYL,(5,2)),
//            DCB=(RECFM=FB,LRECL=80,BLKSIZE=0)
//SYSPRINT DD SYSOUT=*
//SYSIN    DD DUMMY

IEBGENER is a simple sequential copy tool. If I am dealing with partitioned libraries, VSAM, catalog metadata or volume-level movement, I pick something more appropriate.

IEBCOPY: PDS/PDSE libraries

//COPY     EXEC PGM=IEBCOPY
//SYSPRINT DD SYSOUT=*
//SYSUT1   DD DISP=SHR,DSN=YOURHLQ.SOURCE.PDSE
//SYSUT2   DD DISP=SHR,DSN=YOURHLQ.TARGET.PDSE
//SYSIN    DD *
 COPY INDD=SYSUT1,OUTDD=SYSUT2
/*

IEBCOPY understands libraries and members. It is the normal utility I expect when copying or reorganizing PDS/PDSE content, subject to the library type and product requirements.

IDCAMS: catalog and VSAM administration

//AMS      EXEC PGM=IDCAMS
//SYSPRINT DD SYSOUT=*
//SYSIN    DD *
 LISTCAT ENTRY(YOURHLQ.DATA.SET) ALL
/*

IDCAMS covers catalog services and VSAM definitions/operations such as DEFINE, LISTCAT, REPRO and DELETE. Because it can change catalog structures and data-set definitions, I treat recovery-oriented commands with much more care than a simple LISTCAT.

DFSMSdss / ADRDSSU

ADRDSSU is the DFSMSdss program used for data-set and volume dump/restore/copy operations. I reach for it when I need storage-aware movement or backup at a broader level than a record-by-record copy. SMS, serialization, filtering and target-volume considerations all matter.

Batch TSO/E: IKJEFT1B

//TSO      EXEC PGM=IKJEFT1B
//SYSTSPRT DD SYSOUT=*
//SYSTSIN  DD *
  LISTCAT ENTRY('YOURHLQ.DATA.SET') ALL
/*

IKJEFT1B lets me run TSO commands or REXX/CLIST logic in batch. I use it when the function is naturally a TSO command rather than forcing it through another utility interface.

Sysplex utilities: IXCL1DSU and IXCMIAPU

//REPORT   EXEC PGM=IXCMIAPU
//SYSPRINT DD SYSOUT=*
//SYSIN    DD *
 DATA TYPE(CFRM) REPORT(YES)
/*

IXCL1DSU formats couple data sets. IXCMIAPU defines/reports administrative policies such as CFRM/LOGR-related definitions. I keep those roles distinct because formatting a CDS and changing a policy are different operational steps.

SMF and UNIX utilities

SMF has different dump programs depending on the recording model: IFASMFDP for traditional SMF data-set input and IFASMFDL for SMF log streams. For z/OS UNIX execution from batch, BPXBATCH is useful for invoking shell commands or programs.

//USS      EXEC PGM=BPXBATCH
//STDOUT   DD SYSOUT=*
//STDERR   DD SYSOUT=*
//STDPARM  DD *
SH ls -la /your/path
/*

The rule I use before running a utility

• Know whether the utility is reading, copying, allocating, deleting or changing metadata.
• Know the serialization and restart behaviour if the target is in use.
• Preserve LISTCAT/report output before structural changes.
• For system objects—catalogs, couple data sets, SMF, APF libraries—follow the component’s recovery procedure rather than improvising with a familiar utility.

IBM reference points

Release-specific syntax and limits can change. Use the IBM documentation linked below for the z/OS and IBM Z level installed on the target system before making a configuration change.

• IBM data-set utilities
• IBM system utilities

IPL is a configuration event, not just a restart

An IPL reloads z/OS and rebuilds a large part of the system state from hardware definitions, SYSRES content, LOADxx/IEASYSxx choices and PARMLIB members. That is why an IPL can expose configuration mistakes that stayed hidden while the system was running dynamically.

I separate the work into three phases: controlled shutdown, initialization, and validation. The third phase is the one that gets skipped most often. A console prompt and an active JES2 do not prove that Logger, XCF, TCP/IP, OMVS mounts, APF/LINKLIST, automation and critical STCs are all correct.

From hardware LOAD to z/OS initialization

The hardware load profile identifies the device and load parameters used to begin loading the system. The bootstrap and nucleus initialization path brings in the nucleus, processes initialization modules and establishes core services before normal started tasks and subsystems become available.

LOADxx and IEASYSxx are central because they influence which PARMLIB concatenation and system-parameter suffixes are selected. From there, other members control areas such as master-scheduler commands, program libraries, consoles, SMF, dumps, UNIX, networking and subsystem-specific configuration.

I do not memorize IPL as one fixed list of modules. What matters operationally is knowing which choices are made before z/OS is fully available and which configuration can be changed dynamically afterward.

IPL types and why they matter

A normal IPL and a clear IPL are not equivalent operationally. A clear operation resets more state and is used only when the situation/procedure requires it. Likewise, re-IPL after maintenance, disaster recovery IPL and stand-alone-dump loading all have different objectives even though each begins from the hardware console.

I follow the site’s documented LOAD parameters and HMC profiles rather than constructing them during an outage.

A controlled shutdown

The shutdown sequence is dependency-driven. Application work stops first, then middleware/subsystems, then shared/base services in the order required by the installation. The exact list varies, but the principle does not: stop consumers before the service they consume.

D A,L
D R,R
D OMVS,A=ALL
D XCF
$DJES2

Before shutting down I also confirm that recovery evidence is safe: SMF offload, Logger status, dumps, automation state and any product-specific checkpointing. For a planned maintenance IPL, I want a pre-IPL snapshot of active PARMLIB/PROCLIB, APF/LINKLIST/LPA, mounts, key STCs and sysplex state so the post-IPL comparison is objective.

What I validate after IPL

1. Identity/configuration: IPLINFO, intended SYSRES/load parameters, active PARMLIB and system symbols.
2. Core services: master, JES2, security manager, SMS/catalog availability.
3. Sysplex: XCF membership, couple data sets, CF structures and Logger.
4. UNIX/network: OMVS, zFS mounts, TCP/IP/VTAM and required listeners.
5. Program configuration: APF, LINKLIST, LPA/LLA and product runtime libraries.
6. Operations evidence: SMF recording, dump settings, outstanding WTORs and health-check exceptions.
7. Applications: critical STCs reach their ready state—not merely show ACTIVE.

D IPLINFO
D A,L
D XCF
D XCF,STR
D LOGGER,STATUS
D OMVS,F
D TCPIP
D SMF
D DUMP,OPTIONS
D PROG,APF
D PROG,LNKLST

What usually goes wrong

Post-IPL failures are often configuration-activation failures rather than brand-new defects: wrong member suffix, missing PROCLIB library, APF/LINKLIST difference, a zFS not mounted, a dependency started too early, a CF/log stream unavailable, or a product still pointing to the previous release.

The most useful habit is to find the first failed layer. If ten STCs fail because one shared filesystem never mounted, I have one root problem, not ten product incidents.

IBM reference points

Release-specific syntax and limits can change. Use the IBM documentation linked below for the z/OS and IBM Z level installed on the target system before making a configuration change.

• IBM system IPL sequence
• IBM planned z/OS UNIX shutdown

What SMP/E is really buying me

SMP/E is often introduced as “the tool used to install maintenance.” That is true but incomplete. The part I value is that SMP/E keeps a controlled record of what software elements and SYSMODs belong to the product, what level is installed, what prerequisites exist and what has been applied or accepted.

If I need to answer “is this PTF applied here?”, “what FMID owns this module?” or “what prerequisites block this service?”, the CSI is supposed to give me evidence instead of forcing me to infer the answer from library timestamps.

CSI zones and libraries

• Global zone: information shared across the SMP/E environment, including received SYSMOD information and pointers to target/DLIB zones.
• Target zone: describes the installed target-system elements and service level.
• DLIB zone: describes distribution libraries—the base from which target libraries can be built/reconstructed according to product design.

The zones describe state; the actual load/source/macro libraries hold the executable or build elements. I keep that distinction clear because a library being manually copied does not mean SMP/E’s CSI knows about the change.

SYSMODs and service vocabulary

Products arrive as function SYSMODs/FMIDs and are maintained through PTFs, APAR-related service and, where used, USERMODs. HOLDDATA adds conditions that SMP/E cannot simply resolve by dependency processing—manual actions, PE information, HIPER/security context, FIXCAT categories and other service considerations.

When maintenance fails APPLY CHECK, I read the requisites and HOLDDATA instead of trying to FORCE past them. The point of SMP/E is dependency control; bypassing it removes exactly the protection I am relying on.

The RECEIVE → APPLY → ACCEPT model

External service / product package
        │
        ▼
RECEIVE ──► Global zone + SMPPTS / SMPTLIB
        │
        ▼
APPLY CHECK ──► prerequisites / HOLD / MODID analysis
        │
        ▼
APPLY ──► Target zone + target libraries
        │
        ▼
Runtime activation / validation
        │
        ▼
ACCEPT CHECK / ACCEPT ──► DLIB zone + distribution libraries

RECEIVE makes SMP/E aware of the service/package. APPLY CHECK is my dry run for target-system impact. APPLY updates target-zone state and target libraries. ACCEPT advances the distribution-library level after the maintenance has passed the site’s stabilization policy.

I do not ACCEPT immediately just because APPLY succeeded. ACCEPT changes the backout/reconstruction position, so it belongs after validation according to the installation’s maintenance policy.

Runtime activation is separate

This is where SMP/E work becomes systems programming. A PTF can be successfully applied while the running system still uses the old code from LPA, LINKLIST/LLA, a STEPLIB-loaded address space or an already-loaded module.

For every maintenance change I identify the activation requirement: dynamic LPA update, LLA refresh, STC recycle, subsystem restart or IPL. Then I verify the new level is actually active.

How I investigate SMP/E problems

1. Identify the FMID/SYSMOD and correct target zone.
2. Run/check RECEIVE and APPLY CHECK output.
3. Resolve missing requisites and HOLDDATA.
4. Check MODID conflicts or local USERMOD exposure.
5. Confirm target libraries and CSI agree after APPLY.
6. Perform the documented runtime activation.
7. Validate the product before ACCEPT.

I avoid “fixing” SMP/E by manually copying elements into target libraries unless the documented recovery process specifically calls for it. Manual copies create a split between the code on disk and the CSI record of truth.

IBM reference points

Release-specific syntax and limits can change. Use the IBM documentation linked below for the z/OS and IBM Z level installed on the target system before making a configuration change.

• IBM SMP/E commands
• IBM SMP/E documentation

Why I care about SMF

Almost everything important on z/OS leaves an SMF record somewhere: jobs and steps, dataset activity, security decisions, performance data, software usage and subsystem accounting. I think of SMF as the evidence stream that lets me reconstruct what happened after the console has moved on.

That is why an SMF failure is more serious than “reporting will be late.” Capacity, chargeback, SCRT, security audit and incident analysis may all depend on those records. If recording stops during a problem window, the system may recover but the explanation can be gone.

Record selection and SMFPRMxx

SMFPRMxx controls the recording environment: which record types/subtypes are collected, subsystem-specific options, exits and the recording model. A PARMLIB edit is not enough—I verify the active configuration after a SET SMF=xx or other approved activation.

Before enabling a high-volume type I estimate the effect on recording I/O, storage, offload and downstream consumers. “Turn on everything” is not a safe diagnostic policy on a busy production system.

D SMF
D SMF,O
SET SMF=xx

Traditional MAN data sets

In the traditional model, SMF writes to a set of MAN data sets. One is active while alternates are available. When a data set fills, SMF switches and the full data set must be dumped/cleared so it can return to service. The operational risk is obvious: if the dump process fails long enough, every usable target can eventually become unavailable.

IFASMFDP is the dump/extract program for SMF data-set input.

//DUMP     EXEC PGM=IFASMFDP
//INDD     DD DISP=SHR,DSN=YOURHLQ.SMF.INPUT
//OUTDD    DD DSN=YOURHLQ.SMF.EXTRACT,
//            DISP=(NEW,CATLG,DELETE),UNIT=SYSDA,
//            SPACE=(CYL,(50,10),RLSE)
//SYSPRINT DD SYSOUT=*
//SYSIN    DD *
  INDD(INDD,OPTIONS(DUMP))
  OUTDD(OUTDD,TYPE(0:255))
/*

How I expect MAN data-set offload to be automated

A healthy MAN-data-set setup should not depend on somebody noticing that SYS1.MANx is full and manually submitting a dump job. When the active recording data set fills, SMF switches to an empty alternate automatically. The important part is making sure the data set that just became full is dumped and returned to the empty pool before SMF needs it again.

The standard exit point for that hand-off is IEFU29. SMF invokes it when a recording data set fills and also when recording is switched with SWITCH SMF. The exit does not magically archive the data by itself; an installation can use it to issue a WTO or, more usefully, submit the site IFASMFDP dump job through an internal reader. IBM explicitly recommends using IEFU29 to drive the dump process rather than making the operator the scheduler.

SUBSYS(STC,
       EXITS(IEFU29,IEFU83,IEFU84,IEFACTRT))

IEFU29 is an STC-subsystem exit because the SMF writer runs under the master subsystem. If the installation does not define a SUBSYS(STC,...) statement, IEFU29 can instead be enabled on the SYS statement. Where the dynamic exits facility is used, the routine also has to be associated correctly through PROGxx/SETPROG; defining a routine in PROGxx alone does not make SMF call an exit that SMFPRMxx has disabled.

On a real system I also check whether an SMF management product owns this path. A vendor exit or automation product may already be intercepting the switch and performing the dump/accumulation work. Replacing that with the IBM sample path without understanding the existing chain can break the site's SMF pipeline.

The failure mode I watch for: switching is not the same as offload. The first MAN data set can fill and SMF can switch successfully, but if the dump job keeps failing, the pool of empty alternates gets smaller until recording has nowhere left to go.

SMF log-stream recording

SMF can also record through System Logger log streams. In that design, Logger policy, log-stream connectivity, staging/offload and Logger storage become part of SMF availability. IFASMFDL is used to dump SMF log-stream data.

When log-stream SMF fails I do not troubleshoot it as a MAN-data-set switch problem. I move down into Logger: log stream, connectors, CF/DASD staging and offload path.

The corresponding archive hook for log-stream recording is IEFU29L. It is invoked by SWITCH SMF; a site can use that hand-off to run IFASMFDL and archive log-stream data. I keep the two models separate in my head: IEFU29 is tied to MAN data-set switching/full conditions, while IEFU29L is the log-stream switch/archive path.

Record families I commonly look for

• Type 0: IPL information.
• 14/15/17/18: selected data-set open/close/scratch/rename activity, depending on object and configuration.
• 30: job/step accounting and resource usage.
• 70–79: RMF performance families.
• 80: RACF audit activity.
• 89: software product usage.
• 92: z/OS UNIX file-system activity.
• 99: WLM/SRM information.
• 100–102: Db2 families; 110 is widely used by CICS monitoring/statistics.

I always verify subtypes. Modern consumers frequently need a particular subtype, not just “type 30” or “type 80.”

RMF, SCRT and other consumers

SMF itself is the collection framework, not the final reporting tool. RMF writes/reads performance records, SCRT consumes software/capacity-related data, and products such as MXG/SAS or vendor analytics parse other record families. This is why downstream processing can fail even when SMF recording itself is healthy.

When expected records are missing

1. Confirm the system and exact time range.
2. Check active SMF configuration and TYPE/NOTYPE/subsystem selection.
3. Check exits that may suppress or alter recording.
4. Check whether the system uses MAN data sets or log streams.
5. Verify the writer/offload path was healthy during the window.
6. Inspect actual records to confirm the type/subtype is absent—not merely missing from one report.

The distinction I make is simple: not selected, not recorded, not offloaded and not reported are four different failures.

IBM reference points

Release-specific syntax and limits can change. Use the IBM documentation linked below for the z/OS and IBM Z level installed on the target system before making a configuration change.

• IBM SMF data sets
• IBM SMF log-stream recording
• IBM IFASMFDL parameters
• IBM IEFU29 SMF dump exit
• IBM IFASMFDP dump program
• IBM IFASMFDL / IEFU29L path

Why I treat console logging as infrastructure

During an incident I can usually recover from a missed screen refresh; I cannot recover a message that was never retained. The console log is where I go back to establish sequence: what failed first, what command was issued, which reply was given, whether a subsystem retried, and what changed immediately before recovery.

The piece that confused me when I first looked at this area is that “console”, “hardcopy”, “SYSLOG” and “OPERLOG” are related but not interchangeable. The console is where messages are presented. The hardcopy message set is the subset of console traffic z/OS has selected for durable logging. SYSLOG and OPERLOG are two different media that can receive that hardcopy set.

The hardcopy message set comes first

CONSOLxx controls the initial console and hardcopy environment. The HARDCOPY statement determines which command responses and routing codes belong in the hardcopy message set and which hardcopy medium is active. After IPL, operators can change hardcopy characteristics dynamically with VARY commands.

D CONSOLES,HARDCOPY
D CONSOLES

The first display tells me whether the active hardcopy medium is SYSLOG, OPERLOG or both, along with the command/routing criteria currently being captured. If a tool expects SYSLOG and the system is configured for OPERLOG only, that distinction matters immediately.

V SYSLOG,HARDCPY

I treat changes to hardcopy selection carefully. Narrowing the set can remove evidence; broadening it can materially increase message/log volume on a noisy system.

What SYSLOG actually is

SYSLOG is not an ordinary cataloged sequential data set that keeps growing forever. It is a system-managed SYSOUT data set on the primary JES spool for that z/OS system. It can contain the hardcopy message set plus entries written through mechanisms such as WTL and the operator LOG command.

That per-system scope is useful when I am reconstructing exactly what one LPAR saw. It also means that in a sysplex I may need to inspect more than one SYSLOG unless OPERLOG is available.

How SYSLOG rotates — and what WRITELOG really does

SYSLOG already has a rotation mechanism. When the active system-log data set reaches its configured message limit, z/OS closes it, queues it to the configured SYSOUT class and obtains a new SYSLOG. In other words, the current log does not simply grow until it consumes all remaining spool; the old generations become queued JES output.

WRITELOG lets operations force that boundary earlier. It can schedule the current SYSLOG for output processing, and the command also has forms for changing the output class or closing/restarting the log. I normally use it when I want a clean cut around an incident window or when the site's automation deliberately spins the log on a schedule.

WRITELOG

Message IEE043I is useful evidence because it tells me that a SYSLOG data set was queued and shows the SYSOUT class. That class is the next place to look if old logs are backing up on spool.

The real spool risk: rotation can be working perfectly while the queued SYSLOG data sets are not being processed. If the output class is held, has no writer, or retention is wrong, old logs accumulate on JES spool even though a fresh SYSLOG keeps opening.

External writer: getting queued SYSLOG out of JES spool

The classic MVS mechanism for moving selected SYSOUT to another device or sequential data set is the external writer. The IBM-supplied writer runs IASXWR00 in its own started-task address space and selects SYSOUT using criteria such as output class. That makes it usable for a dedicated SYSLOG class that I want written to DASD or tape instead of left on spool.

This is the shape of a DASD example. I would not copy it into production unchanged—the output class, allocation, DCB, retention and naming standard all belong to the site.

//SYSLOGX  PROC
//IEFPROC  EXEC PGM=IASXWR00,REGION=512K,PARM='PL'
//IEFRDER  DD   UNIT=SYSDA,
//             DSN=YOURHLQ.SYSLOG.HARDCOPY(+1),
//             DISP=(NEW,CATLG,DELETE),
//             SPACE=(CYL,(40,10),RLSE),
//             DCB=(RECFM=FBM,LRECL=133,BLKSIZE=0,BUFNO=2)

S SYSLOGX

In PARM='PL', P selects printer-type control-character handling and L is the eligible SYSOUT class in this example. The class must match the site's SYSLOG output class. More importantly, I prefer that class to be dedicated; otherwise an external writer selecting only by class can pick up unrelated SYSOUT along with the log.

Not every shop uses IASXWR00 for archival. Automation products, JES output management products, log collectors or retention tooling may own the process. The thing I verify is the end-to-end path: SYSLOG rotates → the closed log is queued → something processes that class → the archived log is actually retrievable.

SYSLOG versus OPERLOG

IBM recommends OPERLOG when a permanent operations log across a sysplex is required, and SYSLOG can still be active alongside it. I do not assume that because OPERLOG exists every tool is reading it—some tooling specifically expects SYSLOG, so I check the configured hardcopy medium and the tool's input path.

What I check when console history is missing

1. Was the message supposed to be in the hardcopy set? Check D CONSOLES,HARDCOPY, routing/command-level criteria and whether SYSLOG/OPERLOG was active at the time.
2. Did SYSLOG rotate? Search for IEE043I and WRITELOG-related messages around the period.
3. Where was the closed log queued? Identify the SYSOUT class and inspect JES output status.
4. Was the writer/archive path running? Check the external writer or whatever product/automation owns the class.
5. Was the data retained long enough? Output can be processed successfully and still be useless if the archive/GDG retention does not cover the period being investigated.
6. If the site uses OPERLOG, is Logger healthy? Treat missing OPERLOG data as a System Logger/log-stream problem, not a JES-spool problem.

Commands I keep close

IBM reference points

I use the installed z/OS release documentation for command syntax and writer details because console/JES standards differ by site.

• IBM — The system log
• IBM — DISPLAY CONSOLES
• IBM — Starting an external writer
• IBM — IASXWR00 EXEC statement

What WLM is actually trying to do

Every workload on a z/OS system is competing for finite CPU, storage and I/O. WLM’s job is not to make everything fast; it is to decide how the system should favor work so the most important business goals are met when resources are contested.

The useful mental model is: classify the work → assign it a service class and goal → measure how it is performing → let WLM/SRM adjust resource preference toward the defined goals. If the classification or goal is wrong, WLM can very efficiently optimize the wrong thing.

Service definition
  ├─ service policies
  ├─ workloads
  ├─ service classes / periods / goals / importance
  ├─ report classes
  ├─ resource groups
  └─ classification rules by subsystem type

Classification comes first

Subsystem classification rules decide where work lands. JES batch, started tasks, TSO, CICS, IMS, Db2 and enclave-based workloads have different classification attributes. Before I tune a service class, I verify the affected work is actually in the class I think it is.

A bad rule can look exactly like “WLM is starving my job” because the job may have been placed in a lower-importance class or wrong period. That is a classification problem first, not a resource shortage.

Service classes, periods, goals and importance

A service class can contain periods so long-running work can move from an initial goal to a different goal after consuming a defined amount of service. Goals can be response-time oriented, execution-velocity oriented, discretionary or system-defined depending on workload type.

Importance matters when multiple service classes are missing goals and WLM must decide where scarce resources should go. I do not raise importance casually; giving one class more preference can reduce another class’s ability to meet its goal.

Performance Index

Performance Index (PI) is the simplest high-level signal of how a service class period is doing relative to its goal. Roughly, a PI around 1 means the goal is being met; above 1 means the work is missing it; below 1 means it is doing better than required. The exact calculation depends on the goal type.

PI tells me that a goal is being missed. It does not tell me why. That is where delay samples matter.

Delay analysis is the diagnostic step

When a workload is slow I use RMF Monitor III or postprocessor data to identify the dominant delay. CPU delay, I/O delay, storage delay, enqueue delay and subsystem-specific waits point in completely different directions.

• CPU delay: check LPAR capacity/capping, processor contention and whether higher-importance work is legitimately winning.
• I/O delay: check devices/storage path and workload I/O pattern.
• ENQ delay: find the holder through GRS.
• Storage delay: examine paging/real-storage pressure rather than changing WLM first.

If the service class has poor PI but CPU is not the dominant delay, changing importance may simply move the symptom.

Enclaves, report classes and resource groups

Modern transaction work may execute as an enclave across multiple address spaces. In those cases I follow the enclave/service class rather than assuming the front-end STC owns all the resource consumption.

Report classes give me additional reporting separation without necessarily changing service management. Resource groups can bound or protect resource use, so they are another place to check when work appears capped despite available system capacity.

Useful displays

D WLM
D WLM,SYSTEMS
D WLM,IRD
D WLM,APPLENV=*
D A,L
D M=CPU

The displays confirm policy/system state. RMF is what I use to understand PI and delays over time.

The sequence I use for a “WLM issue”

1. Identify the actual affected work and service class/period.
2. Confirm classification.
3. Check goal, importance and PI.
4. Read the dominant delay.
5. Check LPAR cap/capacity if CPU delay dominates.
6. Only then consider whether the policy is wrong.

A WLM policy change is shared-resource redistribution. I want a clear hypothesis and rollback before I activate one during a live performance incident.

IBM reference points

Release-specific syntax and limits can change. Use the IBM documentation linked below for the z/OS and IBM Z level installed on the target system before making a configuration change.

• IBM: What is workload management?
• IBM WLM service classes

A dump is only useful if it captures the failing state

A dump is a snapshot of storage and control information taken so the failing program or system path can be analyzed after the event. The important part is not “getting a dump” in the abstract; it is getting the right dump at the time the defect is present, with enough address spaces/storage included to answer the question.

I therefore choose the dump mechanism based on scope: one application step, one address space, a subsystem, or the whole system when normal z/OS dump services cannot be trusted.

Common dump mechanisms

• SYSUDUMP / SYSABEND: traditional job-step abend dumps written through DD statements. SYSABEND contains more system-oriented information than SYSUDUMP.
• SYSMDUMP: machine-readable dump intended for IPCS-style analysis, commonly used for address-space debugging.
• SVC dump: system dump service used by z/OS components/products and operator requests; can capture multiple address spaces and system storage.
• Transaction dumps: middleware-specific dump facilities such as those provided by transaction managers; scope/content is controlled by the subsystem.
• SNAP: application/program-requested snapshot output.
• Stand-alone dump: separate IPLable dump program used when the system is too damaged for normal dump services.

JCL dump DD examples

//STEP2    EXEC PGM=A
//SYSUDUMP DD SYSOUT=*

//STEP1    EXEC PGM=PROGRAM1
//SYSABEND DD DSN=YOURHLQ.SYSABEND,
//            DISP=(NEW,CATLG,DELETE),UNIT=SYSDA,
//            SPACE=(CYL,(50,10))

//STEP     EXEC PGM=PROGRAM1
//SYSMDUMP DD DSN=YOURHLQ.SYSMDUMP,
//            DISP=(NEW,CATLG,DELETE),UNIT=SYSDA,
//            SPACE=(CYL,(100,20))

The product/runtime can override or supplement these mechanisms. Language Environment, Fault Analyzer, Abend-AID and other tools may change what is produced, so I check the actual job log and product configuration instead of assuming a DD statement guarantees a specific artifact.

SVC dump and SLIP

SVC dump is the workhorse for many z/OS and subsystem failures. A product can request it, an operator can request it, or a SLIP trap can be used to capture a dump when a precise event occurs.

SLIP is powerful because I can target an abend, module, address-space or other condition and define an action such as SVCD. It is also dangerous if the trap is too broad, so I build the narrowest trap possible and remove it after capture.

D SLIP
D DUMP,OPTIONS

Why “a dump was requested” can still mean “no dump exists”

I think of the capture path as a chain:

Failure / operator request / SLIP
          │
          ▼
    dump request created
          │
          ├─ dump mode / eligibility
          ├─ DAE duplicate suppression
          ├─ available dump data set / dynamic allocation
          └─ product recovery logic
          │
          ▼
       dump written

If support says “there is no dump,” I check each layer. DAE may have suppressed a duplicate. CHNGDUMP may have disabled a dump type. The application may never have requested one. A SYS1.DUMPxx target may have been unavailable. These need different fixes.

DAE, CHNGDUMP and SLIP are different controls

DAE recognizes duplicate symptoms and can suppress repeated dumps. CHNGDUMP controls dump options/modes. SLIP creates event-driven traps/actions. I do not disable all three just to obtain one failure. The safest approach is the narrowest targeted change that captures the required event.

IPCS starting workflow

Once I have the dump, IPCS is where I establish what actually failed: dump title/time, failing ASID, system/abend context, PSW/registers, storage, control blocks and component-specific formatting.

IPCS SYSTRACE
IPCS CTRACE
IPCS VERBX MTRACE

The exact IPCS commands depend on the dump type and component. I preserve the original dump and work from a copy when the investigation is likely to be long or shared with support.

What I preserve for escalation

• Dump data set name, title, system and timestamp.
• Abend/reason/module and failing job/STC.
• Messages showing whether DAE or dump-mode controls were involved.
• SLIP/trace settings active at capture time.
• Software level and recent maintenance.

A dump without the failure context can become a very expensive binary file that nobody can reproduce.

IBM reference points

Release-specific syntax and limits can change. Use the IBM documentation linked below for the z/OS and IBM Z level installed on the target system before making a configuration change.

• IBM common diagnostic aids
• IBM z/OS problem determination

Why DAE exists

Dump Analysis and Elimination protects the system from producing the same dump over and over for the same failure symptom. Without it, a looping defect could fill dump storage and create more damage than the original problem.

DAE does this by building a symptom string from identifying information about the failure, comparing that symptom with stored history, and then applying the active matching/update/suppression rules.

Failure / dump request
        |
        v
DAE builds symptom string
        |
        v
Compare with DAE history
       / \
 No match  Match
    |        |
Take dump  Apply active suppression rules

MATCH, UPDATE and SUPPRESS are separate ideas

MATCH tells DAE to compare symptoms with history. UPDATE lets DAE record/update symptom history. SUPPRESS controls whether a matching dump can be eliminated. Keeping these concepts separate matters when I temporarily want one new dump: I may want matching/history to remain active while removing suppression for the next occurrence.

ADYSETxx configuration

DAE=START,RECORDS(400),
SVCDUMP(MATCH,SUPPRESSALL,UPDATE,NOTIFY(3,30)),
SYSMDUMP(MATCH,UPDATE)

ADYSETxx members live in PARMLIB and define the DAE operating options. I activate the intended member with SET DAE=xx and then verify the active state rather than assuming the edited member is in effect.

Temporary control

SET DAE=AA

/* Example stop member */
DAE=STOP

Stopping DAE globally is a broad action. If my only objective is “capture one duplicate failure again,” I prefer a targeted method that keeps the rest of the protection intact.

Shared DAE history in a sysplex

DAE can use shared history so systems in a sysplex recognize the same failure consistently. That improves duplicate suppression across systems, but it also means a symptom captured on one LPAR can influence dump behaviour on another. I include the DAE data-set sharing model when I troubleshoot “why did this system suppress a dump?”

 Shared DAE data set
          |
     +----+----+
     |    |    |
    MVSA MVSB MVSC
     |    |    |
    DAE  DAE  DAE

The DAE data set and history

The DAE data set contains the symptom history. Clearing it can make future duplicates look new again, but that is intentionally broad—it removes useful history for unrelated failures too. I do not use “clear SYS1.DAE” as the normal way to obtain one repeat dump.

IPCS DAE panels or ADYDSP can be used to inspect the history.

IPCS 3.5 DAE

/* or */
ADYDSP 'SYS1.DAE'

My flow when an STC abends but no dump appears

1. Confirm what dump type the product was expected to request.
2. Search SYSLOG for dump/DAE messages and the exact symptom.
3. Check DAE history for a match.
4. Check D DUMP,OPTIONS / CHNGDUMP state.
5. Check available dump data sets/resources.
6. If support needs a fresh duplicate, change the narrowest DAE setting or use the approved targeted method.
7. Restore normal suppression immediately after capture.

DAE vs CHNGDUMP vs SLIP

I remember them this way: DAE decides whether a repeated symptom needs another dump; CHNGDUMP controls dump-service options; SLIP decides what event should trigger an action. They often appear in the same investigation but they are not interchangeable.

CHNGDUMP SET,SYSMDUMP,NODUMP
CHNGDUMP SET,SYSMDUMP
D DUMP,OPTIONS

14. Reference points

The source guide was cross-checked against IBM z/OS documentation. Relevant IBM documentation topics include using DAE to suppress dumps, ADYSETxx statements, defining a DAE data set, IPCS Option 3.5 DAE, TAKEDUMP, DISPLAY DUMP/CHNGDUMP, and SYSZDAE serialization.

Always use documentation for the z/OS release installed on the target LPAR when executing a change.

Tracing is about sequence, dumps are about state

A dump gives me a snapshot of storage at a point in time. A trace gives me a sequence of events leading up to or occurring during the problem. I use tracing when the question is “what path did the system take?” rather than only “what did storage look like when it failed?”

Tracing can be extremely low overhead or very expensive depending on the component, options and volume. I therefore do not start broad traces first and filter later. I decide what event I need, how long I need it, and how I will stop/format it before enabling it.

System trace

System trace records selected low-level system events continuously in trace tables. It is valuable in dumps because it can show dispatching, interrupts, SVCs and other events immediately around a failure. IPCS SYSTRACE is a common way to format it from a dump.

IPCS SYSTRACE

Master trace

Master trace captures console/message traffic and is useful when I need the message sequence around system activity. It is especially helpful when the operator log has rolled or I am working from a dump and need a time-ordered view of messages around the failure.

IPCS VERBX MTRACE

Component trace (CTRACE)

CTRACE is the mechanism many z/OS components use for detailed component-specific tracing. Each component defines its own options and formatting. Examples include XCF and BCPii-related tracing. I follow the component documentation because the correct PARM/options, writer and buffer size are component-specific.

D TRACE
IPCS CTRACE

A vendor may ask for a particular CTRACE name/options and a defined collection window. I keep their exact request with the evidence package so the trace can be interpreted correctly.

GTF

Generalized Trace Facility can collect trace records from system components and applications that write GTF trace data. It is useful when the requested diagnostic source explicitly uses GTF. Buffering, event selection and external writer data sets determine both overhead and how much history survives.

IPCS GTFTRACE

Transaction and product traces

Subsystems such as CICS, Db2, networking products and ISV software often have their own transaction or internal traces. I treat those as product diagnostics rather than assuming a system-wide trace will contain the needed detail. The owning product determines the trace classes, destination and formatter.

GFS and storage-oriented tracing

GETMAIN/FREEMAIN/STORAGE-related diagnostic tracing is specialized and can generate significant data. I only enable it when storage ownership/allocation behaviour is the actual question and the IBM/vendor procedure specifies the required scope.

How I choose a trace

• Need CPU/system event sequence from a dump: system trace.
• Need message sequence: master trace.
• Need one z/OS component’s internals: CTRACE.
• Need a GTF-producing component/application: GTF.
• Need middleware transaction internals: product transaction trace.

Collection discipline

1. Write down the exact symptom and reproduction window.
2. Record current trace state.
3. Enable only the requested component/classes.
4. Reproduce once if safe.
5. Stop/restore the previous setting immediately.
6. Preserve trace data with system/product level and exact timestamps.
7. Format with the component-appropriate IPCS/product tool.

The biggest tracing mistake is leaving a broad trace active because “support may need more data.” A diagnostic setting should always have an owner and a stop condition.

IBM reference points

Release-specific syntax and limits can change. Use the IBM documentation linked below for the z/OS and IBM Z level installed on the target system before making a configuration change.

• IBM z/OS traces

DR is a recovery operating model

I do not think of disaster recovery as “can we IPL the backup LPAR?” A useful DR test proves that the business service can be rebuilt from the alternate site with an understood data point, dependency order and operating procedure—and that we can later fail back without improvising.

The two numbers that drive the design are RPO (how much data loss is acceptable) and RTO (how long recovery can take). Storage replication, catalog recovery, database restart, network cutover and application validation all have to fit those objectives.

Dependency map before the test

Before a DR exercise I want a dependency map covering:

• SYSRES and system volumes.
• Master/user catalogs and SMS configuration.
• PARMLIB, PROCLIB, APF/LINKLIST/LPA libraries.
• Sysplex couple data sets and Coupling Facility structures.
• System Logger/SMF resources.
• RACF/security databases, certificates and cryptographic dependencies.
• TCP/IP/VTAM, DNS/VIPA/load-balancer changes.
• Product databases, zFS filesystems and application data.

If one of those layers is missing, an LPAR can IPL successfully and still be unusable.

Storage and data consistency

Replication has to provide a recoverable consistency point, not merely copied volumes. Multi-volume databases, catalogs and log streams need the replication technology and recovery procedure to preserve whatever ordering/consistency the product requires.

I verify which volumes are replicated, their DR device mapping and what happens to in-flight data at the cut. “The storage team replicated it” is not enough evidence for application consistency.

Hardware and LPAR preparation

The DR CPC/LPAR definitions must provide sufficient processors, central storage, channel paths, devices and network connectivity. HMC profiles, LOAD addresses and IODF/device mappings are checked before the event. Coupling Facility capacity and connectivity are equally important for a sysplex recovery.

Controlled bring-up sequence

The exact order is site-specific, but I bring services up from shared infrastructure toward consumers: system/core services, catalogs/SMS/security, sysplex/XCF/CF/Logger, UNIX/network, middleware, then applications.

I pause at each layer long enough to verify readiness. Starting everything at once makes it difficult to tell whether a later product failed because of its own configuration or because an earlier dependency never became healthy.

Validation

• System identity, IPL configuration and active PARMLIB are correct.
• Catalogs, volumes and SMS allocation work.
• XCF/CF/CFRM/LOGR resources are healthy.
• OMVS/zFS mounts are correct and writable where expected.
• Network paths/listeners resolve to DR endpoints.
• RACF/security and certificates work.
• Critical applications complete functional transactions, not just START successfully.
• SMF, dumps, monitoring and automation are active so the DR environment is supportable.

Stand-alone dump readiness belongs in DR

A DR site is not fully ready if it can run production but cannot capture a stand-alone dump during a severe failure. SADMP residence programs, output data sets, HMC profiles and console procedure should be validated at the alternate site too.

Failback / go-home

Failback is its own recovery event. I plan how data is resynchronized, which site becomes authoritative, how network routing is reversed, what outage is required, and how I prevent both sites from accepting conflicting updates.

After every exercise I want a defect list with owners and exact evidence: missing dataset, stale procedure, wrong port, insufficient CF capacity, certificate issue, manual step or timing problem. A DR test that ends with “successful” but does not update the runbook wastes much of the value.

IBM reference points

Release-specific syntax and limits can change. Use the IBM documentation linked below for the z/OS and IBM Z level installed on the target system before making a configuration change.

• IBM planning disaster recovery
• IBM sysplex availability and recovery

What SADMP is for

Stand-Alone Dump is the fallback I need when normal z/OS dump services cannot be trusted because the system is hung or severely damaged. It is a small independent dump program loaded from the HMC/console path, and it writes real storage to pre-prepared dump output.

The important word is prepared. A SAD procedure cannot be invented after the system is already down. The residence program, output data sets, HMC load profile, console path and operator steps all have to be tested beforehand.

Release compatibility

I generate the SADMP residence program for the z/OS release it is expected to dump. I do not carry an old residence program forward indefinitely just because the volume still IPLs. Release mismatch is exactly the sort of failure I do not want to discover during a real outage.

Residence program versus output data set

These are two separate objects. The residence program is what is IPLed to run SADMP. The output data set is where the storage image is written. Preparing one does not prepare the other.

Sizing the output

I size SAD capacity for the largest target LPAR and the options actually used. Real storage can change over time, so a dump area sized years ago may no longer be adequate after an LPAR memory increase.

D M=STOR

The output must also remain available at the disaster/failure point: correct device type, volumes online/accessible to SADMP, no migration assumption and enough initialized capacity.

Allocate and initialize the output

//SADDEF   EXEC PGM=IKJEFT1B
//SYSTSPRT DD SYSOUT=*
//SYSTSIN  DD *
  EXEC 'SYS1.SBLSCLI0(AMDSADDD)' +
       'DEFINE (SAD001,SAD002,SAD003,SAD004) +
        (SYS1.SADMP) 3390 <TRACKS> Y LARGE'
/*

The exact AMDSADDD operands and capacity values must be calculated for the target release/configuration. I keep the job as a repeatable build artifact rather than relying on manual one-time allocation.

Generate the residence program

//OSG      EXEC PGM=AMDSAOSG
//STEPLIB  DD DISP=SHR,DSN=SYS1.LINKLIB
//SYSLIB   DD DISP=SHR,DSN=SYS1.MACLIB
//         DD DISP=SHR,DSN=SYS1.MODGEN
//DPLTEXT  DD DISP=SHR,DSN=SYS1.NUCLEUS(AMDSADPL)
//IPLTEXT  DD DISP=SHR,DSN=SYS1.NUCLEUS(AMDSAIPL)

Sites can use one-stage or two-stage generation procedures depending on the release and design. I keep the IBM-documented procedure for the active release with the DR/operations runbook.

HMC and console readiness

The HMC profile identifies the SADMP residence device/load information. I verify that operators can select the profile, the residence volume remains IPLable and the expected console/input path works. A technically correct SAD program is useless if the night operator cannot locate or start it.

Sysplex partitioning and SSD

In a sysplex failure I also care about how the failed system is perceived by the remaining members. System Status Detection/partitioning behaviour can determine whether shared resources remain blocked. Before a planned SAD test I review the active sysplex/couple-data-set configuration and the installation’s failure-management procedure.

D XCF,COUPLE
D XCF,COUPLE,TYPE=SYSPLEX
F HZSPROC,DISPLAY,CHECK=(IBMXCF,XCF_SYSSTATDET_PARTITIONING)

Running SADMP

The exact prompt sequence is release/configuration specific, but operationally I want the runbook to answer: which HMC profile, which console, which target system/storage selection, which output volumes and what confirms the dump completed successfully.

During a real outage I avoid experimenting with optional scope reduction unless the IBM/vendor procedure calls for it; the value of SAD is capturing enough system state to diagnose a catastrophic failure.

COPYDUMP after z/OS is back

//COPY     EXEC PGM=IKJEFT01
//SYSTSPRT DD SYSOUT=*
//IN       DD DISP=SHR,DSN=SYS1.SADMP,
//            UNIT=3390,VOL=SER=SAD001
//OUT      DD DSN=YOURHLQ.SADMP.COPY,
//            DISP=(NEW,CATLG,DELETE),UNIT=SYSDA
//SYSTSIN  DD *
  COPYDUMP INDD(IN) OUTDD(OUT)
/*

I copy the dump into normal z/OS-managed storage for IPCS/support handling, preserve the original until the copy is verified, and record which SAD volumes contain the source.

Clear and re-arm

//SADCLR   EXEC PGM=IKJEFT1B
//SYSTSPRT DD SYSOUT=*
//SYSTSIN  DD *
  EXEC 'SYS1.SBLSCLI0(AMDSADDD)' +
       'CLEAR (SAD001,SAD002,SAD003,SAD004) +
        (SYS1.SADMP) 3390'
/*

After the dump is safely copied, I clear/reinitialize according to the procedure so the facility is ready for the next emergency. “We successfully took a SAD last year” is not the same as being ready today.

Readiness checklist

• Residence program level matches the active z/OS release.
• Residence volume/profile is still IPLable.
• Console/input path is known and tested.
• Output capacity matches current LPAR storage.
• Output volumes/data set are initialized and accessible.
• Sysplex partitioning/SSD procedure is understood.
• COPYDUMP job is tested.
• Operations and DR documentation contains the exact procedure.

IBM reference points

Release-specific syntax and limits can change. Use the IBM documentation linked below for the z/OS and IBM Z level installed on the target system before making a configuration change.

• IBM running stand-alone dump
• IBM AMDSADDD utility
• IBM AMDSADMP syntax