MVS TOOLS AND TRICKS OF THE TRADE November 1997 Sam Golob MVS Systems Programmer P.O. Box 906 Tallman, New York 10982 Sam Golob is a Senior Systems Programmer. HELP IN KEEPING YOUR SYSTEM UP Twenty-four by seven operation, or having your MVS system (almost) always up, is a requirement in more and more places nowadays. Even if your shop isn't handicapped by the necessity of such a requirement, the amount of "scheduled down time" for your system, is usually minimal. In the heat of the day's production work, you certainly don't want the system down because of a small problem, and you'd rather have a way to fix the problem while the system is running. Today, we'll have a discussion of a few things you can do, to make dynamic changes to your system. IBM has also gone this route during the last few years. Dynamic APF list changes, dynamic Link List changes, dynamic LPA list changes, dynamic UCB changes, and other ways of reconfiguring your system while it is running, have either already been made a part of MVS (OS/390), or soon will be. The reason for this is obvious, and has been expressly stated by IBM. With more of a requirement, in many shops, for continuous operation of the system, there has to be a way of making sizable changes without an IPL. The general principle of a dynamic change is quite easy to understand. It used to be, in older versions of MVS, that certain system resources, lists of resources, or pointers to resources, would be created at IPL time. These would occupy a fixed place in the nucleus area or another common area of machine storage, where their information would be accessible to all jobs and tasks running in the system. A typical example of such an arrangement, is the configuration of peripheral devices available to the MVS system. It used to be, that you'd have to statically define a list of all devices that your system would conceivably have access to. Then you'd have to run a process (now obsolete) called IOGEN which is a subset of SYSGEN or System Generation, where all of these devices would be represented by a fixed area of Unit Control Blocks (or UCB's), built into the system Nucleus. There would be (as there still is) one UCB for each device. These could not be changed, except by a complete rebuilding of the system Nucleus through another SYSGEN process. And it was then inconceivable to add new devices to the system, except with considerable difficulty, and with much prior planning. IBM worked to make the arrangement and creation of UCB's more flexible. First, the large, contiguous fixed block of UCB's was moved out of the Nucleus itself. (This was at the MVS/XA 2.2.0 level.) Thus, you could replace a Nucleus without altering the I/O configuration. Then, at the MVS/ESA 4.2 level (approximately), the large, contiguous block of UCB's was replaced by a flexible definition of UCB's in common storage, where the UCB's were not necessarily contiguous to each other, but were pointed to by a much smaller, contiguous set of pointers--a UCB lookup table. By the MVS/ESA 5.1 level for sure, the original UCB lookup table could be dynamically replaced by another one, allowing for the addition of new devices "on the fly", without an IPL of the system. All of this allowed for the avoidance of a system IPL, when a system configuration needed change. There is a general principle behind this. Old systems were storage-constrained, and there wasn't much room to put system resources that were required by everyone. Therefore these resources were put into fixed blocks of storage in some common area at IPL time. With the advent of XA (a 2-gigabyte potential region) and ESA (easy cross-access to more than one of these regions), together with bigger real storage facilities to support these virtual storage potentialities, it became possible to think of creating multiple copies of a system configuration, that could be changed. There was now enough "room" in the computer. It also helps if the contiguous block of resource information can be broken up into many single pieces that are chained together, so that if you add one new piece, you don't have to replace the entire collection of information. You just chain the new piece in with the others. For example, if you add one new resource, you can add it into the existing chain of resources, without having to create an entire new chain. IBM has been gradually restructuring its various system resources in this way, so that formerly unchangeable resources, could be altered "on the fly", without an IPL. The system could then be altered, but it would remain up continuously. WHAT CAN WE DO? We ourselves are running various levels of MVS, or OS/390. Some of us, who are running the newer systems, can take more advantage of IBM's improvements. There are more dynamic facilities available on the newer systems. However, I wish to discuss some tools today, which are available for many of the older systems, as well as the newer ones. By knowing about these tools, and the techniques and principles they are based on, you can improve your "up time" in any MVS shop, and often avoid IPL's when you don't want to have them. In this column, I always advocate that you should do any job by whatever means you have. If your installation has purchased a vendor product, then by all means, you should use it. However, I make it a practice not to write about a vendor product, when a free one will also do a similar job, even if the vendor product is better. (Sometimes the free one happens to be better.) The reason for this is that the shops with low budgets, or without that particular product, should also be able to benefit from the information presented here. An enormous collection of free system tools is available on the CBT MVS Utilites Tape, which is independently produced, and is available through the NaSPA office. The CBT Tape is updated quite often. A recent version of the CBT Tape is included on the NaSPA CD-rom disk, together with other collections. There are quite a few tools, packages, and facilities available on the CBT Tape, which can be used to fix a system problem "on the fly", and to avoid an IPL. The facilities we'll talk about today are: XACORZAP, LOADMLPA, SVCUPDTE, and Fullscreen ZAP. XACORZAP has an interesting story. Old-timers among us will remember IBM's INCORZAP program, that was distributed as object code by the Field Service Representatives. INCORZAP will zap the in-storage copy of a Nucleus or LPA module, if the exact same program exists also on disk, in SYS1.NUCLEUS or SYS1.LPALIB. INCORZAP takes AMASPZAP-type control cards; the only difference is that you execute PGM=INCORZAP instead of PGM=AMASPZAP in the batch job. INCORZAP will check the load module structure of the copy on DASD, and then find the entry point of the in-storage copy. INCORZAP then goes to the proper displacements in the in-storage copy, and does the VERifies and REPlacements in machine storage. With XA and later systems, INCORZAP will not always work. This is because MVS/XA has some parts of storage which are "page protected", so that you cannot alter any of that storage. Since earlier OS and MVS systems did not have this facility, INCORZAP did not have any code to get around it. The INCORZAP object code was disassembled by Bill Godfrey around 1980, and the assembler source was sent to the CBT Tape around then (by Jim Marshall). A few years later, the original author of INCORZAP, Robert Budge, was looking through a CBT Tape and saw his own program, written in assembler source, instead of the original PL/S. Budge, who no longer worked for IBM, and who no longer had access to the PL/S compiler, took the assembler source for his program, and fixed it to get around the XA (and ESA) page protection. The result was XACORZAP, which can be found on File 421 of the CBT Tape. Budge even wrote a new user manual for XACORZAP. XACORZAP can be used to keep your system up in an emergency, if there is a serious problem, which depends on a zap fix to correct. Even though an IPL would normally be recommended, the zap could be directly applied to the working code in storage. Since XACORZAP is run as a batch job, I'd recommend saving the exact JCL from all runs of this program in a special library, for reference in your shop. Also, I would recommend making corresponding changes to the DASD copy with SMP/E as soon as possible afterwards. Anyway, in an emergency, this technique might save your system. XACORZAP page-fixes all the in-storage code that it alters, so the change will not disappear with a page-out and re-load from PLPA. Of course, XACORZAP has to be run from an APF authorized library. LOADMLPA and SVCUPDTE are two programs that are available on File 183 of the CBT Tape. That file is from Gilbert Saint-flour. These programs are modernizations of the old MODREP program (still on the CBT Tape) that worked for MVS/370 and older systems. The idea is to replace the working copy of an LPA module (in the case of LOADMLPA) or an SVC (in the case of SVCUPDTE) from a fresh version on DASD. The principle involved is the following: Some common storage, in the proper subpool, is obtained by these programs. Then, the new copy of the program is loaded into this storage. Finally, the system pointers to these modules are altered to point to the new code, instead of the old code. In the case of LOADMLPA, it doesn't quite work that way, but after the common storage is obtained and loaded with the new program, LOADMLPA adds an entry (a CDE) to the Active LPA queue, which overrides the LPDE (Link Pack Directory Entry) in the PLPA Directory. In the case of SVCUPDTE, the SVC Table pointer to the module entry point is altered. In either case, the new copy of the program starts working with every new invocation. What problems can occur with this procedure? In the case of loading a new copy of an LPA module, there is sometimes a problem with aliases. Suppose an LPA module has several aliases, and when the main module is replaced, the alias may not be replaced. LOADMLPA correctly processes just one alias. That's it. With most other of these types of programs, each alias name has to be re-loaded separately from the new copy, in order for that alias name to be used. IBM's new LPA loading scheme (in OS/390 Release 4) also has that drawback. However, OMEGAMON (from Candle Corporation), which is a vendor product, has a similar LPA and SVC loading facility called LPAM, which substitutes a new copy of an LPA or SVC module, and which also loads all the alias names at once. With OMEGAMON's LPAM command, you don't have to worry about reloading all the alias names individually. There is another possible drawback to this procedure, which does not occur often. Sometimes, an LPA module, as it is being initialized, checks to see if it has been loaded into the PLPA region of storage. The common storage which a program like LOADMLPA obtains, is out of this storage range, and the module's operation will fail when it detects this condition. Most LPA modules don't execute such storage range determinations, so there's no problem. You just have to be aware that rarely, this problem might happen. Occasionally an SVC might do this also. Statically loaded SVC's are usually put into SYS1.LPALIB or an LPALIST library. Modern, dynamically-loaded SVC's are usually not, and therefore will not contain such coding. Coding to determine whether the module is loaded into the PLPA region of storage, is generally done as a security measure, to stop someone from dynamically replacing an existing SVC. We'll conclude with a discussion of another program that can sometimes help to keep your system running. This is the Fullscreen ZAP program, from File 134 of the CBT Tape, as enhanced by Greg Price. Fullscreen ZAP is a program which allows you to look at a dataset on DASD, and change anything. If your security program is in place properly, the security may allow looking, but no changing of data. Fullscreen ZAP contains an enhancement which is invoked with the FULLVOL keyword, that allows you to look at the entire disk pack, instead of just at the one individual dataset. When you invoke Fullscreen ZAP with the FULLVOL keyword, the program points you initially to Track 0, Record 1 of the Disk Volume that the dataset is on. This allows you to easily "clip" the pack, or change its volume id, while the pack is still online. IBM's program for this, ICKDSF, requires that the pack be offline while the "clipping" is taking place. Sometimes, you can't get the pack offline, and you still need to clip it. With Fullscreen ZAP, you can invoke the program by pretending to zap dataset 'FORMAT4.DSCB', or the VTOC, pointing to VOL(volser), and using the keyword FULLVOL in addition. You then get to Track 0, Record 1, of the pack you want to clip. The Disk Id record is Record 3, so you invoke ZAP's "R" command twice, to advance to the third record of track 0. The volume id starts at +8 from there, for six bytes, so you can then alter it to another name. When you later bring the pack offline and then online, it will come online with the new volser name. On older systems (XA and below), after you bring it online again, you have to issue a MOUNT operator command, mentioning the new volser name. This technique adds a lot of flexibility, in situations when ICKDSF will not work properly, or can't be used for some reason. Fullscreen ZAP is a program that every "system doctor" ought to be familiar with, because it easily allows DASD manipulations that are much harder, using conventional IBM means. Also, it works on many of the "system" parts of a DASD volume, and on deleted data, which are not easy to reach by other means. Fullscreen ZAP has a string search capability, through its FIND keyword, that will save you a lot of work if something needs to be found and fixed on a disk volume. I hope that this month's discussion will expand your mind and help you to deal with a greater range of emergency situations. I really hope you won't have too much occasion to use these techniques, but they're definitely life-savers when you do need them. Good luck. See you next month.