MVS TOOLS AND TRICKS OF THE TRADE October 1998 Sam Golob MVS Systems Programmer P.O. Box 906 Tallman, New York 10982 Sam Golob is a Senior Systems Programmer. He also participates in library tours and book signings with his wife, author Courtney Taylor. Sam can be contacted at sbgolob@cbttape.org. Information about the CBT MVS Tapes can be found on the web, at http://www.cbttape.org. AN IMPORTANT Y2K TECHNIQUE As the "magic deadline" of January 1, 2000 approaches, it may be necessary for us systems programmers to make some sweeping program changes, very quickly, on that morning. There is one technique that once literally "saved my company" in such a situation, and I feel morally bound to teach it to every systems programmer in the world, if I can get to them, and if they'll listen. It is absolutely essential that each and every one of you knows about this technique. If you don't have it (or an equivalent) in place and ready to use, you could be lucky, but you may be very sorry. It could cost your company many hours or days of production time, and possibly, for some people, their jobs. This is serious business, and it's our business! As far as I know, IBM doesn't have an easy way to do this job either, so I don't think you'll get much help from them, unless they change things very much between now and that big January 1st. I do have an easy and quite painless way to do this almost impossible-looking task in a very short time. Here it is. Please listen well, for your own good. THE SUBROUTINE REPLACEMENT PROBLEM. I faced this problem on January 1, 1991. A time-dependent subroutine which did date conversion for our company's production processing, failed on that January 1st. The consequences of the failure were catastrophic--a user abend was produced by the subroutine when it didn't work. Batch programs which used this subroutine, failed with the user abend. CICS and other online transactions which used this subroutine, did worse. At that time, each CICS region used only a single TCB--remember? In other words, a failure of the subroutine in any transaction, would bring the entire CICS region down. Moreover (and this is the kicker), this subroutine was directly linkedited into many of the company's batch and online production programs. We didn't know which programs contained it. There were upwards of 3300 load modules in the batch load library. And there were more than 1100 load modules in the online load library. Using "standard IBM methods", this problem would take months to fix! Can you imagine the production impact? It didn't take long for a programmer to fix the actual subroutine, which was a small assembler program, and relatively simple. The big question was, how could we find and replace the bad version of that subroutine in all the load modules which contained it? IBM doesn't help. Load module names don't indicate any of the subroutine names, and ISPF member lists only list load module names, not any subroutine names. The standard "brute force" IBM solution would be to take an AMBLIST listing of all the load modules to find the presence of the bad subroutine. (4500 load modules?) Then you'd have to write re-linkedit JCL to include the new subroutine that would replace the bad one--by hand! How could you make sure the linkedit JCL was right? You could cause far worse production problems, if it was wrong. You'd have to carefully examine the linkage editor report for each reworked load module. This sticky situation would take a long time to fix, and you'd never be sure you didn't miss a load module somewhere. I fixed the problem, with complete accuracy, in two hours flat, on January 2, 1991. And I didn't miss one load module. Do you want to read on, to see how to do it? SOLVING THE PROBLEM. My quick solution to replacing a bad subroutine in many places, requires the installation of a product. The up-side, though, is that the product is free. Many of you have this product already installed at your shops, however. I've also posted the latest source code for it on a web site, so you can download and install a new version for yourselves. With this product in place, you can solve the "global subroutine replacement problem", and many others as well. The product we need, is the PDS program package, currently at version 8.5, which resides on File 182 of the CBT MVS Utilities Tape, or which can be downloaded from a web site at: http://members.aol.com/freepds The CBT Tape may be obtained from the NaSPA office, and a (somewhat backleveled) version of it can be found on the NaSPA cd-rom. The CBT Tape is a huge conglomeration of system programmer goodies, which no MVS or OS/390 systems programmer should be without. There is a vendor-supported version of PDS, called STARTOOL, from Serena International in Burlingame, California. STARTOOL contains all the functionality of "free PDS", and much more, so if your installation is licensed for STARTOOL, you don't have to install anything else. The PDS package works as follows: PDS is a TSO command, that can be used with ISPF or without ISPF. You point PDS at a dataset, and then you have a very large number of choices what to do afterward. PDS, even the free version, has perhaps 50 separate subcommands, that allow more than 1000 separate utility functions to be done to the pds, or to some of its members. PDS automatically treats "load module" datasets (RECFM=U) differently from "source type" datasets (RECFM=FB or VB). The PDS package (and of course, STARTOOL) has a powerful facility for keeping track of "member subgroups" of the current partitioned dataset it is pointing at. One of the main purposes of having member subgroups, is to be able to do an operation to all the members of the subgroup, as if they were one member. In other words, PDS has the capability of dealing with all the members from the current subgroup as one unit. You just refer to an asterisk '*' instead of the member name, and a selected operation will be performed on ALL members of the current "member subgroup", instead of on one individual member. This capability will help us very much. Choosing the proper member subgroup in each load library, is an important key to solving our sticky problem. To be specific, we'd like to isolate a subgroup of load module members which all contain a CSECT having a given name (the name of our subroutine). Then we can re-linkedit all those load modules, and those only, to fix them. In other words, we've got to find them, then fix them. Sometimes even that is not enough. Suppose there are two different versions of the subroutine floating around in our libraries, with the same CSECT name. Or perhaps (in a tougher case) there are two entirely different programs with the same CSECT name. We might want to further refine our member subgroup to contain only one of those versions or programs, but not both. Using the PDS package, that can be done also, without much additional effort. We're getting an indication at this point, that with the PDS command, we have a lot of power. It remains, once we have a proper and complete list of load modules to rework, that we need to re-linkedit them and replace the version of the subroutine which is wrong. The PDS package comes up strong in that department also. PDS has a facility, where it can look at a load module, and generate accurate JCL and linkage editor control statements, to re-linkedit the load module exactly as it was before. All the proper attributes are generated, such as reentrancy, refreshability, APF authorization, and so forth. Also, ORDER statements are generated by PDS in the linkage editor control statements, so that the re-linkedited module has all its CSECTs in the same order that the original module had them. It is now obvious that we have all the ingredients in place, to solve our problem. Besides that, there are a few other considerations. If your load library is a PDSE, copy it with IEBCOPY to a pds, and then you can use the free PDS package on it. Free PDS doesn't have PDSE support, but STARTOOL does. If you're using STARTOOL for this, it doesn't matter if the library is a pds or a PDSE. The PDS package is written in assembler language. It runs at assembler speed (i.e. "quick"), and has optimized I/O routines to fetch the members of any partitioned dataset with the greatest speed. For most of this, we have to thank Bruce Leland and Steve Smith. They are the principal "enhancers" and builders of this product, and they maintain the vendor version, STARTOOL. John Kalinich is maintaining the free version of PDS at this time, and he's doing a great job, too. So far, you've seen that PDS can do this formerly tedious job very quickly, and now we'll look at the specifics. CORRECTLY FIXING ONLY THE BROKEN LOAD MODULES In fixing our "globally scattered subroutine" problem, we must first determine which libraries are affected. These, most likely, will be application load module libraries. First, you must determine the production libraries involved, and then, you should probably make the same changes to the programmer libraries and development libraries, so that the old error will never work its way back into production again. If you have a change control system, you'll have to work within its restrictions and parameters. In any case, if you've anticipated the problem, you'll be able to be orderly. If not, you'll have to be just as orderly, but also fast. Once you've got the list of libraries, you'll deal with them one at a time. For simplicity from now on, we'll pretend that only one load library is affected. But it usually isn't the actual case, and you'll just repeat the same procedure for all the libraries that you have to rework. The first task is to determine all the load modules which contain the affected CSECT name. This will probably be the name of the subroutine you want to replace. PDS and STARTOOL both have a powerful method of finding only those load modules which contain a CSECT with a certain name. This is the "IF" subcommand of PDS, a little-known but powerful tool. The purpose of the IF subcommand is to create a subgroup of members, based on a certain criterion. The IF subcommand will select a subgroup of members, starting from all members, or it will further refine a subgroup that was previously selected. Let's invoke PDS, and point it to the proper library. Under TSO, you'd say: PDS 'library.name' to invoke the PDS command, and at the same time, to point it to the proper library. Then you deploy the IF subcommand, and say: IF : MODULE(csectnam) THEN(SUBLIST) The IF subcommand will now look at the starting member group, which is the colon ":". The colon means "all members". From this starting member group, IF examines the structure of every load module, and if it finds a CSECT matching the given (partial or complete) name you said in the MODULE() keyword, it selects that load module for inclusion in the new "member subgroup". Otherwise IF excludes the load module from the subgroup. So we now know, that after this operation, every member of the current member subgroup contains a CSECT with the given module name. We've done the "find" part, of the "find and fix" process. Well, not always. Sometimes, there might be two or more versions of that CSECT floating around in our libraries. To further select one of the two, we could do any of a number of things. One thing we could do, is to find a character or hex string which exists in one of the versions, but not in the other. This string should be so unique, that it's not likely to be found in a different CSECT than the one you're dealing with. Under the PDS package, the FIND subcommand not only finds strings, but it can select members for inclusion into a member subgroup, based on whether a certain string is found in the member. FIND, when used in this way, works like IF. Remember that the asterisk "*" refers to all members of the current member subgroup. Then, you can say: FIND * /charstring/ MODULE(csectnam) THEN(SUBLIST) or FIND * xhexstringx MODULE(csectnam) THEN(SUBLIST) This operation will further refine the member subgroup, picking only from members of the former member subgroup, which contain the string somewhere within the named CSECT. There are other ways you can pick and choose members. The ultimate aim is to make sure that you have a member subgroup which contains every member you want, and no others. Now the re-linkedit part gets deployed. The PDS subcommand which looks at a load module and generates re-linkedit JCL automatically is the "MAP" subcommand with its RELINK keyword. So, once you've determined a proper member subgroup, you use the MAP subcommand as follows: MAP * RELINK This produces an output listing to the PDS LOG (under ISPF mode) or to the terminal (under line mode). That output may be written to a file, using one of the keywords of the CONTROL subcommand, so it can be edited. This brings us to the last step. The re-linkedit JCL produced by the PDS program, assumes that the SYSLMOD library and the original library are the same partitioned dataset--the load library that PDS was pointing at. After PDS generates re-linkedit JCL for all the load modules in the member subgroup, you must edit each SYSLMOD DD statement to point to a different, newly allocated output library, so that the re-linkedited load modules do not replace the original members immediately, during this re-linkedit operation. Only when these new load modules are tested and checked out, can you copy them over the old members in the original library, or you should use whatever "safe" production control procedures you have. The resulting JCL usually needs only a JOB card to run it, once the SYSLMOD DD cards are fixed. But in our case, where we need to replace the bad CSECT, we need two more things. A repaired version of the subroutine, as a load module or as an object deck, must be placed in a library, and a DD card must be inserted into each step of the re-linkedit JCL, to point to that library where the repaired module resides. Then, an INCLUDE linkage editor control card, pointing to the name of the module, and the DDNAME referring to its library, must be inserted before the main INCLUDE cards, so the new CSECT will replace the old one having the same name. This must be done for each module you want to re-linkedit. That being done for all the generated JCL, for all the load modules in the member subgroup, the JCL is run, and all the necessary load modules will have been found and fixed. It only remains to test the new load modules, and (once tested), to copy them back to the original libraries. Or you can run production from new libraries containing the newly fixed modules, in addition to all the other ones. DETERMINING YOUR VULNERABILITY IN ADVANCE It is important to look at all your production programs, and to determine if any subroutines are used often, during the processing. Once it is found that there are frequently used subroutines in production processing, you can do all of the IF processing in advance, to determine where these subroutines are. Then, you can generate the re-linkedit JCL in advance, so that it will be ready in case of emergency, on January 1, 2000. As you become more familiar with the process, it should be redone whenever there are significant changes to the production load libraries. In case any of these subroutines has to be fixed, you've got the mechanisms already there, so the fixed versions can be put into place quickly and accurately, wherever they belong. Just for reference, the SMP/E language for this situation, in a system module context, is having multiple LMOD entries for a single MOD. But we're dealing with application load modules here, and we have to do this job without the help of SMP/E and its record-keeping. I sincerely hope that every one of you learns this technique. With the Y2K situation looming over us, this technique is so important, that I cannot, in good conscience, neglect to teach it. You might be able to easily fix a program, but replacing it wherever it is found in production processing, may be a far harder matter. Now, it doesn't have to be so hard. You've just got to know how to do it right. Good luck. Please learn this technique well! We'll see you next month.