HITCHHIKER'S GUIDE TO RSX An Introduction to RSX for Business-Application Developers Revision 0.0 Disclaimer This is a preliminary version and is not quite one-hundred percent complete. Please send comments, questions, and recommendations -- on this document -- to A-to-Z Base Product Marketing, MKO1-2/E25, Digital Equipment Corporation, Continental Boulevard, Merrimack, New Hampshire 03054. The information in this document is subject to change without notice and should not be construed as a commitment by Digital Equipment Corporation. Digital Equipment Corporation assumes no responsibility for any errors that may appear in this document. The software described in this document is furnished under a license and may only be used or copied in accordance with the terms of such license. No responsibility is assumed for the use or reliability of software on equipment that is not supplied by Digital Equipment Corporation or its affiliated companies. Copyright (C) 1986 by Digital Equipment Corporation All Rights Reserved. Printed in U.S.A. September, 1986 Hitchhiker's Guide to RSX Revision 0.0 Table of Contents Chapter 1 Introduction 1.1 Purpose of this Guide 1.2 Intended Audience 1.3 On the Synergy of Operating Systems and Applications 1.4 Elements of Application Tuning Chapter 2 Overview of RSX 2.1 History of RSX 2.2 RSX Version 3.0 Chapter 3 Getting Started 3.1 The Three Faces of RSX 3.1.1 Micro/RSX 3.1.2 Pregenned M-PLUS 3.1.3 RSX-11M-PLUS 3.2 Installation and Startup 3.2.1 Installing Micro/RSX 3.2.2 Installing Pregenerated M-PLUS 3.2.3 Installing M-PLUS 3.3 Logging in and other First Day Tasks 3.3.1 Logging into the Prebuilt Accounts 3.3.2 Post Installation Cleanup 3.3.3 Customizing Startup 3.3.4 Configuring Terminals and Printers 3.3.5 Creating User Accounts 3.4 Installing Layered Products 3.4.1 Installation on Micro/RSX 3.4.2 Installation on M-PLUS Chapter 4 Terminals 4.1 Cost of I/O 4.2 Specific Cost of Terminal I/O 4.2.1 Interrupt Service 4.2.2 Scheduler Overhead 4.3 Minimizing Costs 4.3.1 Blocking Output Requests 4.3.2 Blocking Terminal Input 4.3.3 Using DMA Terminal Hardware Chapter 5 Record Management Services (RMS) 5.1 Introduction to RMS 5.2 File Organizations 5.2.1 Sequential 5.2.2 Relative 5.2.3 Indexed 5.2.3.1 Structure 5.2.3.2 Population 5.2.3.3 Index Activity 5.2.3.4 Storage Overhead 5.2.3.5 Interlocks 5.3 File Design 5.3.1 Which Files to Design 5.3.2 Selecting an Organization 5.3.3 Common Design Factors 5.3.3.1 Allocation 5.3.3.2 Extend Quantity 5.3.3.3 Contiguity 5.3.3.4 Carriage Control 5.3.4 Designing Sequential Files 5.3.5 Designing Relative Files 5.3.6 Designing Indexed Files 5.3.6.1 Specifying Key Fields 5.3.6.2 Sizing Buckets 5.3.6.3 Areas 5.4 Tuning RMS Files 5.4.1 Avoid File Opens 5.4.2 Use the Simplest Possible File Structure 5.4.3 Beware of Overloading Directories 5.4.4 Pre-Allocate the File 5.4.5 Make Critical Files Contiguous 5.4.6 Substitute Code for Mechanical Movement 5.4.7 Distribute the Load 5.4.8 Avoid Going Overboard Chapter 6 RMS Utilities 6.1 RMSDES File Design Utility 6.2 RMSIFL Indexed File Load Utility 6.3 RMSCNV File Conversion Utility Chapter 7 RMS for CTS-300 Users 7.1 Record Locks 7.2 Access Modes 7.3 Interchanging Sequential and Relative Files 7.4 Extending Files 7.5 Print Files Chapter 8 Flow Control 8.1 Concept of a Task 8.1.1 Installing a Task 8.1.2 Task Names 8.2 Requesting an Installed Task 8.2.1 RUN a Task 8.2.1.1 Running an Installed Task 8.2.1.2 Running a Task from a File 8.2.1.3 Running a System Utility 8.2.1.4 RUN as a Scheduling Command 8.2.2 Chaining between Tasks 8.2.2.1 Indirect Chaining 8.2.2.2 Direct Chaining 8.2.2.3 Chaining to a Command File 8.2.3 Spawning an Offspring Task 8.2.3.1 Indirect Spawn 8.2.3.2 Direct Spawn 8.3 Parent-Offspring 8.3.1 Command Line 8.3.2 Status Code 8.4 Intertask Communication 8.4.1 Send/Receive 8.4.2 Global Event Flags Chapter 9 Linking and Overlays 9.1 Invoking TKB 9.2 Disk Overlays 9.2.1 Trees 9.2.2 Co-Trees 9.3 Memory Resident Overlays 9.4 Guidelines for Use 9.5 Linking to RMS 9.5.1 Disk Resident RMS 9.5.2 Memory Resident RMS 9.5.3 Clustered RMS 9.5.4 Supervisor Mode RMS 9.6 Useful Options 9.6.1 Privilege 9.6.2 Task Priority 9.6.3 Task Name 9.6.4 Cross Reference Map Chapter 10 Packaging Applications for RSX 10.1 Kitting for Micro/RSX 10.1.1 Micro/RSX Layered Product Installation Architecture 10.1.2 Creating the Kit 10.1.3 Developer's Note 10.2 Kitting for M-PLUS 10.3 Kitting Applications for A-to-Z Chapter 11 System Operation 11.1 Terminals 11.1.1 CLI 11.1.2 BROADCAST 11.1.3 CONTROL=C 11.1.4 INQUIRE 11.1.5 FULL_DUPLEX 11.1.6 ESCAPE 11.1.7 EIGHTBIT 11.1.8 HOSTSYNCH 11.1.9 SERIAL 11.1.10 SLAVE 11.1.11 LOWERCASE 11.1.12 WRAP 11.1.13 TYPEAHEAD 11.2 Partitioning Memory 11.2.1 Exec and Primary Pool 11.2.2 Secondary Pool and Extension 11.2.3 Task Space 11.2.4 Cache Region 11.3 Disks 11.3.1 Initializing 11.3.1.1 Preparing a Disk 11.3.1.2 Initializing a Volume 11.3.2 Mounting 11.3.3 Dismounting 11.4 Creating Directories 11.5 Files 11.5.1 File Performance 11.5.2 File Protection 11.6 Logical Names 11.7 Disk Caching 11.8 Printers and Queues 11.8.1 Queue Mechanism 11.8.2 Submitting Jobs 11.8.3 Startup 11.8.4 Shutdown 11.8.5 Transparent Spooling Chapter 12 Troubleshooting 12.1 Before You Start 12.2 Diagnostic Tools 12.2.1 Resource Monitor Display (RMD) 12.2.1.1 Memory 12.2.1.2 Active Task Display 12.2.1.3 Task Display 12.2.1.4 I/O Counts Display 12.2.1.5 System Statistics 12.2.1.6 Cache Region Display 12.2.1.7 Detailed Cache Statistics 12.2.2 Error Logging 12.2.3 Resource Accounting 12.3 What to Look For 12.3.1 Disk Saturation 12.3.2 Insufficient Memory 12.3.3 Unbalanced Load 12.3.4 Insufficient Pool 12.3.5 File Contention 12.3.6 Low Cache Hit Rates 12.3.7 Insufficient Checkpoint Space 12.3.8 Excessive File Opens 12.3.9 Overloaded Directories 12.3.10 Jobs at Wrong Priority 12.3.11 Misuse of Batch Hitchhiker's Guide to RSX D O N ' T P A N I C Revision 0.0 Chapter 1 Introduction This guide introduces the RSX operating system to the Small Business Application Developer. It contains information about those aspects of RSX which are of particular interest and importance to application developers who are considering or are in the process of migrating commercial applications to RSX or creating new ones. Much of the information will also be of use to developers considering a move to VAX/VMS since the two systems share many characteristics. The guide describes general principles for developing a new application or adapting an existing one to RSX. It also covers certain areas of special importance to small business application designers. RSX is a mature operating system and there are many case histories of successful use in a small business environment. Unfortunately, there have also been instances in which developers have encountered difficulty in making the transition from simpler environments. In many cases the effort required to correct the problems proved to be trivial. All that was lacking was information. There is nothing about RSX which makes it unsuitable as a base for small business software. Developers who are able to invest the time and energy in developing an understanding of the strengths and weaknesses of RSX will maximize their chances at a smooth, successful and inexpensive conversion. 1.1 The Purpose of this Guide. Operating systems are all different from one another. The most frequent problem in migrating an application from one to another is moving a set of programs which are designed around the strengths and weaknesses of the system of origin without taking into consideration the different characteristics of the target. In many cases, such oversights are the result of time pressure. In others, they are due to lack of information about which areas need the most attention and how to attain the most improvement for the least amount of work. This guide will help point the way. 1-7 This is not a substitute for the various developers materials available for RSX but is rather a summary intended to assist third party software developers in assessing the desirability of moving to RSX and to minimize the number of pitfalls into which such a developer might fall. If you are new to RSX it will be of assistance during those first days in which adjusting to any new operating system is difficult. The principles and techniques described are largely independent of the implementation language to be used. An efficient RMS file design will improve the performance of an application program whether it is written in Fortran, Basic, or Pascal. Many of the persons reading this guide will be doing so in preparation for a migration of an existing application from CTS-300 or CTS-500 to RSX and VMS, and some special notice is taken of this path. 1.2 Intended Audience This guide is addressed to a number of different people with a variety of backgrounds. You might be a third party software developer who is considering a move to RSX. You might have already decided to give RSX a try and are wondering how to plan the migration of your software. You could be part way through a conversion and are finding difficulties in locating the information you need. You might be facing a particular problem and are looking for a quick solution. It is assumed that you know something of Commercial Application development and have a favorite programming language. If you are coming from a CTS-300 environment you may find that many of the facilities and concepts available with RSX are complex. If you are coming from VMS on the other hand, RSX will seem more familiar. 1.3 On the Synergy of Operating Systems and Applications Computer operating systems are alike in that in one way or another they serve as a mechanism by which system resources are allocated to various objects, terminals, or application programs. They are different, however, in the overall design point which in turn determines the type of application that is expected to be layered on top. Some are designed to react as quickly as possible to an event, some are designed for security against power or component failures, some are optimized for a particular language. In all cases there is a form of synergy between an application and the underlying operating system. Since each system has a distinct set of strengths and weaknesses it behooves the application developer to make such changes as will take advantages of the strengths and avoid the weaknesses. Problems 1-8 do not usually reside exclusively in the operating system or the application and most often result from attempts to treat one O/S as though it had the properties of another. The importance of an application designer making the investment to understand the characteristics of a given operating system cannot be overemphasized. The return on such an investment can be staggeringly large. In one case a simple change of subroutine linkages (which took 20 minutes) resulted in a 2X performance improvement. In another case an optimization of a single subroutine resulted in a 10X improvement of a critical module. 1.4. Elements of Application Tuning There are two, distinct aspects of design and tuning of a particular application in a commercial environment. The first is the number of resources which are required by a particular program such as the number of files, CPU cycles or terminal operations. The second is speed and efficiency with which the operating system can deliver those resources. Identifying the contributions that an application makes to system load is difficult for many developers. A module may work perfectly well with small numbers of users but when a full production load is attempted the overall system deteriorates quickly. All too often this condition will not occur until the product is actually installed at a customer site where there is less time and fewer opportunities to investigate the problem. The purpose of this document is to help you anticipate such problems and to apply corrective action as early and inexpensively as possible. When a performance problem occurs, the solution must be to reduce the number of requests for resources or to increase the speed with which the requests are satisfied. Since the delivery of a resource can never be made instantaneous, the first effort should be to eliminate the requirement. The sections which follow describe ways that you may reduce or eliminate the demands made upon an operating system by your application. 1-9 Chapter 2 Overview of RSX Announcement of the PDP-11 resulted in an industry demand for a high quality, high performance and comprehensive multi-tasking operating system. The RSX family began with an initial adaptation from the PDP-15 and grew with additions of RSX-11D, -11A, -11B, -11S, -11M, IAS, P/OS, -11M-PLUS and, preserving the family ties with the 32 bit big brother, VAX-11 RSX. Many of the concepts of VMS can be traced to previous work done on RSX. 2.1 Historical Overview. A number of operating systems were developed for the PDP-11 including RT-11 as a 16 bit follow-on to OS/8, DOS-11 for larger machines and more sophisticated customers, and RSTS for the Education market. DEC mythology has it that at one time or another there have been 27 different PDP-11 operating system projects funded and underway. But in the early years the scientific/industrial marketplace was of primary importance to Digital and RSX was the flagship product. Most of the early minicomputers were used for process control and data collection and it was essential that the system software offer the greatest possible degree of flexibility. Hardware was expensive and if a month or two of bending and squeezing the software meant that memory requirements could be reduced by 4KW it was well worth the effort. Furthermore, there was no end to the variety of configurations which could be assembled. OEM's and end users were willing and able to create their own hardware components which could have characteristics ranging from a simple parallel digital interface to DMA communication ports. And, since their world was primarily real time, they required that response times to external events be both extremely fast and highly predictable. RSX was designed to satisfy the most exacting real-time requirements and could be tailored to almost any configuration of off the shelf and custom hardware. Every feature has an associated cost. The extreme modularity of RSX meant that the process of creating a working system was extremely complex even if the hardware configuration was comparatively simple. Users who didn't have critical real time requirements were nevertheless obliged to deal with a human interface designed for operating four or five processes at once. This was particularly true for the less technically oriented commercial users who were beginning to experiment with minicomputers as an alternative to batch processing. Multi-user protection was not supported in the early versions and the 2-10 so-called command language bore an unfortunate resemblance to certain UNIX shells. Memory was divided up into partitions which always seemed to be mismatched with the job at hand, the system would simply stop if a particular resource were exhausted, there was no data management facility other than block I/O and the only implementation languages were Macro and Fortran. Terminals seemed to be supported only as an afterthought. But even scientists tire of creating their own record management routines and over the years RSX was improved in ways which were also attractive to non-technical users. As the underlying hardware became less expensive it was possible to ease the extreme requirements for partitions in memory and provide more facilities in the executive. RMS finally appeared with multi-key ISAM. Partitions and pool management became dynamic, round robin scheduling and swapping support were added. DCL was provided (albeit only as an "alternative" to MCR), DECnet was put in place and, wonder of wonders, the terminal driver was made full duplex and could even support type-ahead, trap CTRL/C's and process escape sequences. The SYSGEN problem remained, and it was finally decided to produce a pre-SYSGENed, bounded version of RSX with commercial features which could be handled by a computer naive customer. The Micro PDP-11 was the first PDP-11 intended for installation by a first time user. Since the hardware was user-installable it was felt that the software should be as well. Moreover, it was considered that the Micro system customer represented a new class of prospective RSX user - a person not concerned at all with the traditional scientific/real time features of RSX but simply desiring a system which could quickly and easily be put to work. Rather than require the user to describe the target hardware configuration in detail, the software would accommodate itself to whatever was available. 2.2 RSX Version 3.0 Micro/RSX was the first member of what is now the RSX Version 3.0 family and has been joined by RSX-11M-PLUS in both Pre-genned and full kit form. Micro/RSX and Pre-genned M-PLUS eliminate the SYSGEN process altogether and will adapt themselves to any legal hardware configuration during system startup. M-PLUS still requires a SYSGEN to move from the Baseline (distribution) executive to full timesharing but even this process has been considerably simplified. The features available with RSX V3.0 include: o Full support for all of today's PDP-11 high performance CPU's, memory to 3.8 MB and I/D hardware where available. o Support of all modern disks and peripherals. 2-11 o Disk structure, Named Directories and Logical Names compatible with VMS. o RMS including multi-key ISAM compatible with VMS. o Terminal Emulation and Remote file access via DECnet. o Shareable, Clustered and Supervisor-Mode libraries. o Shareable, checkpointable Common Regions. o Multiple Stream Print and Batch Queues. o Implicit device spooling. o Synchronous and Asynchronous I/O on up to 256 channels per task (not supported by all languages). o DCL with Extended Help and command line continuation. o Disk Data Caching. o Password encryption. o Multi-volume backup to disks and tape. Application implementation languages and tools include: o FORTRAN-IV and FORTRAN-77 o DATATRIEVE-11 o DIBOL-83 o COBOL-81 and COBOL-11 o DBMS-11 o FMS-11 o Pascal o RPG-II o Sort/Merge o MACRO-11. With the availability of Version 3.0, RSX has finally become a high performance, high functionality operating system base for commercial applications. 2-12 Chapter 3 Getting Started This section is for those users who have never installed or used an RSX system. It is intended to get you on the air as quickly as possible beginning with that uncertain moment your system has been installed and checked-out and you're about to open the first of the boxes containing the software. It will also outline the general process by which the software is installed and customized so that you may estimate the amount of time that will be required. 3.1 The Three Faces of RSX Beginning with the newly released Version 3.0, RSX comes in three, highly compatible flavors (there are also two other members of the RSX family, RSX-11M and -11S but these have become special purpose products and are only of interest to the traditional RSX customer audience). 3.1.1 Micro/RSX The low end member of the V3.0 family is Micro/RSX which was the first release of RSX with a full set of commercial features. Micro/RSX was specifically tailored for the MicroPDP-11 and is the first version which could be taken out of the box, installed and used by a non-technical person. Micro/RSX served as the basis for A-to-Z V1.0 and is usually the system of choice for Micro-11 configurations. 3.1.2 Pregenned M-PLUS The intermediate family member is the "Pre-Sysgenned" or RL02 version of M-PLUS. This is actually a variant of M-PLUS which has been so configured that it will run on a wide variety of hardware configurations including the MicroPDP-11. It is self tailoring and, with the exception of certain features specifically selected through SYSGEN, is identical to the full M-PLUS system. This is the most appropriate system for commercial users on configurations other than the Micro-11. This version is only distributed on RL02 disk packs and there are certain restrictions about which disk types may be used on the target system. Otherwise, it is usable as soon as it can be copied to the system disk. Unless you require special hardware support or executive features which are not part of this version, you should use this version of M-PLUS. Due to the size constraints of the RL02 media, certain parts of the full M-PLUS system are missing. If it is found that any of these components 3-13 are required on your system they may be copied from an M-PLUS kit. 3.1.3 RSX-11M-PLUS M-PLUS is the high end, full functionality system and must be tailored (SYSGENed) before it can be used. It is distributed in source form along with a Baseline System (minimal executive) with which a SYSGEN may be performed. The SYSGEN process is an interactive, question and answer process by which the user describes the exact combination of executive features and device support desired. The executive and associated utilities are then assembled and linked together to form a bootable system image. This image is further tailored by setting up partitions, installing tasks, etc. until an executable system executive is finished and ready to be installed. More on SYSGEN later. 3.2 Installation and Startup Once the three versions of RSX have been completely installed there are only a few differences between them. The three versions do, however, differ in the way they are shipped to the customer and in how they are installed. These differences are largely due to which assumptions can be made about the underlying hardware. 3.2.1 Installing Micro/RSX Installing Micro/RSX on a MicroPDP-11 requires only that the software be copied onto the system disk. The distribution media consists of either a set of diskettes or a single cartridge tape. To be a legal MicroPDP-11 you must have either an RX-50 dual diskette drive or a TK-50 cartridge magnetic tape. In either case, the first unit of the media contains a bootable, stand alone version of BACKUP. All that is necessary to install the software is to insert the first media unit into the appropriate drive and turn the machine on. Once the PDP-11 completes it's self check it will notice that the media is in place and will bootstrap BACKUP into memory. BACKUP will announce itself and will then determine the processor type. The distribution kit actually contains two complete executives. One of the system images is intended for use on PDP-11 processors that support separate instruction-space and data-space mapping such as the 11/73 and 11/83. The other is for use on processors which do not support I/D space mapping such as the 11/23. The disk drive which is to be used as the system disk will be 3-14 initialized (and cleared of all existing data!) and then the system software will be copied into place. When the copy phase of the operation is complete, you will be directed to remove the media and restart the system. The restart will request the time and date and will then run automatically through the remainder of the startup procedure. When the startup has completed the console terminal will log itself out to prevent an unauthorized person from gaining access to your system as the manager. The system is now fully operational and ready for use. The entire process requires less than an hour. Micro/RSX was designed around systems with limited disk capacity and consequently there are some optional software packages which are distributed separately. These options are primarily those required for software development. If you have these options they may be installed right away or at some future time as needed. The procedures for installing optional software on Micro/RSX are described below. 3.2.2 Installing Pregenerated M-PLUS The pregenerated M-PLUS kit is the quickest and easiest way to get RSX-11M-PLUS running on your system and, for most commercial customers, it is the most appropriate combination of SYSGEN parameters. Installation involves copying the software onto the target system disk, selecting the appropriate system image file and installing the system disk handler. This process is not as highly automated as that for Micro/RSX and the appearance of the dialog is somewhat more mysterious, but the procedure is only slightly more complicated. The entire installation is described, step by step, in a special chapter in the RSX-11M-PLUS System Generation and Installation Guide. The process does not require any great degree of computer experience as long as you have this guide handy when you begin the installation. The kit consists of a single RL02 which contains two, ready to run system images along with all the necessary system files and utilities. The RL02 pack can be bootstrapped and (except for the lack of free space on the distribution volume) used straight away. One of the system images is intended for use on PDP-11 processors that support separate instruction-space and data-space mapping such as the 11/73 and 11/83. The other is for use on processors which do not support I/D space mapping such as the 11/23. The non-mapped executive can also run on mapped systems and it is, in fact, the system that runs when you first boot the distribution disk. The installation begins with bootstrapping the distribution disk on the target system. When the startup procedure asks for the 3-15 time and date, the procedure is aborted and a special stand alone Backup/Restore is bootstrapped from the RL02. This special task will allow you to initialize the target disk, locating and marking any bad blocks, and then copy all the contents of the distribution kit onto the target disk. At this point all the files are in place but the target disk is not yet bootable. What remains is to choose the appropriate executive (mapped or un-mapped), configure it for the target system hardware, and then make the executive image bootstrappable. These final procedures are largely automated although they will require you to type in some very strange looking commands and will produce even stranger looking output on the console. It is from this type of dialog that RSX acquired it's evil reputation. As of Version 3.0, however, it is no longer necessary to understand the process in any detail. You will not be required to make any decisions other than selecting an executive based on your processor type. Simply follow the instructions carefully. The last step begins with re-bootstrapping the kit disk. Once again interrupt the startup procedure and, following the instructions provided in the Sysgen and Installation guide, load the driver for the system disk, bring it on line, and set default to the desired system area. Selecting the proper system is the only decision you have to make and, unless you have good reason to do otherwise, will be determined by your hardware as described above. You then create a copy of the system image and invoke VMR which performs an automatic procedure to tailor the selected executive to your system disk. You then make the image bootable by "bootstrapping" the just tailored image and then "saving" it in such a way that the bootstrap code can locate the image file. The installation is now complete. Remove the RL02 Distribution pack and restart your system. The entire installation process requires less than two hours. 3.2.3 Installing M-PLUS Installing M-PLUS is a two stage process. First, you must copy the information from the distribution kit to the target system disk. Then, you must execute a SYSGEN to build an executive which is tailored to your specific hardware configuration and software needs. This process can either be performed on an existing RSX system or on a new system. Compared to previous versions, SYSGEN for RSX V3.0 is a painless and largely automatic process. If your hardware configuration is standard and you have no special requirements of the operating system, you can be finished in a matter of hours. If you are working on the target system you can direct SYSGEN to 'Autoconfigure' your hardware and it will use the results as the 3-16 default answers to the most difficult questions. Furthermore, you can direct SYSGEN to create a 'Full Functionality' executive thereby eliminating a large number of intricate and confusing questions about executive options. If you accept all the default answers, the dialog section of SYSGEN is largely a matter of confirming that little used options are, in fact, not desired. If you are an inexperienced M-PLUS user, you are strongly advised to use the Autoconfigure option and to accept all the defaults. In this way you will get your system up and running as quickly as possible. Once you have acquired more experience you may repeat the SYSGEN to create an executive which is more closely tailored to your needs. Most commercial users will find that the Full Functionality executive suits their needs perfectly. On the other hand, if you have an unusual hardware configuration or if you have very special needs, you may wish to contract for enough support to get you through the first installation. RSX can be tailored to fit just about any hardware configuration which can be manufactured, but the process of discovering all the ins and outs of CSR and Vector assignments is a complex one. This is no time to play the hero. If you have any doubts about your own capabilities, call for help. The full details of the installation and SYSGEN process are beyond the scope of this section and are described in the RSX System Installation and Generation Guide. There are a whole series of different paths depending on what existing systems are available but the simplest is installation on new hardware. The sequence of events for installing on a new system is roughly as follows: Boot the distribution media. This will startup a special, stand alone version of Backup (actually an RSX-11/S system image) which you can use to format the target disk and scan it for bad blocks. You then copy the first of two backup save sets from the distribution media to the target disk. The result is a disk which is hardware bootable and contains a Baseline executive which will run on almost any hardware and which contains just enough functionality to allow you to complete the SYSGEN. Boot the target disk. The Baseline system will automatically copy the second backup save set from the distribution kit to the target disk. Apply any necessary Updates. Unless you have obtained your software shortly after a major release of RSX there will likely be one or more Update kits which must be applied to the target disk before you proceed with the SYSGEN. This process is largely automatic. Invoke the UPDATE command procedure to apply the corrections. 3-17 Invoke SYSGEN. Again, there are all manner of paths and options but the best plan for a beginner is to go straight through from the beginning to the end, accepting the defaults wherever possible. While the simplest path does not require you to answer many questions, a copy of the dialogue which takes place is useful in the event that anything goes wrong. If you do not have a hardcopy terminal attached as the system console it would be wise to connect a printer to the CRT so that all the output will be saved. SYSGEN will ask a number of questions about how the executive is to be built. Unless you know better you should take the default in each case. The exception is the question labeled SU100 which asks if you wish to run Autoconfigure. If you are running on the target hardware you should answer YES. Doing so will result in an Autoconfigure operation in which the Baseline system will interrogate all the devices it can find on the bus and will configure your peripherals accordingly. The only other questions you will be asked will concern hardware which was expected but is missing (why SYSGEN asks if you have a card reader is a mystery) or about options that are costly or infrequently used (modem support on terminal lines). After about 15 minutes of dialogue SYSGEN will proceed to assemble and link the executive and all the privileged tasks. Watch the dialog as it unfolds and compare it to the example in appendix D of the SYSGEN guide. The remainder of the SYSGEN will take about three hours. The process is automatic but you will be asked an occasional question so it's a good idea to check in from time to time to see how things are progressing. When SYSGEN completes you will have a fresh copy of the executive but it will not be hardware bootable. Follow the directions in the Guide to software boot the new executive and enter the time of day. This action is only a sanity check to see that everything worked properly. Then, SAVe the executive, again to see if anything untoward occurs. The SAV will cause the new executive to be rebooted and begin the standard system startup procedure. If everything still looks all right, abort the startup as directed and SAVe the system again, this time with the /WB (Write Bootstrap) switch. This final step will change the bootstrap blocks of the target disk to point to the newly generated executive. The system will restart automatically and you are on the air. 3.3 Logging In and other First Day Tasks Once your system is installed and operational the first thing you should do is log in as the system manager and make any additional adjustments that you might feel are necessary. Typical chores involve post-installation clean-up, selecting appropriate startup options, creating user accounts, configuring terminals and 3-18 printers, and installing layered products. 3.3.1 Logging Into the Prebuilt Accounts RSX is distributed with a System account and a User account. You can log into either one from an unused terminal (such as the console terminal once the system startup has completed) by pressing the Return key until you get the system prompt (which might be either ">" or "$"). Type "HELLO" and press Return. You will then be prompted for a username and password. When these have been entered and verified against the system accounting file the login procedure will complete. The prebuilt account username/passwords are SYSTEM/SYSTEM (or MICRO/RSX on Micro/RSX) and USER/USER. Once the login task has validated your account it extracts other information about your account from the authorization file and then executes two command files on your behalf. The first of these, LB:[1,2]SYLOGIN.CMD, is executed for all users and, in it's most common form, will determine your terminal type with SET TERMINAL/INQUIRE and then chain to the LOGIN.CMD file in your default directory. If this second command file exists it is also executed. In the case of the USER account it will print out a welcome message and some introductory information. These two login files can be used to control the accounts in your system. In the case of A-to-Z they are used to direct the user directly from login into the appropriate A-to-Z menu. RSX supports two Command Line Interpreters (CLI). MCR is the traditional CLI still favored by the RSX old timers, and DCL which will probably be more familiar to newcomers. The account entry in the authorization file determines which of the two is established for a particular user during login. You may switch from MCR (usually identified by the ">" prompt) by typing > SET /DCL=TI: You may switch from DCL (usually identified by the "$" prompt) by typing $ SET TERMINAL/CLI=MCR Most users learn one of the CLIs and stick with it. You should modify the authorization file entry for each account according to the user preference. 3.3.2 Post Installation Cleanup The Pregenerated (RL02) version of M-PLUS contains both the mapped and the un-mapped executives. Once the installation has been completed you may wish to delete the unused files to recover 3-19 the disk space. A command procedure is provided for this purpose. The System Generation and Installation guide details how to proceed. The procedure is completely automatic. It will determine which executive is in use and will delete the unused (and completely unnecessary) files. At its conclusion the procedure will report the number of disk blocks recovered. You may postpone this task if you have sufficient disk space for the early days of usage. 3.3.3 Customizing Startup The Micro and RL02 systems are shipped to you with certain system parameters, such as the number of batch and print queues, set to values appropriate for an "average" configuration. These parameters are stored in a file (LB:[1,2]SYSPARAM.DAT) which is readable by non-technical users and which is interpreted by the system startup procedure. The statements within this file determine such things as the amount of checkpoint space and secondary pool allocated, what combination of batch and print queues are to be started, what to do about error logging, and so on. Provision is made for controlling every commonly used element of the system. If you can use an editor, such as EDT, you will have little difficulty modifying this file to your own taste. It is unlikely that you will have to do anything further. If you have chosen M-PLUS you will be supplied with a skeleton system startup control file, LB:[1,2]STARTUP.CMD. This sample file is sufficient to get you on the air but it is expected that you will want to modify it and will know how to do so. Regardless of the system, if you have added A-to-Z and are running primarily A-to-Z applications you will find that A-to-Z manages all of the usual system startup tasks. If you have requirements beyond those managed by A-to-Z you will most probably know how to take care of them. 3.3.4 Configuring Terminals and Printers When RSX configures your system it can detect the presence of the various device controllers on the system, but in the case of serial and parallel ports it makes no attempt to determine the type of device, if any, which is attached. You have several options: If the devices are all terminals, you can let SYLOGIN.CMD do the work with it's SET TERMINAL/INQUIRE which will poll the terminal to determine it's type whenever a user logs in. If the devices are a mix of printers and terminals and you are 3-20 running Micro or Pregenerated flavors, you can edit the parameter file LB:[1,2]SYSPARAM.DAT and, following the examples contained within, designate each of the lines or ports with the appropriate device type and line speed. The next time your system is re-started the devices will be identified and brought on line. If you don't want to be bothered, or if you are installing a system for a customer who may not want to be bothered, you should consider adding A-to-Z to the system. A-to-Z includes a process which runs at system startup which will poll all the available ports and determine the device type and speed of each line. This information is stored in a table where it is available for inspection or modification by the customer. The advantage of this is that if a terminal is moved or the speed of a device is inadvertently changed, the customer can deal with the problem directly without any special training. 3.3.5 Creating User Accounts The procedures for adding, changing, and removing accounts are the same for all three systems. A utility (ACNT) may be invoked by any privileged user to make any desired changes to the system accounting file (LB:[1,1]RSX11.SYS). Take care that this file is not inadvertently deleted. If the file is deleted you may have some difficulty regaining control of your system. Before you enter a new account you should have available certain pieces of information which will be required by ACNT. You will be asked to supply a username, typically the user's last name, which must be unique on the system, a password which is a "hard to guess" word of up to 39 characters that validates the identity of the user to the system, a User Identification Code (UIC) which classifies the user according to group and access privileges, a default CLI (DCL is easiest for most new users to learn), and a disk and directory to contain the user's files. The items are entered one at a time and the default directory will be created if it does not already exist. The new user may log in immediately after you have entered the information for the account. There are several other items which you may enter including a session identifier, first name, and so on. You may also modify an existing account if you wish to change any parameter. Note that beginning with Version 3.0 of RSX you may opt for password encryption on an account by account basis. Such passwords may not be examined once they have been entered but they may be specified anew if the original has been lost. If you have A-to-Z you may wish to manage your accounts with the facilities provided for the A-to-Z Manager. Unless you have special needs you should find that the facilities provided by A-to-Z are adequate and are much easier and safer to use. 3-21 3.4 Installing Layered Products Installation on any version of RSX is largely a matter of copying files into the appropriate directories, possibly task building certain modules against the symbol table for the executive on the target system, and adding the necessary initialization commands to the system startup procedures. On M-PLUS this is usually accomplished by a command file, supplied with the layered product, which will conduct a dialogue with the system manager to determine which, if any, options are to be selected and then will perform all the necessary tasks. Some products require that the system manager manually edit the system startup file and provide the appropriate commands. Micro/RSX has a much more automatic installation architecture which makes it possible for a person with little or no computer expertise to manage the installation. A-to-Z has combined elements of both of these architectures. If you are installing A-to-Z layered products you may do so via an option on the A-to-Z Manager's Menu. The procedure is the same regardless of which of the versions of RSX lie underneath. 3.4.1 Installation on Micro/RSX The Micro/RSX user does nothing more than choose the product to be installed (in case there is more than one product module in the kit) and to provide physical assistance with handling the media. The architecture and the layered product developer do the rest including making decisions about the underlying hardware, product specific installation and startup procedures, and integration with other components on the system. To install a product the user logs into the system manager's account (or the manager's account for A-to-Z) and invokes the optional software installation procedure OPTION.CMD (or selects the application installation option from the A-to-Z manager's Auxiliary Functions Menu). The system software then requests that the installer identify the appropriate device drive according to the media type, insert the media, select the application to be installed, and then insert more media as required. While the particular product may require some customization there is nothing further required to get it running on the underlying system. 3.4.2 Installation on M-PLUS Layered product installation on M-PLUS (both the Pregenerated and full Sysgen flavors) assume a more sophisticated system manager who has more particular requirements. In this case the layered product developer supplies a kit which contains a collection of all the components of the product and (with the nicer products) a 3-22 command procedure with which the manager picks and chooses from the different combinations. The developer is still required to be sensitive to those aspects of the product which may change from system to system such as re-linking privileged tasks against the target executive or relocation of files and modules across disks. The developer is also responsible for providing all the information regarding the installation, usually in the form of an installation guide. The installation guide and the release notes are the basis from which the system manager decides between various options or, in some cases, redesigns the installation procedure altogether to satisfy specific system conditions. There is no set pattern for installation on M-PLUS. Generally, the Installation Guide provided with the product will direct the manager to log in and create a temporary directory. The guide will also describe the conditions that must be set up before installation can proceed. Certain tasks may have to be installed or optional software may have to be moved into place. The manager will then mount the media, usually a magnetic tape or a disk pack, and copy the installation control file (traditionally INSTALL.CMD) into the temporary directory. The command file is then invoked and, through a dialogue with the manager, determines what actions to take. Once the dialog phase is completed, the command file will perform the remainder of the installation: copying and converting files, linking tasks, and setting up the database. The installation guide will also usually provide the manager with information about what modifications must be made to the system startup procedure in order for the product to operate properly. Products for M-PLUS differ considerably in the degree to which the installation is automated. Some come with a list of instructions for the system manager including general directions on what to add to the system startup command file. Others are more automatic. The trend is towards automation as the available hardware resources become more plentiful and squeezing the last free byte out of a file becomes less important. An example of such automation is the A-to-Z Base System which not only controls it's own installation to a high degree but carries with it elements of the Micro/RSX layered product architecture. This makes it much easier for a computer naive system manager to install and manage an application mix. It also makes it easier on the product developers since the kitting procedure for A-to-Z applications on Micro/RSX and M-PLUS is the same. Only the media is different. 3-23 Chapter 4 Terminals Terminal I/O is a prominent aspect of modern commercial applications on timesharing systems. It is, in fact, one of the primary reasons that the old batch processing systems have been replaced. It is also, unfortunately, a major source of system overhead. Problems related to this aspect of application design are particularly insidious because the cost of moving characters to and from a terminal is so easy to overlook. Most programmers understand that opening and closing files is costly. Far fewer consider what is happening when a screen is being painted. 4.1 Cost of I/O The cost of a particular I/O operation can be divided into two general categories. The first is the number of instructions which must be executed to guide the data between the program and the device controller (overhead). The second is the time the device requires to move the data between the controller and the outside world (latency). With disks, much of the time required to fulfill an I/O request is the latency due to the mechanical movement of the head and spindle. Once the disk head has been moved to the proper cylinder, large quantities of data can be transferred very quickly, usually without CPU intervention. More on disks and files in the next section. 4.2 Specific Cost of Terminal I/O Terminals are another problem altogether. There is a short latency period while a character is shifted into or out of a UART register but the real problem is that there are ever so many more I/O operations involved in moving a block of characters to or from the terminal. A disk can transfer thousands of characters in a single operation whereas most terminals make the transfers one byte at a time. 4.2.1 Interrupt Service Terminals can cause almost 1000 CPU interrupts per second and each interrupt requires the executive to suspend processing to service the device. Even worse, terminals are unpredictable on the input side. Operating system software can keep track of the head position of a disk drive and can make a guess at how long a given operation will take. This permits more advanced operating 4-24 systems, like RSX, to optimize the queue of waiting I/O requests. But there is no way that the system can predict how long it will take an operator to press a character key - the computer simply has to wait for the interrupt. 4.2.2 Scheduler Overhead RSX considers the completion of any I/O request to be a "Significant Event". Such events signal to the system scheduler that there may have been a change in various lists of tasks waiting to execute - the I/O completion may have unblocked a task of higher priority than the current task thereby necessitating a context switch. With even a small number of commercial applications painting screens the scheduler may be executing hundreds of times per second. Excessive context switching (thrashing) is a factor in almost every system performance problem encountered thus far. The worst possible circumstance is when the various interactive tasks are competing for memory and are being checkpointed to disk in which case each character moving to or from a program causes several, far more costly, disk reads and writes to take place. So the cost of moving a single character is the sum of the code to actually move the characters to or from the device plus any program overlays which may have occurred plus the interrupt service routines plus the context switching. When many hundreds of characters are being moved back and forth every second the system overhead is considerable. 4.3 Minimizing Costs There is little you can do to reduce the number of characters which must be moved to and from the terminal. You can, however, do a great deal to reduce the operating system overhead associated with terminal intensive applications by reducing the number of I/O requests necessary to move the requisite characters. 4.3.1 Blocking Output Requests Whether or not DMA hardware is available it is very important to gather small groups of characters together before sending them to the terminal. Throughput improvements of 10 to 1 have been reported when scattered output requests have converted to calls to a central routine which assembles the characters strings into a single buffer. The amount of processing to move the characters to the terminal is the same in either case but if the number of I/O requests is reduced there will be a corresponding reduction in scheduler overhead. 4-25 It is not always easy to determine just how an implementation language such as BASIC or DIBOL performs terminal I/O but making the effort to find out will be well repaid. For example, the following three DIBOL code fragments will produce output that looks exactly the same to the operator: WORST, DISPLAY (CHAN,'A') DISPLAY (CHAN,'B') DISPLAY (CHAN,'C') This results in three I/O requests and some unnecessary interpreter overhead. BETTER, DISPLAY (CHAN,'A','B','C') This eliminates some of the interpreter overhead but there are still three I/O operations (and consequently three calls to the scheduler, etc.) involved. BEST, DISPLAY (CHAN,'ABC') Best of all. It is admittedly unlikely that the output data will be so easy to assemble. Most often it will be necessary to feed the data fragments to a subroutine along with a flag which serves as a 'Do It Now' or 'More Coming' signal. But it's worth it. 4.3.2 Blocking Terminal Input It wasn't so long ago that many operating systems were designed to only accept data from an operator in line mode - a series of characters terminated with the Return key. No Fortran programmer in his/her right mind would want to see the characters one at time or, Heaven forbid, a five character escape sequence. And the operating system people were glad to avoid the headaches of having to worry about mapping all the ANSI sequences into terminator codes. They preferred to tell any developer who wanted to know whether an input string had been terminated with Up-Arrow or Down-Arrow to figure it out themselves. But terminals kept getting smarter and even people working in laboratories type with their elbows now and again so eventually the operating systems began to support commercial features, beginning with single character input and some going so far as providing elementary block mode facilities. There is, however, a certain amount of holdover code in the language processors which was put in place, in those good old days, to give the underlying operating systems the appearance of block mode input and in some cases, this code can get in the way. 4-26 Thus, it is once again important for the designer to understand how a high level language translates the input statements into RSX directives. It is the case that some languages will convert what appears to be a field level request into a series of requests for single characters. DIBOL, for example, converts a READS into a series of single character input requests. The latest versions of such languages may, however, provide a means of control over such actions. This is important because grouping input characters or data has the same benefit that blocking output has. It often makes the job of the developer more difficult, and sometimes impossible, if the functionality of an application is going to be preserved. If you cannot avoid processing the input stream a byte at a time, then so be it. But if it is at all possible to change the character-at-a-time to post entry validation then you will enjoy considerable improvement in overall system efficiency. 4.3.3 Use DMA Terminal Hardware The more expensive DMA terminal interface hardware will reduce the number of interrupts the executive must service to move a block of characters to a device. The terminal driver is able to treat the output side of the device much as it would a disk. The output operation is initiated and, except for the bus cycles stolen by the interface to fetch the next character, the CPU is free to perform other tasks. However, there will be no great improvement with such an interface if characters are fed to it individually or in small groups. The greatest gain will be seen when output data is gathered into a single block and sent to the device with a single I/O request. It is possible to output an entire screenful of information including escape sequences for formatting and setting character attributes. 4-27 Chapter 5 Record Management Services (RMS) In the early days of RSX there was little in the way of facilities for file or record management. The File Control Services (FCS) offered routines to open and close files and a Macro program could issue QIO requests directly to the disk driver. There was provision for rudimentary sequential access to records and random access to disk blocks but otherwise application developers were left pretty much to themselves. The need for something more was apparent and eventually RMS-11 was developed. 5.1 Introduction to RMS The RSX Record Management Services (RMS) is a language independent collection of file and record management routines which may be linked into a user program and called upon to perform a variety of file and record related functions. Since the in-file format is determined by RMS and not by the host language processor the records within the files are also language independent thereby providing a ready mechanism for exchange of data between applications. RMS has undergone several revisions since it's inception, primarily to take advantage of more advanced hardware and operating system features. The first versions of RMS were actually a collection of object modules that had to be linked into, and carried about with, the task image. The more modern flavors take full advantage of memory management hardware and CPU operating modes for improved speed and reduced disk space requirements and activity. The current version of RMS may be linked as a shared library which reduces disk requirements for task images and eliminates the overhead of disk overlays for the RMS routines. Or the RMS library may be clustered with the language OTS which increases the amount of virtual address space available to the application. Lastly, it may be linked as a Supervisor Mode Library (on processors which support Supervisor Mode operation) which does all of the above and reduces the time required for context switching between libraries. For further details, see the chapter on linking. Full access to all of the RMS features is really only available through Macro. Each higher level language has had to make trade-offs in mapping the traditional semantics of the language statements to the facilities provided by RMS. Some languages allow the developer to specify the characteristics of a file being created with great precision while others do not. Some permit update of files opened for Input and some do not. Some 5-28 language statements may cause the run-time library to execute hidden RMS operations. The DIBOL WRITE statement, for instance, will store a record into both occupied and un-occupied cells, executing either an RMS PUT or UPDATE operation, as required. Utility programs are provided with RMS which provide for creating, converting, and re-organizing RMS files. These utilities can be used interactively or can be spawned from an application to perform their various functions. 5.2 File Organizations RMS provides support for three major and two minor file organizations. The major types are Sequential, Relative and Indexed. The minor types are Stream and Undefined and will not be discussed further here. The types of access which are allowed to a file depend on it's organization. 5.2.1 Sequential Sequential files are the simplest format and, in some circumstances, offer the highest performance. A sequential file consists of a header and a number of data records. The file is loaded by writing a series of data records. Later, the records may be retrieved in the order in which they were written. When you create a sequential file you may specify that the file will contain fixed length or variable length records. Fixed length record format offers the most efficient packing. Variable length record format offers the greatest flexibility and is the more common of the two formats. Sequential files are most appropriate for serial writing or reading of records. Very high performance may be attained with languages which support the multiple buffering and deferred write features of RMS. There is no provision for deleting, replacing or updating records once the file has been created. Some languages permit opening a sequential file to append data to existing records. Sequential files with fixed length records allow limited facilities for random access but record interlocks are limited and such use is not encouraged. 5.2.2 Relative Relative files consist of a file header, a prolog and data records. The header and prolog contain information about the file and the records. Data records are stored in groups known as Buckets. The size of the bucket is declared when the file is created and each bucket is divided into as many record cells as will fit. Cells are all 5-29 the same size and can hold the largest record. Maximum record size must be declared when the file is created. Data records within the cells may be fixed or variable length but the unused space in a cell containing a short record cannot be put to other uses. Once the file is created, the records may be loaded serially or randomly. There is a flag byte in each record cell which indicates whether a record is present, has never been stored or has been stored and then deleted. Records may be stored, retrieved, updated and deleted, randomly or sequentially. RMS automatically maintains a current record context which determines which record is selected for the next sequential operation. If a record is written to a cell beyond the end of the existing data, the file is automatically extended. New buckets are created and initialized by RMS, as required. Relative files represent a good compromise between convenience and efficiency. As long as buckets are densely populated data is stored with comparatively little overhead and access to any record in the file requires only a single I/O operation (not counting window turns). Interlocks are provided for files which have been opened for write access by one or more programs. Interlocks are maintained on a bucket basis. 5.2.3 Indexed Indexed files offer the most flexibility to the programmer. You may access records directly or sequentially by a primary key and any of up to 254 secondary keys. The keys may be located anywhere within the record, may be contiguous or segmented and may be one of several data types. The index structure is dynamic and will increase in size and number of levels to accommodate new records as they are added to the file. Access time for all records is uniform whether the file is loaded in sequence by primary key, randomly across the file or in clusters. The number of I/O operations required to reach any record in the file is a minimum and is approximately equal to the number of levels in the index. In effect, RMS exchanges the CPU time required to manage the (comparatively complex) index for the far more costly mechanical movements of the disk head. For small files there is little benefit to using this organization but for files which contain tens of thousands of records the savings are substantial. There is a substantial amount of overhead carried with indexed files, both in terms of disk space and the code required to manage the file. Indexed files require the most attention to their design. There are a large number of design parameters associated with indexed 5-30 files and unless values for these parameters are selected carefully, the file performance may prove disappointing. Due to the complexity of indexed files and the great variety of usage patterns it is impossible to provide defaults which are universally suitable and the burden of design is left to the developer. 5.2.3.1 Structure An Indexed file consists of a header, a prolog and two or more index levels for each key. The header describes the location of the file extents and the prolog describes the structure of the records and the index levels. The lowest level of the primary index contains the data records themselves which are grouped together into buckets. Higher levels of the primary index contain copies of data record keys and pointers to the appropriate buckets at the next lower level. The highest level of the index is called the Root and consists of a single bucket. Each alternate key defined for the file has it's own index structure. The structure of an alternate key index is the same as the primary key index except that, in place of user data, the lowest level contains Secondary Index Data Records (SIDR). SIDR's contain alternate key values and pointers to data records in the primary index. There is only one copy of a data record in a file but there may be several copies of the key field. Data records are collated in the buckets by primary key and the buckets are linked together so that sequential access can take place without any references to the index. 5.2.3.2 Population A newly created file may be populated by writing records sequentially or randomly throughout the file. The index is dynamic and will expand both horizontally and vertically as required. When the initial allocation is exhausted RMS will expand the file automatically by allocating buckets as required. As buckets fill up they will be split to make room for new records. Populating a file with an application program is not necessarily the best way to proceed, however. There are certain features of RMS which pertain to Indexed files, such as Bucket Fill size, which are only available to Macro. In addition, secondary key indexes will often be convoluted since even if the records are added to the file in order of ascending values for primary key, the values for the secondary key are unlikely to be in an optimal order. Both these problems can be overcome with the Indexed File 5-31 Load utility. 5.2.3.3 Index Activity When a new record is added to the file, the primary index is scanned to find the proper bucket and the record is inserted according to it's primary key value. If there is not sufficient room to store the record, the bucket is split and half the records are moved to a newly created extension bucket. The next higher level index is updated to indicate the existence of the new bucket and the record is inserted. If the index bucket is filled, it too is split and if the bucket being split is the Root, a new index level is formed. If the file contains alternate keys, then each alternate key index must be searched until the lowest level is reached, a SIDR record added or updated, splitting the SIDR bucket if necessary, and so on. The amount of I/O required to store a new record or update an old one is a direct function of the number of keys. 5.2.3.4 Storage Overhead The storage overhead in a bucket is a significant factor in indexed files and, unlike relative files, may increase over the life of a file. One of the features of RMS is that each data record in a file has a unique address and if the record is deleted the address is never re-used. Because of this, deleted records leave behind a residuum which can be as little as two bytes (fixed length records, no duplicates for the primary key) or as much as the entire record (fixed length records, duplicates allowed for primary key). Similarly, if a record moves from it's original location in a file when a bucket is split, a Record Reference Vector (RRV) is left in the original location. If you expect to be splitting buckets or deleting records frequently you must carefully consider the record type and key locations and the implications for bucket overhead. 5.2.3.5 Interlocks RMS provides full interlock support on a bucket-wide basis. Each program which opens a file declares the operations that it wishes to perform and also the operations that it will allow other programs. These declarations are often supplied by the language runtime system depending on the particular statement used and the default values may differ from one language to another. If the access declarations of all programs accessing a file are compatible, processing proceeds. Otherwise, the program opening the file with the incompatible access code receives a protection violation. 5-32 5.3 File Design The design and tuning of RMS files is an area of study unto itself. The number of possible usage patterns for files is large and it is not possible for RMS to anticipate the manner in which a particular dataset will be accessed. There is no obvious limit to the amount of effort you may put into improving your files but there is usually a point beyond which additional effort might be better expended elsewhere. You must decide for yourself when you have reached the point of diminishing returns. This section is not intended to be a comprehensive tutorial on file design but rather to point the way to those areas where small changes may have major impact on the performance of an application as a whole. See the RMS User Guide for further details about file usage and design. 5.3.1 Which Files to Design Although it may seem obvious, you should limit your design efforts to those files which are most critical to the performance of your application. If a particular file is used only on occasion there is little benefit to spending a great deal of effort on optimizing it's performance. Concentrate your efforts on files which are large, which have many simultaneous users, or which have high levels of insertion and deletion activity. 5.3.2 Selecting an Organization In general, the amount of processing and overhead necessary to manage a file increases with the flexibility of the available access methods. Sequential and Relative file organizations are very fast and carry little or no overhead but they do not provide all the access paths of the Indexed organization. You will almost always attain the highest performance by choosing the simplest file organization which can be made to provide the access features you require. 5.3.3 Common Design Factors The design parameters of a file are set when the file is created. The more complex the file organization, the more parameters there are to consider. There are some parameters, however, which are common to all file types. 5.3.3.1 Allocation All files allow an initial allocation of disk blocks to be 5-33 specified when the file is created. In the case of Relative and Indexed files this allocation is mandatory since storage space must be provided for the file prolog. Allocating space for a file when it is created has two benefits. The allocation takes place all at one time and the disk blocks are likely to be located close together. Also, the file can be loaded without the interruption of extend operations. Unused space which has been allocated to a file cannot be used for anything else. Once the file has been fully populated you may return any excess blocks to the operating system by truncating the file. If your language does not support the RMS truncate facility, you may spawn DCL with a SET FILE/TRUNCATE command. 5.3.3.2 Extend Quantity If a file is created with less than the amount of space which will be required for storage of the data you may wish to specify an extend quantity. This is the amount by which the size of the file is increased when additional space is needed. The extend operation is carried out automatically by RMS so your application may not be aware that it is happening. The default extend quantity for all files on a disk is set when the volume is initialized and this value may not be appropriate for a given file. If the extend quantity is too big you may be wasting space. If it is too small the file will have to be extended frequently which will mean frequent interruptions in file processing, scattered file fragments and reduced performance of subsequent file processing. A commonly used rule of thumb is to specify an extend quantity derived from some unit of processing such as a day's worth of records. 5.3.3.3 Contiguity The best single action you can take to assure optimal file performance is to make the file contiguous. To make a file contiguous you must allocate all the required disk blocks when the file is created which may result in some wasted space until the file is filled. It may also require that you re-organize the disk with Backup to gather the unused fragments into a larger, contiguous space. 5.3.3.4 Carriage Control If the file is ever to be printed or otherwise output to a record I/O device (printer or terminal) you may wish to specify a Carriage Control attribute for the file. Carriage Control 5-34 determines the way in which RMS will separate the records as they are being output. Instead of carrying the formatting information with each data record, RMS stores a single format setting in the file header and this setting is interpreted by the system utilities when the file is printed. The default setting is Carriage_Return which means that as each record is output it is preceded by a Line Feed and followed by a Carriage Return. You may override this setting by specifying None. If you specify None you must store the formatting characters as data within your records, that is, you must explicitly include Carriage Return or Form Feed characters within your data records. Some languages have a special file qualifiers which are used for this purpose. 5.3.4 Designing Sequential Files Sequential file design is primarily a matter of allocation. Most such files are created with variable record lengths and that's the end of it. You are allowed to specify that records shall not span block boundaries but in most cases this practice is wasteful and not recommended. There is a special case situation involving fixed length records which you may wish to use. If you declare the records to be of fixed, defined length when you create the file you may use a limited form of random access to the file. You may store and retrieve records either sequentially or randomly. Access to this facility depends on which language you are using and there is no record interlock protection. 5.3.5 Designing Relative Files Designing Relative files is also comparatively simple. Like sequential you choose an allocation and extend size. Records may be of fixed or variable length but the record cells which are allocated will be of uniform size and large enough to contain the longest record. RMS keeps track of the byte count of variable length records so the application never sees anything except real data. The one additional design factor is determining the bucket size. A bucket is the logical structure within which records are grouped and the size you choose can affect the performance of the file. RMS reads and writes buckets in their entirety so that when you read a record from a file the entire bucket will be loaded into a buffer in your program space. Bucket sizes are specified in blocks (512 bytes). RMS will fit as many record cells in a bucket as possible. Record cells are equal to the defined maximum record length plus one flag byte plus two length bytes if the records are variable length. 5-35 Large buckets are good for access which is generally sequential or clustered. If your program reads a record and then reads an adjacent record which is in the same bucket, RMS will fetch the record without re-reading the bucket thus saving an I/O operation. Small buckets give better performance if the access patterns are random or if the file is shared between two or more programs. Interlocks are maintained by RMS on a bucketwide basis so that reading a record from a file with large buckets will tie up more records than if smaller buckets are used. RMS also keeps a flag byte for each record cell in the bucket. This byte tells whether the cell has ever been used and whether it is presently occupied. RMS can therefore distinguish between records which have never been stored, have been stored, and have been deleted. 5.3.6 Designing Indexed Files Designing Indexed files can be a complex process. You must define key fields, decide on optimal bucket sizes, partition the file into areas and populate the file. Any or all of these decisions may affect the performance of the file. Here, more than anywhere else, you must consider each feature in light of it's associated cost and use only those facilities which are really necessary. 5.3.6.1 Specifying Key Fields When you create an indexed file you must specify how many keys you will use and a number of parameters for each key. You must indicate how many segments each key has and where they are located, whether the key may be duplicated, what type of data it is, and an optional null value. Key placement within the record is not critical for static files. If, on the other hand, records are to be deleted, the primary key should be located at or near the beginning of the record to reduce the amount of residual overhead in the bucket. The length of the key is also important. Index buckets contain key values and bucket pointers. The shorter the key is the more index entries will fit in a bucket and the shallower the index as a whole will be. Shallow index structures mean fast access. Keys may be string, integer or packed decimal fields. Whether or not you allow duplicate values depends on the application but you should be aware that allowing duplicates can slow down writing of records and increases the amount of overhead due to deleted records. 5-36 Null values can be specified for alternate key fields. If a record is stored and an alternate key field contains only the null character then no entry is made for that record in the alternate key index. If many such records are stored the overhead of the alternate key index processing can be kept to a minimum. If at some later time the record is updated and a real key value is supplied, the alternate key index will be updated accordingly. 5.3.6.2 Sizing Buckets The same general rules govern bucket sizes in Indexed files that pertain to Relative files. If access is expected to be primarily sequential or clustered around certain key values, make the buckets large. If access is expected to be random or the file will be heavily shared, make the buckets small. The memory penalty for large buckets is greater for Indexed files since RMS will allocate two, bucket-sized buffers when an Indexed file is opened. Furthermore, you should consider the relationship of bucket size and the number of levels in the index. The time required to process an index bucket (computation) is negligible compared to the time required to move to the next index level (I/O). Therefore, the time to access a record will be directly proportional to the number of levels in the index. Larger buckets mean a shallower index and faster operations. You should set the bucket size to the smallest size (program size considerations) that will result in an index with three or four levels including the data. Very large files may require five levels total. You must anticipate the amount of activity overhead in the bucket due to splits and deleted records. If you expect the file to be static, that is, it will be loaded once and then read many times you need not allocate space for the overhead. If, on the other hand, you intend to insert and delete a lot of records you must allocate additional space. Finally, you may specify a Bucket Fill Size when you create the file. This factor is the percentage of each bucket that will be used if the Mass Load bit is turned on during the initial load of the file. This is of most benefit when records will be added uniformly across the file. Leaving the extra space in each bucket will reduce (but not necessarily eliminate) the number of bucket splits which occur and therefore the RRV overhead. There will most probably be unused space in some buckets, however. 5.3.6.3 Areas 5-37 RMS Indexed files may be partitioned into Areas as part of the initial allocation. An Area is a portion of the file which is set aside for a specific purpose. Each index may have up to three areas allocated to it. Each Area may have it's own allocation, extension size and bucket size. The primary advantage of this partitioning scheme is that logically related buckets are grouped closely together in a file and head movement while traversing an index tree or scanning sequentially through data buckets is minimized. In the simplest case a file could be divided into an Index and a Data area. When bucket splits occur, the extension buckets are allocated from the appropriate area. Without Areas, index buckets will be scattered throughout the file in such a way that moving from one index level to another may require moving the disk head a considerable distance. Areas also allow a certain amount of optimization. You may be able to improve performance by specifying a smaller index bucket if data records are very large and access is primarily random. Large index buckets should be used if records are small or access to the file is clustered. 5.4 Tuning RMS Files File design is only one aspect of what makes an application mix go fast or slow. But it's a highly visible aspect and is one with which developers feel most familiar and comfortable. It is often the case that poor file performance results from a lack of understanding that every feature of a system has an associated cost and, in some cases, the cost may not be obvious or may be described in such a way that it's significance is not properly marked. This section will point out some of the more common areas in which unneeded features can burden an application. See the RMS-11 User Guide for more details. 5.4.1 Avoid File Opens One of the most expensive things you can do with a file is to open it. RSX supports lots and lots of files in any given directory and developers who are moving over from CTS-300 may not be making effective use of the directory structure. Once you have a file open you should avoid closing it since the close usually means another open at some later time. There is more program space available on RSX than any other PDP-11. You may be able to keep files open on RSX that memory constraints caused you to close on other systems. 5-38 5.4.2 Use the Simplest Possible File Structure If you use a multi-key index structure the file will necessarily be larger and slower than if you use a single key. A single key file is more complex than a relative file and access to a record with a known key is slower. Sometimes developers discover that they can live without ISAM after all and the performance difference is considerable. 5.4.3 Beware of Overloading Directories Directory searches are no different from scanning any other sequential file and records at the end take the longest to find. Keep your directories small and balanced for fast access. Also, you should realize that when you create a new file the system has to search through the entire directory to see if a file of the same name already exists. Consider re-using an old file in place of creating a new one. 5.4.4 Pre-Allocate the File If the file is to be static and the amount of data to be stored therein can be estimated then you may wish to allocate the entire file when it is created. This will keep fragmentation to a minimum and improve processing speed. Such allocations must be judged in light of total disk capacity and their affect on free space. 5.4.5 Make Critical Files Contiguous The cost of locating the different parts of a file can represent up to 30% of the disk activity. Make your file contiguous. Failing that, make the extents of the file as large as possible. If a file which is contiguous is extended then it is no longer strictly contiguous, although some of the benefits of the initial contiguity still pertain in that the initial allocation is a single large extent. If the file is further extended over time you may wish to periodically restore it to a contiguous state. You can do this from your application by spawning a COPY/CONTIGUOUS command. 5.4.6 Substitute Code for Mechanical Movement Disk activity almost always involves some mechanical movement, often requiring tens of milliseconds to complete. It is often possible to replace disk operations with program code. If you are suffering a performance problem, try to determine whether you could simplify the file by doing a little more work in your 5-39 program. 5.4.7 Distribute the Load Multiple disk spindles very often can provide more throughput than a single drive for a given total storage capacity. The disk head itself is subject to a number of demands in addition to data transfer such as directory manipulation and program overlays and consequently may become a significant bottleneck. Even though timesharing will continue while the head is being moved all the other disk I/O requests will be delayed and since commercial systems depend heavily on I/O this is a serious problem. RSX supports overlapped seeks if multiple disk drives are available so that a number of disk requests can be processed simultaneously. 5.4.8 Avoid Going Overboard Multiple keys are costly if records are being stored or updated in a file. The number of disk operations required to store a record is proportional to the number of keys. Very long keys consume extra storage and can increase the number of levels in an index by reducing the number of key entries in an index bucket. Keep the keys near the beginning of the record. For certain types of files, deleted records leave behind a fragment which includes the record from the beginning through the end of the primary key. Load the file in order of ascending key value. Doing otherwise will cause more frequent bucket splits and a more complex index. Avoid duplicate keys, if possible. Duplicate primary keys increase the overhead of deleted records. Duplicate alternate keys can result in long chains of pointers in the SIDR records. 5-40 Chapter 6 RMS Utilities Most programming languages on RSX lack support for one or more RMS features. Moreover, there are a number of operations which should be carried out on a regular basis for creating and maintaining RMS files. Therefore, special utility programs have been provided which may be invoked interactively or may be spawned with a command file. What follows is a brief description of three of these utilities. For details about the operation and use of these programs see the RMS-11 Utilities Manual. 6.1 RMSDES File Design Utility RMSDES is an interactive utility for designing and creating RMS files. You design a file by issuing commands to set, clear and display file attribute values in a file design buffer workspace. Attributes are arranged in functional groups which pertain to the target system, file, record, key and areas for the file. You may enter or change the attributes in order from beginning to end, by section, or individually. You may enter the attribute values directly, load them from a description file or copy them from an existing data file. You may create a file directly or save the workspace in a description file for use at another time. When the attributes are arranged to your satisfaction you may create an empty file. RMSDES will check to see that all required values are present and will perform sanity checks on the values. 6.2 RMSIFL Indexed File Load Utility RMSIFL reads records from any type of RMS file and loads them into an empty Indexed file. RMSIFL is superior to other loading methods in that it bypasses the standard RMS mechanism and exploits the underlying file structure to produce an output file quickly and with optimal packing of buckets. Use of RMSIFL is the most common way of honoring the Area Fill values specified for the indexed file when it is created. RMSIFL operates in a number of phases. It will optionally sort the input records by primary key value before loading them into the indexed file. During the load it will extract any alternate key values and store them in a temporary file. When the data records are completely loaded, RMSIFL will sort the values for each alternate key and then build the alternate key indexes. The resulting file is optimized for the fastest possible access along all index paths. 6-41 RMSIFL has some limits. The index structures are created without the assistance of RMS which means that the output file must be empty at the beginning of the load. RMSIFL can handle no more than 20 keys per record and may be limited to bucket sizes of five blocks or less depending on the format of the record and the keys. These restrictions aside, RMSIFL is absolutely the fastest way to load a high performance indexed file. 6.3 RMSCNV File Conversion Utility RMSCNV can read data records from any RMS file organization and load them into any other, either locally or over a network. RMSCNV uses the standard RMS facilities to read the data and build the output file. It is not subject to the same limitations as RMSIFL. RMSCNV can be used to append records from one file to another, re-establish contiguity and convert data from one file format to another. If the output file is to be sequential, RMSCNV can create it. Otherwise the file must be created before RMSCNV can be used. A number of switches are available (and may be required) which determine such operations as record truncation and padding, recognition of bucket fill size and mass insertion mode, block mode operation and so on. When used with Indexed file structures RMSCNV is not subject to any of the restrictions of RMSIFL but does not provide the same level of performance. 6-42 Chapter 7 RMS for CTS-300 Developers Since many commercial users are migrating their applications from CTS-300, this section is set aside to point out some of the more commonly encountered problems caused by the differences between DMS and RMS files. 7.1 Record Locks The biggest single difference encountered by developers moving from CTS-300 to RSX is the way record interlocks are managed. CTS-300 (and VMS) were created with the record management facility tightly integrated with the operating system and it is possible, therefore, to control access to shared files in such a fashion that potentially dangerous operations can be avoided. The most common such case is allowing one program, which has opened a file for input, to read a record or bucket which has been locked by a second program which has the file open for update. This is permitted on both CTS-300 and VMS such that read operations issued to a file opened for Input will never encounter a record lock. There may, in fact, be delays in the read operation if the request occurs at a difficult time but these delays are temporary and are invisible to the application. RMS on RSX is more of a layered application than an integral part of the operating system. It is possible, therefore, that a program which has a file open for input might try to traverse an index tree which is being updated by another program and if this happened, the program issuing the read might crash. To prevent such an occurrence, any access to a file which has been opened for update or write operations is subject to record locks. 7.2 Access Modes As explained earlier, DIBOL makes certain assumptions in mapping the various language statements to RMS facilities. One commonly encountered example has to do with access modes declared when a file is opened. When a program calls upon RMS to open a file it must declare both the operations it intends to perform (Read, Write, Update, Delete) and also the operations it will allow other programs to perform. RMS matches these access declarations with those of any other program which has the file open and will only permit the latest program to proceed if the access codes are compatible. This has certain side effects. A common example is the complaint that DIBOL on RSX is much slower than DIBOL on RT-11 or RSTS since it takes longer to read sequentially through a file. For some reason that is not clearly understood the sample file chosen 7-43 for such experiments is always a relative file which results in an unexpected condition. When DIBOL opens an existing file for input it tells RMS that it wishes Read only access and, because of the traditions of CTS-300, it will allow other programs write access. RMS interprets this as a hint that the data in the file may change (even though no other programs have the file open at that moment) and consequently will re-read the entire bucket each time the DIBOL programs tries for the next record. If the DIBOL program were to open the file in Update mode, the bucket would be locked when the first read occurs, and subsequent reads take place directly from the bucket in memory. There would only be I/O when moving from one bucket to the next. The difference in processing speeds between input and update mode will be directly proportional to the number of records in each bucket. That the open mode should affect the speed with which a program may scan a file is just one example of how "poor performance" may be due to a simple misunderstanding. 7.3 Interchanging Sequential and Relative Files It is a common practice on CTS-300 to create a file as sequential, fill it with records and then close the file and re-open it for random access. This is also possible with RMS except that the file must be created with a Relative structure. While RMS permits limited random access to a sequential file with fixed length records, the use of Relative files offers more capability and is the preferred method. There is one consideration in transporting such an application from CTS-300 to RSX or VMS. It is sometimes the case that such files begin with a header record which is a different length than the transaction records. Since RMS relative files are allocated in record cells, each large enough to contain the largest possible record, it would be a wasteful to write a very large record followed by a number of smaller ones. It is much more efficient to write the header data into a number of smaller records and adjust the transaction record numbers accordingly. 7.4 Extending Files RSX permits files to be extended regardless of the internal structure while CTS-300 does not. This means that you need not allocate all the space required for a file initially. Performance will suffer somewhat if the file is extended many times but you can overcome this by periodically using RMSCNV to restore the contiguity. 7-44 7.5 Print Files It is common practice on CTS-300 to write records in fragments using mixtures of WRITES and DISPLAY statements. This is done partly for coding convenience and partly due to the increased flexibility of formatting. This practice is possible because DMS files are really an unstructured stream of characters whereas RMS files (on VMS as well as RSX) are a series of records. Formatting of reports on RMS is provided through the Print Mode files (O:P) which use the NONE declaration for Carriage Control when the file is created. Each subsequent output operation, whether a WRITES, DISPLAY or FORMS creates an individual record and the application program is responsible for adding all the carriage control characters. While these files are somewhat lacking in storage efficiency, the programmer has full control over the output format. A consideration when using such files is that attempts to re-read the file must be made with care. On CTS-300 it is possible to create a record in a file with a number of DISPLAY statements followed by a WRITES. On re-reading the file, all of the information will be returned as though it had been written as a single record. With RMS this will not be the case. Each field will be stored as an individual record and will therefore be returned as individual records in exactly the order with which they were written. 7-45 Chapter 8 Flow Control Most small system commercial applications are broken down into a (possibly very large) number of program modules, each of which will fit within the bounds of a limited hardware configuration and CPU architecture. Flow control is that aspect of application design which determines the manner in which these logic modules exchange information and control. One of the dimensions along which operating systems differ most widely is the variety and comparative efficiency of the flow control mechanisms. CTS-300 (RT-11) and RSTS are limited to program Chaining. VMS allows a primary program module to Chain to a secondary, spawn a secondary in a sub-process or, when the secondary module permits, call it as though it were a subroutine. RSX permits both chaining and spawning - each in a variety of flavors. Each of these mechanisms has it's advantages and disadvantages. RSX had it's origins in the Scientific/Industrial marketplace where Real Time response is a top priority. The design point was to create a system in which a particular program module (Task) could be activated as quickly as possible upon the occurrence of some event. Consequently, the flow control architecture is more complex than the other systems, and it requires a bit of understanding before it can be used to greatest advantage. 8.1 Concept of a Task The fundamental unit of execution on RSX is the TASK. Compilers and editors are tasks. Application programs are tasks. Even device drivers are tasks. Activation of a task is a two step process. 8.1.1 Installing a Task Before RSX can do anything with a task it must be Installed. Installation is the process by which the location and characteristics of the executable module are made known to the system executive. Once installed, a task is available for execution and remains so until it is removed. Each task on the system must be installed with a unique, six character name. During it's installation the executive records the task's location on disk (by File ID for very fast access), it's priority, the name of the partition in which the task will execute, and several other items which reduce to an absolute minimum the number of operations which remain before the task can actually begin to execute. Contrast this with other systems in 8-46 which a RUN command will cause the executive to begin looking through disk directories just trying to find the program image. 8.1.2 Task Names Installed tasks are identified by name and therefore the name must be unique on the system. Multiple programs may have the same task name but only one of them may be installed at any time. The name of the task may be established when it is linked, when it is installed, or when it is requested and run. A special case of task naming is the 'Prototype' name which consists of three periods followed by three alphabetic characters such as "...PIP". This is the mechanism by which multiple terminals can be executing independent copies of the same program. When a particular terminal invokes a task which was installed with a prototype name, the executive creates a private copy of the task for the requesting terminal and gives it a unique task name by combining the prototype name and the terminal name. Thus, a copy of PIP running at terminal 11 becomes "PIPT11". The prototype task name is an important facility for systems where multiple copies of job streams may be executing simultaneously. 8.2 Requesting an Installed Task Once a task is installed and brought to the absolute brink of execution, all that remains is for some entity on the system to request it's execution. Some tasks are requested as a result of a command typed at a keyboard. Others are requested via system directives issued by another program. There are a number of different mechanisms involved but the functionality of all of them falls into one of three categories. 8.2.1 RUN a Task The simplest way to invoke a task is to RUN it. The RUN command has four forms. 8.2.1.1 Running an Installed Task If a task is already installed you may invoke it with the RUN command and the task name: RUN taskname 8.2.1.2 Running a Task from a File If a task is not installed you may invoke it directly from the 8-47 task image file: RUN filespec This form of the RUN command invokes a special facility known as Install-Run-Remove. Before the task actually runs it is installed with your terminal name (e.g. "TT11") as the taskname. The task is then run and when it exits, it is removed. You may pass a command to the task with the /COMMAND:"command string" qualifier. 8.2.1.3 Running a System Utility You may invoke the system utilities by preceding the task image filespec with a dollar sign: RUN $RMSDES This tells RSX to look in the system library for the image file. You may pass a command to the utility with the /COMMAND:"command string" qualifier. 8.2.1.4 RUN as a Scheduling Command The last form of RUN allows a privileged user to schedule the execution of an installed task for some future time: RUN/SCHEDULE:hh:mm:ss taskname 8.2.2 Chaining between Tasks The chain mechanism on RSX is the RPOI$ directive - Request and Pass Offspring Information. All the popular implementation languages provide support for this directive in one form or another. Check the User Guide for details. There are two options associated with the RPOI directive of which you should be aware. One option determines whether the program issuing the directive exits. The other determines whether RSX Parent-Offspring connections are passed to the target task. Unless there is a good reason to do otherwise, both these bits should be set. The directive itself may be used in two ways according to whether the offspring task has been installed. 8.2.2.1 Indirect Chaining (to a non-installed task) You may chain to a non-installed task image file indirectly by 8-48 forming the task image filespec into a pseudo RUN command and then RPOI to either MCR or DCL passing the string as a command. This is how DIBOL and BASIC implement the STOP 'filespec' and CHAIN 'filespec' statements. This form will be the most familiar to developers who are used to working with CTS-300 and RSTS but it is also the least efficient. This is really a special case of the RUN filespec command and because of the number of intermediate tasks which must become involved it is the slowest form of chain. 8.2.2.2 Direct Chaining (to an installed task) You may chain directly to an installed task by using the installed task name as the object of the RPOI$ directive. DIBOL supports this form with the CHAINI subroutine. You may pass a command string to the offspring and the performance is much, much better than the indirect chain. 8.2.2.3 Chaining to a Command File You may chain to a command file by issuing an RPOI$ to the Indirect Command File processor and passing the the name of the command file to be executed. The last action of the command file should be to invoke the next section of your application. 8.2.3 Spawning an Offspring Task Developers who are moving from CTS-300 or RSTS may, at first, have difficulty understanding the significance of the Spawn facility but it is one of the principal reasons that RSX was chosen as the first operating system for A-to-Z. It is the foundation of the A-to-Z application integration architecture in which commonality of form and function is attained by spawning the appropriate module to perform an operation such as spawning Business Graphics to display some data. It also serves as the mechanism by which otherwise unrelated applications may be nested such that the operator may request an ongoing task to be suspended while another function is performed. That function may also be interrupted and so on. With or without A-to-Z, Spawn allows an application to invoke almost any facility on the system without losing any context other than, possibly, the contents of the screen. You can spawn a system utility with a command to create an ISAM file, for example, and wait for the exit status to determine the success or failure of the operation. You can spawn infrequently used sections of your own code as independent tasks to avoid having to build them into the mainline images. You can spawn the CLI tasks 8-49 to execute commands as though they were being entered from the keyboard. The possibilities go on and on. Like Chain, Spawn has two forms depending on whether the offspring task is installed or not. 8.2.3.1 Indirect Spawn (of a non-installed task) You can spawn a non-installed task indirectly by spawning your favorite command line interpreter with a 'RUN Taskname' command. You can include the /COMMAND:"command string" qualifier for the RUN command to pass a command string through to the task. You should also use the /STATUS:TASK qualifier as this will assure that the exit status which is passed back to the parent task will be that of the task and not the RUN command itself. Because it carries all the overhead of the RUN command entered from the keyboard, this is the slowest of the two forms of spawn. 8.2.3.2 Direct Spawn (of an installed task) Direct Spawn of an installed task is the fastest way to invoke an external facility on RSX. You may pass the task a command string and you may wait for exit status. You can spawn more than one offspring task at a time if your implementation language interface to spawn permits. The designer of the parent task is responsible for deciding whether to wait for the offspring task to exit and for correctly interpreting any status which is returned. If task A spawns task B which in turns chains to task C passing all connections and task C exits, the status returned to A is that of C, not B. 8.3 Parent-Offspring You may choose to create a library of spawnable tasks for your application. These tasks may perform infrequently used functions or they may serve to centralize certain functions as in the case of multiple front end tasks feeding transactions to a single (or multiple) background processor. The two most commonly used means of communication with the parent task are the command line buffer passed from the parent task to the offspring and the status code passed from the offspring back to the parent task. 8.3.1 Command Line RSX provides for passing a 255 byte command line to any offspring task. The offspring must retrieve this buffer through use of the GMCR$ directive. Access to this directive is available in the more popular implementation languages. This buffer may contain function codes, filenames or any other data in a format agreed 8-50 upon by the parent and offspring task designers. 8.3.2 Status Code RSX provides for a task which is Exiting to pass back a 16-bit status code to the parent task. The directive is EXST$ and access to this directive is available in the more popular implementation languages. The low order three bits of this code are defined (by convention) to be the error code and severity level. The remaining bits may be used to communicate any other information in a format agreed upon by the designers of the parent and offspring tasks. A-to-Z has established a convention by which this 16 bit value is divided into fields, each with a particular meaning. You might consider adopting this convention. 8.4 Intertask Communication RSX provides two principal mechanisms for communicating between tasks running on the system. 8.4.1 Send/Receive Send and Receive can be used to pass blocks of information between cooperating tasks. The block can consist of up to 500 bytes of information. If more data is to be passed it can be done with multiple blocks or by storing the information in a file and passing the name of the file in a message. A second, more complex mechanism allows passing of a memory common between two tasks. This feature is more complex than most applications require and the memory mapping requirements make it's use in commercial applications uncommon. Support for Send Data and Send By Reference varies from one language to another. Some languages provide support through library subroutines while others allow access to the requisite system services. 8.4.2 Global Event Flags RSX supports a second intertask communication mechanism which is specifically intended for synchronization of task execution. It is possible for a program which has access to system services to define and manipulate event flags on a task, group or system wide basis. Event flags are one bit registers which can be set, cleared or 8-51 interrogated via system services. Typically one application will set a flag whenever data is available for processing by another. The waiting application will clear the flag when all available data has been processed. Support for event flags is limited or non-existent in some languages. It is mentioned here because it is the highest performance intertask signaling mechanism available on RSX. 8-52 Chapter 9 Linking and Overlays Probably the single biggest surprise to developers who are moving to RSX from other environments is the linker - The Infamous Task Builder (TKB). It's very, very slow. We know it's slow, we use it too. There's no getting around it. No matter which implementation language you have chosen you will sooner or later have to deal with the Task Builder. It's also an exceptionally powerful linkage editor. With it you can form your program into a monolithic or overlaid structure, hook in shared common regions, set execution priority, pre-open I/O channels, make use of Supervisor Mode shareable libraries, patch global references, set stack sizes, even get cross reference tables. If your language allows, you can create Multi-User (shareable task images). You can even arrange that disk overlay segments be loaded asynchronously. The cost of all this functionality is that the Task Builder itself executes very slowly compared to RT-11 or VMS. Again, this section is not intended to make you an expert but rather to get you started as quickly and as easily as possible. See your language RSX User Guide for further details. 9.1 Invoking TKB TKB commands and parameters can be entered interactively but, for any but the simplest programs, you will find it much easier to use it with command and map files. The Command file (.CMD) contains the input and output file specifications and all the option statements. The Map file (.ODL) describes the location of the object files and the manner in which they are to be arranged and/or overlaid. If an ODL file is to be used the name of the file is supplied in the CMD file in place of the list of input modules. Most of the high level languages (DIBOL, BASIC, COBOL) have a 'build' option of some type which will construct prototype CMD and ODL files for you. The CMD files can often be used as is. Unfortunately, it is not possible for the various compilers to anticipate the way your subroutines will call each other so it is left to you to design and describe the module overlay structure, and edit the ODL file accordingly. 9.2 Disk Overlays 9-53 As an application designer you may choose from a number of options with regard to overlays. You may use a single overlay tree structure or you may use multiple, co-trees. You may specify that the overlay segments be loaded manually or automatically. You may direct that the overlays be loaded from disk or be mapped from memory. Overlaying subroutines from disk on RSX is done as on most other systems - a call to an entry point in the routine is actually diverted to a system (linker) supplied autoload vector which checks first to see if the segment is already loaded and if not, issues a read request to the program image on disk to load the blocks which contain the desired routine. Once the routine has been loaded into the proper address range, the call is allowed to complete. Note that this mechanism is only intended to work for a call. Attempting to reference data in overlay segments will generally not work unless you load the segment manually. Overlay segments may be specified as manual load or autoload. Only the latter mechanism is of interest to most commercial application developers. 9.2.1 Trees TKB requires that overlays be strictly tree structured, that is, the Main Program module forms the root or trunk of the tree and the subroutines are grouped into segments which form the branches. When a module which is lower in the tree calls a module which is higher up, the intervening segments are also loaded into memory. Subroutines may not call each other across branches and TKB assures that global symbols are resolved uniquely in each path of the tree. Thus, if routines in two or more branches of the tree call a common routine, and that routine is not linked into the root (routines in the root are always available to be called from everywhere), then there may be two or more copies of the routine in different places in the task image on disk. If such a single tree structure is used, TKB will assure that the references throughout the tree are correct and proper. It is, in fact, the searching up and down the branches of the tree to assure that there are no ambiguities in the intercall structure and that the return paths will always be protected, that consumes much of the execution time of the task builder. 9.2.2 Co-Trees Developers who are used to other sorts of overlay structures, 9-54 such as the Regions used on RT-11 may find that task images become monstrously large due to the requirements for unique pathways and hence duplication of code many times throughout the structure. It may, in fact, be impossible to attain certain kinds of functionality, particularly if subroutine context is expected to be maintained across calls to the routine. In such a case you may use Co-Trees to emulate the RT-11 region structure. To do this, you define multiple root segments, one for the root segment and one for each 'region'. The modules which are to reside in each region will then be linked to the appropriate root. These extra root segments may contain application code or they may be dummy (null) segments created with NAME directives in the ODL file. Co-Trees allow a much greater degree of flexibility in terms of calling segments from different parts of the program and following different paths to a common routine. However, it is impossible for the task builder to determine what order calls will occur in and there is no protection against calls which might cause the return path to be destroyed. The following (oversimplified) command sequence will direct the RT-11 linker to create a program with a root and two overlay regions. MAIN can call any of the four subroutines directly and the routines can call each other as long as the return path is preserved. PROG=MAIN/C SUBRA/O:1/C SUBRB/O:1/C SUBRC/O:2/C SUBRD/O:2 This (oversimplified) ODL structure will direct the task builder to create an overlay structure which is functionally identical to the one shown above. .ROOT MAIN-REG1-REG2 .NAME NULL1 REG1: .FCTR NULL1-*(SUBRA,SUBRB) .NAME NULL2 REG2: .FCTR NULL2-*(SUBRC,SUBRD) The RT-11 overlay regions and RSX co-trees are so similar that translation from one to the other is almost entirely mechanical. 9.3 Memory Resident Overlays Memory Resident overlays work by changing the virtual address mapping registers for the task. When the program is initially loaded into memory the entire task image is loaded from disk. 9-55 When a call to an overlay segment takes place, the call is once again diverted to the autoload vectors, only in this case the segment load is accomplished by changing the memory management mapping control registers so that the subroutine 'appears' in the correct place. Since there is no wait for the disk transfer to complete, the overlays are much faster. There are, however, a number of restrictions. Because of the way the memory management hardware works the memory resident overlay segments must begin on 8 KB memory boundaries and the segment size may not exceed 8 KB. Also, the task builder locates the overlay segments in such a way that the task 'upper limit' cannot be extended in memory. The former restriction is a nuisance but the latter is more severe, particularly for language processors which allocate 'heap' storage dynamically. Of the popular languages on PDP-11's, only Fortran allows the use of memory resident overlays. The use of Disk Data caching in RSX V3.0 will, however, improve the speed of disk overlays to almost that of memory resident. 9.4 Guidelines for Use Overlays do not come free. Even the memory resident overlays require a fair number of machine instructions to be executed. You will find, therefore, that if overlays are misused there may be a problem with performance. In particular: Disk overlays are just like any other kind of disk I/O. There is a delay while the autoload code waits for the QIO to complete and there is an opportunity cost when the disk head is moved away from the data files. You should carefully consider the flow of control through your subroutine structures. The best possible design is where the flow is sequential through a series of segments in the same region with each performing it's particular function in turn. Many application root segments are nothing more than a common data region for context and a series of calls which invoke a stream of modules moving through the segment. The Worst possible design is one in which two, co-resident routines are called repetitively. There was a case in which the keyboard input and output routines of a program occupied the same region but were in different segments. Every time the operator pressed a key there were at least two disk read operations. The performance of the application doubled when these two routines were moved to the root. 9.5 Linking to RMS RMS is provided in the form of one or more subroutine libraries 9-56 which are, insofar as is possible, language independent. All of the high level language compilers translate the various I/O statements into a call to one of the RMS routines and provide such mapping of arguments as may be required for proper communication. The first version of RMS was nothing more than a collection of object modules which had to be linked directly to the application program. As hardware and software technologies improved it became possible to shortcut many of the linkage processes while at the same time improving task sizes, linking time and processing speed. RMS is provided in several different forms and each has it's own linkage method. You must select the flavor and the linkage when you task build your program. You may chose between a variety of linkages, usually without changing any aspect of your application code. 9.5.1 Disk Resident RMS It is still possible to link the RMS modules directly to your program. The code is quite large, however, and arranging a suitable overlay structure may become burdensome. Unless you have very special requirements you should select the clustered library. 9.5.2 Memory Resident RMS RMS is provided in a special library which uses memory resident overlays to keep the virtual program space requirements to a minimum. When you use this form you actually link only a stub routine into your program. This stub intercepts the calls to the various RMS routines and causes the appropriate RMS segment to be mapped into your program space. The RMS code itself is stored in a special library which must be independently installed on your system. Since you are only linking to the RMS stub the link process itself is much faster. This form of RMS is considerably faster than the disk overlaid version since the RMS segments are shared between all applications and are usually already in memory. All of the disk overhead associated with the RMS overlays is eliminated. Use of this library also has the advantage that there is only one copy of the RMS code on the system. This results in a very great savings of space required to store task images. Finally, it means that the system manager can upgrade to a newer version of RMS by simply replacing the common library. There is 9-57 no need to re-link all the application programs. You select this form of RMS by including a LIBR declaration in your CMD file and a pointer to one of the secondary RMS ODL files in your primary ODL file. The RMS library will use two of the eight memory segments your task is allowed, thus consuming 16KB of address space, about what the full, disk-overlaid version requires. 9.5.3 Clustered RMS Most languages require a run-time library of some sort and, in most cases, this library is similar to the memory resident form of RMS. Since RMS requires 16KB and most of the language libraries require a similar amount this would limit your program space to 32KB total. It is possible, however, to cluster the language and RMS libraries in such a way that they occupy the same virtual addresses so that the available program space is increased to 48KB. You direct the task builder to cluster RMS and the language library by replacing the LIBR statement in the CMD file with a CLSTR statement. This is the default for most of the Build options in the language processors. 9.5.4 Supervisor Mode RMS Memory resident overlays are considerably faster than disk overlays and they certainly reduce the amount of disk head contention on the system, but they still consume CPU time. On systems which support Supervisor Mode you may link RMS as a supervisor mode library. This option will cause RMS to use two otherwise idle Address Page Registers (APR) and will reduce the time required for context switching between the RMS and language libraries. You specify supervisor mode RMS by changing the LIBR or CLSTR statement in the CMD file to RESSUP and making certain explicit references to special RMS modules in your ODL file. See the RMS-11 User Guide for details. 9.6 Useful Options The Task Builder offers several useful options to the developer. These options are invoked with statements in the CMD file. 9.6.1 Privilege In the early versions of RSX a privileged task was one which was mapped directly to the executive and could read and write the 9-58 various data tables and structures. Privileged tasks are also exempt from the system security mechanisms and can read, write, modify and delete files regardless of ownership or protection codes. It is now possible to build a privileged task which does not map to the executive but which can bypass file protection. You can use this technique to permit a user to perform occasional privileged operations. To Task Build a privileged program, append the /PR:0 switch to the task filespec in the CMD file. 9.6.2 Task Priority You may be able to improve the overall crispness of your application by adjusting task priorities. You may modify a task priority when you install it or when you run it, but you may wish to set the normal operating priority with the Task Builder. You may specify a task priority with the PRI=nnn option. Priority may range from 1 through 250 with higher numbers indicating higher priority. Normal system priority is 50. 9.6.3 Task Name You can set the task name with the TASK=name option in the CMD file. This is the most convenient way to set a task name to other than the task file name. It is also a convenient way to specify a prototype task name. 9.6.4 Cross Reference Map You can direct the Task Builder to append a Cross Reference table to the Map file. This table will show where all the global entry points (usually subroutine names) are defined in your task and where they are referenced. This option is particularly useful when you are restructuring your overlays or trying to locate typographical errors in CALL statements. You request this table by appending the /CR switch to the map filename in the CMD file. The Cross Reference Facility (CRF) must be installed on your system for this to work properly. 9-59 Chapter 10 Packaging Applications for RSX There are two views of layered product installation: That of the person performing the installation and that of the developer who is constructing the distribution kit. The former is discussed in the section on Getting Started with RSX. This section will outline what a developer must do to package an application for installation on a production system. The layered product developer is responsible for creating the distribution kit - media and documentation - that will enable the system manager to install the software as quickly and easily as possible, and with the desired level of functionality. The kitting process takes two forms according to whether the target system is Micro/RSX or M-PLUS. 10.1 Kitting for Micro/RSX Creating a software option for Micro/RSX will be easier if the developer understands something of the underlying architecture. 10.1.1 Micro/RSX Layered Product Installation Architecture Micro/RSX was designed with the naive user in mind. A Command procedure is supplied with the system which controls installation and removal of all layered products. This means that the developer is responsible for packaging the application in such a way that the command procedure will operate correctly. When the installation control file (OPTION.CMD) is invoked, it will mount the media on the drive selected by the user and will look in directory [0,0] for a file INSTALL.DAT. This file is provided by the developer and contains certain information about ALL the options on the media set including the name of the option and a pointer to the installation parameter file for the layered product. The installation parameter file is also provided by the developer and it contains information about the files that comprise the application, where they go on the target system and whether they are to be deleted when the option is removed. It also contains directions for actions that must take place during installation, during system startup, and during removal. There are conditional mechanisms by which the installation can be modified if, for example, floating point hardware is available or I/D space is supported by the CPU. 10.1.2 Creating the Kit 10-60 The kit is nothing more than an image of the application as it will reside on the target system. If a file name APPL.XYZ is to be placed into directory [POOF] on the system then it is stored in a directory with the same name on the kit. When the application is installed, the appropriate directories are created as needed. The simplest case is where the distribution media is diskettes. The developer initializes enough diskettes to contain the product, creates directories and copies files into place, just as they will be on the customer's system. The developer then creates the INSTALL.DAT and installation parameter files and adds them to the disk set. Creating a kit for magtape distribution is approximately the same except that instead of creating an image of the application on disk, the files are distributed as a collection of backup save-sets. The INSTALL.DAT and parameter files are placed in a special save-set at the beginning of the tape. It is usually easiest to develop and debug an installation using disks as media and then convert the procedure to magnetic tape. The conversion from disk to tape kits is largely mechanical. 10.1.3 Developer's Note Underlying this automation is a database of interest to the developer, especially during the debugging of the installation procedures. A list of all the software options installed on a system is kept in the file LB:[1,2]OPTIONS.DAT. You can examine and modify this file with your favorite text editor. Each record contains the name of an application and a pointer to the installation parameter file for that application. If the OPTIONS.DAT file is deleted or corrupted, the layered product database is destroyed and must be rebuilt, usually by deleting all the application files and starting over. Under certain conditions (noted below) OPTIONS.DAT is examined during system startup and the various parameter files are opened and scanned for any actions which must be performed. The scanning process is slow, so as the actions are performed they are also collected in a shortcut startup file LB:[1,2]FASTART.CMD. FASTART.CMD is the key to system startup. If it exists during startup, it is executed directly, since doing so is much faster than parsing the different layered product installation parameter files. If FASTART.CMD does not exist, then it is rebuilt from the information in OPTIONS.DAT. Removing a software option from the system is one of the ways in which this file might be deleted. Since the parsing of the application installation parameter files 10-61 is slow, a system startup after an application has been removed takes considerably longer than normal. There are conditions, such as a product installation which aborts, in which the contents of OPTIONS.DAT and FASTART.CMD no longer match and the system may begin to malfunction. The cure for this is to edit OPTIONS.DAT to assure that the contents are as desired. Then delete all versions of FASTART.CMD and restart the system. FASTART will be built anew during the subsequent startup. 10.2 Kitting for M-PLUS Creating a layered product for M-PLUS is largely a matter of collecting all the components and creating a command file which controls the installation procedure. The Indirect Command File processor (IND) on RSX provides extensive and detailed access to the underlying system and it is extremely rare that a layered product has any installation requirements which cannot be satisfied with IND. It is now fashionable to design the kit in such a way that the installer need only copy the INSTALL.CMD file into a temporary directory and invoke IND to execute it. Writing IND procedures is no more difficult than other kinds of programming but, like all implementation languages, it has a flavor and style of it's own. If you have access to any layered products you may wish to examine a copy of the INSTALL.CMD and possibly even use it as a starting point. When creating this command file the developer must consider the spectrum of hardware and software configurations upon which the product is to be supported. If any of the tasks require access to the system executive and/or data base they will need to be linked on the target system. Provide the required object modules (in a library for convenience) and the command and odl files, and link the task during installation. The command file should be structured with as much of the dialog as possible up front so that once it is complete the more lengthy operations can proceed without further intervention. You should provide for errors during installation and where it is not possible to recover from the errors, provide a means by which the installation can be unwound and the system restored to it's original state. Documentation for the manager should not only describe the average conditions under which the product is supported but should also explain enough of the overall architecture that a knowledgeable system manager can tailor the product to an unusual configuration. You should provide information about the consequences of the different alternatives at each choice point. You should also provide whatever information the manager will 10-62 need to modify the system startup file to create the proper environment for your product. At the least, provide a system startup command file fragment which can be inserted by the manager. Creating a disk kit is straightforward. Initialize the disk and create as many directories as are required with whatever structure is most convenient. Your command file can then copy the files into the appropriate place on the target system. Magnetic tape is a little trickier since it is a strictly sequential medium and since there is a certain difficulty with unusual types of files. The most commonly used magnetic tape transfer utility is FLX which is rather like a PIP designed specifically for magtape. Once your disk installation is working to your satisfaction you can create a magtape kit by putting together a command file which invokes FLX to copy all the necessary files onto the tape, usually with the INSTALL.CMD file going first. It is important to match the order of the files on the tape with the order in which INSTALL.CMD attempts to copy them onto the target system. If the files are in order (even if some of them get skipped), the No-Rewind switch can be used with FLX to greatly reduce the time spent searching along the tape for the next file. There are two problems which are frequently encountered with FLX. The first is that there is no concept of file contiguity on magnetic tape so contiguous files, particularly task images, must be accorded special treatment. The FLX command to copy the task image to the target disk must not only specify that the output file be contiguous but must also specify an initial allocation. This allocation must be the actual size of the image in blocks so it will be necessary to update INSTALL.CMD anytime a task size changes. The other problem with FLX is that files with certain types of contents, such as message files, cannot be transferred. The only way around this is to convert the file to a simpler format before building the kit, and transfer the source file and any necessary commands or parameters for converting it back to the desired format once it's on the target system. You can make use of the temporary directory for such operations. 10.3 Kitting Applications for A-to-Z A-to-Z was designed to be installed on Micro/RSX and has extended the architecture somewhat to provide mechanisms by which a layered product could update the A-to-Z database and integrate itself into the A-to-Z environment. A-to-Z has more recently been made available on Pregenned and full M-PLUS and has carried with it a variation of the Micro/RSX layered product installation. This makes it much easier for a 10-63 computer naive system manager to install and manage an application mix. It also makes it easier on the product developers since the kitting procedure for A-to-Z applications on Micro/RSX and M-PLUS is the same. Only the media is different. 10-64 Chapter 11 System Operation Once your application has been modified to take full advantage of the features of RSX you may turn your attention to the details of the underlying system. It is important to tune the application first, since a job mix which misuses the underlying system or which consumes an inordinate amount of resources will make it difficult if not impossible to attain a satisfactory level of performance no matter how much effort is put into the system. This section describes the various dimensions along which you can tailor RSX for a commercial application environment. This section is not intended to be comprehensive but rather to get you started as quickly as possible. Consult the RSX Command Language Manual and the System Management Manual for more details. 11.1 Terminals Traditionally, RSX is oriented towards expert operators who require only the barest assistance or prompting from the Command Line Interpreter. The terminal driver reflects this heritage and developers who are moving to RSX from other operating systems may be startled by some of the default terminal characteristics such as being able to submit multiple commands for parallel processing by the operating system. Terminal characteristics are most commonly set during system startup. On Micro/RSX and Pregenned M-PLUS the terminal attributes may be set with suitable statements in SYSPARAM.DAT. On M-PLUS you may include SET TERMINAL/attribute commands in STARTUP.CMD. You may also set terminal attributes as part of the user's LOGIN.CMD. A-to-Z offers a comprehensive terminal management architecture which you may wish to use in place of individual system tailoring. The following list of terminal attributes may be set or cleared with the DCL SET TERMINAL/attribute command. This list is not inclusive but contains the terminal settings which are typically of importance on a commercial system. See the RSX Command Language Manual for further details. 11.1.1 CLI You may set or change the CLI attached to a terminal. Normally, you set the default CLI (most commercial people prefer DCL) when you create the account but there may be circumstances under which you wish to change the CLI for some special purpose. 11-65 11.1.2 BROADCAST Terminals which are set /BROADCAST will be subject to message delivery when messages from a BROADCAST command arrive. This may disrupt information on the screen but the message may be important enough that this is acceptable. Certain messages, such as those issued during the last five minutes of SHUTUP cannot be intercepted. Setting of this attribute is left up to the developer. 11.1.3 CONTROL=C This attribute determines whether typing CTRL/C at the keyboard aborts any program currently in progress at the terminal. If the terminal is set /NOCONTROL=C typing CTRL/C will allow any ongoing task to continue while the operator enters another command to the CLI. This is the means by which you may invoke multiple tasks simultaneously. The recommended setting for users in a production environment is /CONTROL=C. 11.1.4 INQUIRE Using the /INQUIRE attribute in a SET TERMINAL command will cause the operating system to send the Request Device Attributes escape sequence to the terminal and then to listen for an identification string. If the response from the terminal is one of the DEC supported terminals then the terminal type is set accordingly. This attribute is useful when the terminal type on a particular line might change from time to time such as when a LAN is in use. You may override the terminal type setting at any time with the SET TERMINAL/type command where type is one of VT100, VT200, etc. 11.1.5 FULL_DUPLEX Terminals which are set /FULL_DUPLEX may accept input and receive output at the same time and will allow overlapping terminal input and output requests for languages which permit such operations. The recommended setting is /FULL_DUPLEX. 11.1.6 ESCAPE Setting a terminal /ESCAPE enables escape sequence processing. Normally, an escape character in the input stream will be treated as a terminator but if escape sequence processing is enabled any characters which follow the escape will be examined. If the characters form a proper escape sequence then the entire sequence is passed as the input stream terminator. Some commercial 11-66 language runtime systems assert this attribute regardless of the setting. The recommended setting is /ESCAPE. 11.1.7 EIGHTBIT This attribute determines whether seven or eight bit bytes are exchanged between the terminal and the operating system. If your application makes use of Digital's Multinational Character Set you should set the terminal /EIGHTBIT. You may also wish to use this attribute if you have a modem attached to a particular terminal line. Many of the asynchronous file transfer protocol's require eightbit support. In this case you must set the terminal characteristics in the system startup command file. 11.1.8 HOSTSYNCH The hostsynch attribute is used with block mode terminals or with any device which may send a long string of characters to the operating system. If hostsynch is enabled and if the typeahead buffer is in danger of overflowing, the terminal driver will send an XOFF character to the terminal. When the characters have been processed and there is more room in the buffer, the driver will send an XON character. This attribute causes extra processing in the terminal driver and should only be used when necessary. The recommended setting for interactive terminals is /NOHOSTSYNCH. 11.1.9 SERIAL Normally RSX permits parallel command execution. If a first command typed at a terminal is followed by a second before the first has completed, the two will execute in parallel. Setting the terminal /SERIAL will cause the execution of the second command to be postponed until the first has completed. The recommended setting is /SERIAL. 11.1.10 SLAVE A terminal which has the /SLAVE attribute will not accept any input unless it has been attached by an application program. This attribute is useful in situations where very inexperienced operators are using the system. If such a user is running at an unslaved terminal and the application should abort the user would be confronted with the CLI and might do some inadvertent damage. If the terminal is set /SLAVE, input characters will only be accepted while the application is alive and well. If the program 11-67 aborts, the terminal will ignore any further input until it is set /NOSLAVE from another terminal, or the system is restarted. Note that the /SLAVE attribute should be set as part of the login process and /NOSLAVE set as part of logout. A-to-Z automatically sets terminals /SLAVE for all the user accounts. 11.1.11 LOWERCASE Terminals set /LOWERCASE will echo characters just as they are typed in. If the terminal is not set /LOWERCASE characters will be converted to their uppercase equivalent before they are echoed. /LOWERCASE is the recommended setting. 11.1.12 WRAP Terminals which are set /WRAP will automatically wrap lines which are longer than the line width setting. The terminal driver is not quite smart enough to keep track of cursor positioning, however, and if the terminal is set /WRAP, application screens may be corrupted by the automatic wrapping. /NOWRAP is the recommended setting. 11.1.13 TYPEAHEAD Setting a terminal /TYPEAHEAD directs the operating system to collect characters which are typed as input and to save them for a subsequent read request by an application. If the terminal is set /NOTYPEAHEAD, the characters will be discarded or will be passed to the default CLI. /TYPEAHEAD is the recommended setting. On M-PLUS systems the command may be used to set the size of the typeahead buffer as in SET TERMINAL/TYPEAHEAD:nn. If you are using a block mode terminal or have other special requirements you may wish to change the typeahead buffer setting from it's default size of 86. This option is not available on Micro/RSX. 11.2 Partitioning Memory The earliest versions of RSX required that the developer determine the amount of memory available and divide it up into fixed partitions. With the introduction of the System Controlled partition (GEN) this is no longer necessary and most commercial applications will require nothing more. There are, however, three, comparatively simple, allocation decisions remaining. Memory usage may be divided into four general classes -- three of which the developer can allocate -- and the developer should assure that the memory allocated to each class is sufficient to the task. The four classes are the 11-68 Executive, Secondary Pool, Task Space and the Disk Cache region. You may have to add memory to your system to attain acceptable performance. 11.2.1 Exec and Primary Pool The RSX executive and it's dynamic memory region (Primary Pool) are fixed in a special region during the system generation process. The size of this region is fixed and pool space is set to the highest possible value. There is little that you can do to change this allocation. 11.2.2 Secondary Pool and Extension Secondary Pool is another region of dynamic memory which is used by the executive for a variety of purposes including task header and logical name storage. The size of the Secondary Pool is established during system startup. Because Secondary Pool is used for so many different purposes it is impossible for RSX to predict a suitable allocation figure with any accuracy. You should take care to examine your application in a production environment and assure that the amount of memory allocated to Secondary Pool is adequate at peak load. The RMD memory Display will indicate current utilization and high water mark figures for Secondary Pool usage. If Secondary Pool runs low or is exhausted, performance will suffer and the system may malfunction. Excess pool allocation is wasted memory. The default size of Secondary Pool is set during system generation. You may increase the allocation by extending the pool. On Micro/RSX and Pregenned M-PLUS you extend Secondary Pool with the SECONDARY_POOL statement in SYSPARAM.DAT. On M-PLUS you must include LOA /EXP=SEC/HIGH/SIZE=nnn in your startup file. The size parameter is the extension quantity expressed in words (not bytes). 11.2.3 Task Space Task space is the area of memory in which tasks, libraries, common regions and loadable drivers reside. If there are more tasks installed on the system than will fit in this region, some of them will be moved to the checkpoint file on the disk. Checkpointing active tasks against each other is a common source of performance problems. You can use the RMD Memory display to examine your system memory requirements under load. You will need approximately 256KB for RSX plus 64KB for each interactive user plus 64KB for each background processor or batch stream. 11-69 Tasks which have memory overlays require more memory and must be included in these calculations on a case by case basis. Space for tasks is not allocated but is, rather, what remains after the executive, Secondary Pool and Cache allocations have been made. Nevertheless, the amount of memory required for tasks will affect the allocations you make for pool and cache. 11.2.4 Cache Region When you have allocated sufficient Secondary Pool and have determined how much memory you need for tasks, throw the remainder into the Cache region. Cache is a means of substituting (comparatively) cheap memory for much more expensive, high performance disks. The bigger the cache is, the better performance you will attain. On Micro/RSX and Pregenned M-PLUS you set the size of the cache region with the CACHE= statement. On M-PLUS you must insert a SET CACHE command in your startup command file. For hints on getting the most from you cache memory, see the section on caching, later in this chapter. 11.3 Disks One of the principal determinants of system performance is disk I/O. Because the movement of the disk head and the rotation of the platter are mechanical operations there are inherent delays during which no processing can occur. This condition is further aggravated on small system configurations where the number of spindles is limited and the contention for head positioning is intensified. There are a number of system components which make demands on the disk including system services, task activation and loading, overlays and, of course, reading and writing data files. Understanding each of these areas is important to understanding the overall system throughput - moving the disk head away from a data file to load an overlay segment from a task image incurs not only the delay in loading the overlay but also the delay in returning the head back to it's position over the data file. Making the most efficient use of the available disk capacity on a system requires you to have a basic understanding of the disk structure On Disk Structure Level 1 (ODS-1) and the file system (RMS). The principles described here pertain whether or not RMS is used for record management. Furthermore, this is an aspect of application development in which there is a great deal of similarity between RSX and VMS. Any improvements made to the performance of an application on RSX will also improve the performance of the application on VMS. 11-70 11.3.1 Initializing Before a disk can be used by RSX for any file structured operation it must be initialized. Initialization is the process of creating an empty ODS-1 directory structure on the disk such that user files can be created and managed. The system disk is usually initialized during installation of RSX. Any additional disks must be initialized manually, an operation which can be performed on-line. 11.3.1.1 Preparing a Disk Before a disk can be initialized it must be formatted and scanned for bad blocks. All of these operations require that you be logged in as a privileged user and that the disk drive be allocated as a private device. This will insure that no other user is granted access to the device. Use the ALLOCATE command and then mount the disk as a foreign file structure with the MOUNT/FOREIGN command. If the disk is a very old type, or if it was not manufactured by DEC, you may need to format it. Formatting is the process of writing sector header and address information on the disk surface. Use the FMT utility. Most modern disks come from the vendor already formatted. Once the disk is formatted it should be scanned for bad blocks. You can direct the BAD utility to write data patterns across the disk and record which blocks on the disk fail to return the information correctly. The location of the bad blocks is written into a special area on the disk. During initialization these blocks are collected together into a special file (BADBLK.SYS). Since they are allocated to this file they will not be used by the file processor for storing data. Newer disks, such as the RDxx series, already have the bad blocks information recorded on the disk volume. 11.3.1.2 Initializing a Volume Use the INITIALIZE command to write the skeleton directory structure to the disk. This operation creates the Master File Directory, allocates file headers and creates a storage bitmap for the disk. It also creates the bad block file and a null checkpoint file. All these files are stored in a special system directory ([0,0]) on the volume. When you initialize a volume you must supply a label by which the disk is identified during subsequent MOUNT operations. You also specify a number of parameters which will affect it's subsequent operation such as the maximum number of files and file headers, 11-71 the volume protection which determines who can access the disk as a whole, the default protection and extension size for files created on the volume. The default values for each of these parameters are usually adequate but there are occasional circumstances, possibly a requirement to store a very large number of (small) files, where you may wish to override the defaults with your own values. 11.3.2 Mounting Before a disk can be used during a session it must be MOUNTed. MOUNT is the procedure by which the disk is made known to the operating system and the parameters regarding it's use are determined. The system disk is mounted during system startup but all other disks, be they physically removable or not, must be mounted through commands. In the case of fixed disks which are used in the day to day operation of the system the mount commands are usually edited into the system startup command file. When you mount a removable disk you should supply the volume label which will be checked against the label on the disk itself. You may also supply qualifiers with the MOUNT command which determine whether the volume is to be public (available to all users), shareable (available for MOUNTing by other users), or private. Other qualifiers can be used to override the volume protection, the default file protection and extension quantity, and whether the files on the volume are to be cached in memory. Once the disk has been mounted you may create directories with the CREATE/DIRECTORY command, open and close files, read and write data. 11.3.3 Dismounting If a volume is removable, and you wish to exchange it for another, you should DISMOUNT the disk before removal. Note that you cannot dismount a disk completely if there are any files still open. This can occur with a volume mounted SHAREABLE where more than one program or user has files open. If not all the files have been closed and not all users have dismounted the disk the DISMOUNT command will only "mark" the volume for dismount. The actual dismount will not occur until the last active user dismounts the disk. Be careful if you are allowing sharing of removable disks. 11.4 Creating Directories All files are indexed in directories. A directory is a special file which contains identification information for other files. 11-72 The Master File Directory (MFD) contains a list of User File Directories (UFD) on a disk volume. A UFD contains a list of user data files. All directory files are created and maintained in a special system area ([0,0]). There is one MFD and zero or more UFD's on each disk. A directory contains a series of data records, each containing the name of a file contained within the directory and information about the location of the file header. The MFD contains a list of all the UFD's on the volume. The UFD's contain a list of the data files contained in the user directory. When a file is to be opened the directory is determined, either explicitly or through defaults, and the MFD is searched for the proper UFD entry. The UFD is then located and it, in turn, is searched for the file entry. The data in the entry is then used to locate the file header in the volume index file. Note that because the search through file entries is sequential the average time to find a file will increase with the number of entries. Collecting large numbers of files together into a directory will slow down access time considerably. Directory files have all the properties of data files in that they have protection codes, can be copied, renamed and deleted. The protection of the MFD is the volume protection, that is, it determines which users can access the disk as a whole. If a user is denied read access to the MFD then there is no way to get at any of the UFD's and the data files. The protection of a UFD determines which users can read and/or write the data files listed within that directory. Normally, the system area [0,0] is concealed and will not be listed (such as in a DIRECTORY command) along with the other directories on the disk. If you specify this area explicitly, however, you can gain access to the MFD and the UFD's as though they were data files which, in a sense, they are. An RSX expert can create various kinds of magical effects and illusions such as listing a file in multiple directories. But unless you consider yourself to be such an expert, you should leave this area and it's contents alone. 11.5 Files A file consists of a header and a collection of extents. The header is located in the volume index file ([0,0]INDEXF.SYS) and contains information identifying the file, such as the file name and type, the version, the owner ID, the creation date and the most recent revision date. The location of the header block for a file is reflected in the File ID which is a unique identifier by which the file may be located and opened without searching through the directory structures. Some RSX components manipulate files by their ID's. 11-73 The header also contains a number of extent descriptors which describe each of the segments of the disk which belong to the file. Available disk blocks are listed in the disk bitmap file ([0,0]BITMAP.SYS). As a file is extended the header is updated to include the new extents. If the extent descriptors become too numerous to fit in a single header block, an extension header is allocated. This is why the number of headers allocated for a volume (declared during initialization) must always be greater than the maximum number of files. The number and the size of the extents which comprise a file will determine the access speed. The extents may be physically adjacent on the disk or they may be quite far apart. If the extents are close together the disk heads will spend less time when moving from one to another. When a file is opened RSX will read the first few extent descriptors into a memory structure called a Window. If a request is made to read or write a logical block of the file which lies outside of the window, RSX must return to the file header (more disk head movement and further delay) to get a new set of extent descriptors (a window turn). The more extents there are in the header the more often RSX will have to disrupt file operations to obtain new window data. RSX will normally try to allocate the extents for a file as close together as possible but the pattern of disk usage may preclude any real optimization. 11.5.1 File Performance There is a tradeoff between file performance and disk space utilization, that is, it is often possible to improve performance of a file by 'wasting' a certain amount of space. There are three actions you can take. First, you can pre-allocate the file as a contiguous space. Making the file contiguous will reduce the amount of disk head travel and, equally important, will reduce the number of extents required to describe the file. If a file is contiguous there will be only a small number of extent descriptors and you may be able to eliminate window turns altogether. The disadvantage is that there may not be sufficient contiguous space on a disk and once the space is allocated to a file it cannot be used for any other purpose. Second, pre-allocating a file will improve performance when the file is first written whether or not the file is contiguous. By pre-allocating the extents you eliminate the small extend operations that would otherwise be required as the file grows. Third, you can increase the size of the extension quantity or the number of blocks that RSX will add to a file each time it is extended. It is better (and possibly wasteful) to extend the file less frequently and in larger chunks. Even if the file as a 11-74 whole is not contiguous it is nonetheless helpful if pieces of the file are, since there will be fewer extent descriptors and the blocks will be clustered. 11.5.2 File Protection RSX supports the same, multilevel file protection that is used in VMS. Potential users of a file are grouped into four categories: Owner, Group, System, and World. Membership in these groups is determined by UIC. The Owner of a file is a user with the same UIC as that listed as the file owner in the header. A group member is a user with the same group UIC code as the owner. System is the operating system itself and any user with a privileged account (a group number of 10 or less). World is everyone else. For each class of user you may specify four kinds of access. Read access allows the user to read, copy or execute the file. Write access allows the user to modify data within the file. Extend access allows the user to increase the amount of space allocated to the file. (In VMS, there is Execute access rather than Extend access; "execute" is not a concept contained in RSX.) Delete allows the user to delete the file altogether. The protection of a file is determined when the file is created or when it is set or changed with a SET PROTECTION command. A hierarchy of defaults are in effect beginning with the system default and continuing through the volume default and the user default. Only the owner of a file or a privileged user can change the protection of a file. Thus, it is possible for a non-privileged user to set the protection of a file to prevent inadvertent deletion by the system but not to prevent the system or a privileged user from changing the protection to allow subsequent deletion. Note that the protection of a Directory determines which users may access the files identifiers within the directory but not necessarily the file data. You may allow another user access to selected files within a directory or you may lock them out altogether. 11.6 Logical Names Older versions of RSX supported a very limited logical name facility which could only be used with devices. Beginning with V3.0, however, both Micro/RSX and M-PLUS support a VMS compatible logical name facility. Names may be defined in User, Group and System tables, may be nested, and may expand to either partial or full filespecs. Logical names may be established with the DEFINE command or with calls to system services. Since DEFINE does not check the syntax 11-75 of the equivalence name it is possible to use logical names to convey string information other than file specifications between users and tasks on the system. This facility is used throughout A-to-Z. The logical name translation process follows the same rules on RSX that it does on VMS. When a name is presented to the system for translation the leftmost part is examined. If the leftmost part is terminated with a colon or a space, an attempt is made to translate it and if the translation succeeds, the equivalence name is substituted and the translation is repeated with the new name. When the leftmost portion is terminated with anything except a colon or space the translation process ceases and the resultant name is returned to the caller or used as a file specification. Thus, NAME or NAME:other will result in an attempt to translate "NAME" whereas NAME.other or NAME;other will not. Logical names may be defined with embedded colons but the rules for translation become much more complicated and the beginner is advised to avoid this practice. Logical name tables are stored in the secondary pool. If you plan to make extensive use of logical names you should use the SHOW MEMORY command to see whether the pool is close to exhaustion and more space should be allocated. 11.7 Disk Caching Most applications will benefit from disk caching. Caching is actually a means by which memory may be treated as a very fast disk. Note that it is always better to eliminate the disk I/O altogether or to eliminate as many sources of I/O as possible, but once this has been accomplished you should enable caching. To make use of caching you must first establish one or more cache regions in memory and then notify the devices which are to use them. You may do both of these with the MOUNT command. More than one device may be associated with a region. When you create the cache region you should specify as much memory as you can afford to spare. Generally, the larger the region is, the more effective the cache will be, but this must be balanced against all the other demands made on the system memory. In particular, if you reduce the memory available to tasks on the system you may cause some degree of thrashing which would wipe out all the benefits of caching and then some. There is no particular advantage to allocating multiple regions unless you have divided memory into multiple partitions. When you mount a device and associated it with a cache region you may specify the types of operations that are to be cached and the 11-76 maximum size in disk blocks for each type. Generally the defaults are adequate but you may wish to increase the extent size for VIRTUAL (application data) operations if you use files with bucket sizes larger than five blocks. The extent size should be at least as large as the bucket size or caching will not take place for relative and indexed file operations. If there are a large number of interactive users it might be worth the additional expense of extra memory simply to increase the size of the disk cache. 11.8 Printers and Queues RSX offers full support for multiple Print and Batch Queues. Queues are named and can be specific to a particular device or generic for a class of devices. Included with this facility is Transparent Spooling of physical devices which permits multiple programs to access a single device simultaneously. There is also support for application specific queues. The Queue manager is also somewhat complex and can be confusing to the neophyte. Names of devices, device processors and queues can be hard to distinguish and the sequence of startup commands on M-PLUS is complex. Nonetheless, this is a powerful facility and is one of very great interest to a commercial application developer. For information about submitting jobs, see the Batch and Queue Operations manual. For information about starting and stopping the queues, see the System Management Guide. 11.8.1 Queue Mechanism A Queue is a collection of jobs waiting for processing. A queue may be specific to a device or it may be generic and will pass a job to whichever device is available. The Queue Manager (QMG) is a task which collects queue requests from PRINT and SUBMIT commands and distributes the jobs to the various queues. A Processor is a task which is assigned to a specific device. Jobs are passed from a queue to a processor for execution. A spooled device is considered to be the device itself along with the associated processor. A Batch processor is not associated with a physical device. 11.8.2 Submitting Jobs You may submit a job to a queue with either the PRINT or SUBMIT commands. Both commands allow a variety of qualifiers which control the way the job is executed. 11-77 SUBMIT is used for batch jobs. You may submit a job to the generic batch queue or to a particular queue by name. PRINT is used for print jobs and also for queueing jobs to application processors. You may submit a job to the generic print queue or to a particular queue or device by name. You may also use the PRINT command to direct a job to a particular device or processor by specifying qualifiers such as FORM or LOWERCASE. If you use such a qualifier, the job will be directed to a processor which was initialized with the proper attributes or, if none such exist, it will be held until one is created. There are also a variety of commands for controlling jobs already in a queue. You may HOLD a job and then RELEASE it at some later time. You may also DELETE a job after it has been queued. 11.8.3 Startup Starting up Print and Batch queues on Micro/RSX and Pregenned M-PLUS requires only that you edit SYSPARAM.DAT and include the proper QUEUE_MANAGER, BATCH_PROCESSORS and PRINTER statements. Getting everything going on M-PLUS is a little more difficult. A prototype QMGSTART.CMD file is provided with the system which will start up one print queue and one batch queue. If this is not sufficient for your needs you must modify the skeleton QMGSTART command file. Making such a change will be easier once you understand the general workings of the queueing mechanism. You must install and start QMG. This also initializes the generic PRINT and BATCH queues. Then install the QMG interface QMGPRT. Then, for each device or batch queue you must initialize a queue, install and initialize a processor task and assign the queue to the processor. Each device must have an associated processor which is named after the device. Both print and batch processors are provided with RSX which may be installed with the appropriate task name. Each processor must have an associated queue which has the same name as the processor. You may also initialize generic queues which are assigned to more than one processor. You may also install application processors and initialize the associated queues. Implementing such a processor is a more advanced aspect of application development on RSX and is mentioned here only to indicate that such a capability is available. 11.8.4 Shutdown 11-78 If your application or system makes frequent use of queues you should assure that system shutdowns are conducted in an orderly fashion. Information about jobs awaiting execution is maintained in a file (SP0:[1,7]QUEUE.SYS) and if the system is shutdown abruptly while processing is in progress it is possible that output data may be lost. RSX is provided with a system shutdown command file (LB:[1,2]SHUTUP.CMD) which will, among other things, assure that the system queues are shut down in such a way that they may be restarted properly when the system comes back up. You should assure that your operators use this command procedure or provide the same functionality via an application program. 11.8.5 Transparent Spooling One of the features of the RSX queue architecture is Transparent Spooling. A device which is spooled may be accessed by more than one program simultaneously. When a program Opens such a device, usually for output, the file system passes the data to the processor instead of directly to the device. The processor, which has the same name as the device, accumulates the data in a temporary file and when the program closes the device channel, the data file is queued, along with all other jobs, for the physical device. You may output a file to a spooled device by COPYing the file to the device name. The COPY command will queue the request as though you had used a PRINT command. The advantage of this is that multiple programs may perform output to a device, such as a printer, without assigning the device for exclusive use or even determining if it is in use. The disadvantage of this is that the program cannot determine whether output is going directly to a device or is being accumulated in a file. Programs which generate very large amounts of output may inadvertently fill up the disk. Another problem with transparent spooling is that programs which require specific device control, such as a word processor which wishes to stop printing at the top of a page and await a user command, cannot determine whether it actually owns the device or not. The solution for both of these problems is for the program to suspend the queueing operations temporarily, perform the operations and then restore the queue. You can do this directly from your program by spawning DCL with the following series of commands. This example assumes that the spooled device is a printer. STOP/QUEUE queuename. Following this command, jobs will no 11-79 longer be taken from the queue although they may still be added to it. STOP/PRINT processor_name/JOB_END. This command will direct the processor associated with the printer to stop at the end of the current job. The processor name is usually the same as the printer name, such as LP2. Once the current print job is complete the processor will release the printer. Then, from your program, spawn DCL to ALLOCATE the printer. This command will fail while the printer is still busy but will succeed once it is free. You may then Open the printer and be assured of full, direct control over the device. When your program is finished with the device, you should restore it to spooled status. Spawn DCL with a DEALLOCATE command to release the device. Then spawn the START/PRINT processor_name and START/QUEUE queuename commands to resume normal operations. 11-80 Chapter 12 Troubleshooting The performance of a system as a whole is dependent on the amount of resources required by an application and the efficiency with which the available resources are managed. The tuning process must begin by reducing the resources required by an application to the absolute minimum. It is always better to eliminate the need for a resource than it is to improve it's availability or performance. Once the application tuning is complete, you may begin determining that the resources available on the system are adequate to the demand and that they are being managed properly. There is little that a well tuned system can do to improve the performance of an application which is inefficient or is consuming an inordinate number of resources. On the other hand, there are some ways that a poorly tuned system can slow things down. If the system is not managing available resources well then performance will suffer. The tuning process is one of identifying resources which are insufficient or which are being managed poorly. 12.1 Before You Start A common cause of system performance problems is insufficient hardware. There may not be enough memory. The processor or disks may be too slow. Another possibility is that error rates on malfunctioning devices may be impeding the application by causing excessive numbers of retries. This is not to say that you should blame all your problems on hardware but rather that you should not overlook the obvious. Once you are satisfied that the hardware is sufficient to the task and is working properly, you should begin examining the way in which it is being used. 12.2 Diagnostic Tools There are three tools provided with RSX which can be used to examine a system under load and which may point out which resources are in short supply or where imbalances have occurred. The use of these tools is not something that you would want to leave to an inexperienced user, however, and may require that you be on site. 12-81 12.2.1 Resource Monitor Display (RMD) RMD is the single most useful tool for diagnosing system problems and observing system operation in general. It is invoked from DCL with the SHOW MEMORY command and has a variety of displays which provide information about the behavior of the system and the usage of various resources. If you are having a performance problem, begin your diagnosis with RMD. RMD has a number of different screens which present various sets of information about the system. You can switch from one screen to another as well as changing the rate at which the screens are updated with new information. You can even invoke RMD over a dial-up line to investigate problems at a remote site. The Memory display is the first to appear and you may switch from one display to another on the fly. See the chapter on RMD in the System Management Guide for more information. 12.2.1.1 Memory The Memory display is the most frequently used and contains a great deal of information. The display portrays system memory and shows all the tasks and regions in their approximate locations. You can watch programs move in and out as terminals proceed through an application. The display also shows the current and "high water" usage statistics for primary and secondary pool, the amount of free space on each of up to four disks, the name of the task currently executing, and a summary of how many tasks are in memory versus how many are checkpointed to the disk. The display also shows the latest sequence number being used by the Error Logger. If you have enabled Error Logging on your system and you see the sequence number changing frequently it may indicate that your hardware is marginal. 12.2.1.2 Active Task Display The Active Task display is simply a list of all the tasks which are active on the system. Each task is listed by name along with it's size, priority, and status information. This is, perhaps, less useful to a commercial programmer than the others. 12.2.1.3 Task Display The Task display shows the contents of the header for a particular task on the system. The header is a region of memory that RSX uses to maintain task context and contains various information such as hardware register contents, priority and 12-82 status indicators. You can use the task display to examine the operation of any program on the system in some detail. You can watch the program open and close files, enter and leave certain processing states and change size. In particular, if you observe the list of files in use by the program to be changing rapidly you may have a problem with too frequent file opens. 12.2.1.4 I/O Counts Display The I/O Counts display shows the I/O and error logging counts for up to six mass storage devices. You can use this display to examine the distribution and level of disk activity on your system. You can also check for errors accumulating on a device. If you notice that a particular device is showing a disproportionate amount of activity you may need to redistribute the load by moving programs or files from one device to another. You should also study the individual activity level for each drive. If a particular disk is showing anything over 20 I/O requests per second, the device may be saturated. 12.2.1.5 System Statistics The System Statistics display shows you a variety of general information about the operation of your system. The information ranges from the total number of user logins to the average number of QIO's issued per second. You can obtain information about CPU utilization percentage of memory and checkpoint space being used, and the rate at which tasks are being checkpointed. You should pay particular attention to memory utilization and checkpoint frequency. It is acceptable and appropriate for an inactive or suspended task to be checkpointed to disk, such as a menu driver which is waiting for a selected application to complete. This will free up system memory for tasks engaged in active processing. However, if the display shows that tasks are being checkpointed on a more than occasional basis, it may be that active tasks are being checkpointed against each other. This condition, called thrashing, is the worst performance problem you can have on RSX and must be corrected by decreasing the number of active tasks (users) or increasing the amount of memory. 12.2.1.6 Cache Region Display The Cache display shows general information about the disk caching on your system. By comparing the total number of reads and writes to the number of cache hits you can determine the 12-83 effectiveness of the cache. If the hit rate is below 50% then you are not getting maximum benefit from the cache and you should adjust the cache parameters for better performance. You should also note the Cache Used statistics. If the percentage of a region in use is very high or very low you should adjust the size of the region accordingly. 12.2.1.7 Detailed Cache Statistics The Detailed Cache statistics display shows the finer details of usage of a particular cache region. This display contains more information than is usually required by commercial developers although it may be of use by an expert for very fine system analysis and tuning. 12.2.2 Error Logging RSX supports an extensive error logging facility which can be used for detecting and recording errors occurring on the devices and memory in your system. The error information is detailed and is collected for both soft (recoverable) and hard (non-recoverable) errors. The information collected will assist service personnel in diagnosing and correcting hardware problems. Intermittent errors may affect performance of the system. If you are experiencing a performance problem you may use error logging to either identify a specific problem or to eliminate hardware errors as the source of your problem. You must explicitly enable error logging by toggling the ERROR_LOGGING statement in SYSPARAM.DAT on Micro/RSX and the Pregenned M-PLUS, or by including the appropriate ELI statements in your STARTUP.CMD file for M-PLUS. See the Error Logging Manual for further details. The error logging facility consists of four components. The logging task itself, ERRLOG, accumulates error log packets, generated by the executive when an error occurs, and collects them in a log file, usually LB:[1,6]LOG.ERR. The user interface (ELI) sends command packets to ERRLOG and is used to start, stop and control logging activity. The reporting facility (RPT) is used to analyze the information in the log file and format it for use. The fourth component (CFL) is used for modifying the logging facility, such as when a foreign device is added to the system. It's use is not normally required. You may use error logging for detailed diagnosis or, more simply, as an early warning mechanism. Activity in the ERRSEQ field in the RMD Memory display will indicate that errors are occurring. Note that this field will not be updated unless logging is explicitly enabled as described above. 12-84 Note also that if you have enabled logging, and errors are occurring, there may be considerable growth in the log file. You should check the size of this file from time to time to assure that it isn't gobbling up your disk. 12.2.3 Resource Accounting Resource Accounting provides a record of system usage. Data is gathered for individual users and for the system as a whole. You can use this accounting as part of your customer billing operation but you can also use it to evaluate the operation of your system. You can direct Resource Accounting to collect information about disk activity and throughput, about the usage patterns of individual users of your system or about particular tasks on the system which are suspected of heavy resource usage. Resource Accounting is automatically enabled on Micro/RSX and Pregenned M-PLUS. On M-PLUS you must include the appropriate START/ACCOUNTING statements in your STARTUP.CMD. See the Resource Accounting chapter in the RSX System Management Guide for further details. Accounting works much like Error Logging, that is, packets of information are generated by the executive and are collected in a file by the accounting facility. You can enable and disable accounting with DCL commands in order to collect information only during times of special interest to you. Once some data has been collected you may use the SHOW ACCOUNTING command to examine and format the information. You may specify that only statistics for a particular terminal or task be reported. Normally, Resource Accounting only collects information about the system as a whole. You may optionally direct that statistics be collected on each task. This may be useful in Before/After comparisons of application modules. To enable task accounting you must add the TASK:YES parameter to the START/ACCOUNTING statement. If you are using accounting you should take care that the accounting file does not use too much space on your disk. When accounting statistics for everything on the system are gathered, the data can pile up pretty quickly. This is particularly true if you are collecting task statistics. The default accounting file is LB:[1,6]ACNTRN.SYS. 12.3 What to Look For There are a number of symptoms which you should be looking for if you suspect that your system is performing below it's capability. 12-85 Many of these conditions can be detected by using RMD and interpreting the data in light of your knowledge of the architecture of the application. Once you have identified a problem area you will have to take some corrective measure. Generally the nature of the symptom will suggest one or more actions to take. 12.3.1 Disk Saturation Probably the most common problem encountered on a small system is saturation of the disks. Configurations are limited by price sensitivity to small numbers of drives and the disk head becomes a potential bottleneck. Use the RMD I/O display to examine the QIO rate for each disk. If you see that the rate is close to the maximum as determined by average access times for the drive, then you have identified a bottleneck. More than 20 operations per second for a particular disk drive usually indicates saturation. 12.3.2 Insufficient Memory Use the System Statistics display of RMD to determine the percentage utilization of memory. Memory on the system is specifically allocated for the system executive and the various common, secondary pool and cache regions. Whatever memory remains is available for tasks. If the percentage utilization of memory is very high there is the possibility that there are more active tasks than can fit and the executive will begin freeing up memory by moving otherwise runnable tasks to the checkpoint file. This condition is called thrashing and it is the cause of extreme performance degradation. You must consider the different demands that are being made on system memory. There is little you can do about the size of the executive and the common regions such as RMS and the language processors. Secondary pool usage will vary from time to time but a maximum usage can be established over time and any memory allocated beyond the maximum is wasted. Tasks which have been suspended until some external process occurs may be safely checkpointed and require no memory until they become active once again. Most applications are designed with one interactive task for each interactive user with perhaps a small number of background or batch processes running concurrently. The architecture of the PDP-11 generally limits the memory requirements of disk overlaid tasks to 64KB for each user or process. Tasks which are memory overlaid must be evaluated individually. It is usually 12-86 sufficient, therefore, to allocate about 64KB for each interactive user and for each background process. All remaining memory may be allocated to the disk cache region. 12.3.3 Unbalanced Load While you are in the I/O display of RMD notice whether the disk related QIO's are evenly distributed across all the drives. If one drive shows a much greater level of activity than the others, you should investigate the cause. There are many system operations which require disk I/O and some of them, such as program overlays or implicit spooling, may have escaped your notice. You may be able to even out the load by simply moving commonly used programs or libraries from one disk to another. 12.3.4 Insufficient Pool Use the Memory display of RMD to assure that there is sufficient free space in both the Primary and Secondary Pools. Primary pool space is fixed during SYSGEN and is generally sufficient. Secondary Pool, however, is used for a wide variety of purposes and it may be running low. Note that the percentage figures calculated for secondary pool may not be correct if the pool has been extended. If you suspect that the pool may be running short, increase the allocation. 12.3.5 File Contention File or record contention is sometimes difficult to detect and requires a knowledge of the underlying application. Excessive contention can occur if the application has been converted from CTS-300 or VMS without regard to the differences in the way RMS treats access combinations on shared files. 12.3.6 Low Cache Hit Rates Use the RMD Cache display to assure that the existing cache regions are showing high hit rates and are being used to about 80% of their capacity. Cache regions should be allocated with good measure but not to the extent of reducing memory available to other system facilities. If hit rates are low you should find out why and take whatever corrective actions are necessary. It may be that you have forgotten to enable caching for certain disks or have chosen the wrong combination of operation types. The size of the region may be too small to be effective. If the percent usage of a region is high but the hit rate is low, try increasing the size of the region. 12-87 It may also be that the default extent size for a particular I/O operation type is too small. I/O requests which are larger than the extent size will never be cached. This condition is reported on the Detailed Cache Statistics display of RMD. If any of your high activity files have bucket sizes greater than five blocks you should increase the extent size for VIRTUAL operations to equal that of your bucket sizes. It may also prove that program overlays show a high Extent Too Big rate requiring an increase in the extent size for OVERLAY operations. 12.3.7 Insufficient Checkpoint Space Use the RMD System Statistics display to assure that there is always space available in the system checkpoint files. If no free checkpoint space can be found the system will lock up until enough tasks exit to free up the required space. 12.3.8 Excessive File Opens Use the RMD Task display to examine the major components of your application while the system is under load. The display will show which files are open and you may discover that the list of files changes frequently. This means that files are being opened and closed, and if these operations are occurring often it slows down other disk processing. 12.3.9 Overloaded Directories Use the DIRECTORY command to look at the distribution of files across your directories and disks. The more files there are in a particular directory the more expensive file opens become. If you have one directory with a large number of files, consider separating the files into multiple directories. Better yet, add a disk drive to the system and redistribute the users and files accordingly. 12.3.10 Jobs at Wrong Priority You should check to see that all the tasks on the system are running at the correct priority. If you have inadvertently set the priority of a non-critical task higher than that of more important programs, performance will suffer. You may be able to increase the responsiveness of terminal interactive tasks by increasing their priority somewhat. 12.3.11 Misuse of Batch The scheduling of batch processing on RSX is extremely flexible. 12-88 You may be able to achieve considerable performance gains by scheduling non-critical batch processing for off hours. You may also be able to improve performance by using multiple batch queues. You can use the SHOW QUEUE command to examine the condition of the queues on your system. 12-89 Hitchhiker's Guide to RSX To get an updated copy of this document, when it becomes available, simply return this sheet with the requested information to: A-to-Z Base Product Marketing MKO1-2/E25 Digital Equipment Corporation Continental Boulevard Merrimack, New Hampshire 03054 Name: ____________________________________________ Title: ____________________________________________ Company:____________________________________________ Street: ____________________________________________ City: ____________________________________________ State: ____________________________________________ ZIP: ____________________________________________ 12-90