HIGH SPEED INTERPROCESSOR DATA LINKS USING THE DR11-W P.Heinicke, J. Biel(*), D. Burch, R. Dosen(**), M. Pyatetsky, D. Ritchie, V. White Fermi National Accelerator Laboratory Batavia, Il 60510 ABSTRACT ________ At Fermilab, DR11-W's have been used as high speed data links to interconnect PDP-11's (under both RT-11 and RSX11-M) in data acquisition applications. Using this hardware, several processors can be used to provide distributed data collection, data monitoring and control for physics experiments. This paper discusses the motivation for such a project and for the choice of the DR11-W links. It describes the data link protocol which enables the DR11-W to be used, the philosophy of general process-to-process communication, system independent FORTRAN callable subroutines for inter-process communication and the interleaved dedicated use of the link for extra-high-speed data transfer. The data link protocol is implemented using a device driver on each system. Some performance figures are given. 1.0 INTRODUCTION ____________ The Fermi National Accelerator Laboratory (Fermilab) is a national laboratory dedicated to pure research in the field of high energy physics. Research is performed by a large number of experimental groups, each using a dedicated minicomputer for data acquisition. Many experiments have similar hardware configurations and software needs. Using PDP-11s, some twenty to thirty experiments acquire data from external hardware and log it to magnetic tape. Often, online analysis of the data and diagnostic checks of the hardware are required to ensure correct operation of the experiment. Ninety percent of the systems use RT-11, while the rest use RSX-11M. PDP11's in experiments have proved adequate for over ten years, but upgrades in the accelerator will require new capabilities and performance. To obtain this increased capability we chose to connect additional machines to the existing PDP-11's. Figure 1 shows some possible configurations. 2.0 ADVANTAGES OF CONNECTED MACHINES __________ __ _________ ________ Connecting several machines provides us with quite a few benefits: 1. Allows additional analysis capability (PDP-11 or VAX) while making use of existing software. 2. Allows parallel acquisition of data by more than one computer. 3. Preserves simplicity in the data acquisition system where interrupt response times of the order of 50 microseconds are often crucial. 4. Allows the use of different operating systems for analysis and data acquisition e.g. RT-11 for data acquisition and RSX-11 for analysis. 5. Allows flexible configuration of multiple processors to meet a range of experimental needs. Tightly coupled systems with shared memory regions were considered, but rejected because of their inflexibility and the lack of suitable hardware. DECnet was also considered and rejected because of its large requirements on address space and its time overheads. 3.0 LINK HARDWARE: THE DR-11W ____ _________ ___ ______ As a connection method, we chose pairs of DR-11W's to provide a high speed parallel link. The DR-11W is a DMA controller capable of sustaining DMA transfers under program control at a rate of 333 kW/sec (non-burst mode) and 500 kW/sec (burst mode) between devices up to 50 feet apart. The DR-11W is relatively inexpensive. It is DEC supported, second sourced and readily available. It does have several disadvantages. It has no data parity but in our experience this has not been a problem. Further, we have had to expend considerable software effort to use the device as a general purpose link: the device has only one interrupt vector, transfers proceed in only one direction at a time, and it has a few eccentricities that will cause it to get "hung" unless it is treated very carefully. 4.0 NETWORK ARCHITECTURE _______ ____________ Our network architecture is designed for typical configurations such as shown in Figure 1. We adopted a layered approach with the following ground rules in mind: 1. point-to-point connections only, 2. bi-directional use of the link, 3. direct transfer of data to a program, without intermediate buffering, for maximum speed, 4. interchangeability of processors and operating systems at each node, 5. capability of switching to alternate link hardware with minimal program change. 4.1 Link Layer ____ _____ The link layer executes a series of transactions with its counterpart in the other machine. This protocol accomplishes the sending of a single block of data (a packet) or a single byte of control information (a signal) over the link. In additional it guarantees fair bi-directional use of the link. 4.2 Logical Link Layer _______ ____ _____ The logical link layer, which is minimal in this design, uses a system-wide identifier, termed a "destination address" (destination address), to determine the destination of a message (which may be a packet or a signal). Each packet/signal of data has to be given a code which describes its intended destination. Any program may decide to "listen" for packets of a particular type and thereby declare itself the owner of that particular destination address. For a particular physical link and within a single processor, no two programs may own the same destination address. There is no allocation method for destination address's except coordinated system design. This philosophy allows the sender to ignore the way in which the packet is handled on the remote system. When the remote system is RT, receipt of different destination address's may be handled by different sections of one program. In RSX, this may also be the case; alternatively, receipt may be handled by several different tasks. 4.3 Application Layer ___________ _____ Programs wishing to exchange messages must develop their own higher-level protocols. These protocols must include the destination addresss necessary to reply to service requests if that is desired. 5.0 IMPLEMENTATION OF THE NETWORK ______________ __ ___ _______ 5.1 Drivers _______ For both RT-11 and RSX11-M, we have chosen to implement the link and logical link layer using device drivers/handlers. For the details of the implementations, see the associated papers (1),(2). For RT-11, this enables us to use a Fermilab feature which allows SJ programs to access upper memory. In RSX, we have allowed the possibility for a privileged task to disengage the driver in order to gain control of the DR-11W registers for highly time critical applications. 5.2 FORTRAN Callable Support Routines _______ ________ _______ ________ We have provided a FORTRAN callable package of routines (CDPACK) to interface to the device drivers described above, in a uniform, convenient, and operating system independent way. These routines allow the same FORTRAN program under RT or RSX (and soon VMS) to perform the following driver calls: 1. Declare ownership of a destination address, 2. Send/Receive a packet or signal with a particular packet type, 3. Get operating system independent error and status information 4. Perform no-wait I/O in a simple manner The code was written in MACRO-11 and contains conditionals to allow assembly for RSX and RT-11. Much of the bookkeeping functions are implemented in common code. 5.3 Application Programs ___________ ________ One of the first application programs to use the connected machines software is RTMULTI (3). This is a large (24,000 lines) data acquisition, analysis, and run control package. In this application, data buffers may be sent across the link to an RTMULTI system for additional analysis or to an RSX system where several tasks may examine the data. 6.0 LINK LEVEL PROTOCOL IN DETAIL ____ _____ ________ __ ______ A complete discussion of the protocol is more involved than can be dealt with here. Interested parties are referred to the Fall 1982 DECUS tape code. We will discuss characteristics of the device and the way it was handled, the format of the control and status words exchanged, and the proper handling of the case when interrupts overlay each other. Each end of the link is a finite state automaton with thirteen possible states: -4 Link Down -3 Waiting for Start Retry -2 Starting -1 Waiting for protocol start 0 Idle 1 Requesting link 2 Waiting for destination address 3 Waiting for destination address ACK 4 Waiting for WC or Signal 5 Waiting for Receiver WC or WC NACK 6 Waiting for end-of-message status 7 Transmitting 8 Receiving The state transitions are initiated by either a DR11W interrupt from the other side, an end of DMA DR11-W interrupt, or a time-out on waiting for an interrupt. The system may be loosely divided into three cases. These are: Link Down, Transmitting and Receiving. Figures 2-4 shows the state diagrams for the protocol. The DR11W provides for one end of the link to interrupt the partner computer by manipulation of the bits in the CSR of the device. An interrupt will occur provided the partner DR11W is interrupt enabled, and does not already have a pending interrupt. Single words of control and status information may be passed from one end of the link to the other by means of DR11W registers provided for the purpose of single word transfers. This is the mechanism by which state change information is communicated from one side of the link to the other. The following sorts of things are indicated: - Requests to the partner - ACKS and NACKS - End-of-message status returns - Word counts - Single byte messages (signals) See Table 1 for a more detailed presentation of the data format. The format of the control and status information word is designed to detect single bit errors and to allow more than one state change indication to be passed in a single control and status information word. This format is used to ensure fairness of link access since it allows a computer indicating the receipt of a message to immediately make a bid for link ownership if it wishes to send. It is also used to insure the correct handling of situations in the state diagram where one computer may put a second transition indication into the control and status word before the other side has handled a previous indication. (Without this feature interrupts would be lost.) The protocol rules demand that the software "or"s together successive indications, whenever such a situation could arise. In order to handle possible link ownership bid collisions, a primary/secondary relationship between the ends of the link is established at link startup time. If such a collision occurs, the primary machine gains link ownership. Collisions on startup are handled by retry after random delay. The side which successfully initiates startup (or restart) of the link becomes the primary machine. 7.0 PERFORMANCE AND SIZE ___________ ___ ____ Using the RSX-11M driver, we have measured the per-word DMA transfer time to be 3 microseconds in non-burst mode. The average RSX-11M per-packet overhead was measured to be 5.7 - 5.8 milliseconds (one-way 11/50 to 11/34 transfers). The figures are obviously highly dependent on processor type and memory speeds. These timings are better than those that can be obtained writing to a 6250 BPI, 45 IPS magnetic tape drive. The RT-11 average per message delay was measured to be 7 milliseconds in loopback mode. The RSX driver is 1800 decimal words. The RT-11 driver is 2800 decimal words; effort is underway to reduce this. REFERENCES __________ 1. M.Pyatetsky, P.Heinicke, D.Ritchie, V.White, 1982 "Using the DR-11W DMA Device for Interprocessor Communications in RT-11.", 1982 Fall ____ ____ DECUS U.S. Symposium, Anaheim, Ca _____ ____ _________ 2. D.Burch, V.White, R.Dosen, 1982 "An RSX-11M Device Driver Implementing A Network Protocol For The DR11-W", 1982 ____ Fall DECUS U.S. Symposium, Anaheim, Ca ____ _____ ____ _________ 3. J.Bartlett, J.Biel, D.Curtis, R.Dosen, T.Lagerlund, D.Ritchie, L.Taff, 1979 "RT/RSX MULTI: Packages For Data Acquisition And Analysis In High Energy Physics, IEEE Transactions On Nuclear ____ ____________ __ _______ Science, Vol. NS-26, No 4, August 1979 _______ ____ * Present address of J. Biel: CommaVid, Inc., Aurora, Il. ** Present address of R. Dosen: Hughes Aircraft, Aurora, Co.