HIGH SPEED INTERPROCESSOR DATA LINKS USING THE DR11-W P.Heinicke, J. Biel(*), D. Burch, R. Dosen(**), M. Pyatetsky, D. Ritchie, V. White Fermi National Accelerator Laboratory Batavia, Il 60510 ABSTRACT ________ At Fermilab, DR11-W's have been used as high speed data links to interconnect PDP-11's (under both RT-11 and RSX11-M) in data acquisition applications. Using this hardware, several processors can be used to provide distributed data collection, data monitoring and control for physics experiments. This paper discusses the motivation for such a project and for the choice of the DR11-W links. It describes the data link protocol which enables the DR11-W to be used, the philosophy of general process-to-process communication, system independent FORTRAN callable subroutines for inter-process communication and the interleaved dedicated use of the link for extra-high-speed data transfer. The data link protocol is implemented using a device driver on each system. Some performance figures are given. 1.0 INTRODUCTION ____________ The Fermi National Accelerator Laboratory (Fermilab) is a national laboratory dedicated to pure research in the field of high energy physics. Research is performed by a large number of experimental groups, each using a dedicated minicomputer for data acquisition. Many experiments have similar hardware configurations and software needs. Using PDP-11s, some twenty to thirty experiments acquire data from external hardware and log it to magnetic tape. Often, online analysis of the data and diagnostic checks of the hardware are required to ensure correct operation of the experiment. Ninety percent of the systems use RT-11, while the rest use RSX-11M. PDP11's in experiments have proved adequate for over ten years, but upgrades in the accelerator will require new capabilities and performance. To obtain this increased capability we chose to connect additional machines to the existing PDP-11's. Figure 1 shows some possible configurations. 2.0 ADVANTAGES OF CONNECTED MACHINES __________ __ _________ ________ Connecting several machines provides us with quite a few benefits: 1. Allows additional analysis capability (PDP-11 or VAX) while making use of existing software. 2. Allows parallel acquisition of data by more than one computer. 3. Preserves simplicity in the data acquisition system where interrupt response times of the order of 50 microseconds are often crucial. 4. Allows the use of different operating systems for analysis and data acquisition e.g. RT-11 for data acquisition and RSX-11 for analysis. 5. Allows flexible configuration of multiple processors to meet a range of experimental needs. Tightly coupled systems with shared memory regions were considered, but rejected because of their inflexibility and the lack of suitable hardware. DECnet was also considered and rejected because of its large requirements on address space and its time overheads. 3.0 LINK HARDWARE: THE DR-11W ____ _________ ___ ______ As a connection method, we chose pairs of DR-11W's to provide a high speed parallel link. The DR-11W is a DMA controller capable of sustaining DMA transfers under program control at a rate of 333 kW/sec (non-burst mode) and 500 kW/sec (burst mode) between devices up to 50 feet apart. The DR-11W is relatively inexpensive. It is DEC supported, second sourced and readily available. It does have several disadvantages. It has no data parity but in our experience this has not been a problem. Further, we have had to expend considerable software effort to use the device as a general purpose link: the device has only one interrupt vector, transfers proceed in only one direction at a time, and it has a few eccentricities that will cause it to get "hung" unless it is treated very carefully. 4.0 NETWORK ARCHITECTURE _______ ____________ Our network architecture is designed for typical configurations such as shown in Figure 1. We adopted a layered approach with the following ground rules in mind: 1. point-to-point connections only, 2. bi-directional use of the link, 3. direct transfer of data to a program, without intermediate buffering, for maximum speed, 4. interchangeability of processors and operating systems at each node, 5. capability of switching to alternate link hardware with minimal program change. 4.1 Link Layer ____ _____ The link layer executes a series of transactions with its counterpart in the other machine. This protocol accomplishes the sending of a single block of data (a packet) or a single byte of control information (a signal) over the link. In additional it guarantees fair bi-directional use of the link. 4.2 Logical Link Layer _______ ____ _____ The logical link layer, which is minimal in this design, uses a system-wide identifier, termed a "Packet Type Code" (PTC), to determine the destination of a message (which may be a packet or a signal). Each packet/signal of data has to be given a code which describes its intended destination. Any program may decide to "listen" for packets of a particular type and thereby declare itself the owner of that particular PTC. For a particular physical link and within a single processor, no two programs may own the same packet type code. There is no allocation method for PTC's except coordinated system design. This philosophy allows the receipt of packets to be performed by a single program under the RT-11 SJ monitor, but to be transmitted to one or more tasks under RSX or processes under VMS, transparently to the sender. 4.3 Application Layer ___________ _____ Programs wishing to exchange messages must develop their own higher-level protocols. These protocols must include the packet type codes necessary to reply to service requests if that is desired. 5.0 IMPLEMENTATION OF THE NETWORK ______________ __ ___ _______ 5.1 Drivers _______ For both RT-11 and RSX11-M, we have chosen to implement the link and logical link layer using device drivers/handlers. For the details of the implementations, see the associated papers (1),(2). For RT-11, this enables us to use a Fermilab feature which allows SJ programs to access upper memory. In RSX, we have allowed the possibility for a privileged task to disengage the driver in order to gain control of the DR-11W registers for highly time critical applications. 5.2 FORTRAN Callable Support Routines _______ ________ _______ ________ We have provided a FORTRAN callable package of routines (CDPACK) to interface to the device drivers described above, in a uniform, convenient, and operating system independent way. These routines allow the same FORTRAN program under RT or RSX to perform the following driver calls: 1. Declare ownership of a PTC, 2. Send/Receive a packet or signal with a particular packet type, 3. Get operating system independent error and status information 4. Perform No-wait I/O in a simple manner The code was written in MACRO-11 and contains conditionals to allow assembly for RSX and RT-11. Much of the bookkeeping functions are implemented in common code. 5.3 Application Programs ___________ ________ One of the first application programs to use the connected machines software is RTMULTI (3). This is a large (24,000 lines) data acquisition, analysis, and run control package. In this application, data buffers may be sent across the link to an RTMULTI system for additional analysis or to an RSX system where several tasks may examine the data. 6.0 LINK LEVEL PROTOCOL IN DETAIL ____ _____ ________ __ ______ A complete discussion of the protocol is more involved than can be dealt with here. Interested parties are referred to the Fall 1982 DECUS tape code. We will discuss characteristics of the device and the way it was handled, the format of the control and status words transmitted, and the proper handling of the case when interrupts overlay each other. Each end of the link is a finite state automaton with thirteen possible states: -4 Link Down -3 Waiting for Start Retry -2 Starting -1 Waiting for protocol start 0 IDLE 1 Requesting link 2 Waiting for PTC 3 Waiting for PTC ACK 4 Waiting for WC or Signal 5 Waiting for Receiver WC or WC NACK 6 Waiting for end-of-message status 7 Transmitting 8 Receiving The state transitions are initiated by either a DR11W interrupt from the other side, an end of DMA DR11-W interrupt, or a time-out on waiting for an interrupt. The system may be loosely divided into three cases. These are: Link Down, Transmitting and Receiving. Figures 2-4 shows the state diagrams for the protocol. The DR11W provides for one end of the link to interrupt the partner computer by manipulation of the bits in the CSR of the device. An interrupt will occur provided the partner DR11W is interrupt enabled, and does not already have a pending interrupt. Single words of control and status information may be passed from one end of the link to the other by means of DR11W registers provided for the purpose of single word transfers. This is the mechanism by which state change information is communicated from one side of the link to the other. The following sorts of things are indicated: - Requests to the partner - ACKS and NACKS - End-of-message status returns - Word counts - Single byte messages (signals) See Table 1 for a more detailed presentation of the data format. The format of the control and status information word is designed to detect single bit errors and to allow more than one state change indication to be passed in a single control and status information word. This format is used to ensure fairness of link access since it allows a computer indicating the receipt of a message to immediately make a bid for link ownership if it wishes to send. It is also used to insure the correct handling of situations in the state diagram where one computer may put a second transition indication into the control and status word before the other side has handled a previous indication. (Without this feature interrupts would be lost.) The protocol rules demand that the software "or"s together successive indications, whenever such a situation could arise. In order to handle possible link ownership bid collisions, a primary/secondary relationship between the ends of the link is established at link startup time. If such a collision occurs, the primary machine gains link ownership. Collisions on startup are handled by retry after random delay. The side which successfully initiates startup (or restart) of the link becomes the primary machine. 7.0 PERFORMANCE AND SIZE ___________ ___ ____ Using the RSX-11M driver, we have measured the per-word DMA transfer time to be 3 microseconds in non-burst mode. The average RSX-11M per-packet overhead was measured to be 5.7 - 5.8 milliseconds (one-way 11/50 to 11/34 transfers). The figures are obviously highly dependent on processor type and memory speeds. These timings are better than those that can be obtained writing to a 6250 BPI, 45 IPS magnetic tape drive. The RT-11 average per message delay was measured to be 7 milliseconds in loopback mode. The RSX driver is 1800 decimal words. The RT-11 driver is 2800 decimal words; effort is underway to reduce this. REFERENCES __________ 1. M.Pyatetsky, P.Heinicke, D.Ritchie, V.White, 1982 "Using the DR-11W DMA Device for Interprocessor Communications", 1982 Fall DECUS U.S. ____ ____ _____ ____ Symposium, Anaheim, Ca _________ 2. D.Burch, V.White, R.Dosen, 1982 "An RSX-11M Device Driver Implementing A Network Protocol For The DR11-W", 1982 ____ Fall DECUS U.S. Symposium, Anaheim, Ca ____ _____ ____ _________ 3. J.Bartlett, J.Biel, D.Curtis, R.Dosen, T.Lagerlund, D.Ritchie, L.Taff, 1979 "RT/RSX MULTI: Packages For Data Acquisition And Analysis In High Energy Physics, IEEE Transactions On Nuclear ____ ____________ __ _______ Science, Vol. NS-26, No 4, August 1979 _______ ___ * Present address of J. Biel: CommaVid, Inc., Aurora, Il. ** Present address of R. Dosen: Hughes Aircraft, Aurora, Co.