DS-83 Four RTMULTI Connected Machine Applications V. White, P. Heinicke, and D. Ritchie March 8, 1983 The design of RTMULTI in four different connected machine applications is given. The design is based on an informal note of V. White and addresses the four RTMULTI connected machine applications expected to be required initially: sending unselected events for analysis, sending selected events for analysis, sending all events one-by-one for possible concatenation and logging, and receiving events for analysis. 0.1 File -- CL7VAX::[RITCHIE.CNNMCH]RTMLCD.RNO 0.2 18 July 1982 -- Original. 0.3 26 July 1982 -- Revisions - Significant Changes Have Change Bars. With respect to the 23 July edition given to Vicky, minor changes have been made in the description of the existing data acquisition system. A major revision has been made in the description of the LINK command, however. 27 July -- I have moved the figures to the end of the document and changed LKPROC to LIPROC. I have also made sure that the descriptions of the new additions say "event" where they mean event and "buffer" where they mean buffer. LKDON was changed to LKDUN to parallel FULDUN. FOUR RTMULTI CONNECTED MACHINE APPLICATIONS DS-83 (DRAFT) Page 2 INTRODUCTION 1.0 INTRODUCTION We present in as concrete detail as practical the design of four RTMULTI applications. One possible application, that of receiving events for concatenation and logging, is purposely not addressed--receipt of events for concatenation and logging is expected to be handled (at least initially if not always) by an RSX-11M machine. The applications are presented together because their designs are similar to one another. The applications are: 1. an RTMULTI application sending unselected events to a remote analysis machine (which may be either an RT or an RSX system), 2. an RTMULTI application sending selected events to a remote analysis machine as above (the selection is expected to have been done by user additions to the standard MULTI hooks, such as EUSERA, EUSERB, etc., which then call routines specified by this design to initiate transmission of the event to the remote analysis machine), 3. an RTMULTI application sending all events one-by-one to a remote application doing concatenation (possibly) and logging (definitely), and 4. an RTMULTI application receiving events to be analyzed (this is the partner in the RT case of the first two applications). In what follows, we present the goals and ground rules of these application designs. We next present several pictures to show the data flow in the various cases along with a discussion of important points to note in these data flow pictures. We follow this discussion with a presentation of the processes to be coded and the particular RTMULTI modules in which the processes will reside. 2.0 GOALS AND GROUND RULES OF THE DESIGNS We have agreed upon the following goals and ground rules in the design of these applications: 1. All link transfers are to be done using the RT-11 CD driver. This is certainly the case initially. After the applications are completed and timings done, we may wish to consider going direct to the DR-11W registers (as is being done in certain of the RSX cases) for one or more of the time-critical applications (presumably, the concatenation and logging application is the main candidate). FOUR RTMULTI CONNECTED MACHINE APPLICATIONS DS-83 (DRAFT) Page 3 GOALS AND GROUND RULES OF THE DESIGNS 2. The CDPACK routines are to be used wherever possible. 3. The changes to RTMULTI are to be made in the style of an add-on with the usual ".TXT" source files and ".BAT" batch files (the batch files should begin the addition from the RTMULTI V1K'' with LIMIT Add-On and bulk memory (?) base level. That is, the link application is to be effected by calling link-specific subroutines from the usual standard hooks, such as MULPOL, IUSER, BEUSER, BERUN, etc. However, the changes should be made in such a way that the add-on can be incorporated into the standard base line system. (This is analogous to the way that the LIMIT Add-on has now effectively become part of the standard base line system due to its mandatory nature for efficiency of data acquistion.) In the future when the link add-on becomes usual to have in the system, we plan to be able to "subtract-off" the link for those users who do not wish it in their systems by: 1. some limited amount of re-assembly of MACRO routines with a different setting of an assembly parameter, 2. re-naming of files to substitute dummy routines for the link routines, 4. Initially at least, much of the link functionality will be implemented by routines called from MULPOL, the MULTI polling appendage. It is here that checks for the receipt of requests for events for analysis will be made and here that checks on errors and reporting of such errors will be done. 3.0 THE EXISTING RTMULTI SYSTEM We describe the existing RTMULTI system from a buffer and data flow point of view (see Figure 1). 3.1 Initialization Phase Using the IGETSP function, IUSER allocates buffers and buffer descriptors. This is done according to what has been specified in the CAMAC.DEF BUFFER statement. The buffers are generally allocated out of both low and high memory. FOUR RTMULTI CONNECTED MACHINE APPLICATIONS DS-83 (DRAFT) Page 4 THE EXISTING RTMULTI SYSTEM In the case of low memory buffers (and the buffer descriptors), the allocation comes from the area between the end of the user's code and the beginning of the monitor. (This is called the FORTRAN Free Space and is the area into which the FORTRAN OTS fetches drivers not previously LOAded before the program was RUN.) In the case of high memory buffers, the allocation is effected by the values set into the buffer descriptors and certain COMMON blocks initialized by ISYSTM to the size of memory. No other action is necessary because the RT-11 SJ monitor knows nothing about high memory and MULTI's operations are entirely unnoticed by it. IUSER has previously primed the data acquistion system by calling CAMVL to set up the BISON Box Interrupt Vectors, disable the interrupts, and load pointers to the "compiled" form of the CAMAC Definition Lists (which have similarly been placed in the FORTRAN Free Space by IUSER's call to RDCMC). The initial state of the data acquisition system after this priming is that it is "out-of-buffers". IUSER, using DCLBUF, places buffers on the data acquisition system's empty buffer queue MTQ. DCLBUF calls OLDBUF to do this after setting up the buffer descriptor elements. The first call to OLDBUF turns off the out-of-buffers flag (in COMMON /CAMCOM/) and results in the data acquisition system doing a check to see if there is a pending event which it should read in, now that it has an empty buffer. (Of course, there isn't in this initialization case.) Subsequent calls to OLDBUF just add buffers to MTQ. Nothing more happens to the data acquisition system until the experimenter does a BEGINRUN to begin taking data, whereupon the BISON Box interrupt is enabled. 3.2 Event Phase During Data-taking When an event interrupt from the BISON Box occurs, the BISON Box hardware closes the Trigger Output Gate (usually within 60 nanoseconds) by changing the state of a signal coming from a lemo connector on the front of the BISON Box. The experimenter has connected this signal to the apparatus in such a way that further event interrupts are inhibited. The data acquistion system will open this gate when it has finished processing the event interrupt (during its performance of the STOP, REJECT, or END directive in the CAMAC list). Two comments should be made with respect to event interrupts and the trigger gate output signal. Firstly, both interrupts on a BISON Box close the gate. The latest (MOD III) BISON Box's open the gate on a STOP, etc., only if no interrupt is pending in the interrupt flip-flops (i.e., no LED lights are on in the interrupt section of front panel FOUR RTMULTI CONNECTED MACHINE APPLICATIONS DS-83 (DRAFT) Page 5 THE EXISTING RTMULTI SYSTEM indicators); otherwise, the gate is left close even though the computer attempts to reset it. Earlier models opened it, allowing interrupts to occur when they should still in fact have been held off and leading to a possiblity of several interrupt signals giving only one interrupt. Secondly, there is a one-bit interrupt pending flag which DRINT sets if it gets a subsequent interrupt of different or even the same type while it busy for whatever reason. CAMWHC (see below) is the code responsible for making the interrupt not pend and beginning the DRINT processing of an interrupt so held off. This one-bit "queue" slightly ameliorates the possible interrupt loss problem just discussed. The DRINT module in the data acquisition system gets control upon the event interrupt. It checks a software flag to determine whether the data acquisition system is busy. If it is, it marks the interrupt as pending and RTI's, leaving the interrupt enabled. This is very much _______ ___ _________ ________ unlike most software device handlers which would disable interrupts before RTI-ing. Comments for and against this technique may be made but mainly one should just be aware of the non-standard treatment so as to not run into problems with it. If the data acquisition system is not busy, a software flag is set to show that it now is active doing what is called a "high priority transfer". It then drops the CPU priority. (The first reference to "priority" signifies that the activity takes precedence over program initiated CAMAC transfers, etc. It has nothing to do with CPU priority directly.) DRINT then compares the maximum event length specified for this event in this interrupt list's BEGIN directive to that remaining in the buffer it is currently filling. If the maximum possible event will not fit, DRINT declares the buffer full by putting it on a queue called HISQ. It then trys to get another buffer from MTQ. If it fails, it leaves the event processing in DRINT with the Trigger Gate still closed and goes on to the interrupt level scanner module CAMWHC. Note: The data acquisition system makes this check at the beginning of the event interrupt handling for all interrupt types (A, B, C, and D). Doing so unnecessarily lengthens the time the experiment is gated off via the BISON Box Trigger Gate Output signal mentioned above. The early versions of the CAMAC handler (upon which this data acquisition system is based) were better in this regard and checked at the end of the event interrupt sequence whether there was at least room in the buffer for interrupt A (considered to be the primary interrupt for which quick response was most important). This feature was lost during the evolution from CAMAC handler to complete data acquisition system and it, or a variant, should be re-incorporated in the next version of the system. FOUR RTMULTI CONNECTED MACHINE APPLICATIONS DS-83 (DRAFT) Page 6 THE EXISTING RTMULTI SYSTEM This action causes the experiment to be effectively shut off until the data acqusition system is given an empty buffer to fill (either by the logging process completion routine calling OLDBUF to put a just-taped buffer on the MTQ or by GETEV in MULTI similarly returning a buffer whose events have all been analyzed). If the event will fit in the current or new buffer, then DRINT proceeds with processing the event according to the CAMAC list appropriate for this interrupt (i.e., interrupts A or B--actually, the software is written to be able to handle additional lists and interrupts C and D, but the hardware usually is not present to support this). When the STOP, END, or REJECT CAMAC list directives are encountered, DRINT goes to high priority to protect against being re-entered with additional event interrupts, turns off the "high priority transfer" flag, opens the Trigger Gate, and goes to the module CAMWHC. The experiment responds to the Trigger Gate action by becoming live for an event. If one occurs, the experiment will go dead again via the BISON Box hardware until the above processing has taken place for this subsequent event. In any case, such a pending event interrupt will not actually be taken until the priority is dropped later on in the operation of CAMWHC. 3.3 Interrupt Level "Which Thing To Do" Phase The module CAMWHC (i.e., CAM-WHICH) may be thought of as a "multi-tasking scanner," operating at the interrupt level and dedicated (i.e., coded) to the very specific purpose of handling the two or three fundamental "real-time" operations going on in the data acquisition system. Because of its specificity to the activities at hand, its overheads are minimal. It is analogous in purpose (but not in detail, to be sure) to the "sub-tasking" mechanism invented by the writers of the early RSX-11M data acquisition systems to avoid some of that executive's high overheads. (Warning: Because of its ad hoc nature and the historical way it has __ ___ been developed, CAMWHC is not symmetric in the way it does things and so, for example, JMP's to and from DRINT whereas it JSR,PC's to and RTS PC's from FULBUF.) Running at priority 7, CAMWHC scans various flag bits and flag words checking for data acquisition related things to do in the following order (and hence, intrinsic priority): 1. Pending Bison Box interrupts. If there are any (and the system is not out of buffers), CAMWHC clears the appropriate interrupt pending bit set by the beginnings of DRINT and immediately jumps into DRINT at the point where the latter sets the "high FOUR RTMULTI CONNECTED MACHINE APPLICATIONS DS-83 (DRAFT) Page 7 THE EXISTING RTMULTI SYSTEM priority transfer" flag and drops the CPU priority. (For simultaneously pending interrupts, the processing order is A, B, C, and then finally D as set by a table in CAMWHC.) 2. Full Buffers. If there are any buffers on HISQ, CAMWHC sets a "FULBUF in operation" flag, drops the priority, and calls the FULBUF routine. (This is still all at the interrupt level, of course). The "FULBUF in operation" flag protects the data acquisition system against being re-entered by some event interrupt, for example. If an event interrupt should occur during the FULBUF operation, it is forced to pend much in the same way that it pends when the system is out of buffers. After processing one buffer on HISQ, FULBUF returns. When this occurs, CAMWHC raises the CPU priority to 7, clears the "FULBUF in operation" flag, and goes to the beginning of the CAMWHC module to once again scan to various flags. (It is a rule of CAMWHC's design that after it effects some action the flow of control by one means or another must begin again at the start of CAMWHC, looking for things to do, since event interrupts, for example, may have occurred during that action.) 3. Program (i.e., MULTI) Initiated CAMAC calls. If MULTI, through the standard CAMAC routines, has tried to do CAMAC operations while the system is busy doing one of the other things in this list, the attempt will have been put on a special CAMIO queue of operations to be performed (not like any of the previously mentioned MTQ, HISQ queues of buffers). CAMWHC will initiate it at this time. An interlock bit is set in the PENDNG word of the data acquisition system's COMMON block /CAMCOM/. The priority is then dropped during the execution of this program-initiated CAMAC operation. After doing one enqueued operation, the CAMIO module will jump to the start of CAMWHC to begin the scan again of the flags. This sort of CAMWHC action could happen due to the CPU having executed a CAMAC call from the MULTI mainstream level during the time DRINT had "RTI"-ed pending the CAMAC completion interrupt. For example, DRINT goes back to the MULTI mainstream program during a long DMA transfer initiated by a TRANSFER directive in DRINT's execution of a CAMAC list as the result of an event interrupt. During this time, the MULTI mainstream program could make a CAMAC call which would be put on this queue. In the case of the standard CAMAC routine calls, the mainstream program will wait until this program request is satisfied so there will be only one CAMAC operation on this special CAMIO queue. The lower level non-standard CAMIO program request routine does, however, allow "no-wait" calls which could enqueue several program-initiated CAMAC operations. FOUR RTMULTI CONNECTED MACHINE APPLICATIONS DS-83 (DRAFT) Page 8 THE EXISTING RTMULTI SYSTEM 3.4 Full Buffer Phase FULBUF trys to remove a buffer from HISQ (by calling IREMOV), returning if it is not successful (although because of the way it is invoked, such a thing would actually represent some kind of logical error). Upon obtaining a buffer, it checks whether it is logging and whether any fatal tape error has occurred. If logging isn't specified or if fatal tape errors have occurred (the latter result in a message via MULPOL and a SUSPENDRUN command automatically being invoked), it immediately calls FULDUN to dispose of the buffer and, upon return from FULDUN, FULBUF returns to its caller CAMWHC. If logging is specified and no fatal tape errors have occurred, FULBUF puts the buffer on a queue denoted MTAPEQ in the figure. If there were no buffers on the tape queue before this, then FULBUF calls FULNXT to initiate the tape write. Otherwise, FULBUF simply returns to its caller CAMWHC, knowing that the outstanding tape write, as evidenced by buffers on the tape queue, will result in a FULCOM completion routine call which, after processing the just-completed tape write, will eventually initiate this one. FULNXT and FULCOM work delicately in combination with one another. If there are no buffers outstanding to be written to tape and no directive or magnetic tape errors, then the operation is fairly straight-forward. FULBUF calls FULNXT to initiate the special function write of the buffer, which FULBUF has just placed on MTAPEQ. FULCOM is specified as the completion routine to be called. Upon completion of the tape write, RT-11 calls FULCOM. FULCOM removes the buffer from MTAPEQ and calls FULDUN to dispose of the buffer. Upon FULDUN's return, FULCOM checks whether there are buffers still on MTAPEQ. In the simple case, there are none and FULCOM returns to RT-11, ending the completion operations associated with the tape write. If buffers are outstanding, then FULBUF, as mentioned, will have simply returned after having put them on MTAPEQ. At the completion of the current tape write when FULCOM finds additional buffers on MTAPEQ, it jumps to FULNXT with the buffer descriptor address of the top buffer on _____ MTAPEQ given as a parameter. FULNXT then initiates the tape write. The return executed by FULNXT acts as the end of the completion routine for the tape operation just completed. Now, we describe the operation when various kinds of special function errors occur. If FULNXT's special function operation results in a directive error (i.e., the operation is immediately refused without the completion routine FULCOM ever being called), then FULNXT sets registers to indicate the error in the manner expected by FULCOM and falls through into FULCOM to have it handle the error and the buffer disposition. In this case, the return executed by FULCOM acts as the end of the FULNXT call. FOUR RTMULTI CONNECTED MACHINE APPLICATIONS DS-83 (DRAFT) Page 9 THE EXISTING RTMULTI SYSTEM In certain cases, FULNXT's special function call can obtain an error response which results in a completion routine call directly in the RT-11 processing of ".SPFUN" (i.e., the return from the ".SPFUN" to FULNXT has not even been done; FULCOM has been called instead). FULCOM senses this by a storage location XSPFUN being non-zero (which FULNXT sets immediately before the ".SPFUN" call and clears immediately afterwards). In this case, FULCOM does the first part of its processing and then copies the channel status word into the CSW memory location and then returns. This return is midway through the ".SPFUN" execution done in FULNXT. FULNXT, having zeroed CSW before the ".SPFUN", senses this and operates as in the directive error case: it sets the registers to indicate the error in the manner expected by FULCOM and then falls through into FULCOM, executing the first part now a second time. XSPFUN is now clear so FULCOM proceeds to do the complete processing and buffer disposition. As with the directive error case, the return executed by FULCOM (now a second time) acts as the end of the FULNXT call. 3.5 Buffer Disposition Phase FULDUN is called to decide whether the buffer should be returned to the MTQ to be re-filled by DRINT or whether it should be placed on ANALQ to await analysis by the MULTI mainstream program. The algorithim implemented is well-described in the LIMIT Add-on, PN-130 (?) and so will not be discussed in detail here. We note a few items, however. The first check made by FULDUN is whether or not MULTI is set to obtain data from "EX", the experiment. If it is not, all buffers are returned to the MTQ, regardless of the settings of SCHEME and LIMIT. If it is, then the settings of these MULTI variables effect whether all buffers are sent to MULTI for analysis, or just a sample of them, and, if a sample, how many should be allowed to await analysis (at the expense of fewer buffers to fill with new events). 3.6 MULTI Mainstream Program-initiated CAMAC I/O For completeness, we discuss this aspect of the data acquisition system briefly. Calls by the program to do CAMAC I/O may occur through the standard CAMAC routines (i.e., CFSSA, etc.) or through the lower level routine CAMIO. The former are implemented by having their arguments re-formatted and calls made to the latter. FOUR RTMULTI CONNECTED MACHINE APPLICATIONS DS-83 (DRAFT) Page 10 THE EXISTING RTMULTI SYSTEM A CAMIO call makes an entry in a queue of operations to be performed. (The code for this is contained in the first half of the CAMIO MACRO module.) In the simple case, the data acquisition system is free to perform this CAMAC operation immediately. To do so, it simulates an interrupt by putting a priority on the stack and an address to which it wishes to return after the "interrupt." It then processes the top entry in the CAMAC operations queue as described below. (The code here is in the second half of the CAMIO module.) In the more complex case, the system is busy. Inserting the entry in the operations queue is then all that happens. A return by CAMIO to the caller is done. The caller may either loop on the "done" word specified in the CAMIO call or proceed with other computing. (The standard CAMAC routines always loop.) An example of this busy situation is as follows: An event interrupt has occurred. Its CAMAC list specifies a long DMA. Since it is a long DMA ("long" defined by an assembly parameter as being greater than some number of words), the system RTI's back to the main program, expecting the CAMAC Branch Driver completion interrupt to let it know when the DMA is done so that it can continue with the CAMAC list. During this time, the main program (in this case presumably, a MULTI user routine addition) makes a CFSSA call. This calls CAMIO, which makes a queue entry and finds the data acquisition system busy. It then returns and waits within the CFSSA subroutine. The DMA eventually completes, the event is finished, and CAMWHC is executed. All activities which could busy the data acquisition system (e.g., the CAMAC list WAIT directive) are required upon their finishing their task and un-busying the system to execute the CAMWHC module. This will eventually check the CAMIO queue after making sure that there are no pending event interrupts and no full buffers to process. When that occurs, the top operation in the CAMIO queue will be processed. This processing interlocks the system by setting the "low priority transfer" flag to hold off such things as event interrupts. It then processes the top entry in the queue, indicates in the "done" word that the operation is complete, and removes it from the queue. Continuing at an interrupt level, it jumps off to CAMWHC to check whether any other data acquisition system activities need doing (an event interrupt may have occurred and been made to pend while the "low priority transfer" flag was set). If nothing in the data acquistion system of higher priority is pending, the CAMIO queue will again be checked as the discussion regarding CAMWHC indicates, and the operation at the interrupt level repeated if there are additional entries in the queue. FOUR RTMULTI CONNECTED MACHINE APPLICATIONS DS-83 (DRAFT) Page 11 THE EXISTING RTMULTI SYSTEM 3.7 Branch Demand Interrupt Phase Again, for completeness, we discuss the operation of the system to CAMAC branch demand interrupts (i.e., LAM's). The CAMVL call made by IUSER initializes the Jorway JY411 branch demand interrupt vector to point to BDINT. The handler does very little with such interrupts, however. Upon a branch demand interrupt, it sets a variable in the /CAMCOM/ COMMON block to indicate that such an interrupt has occured and it also places the two-word Graded LAM status in the COMMON block. It is left to the MULTI mainstream program (specifically CMDAUX and LAMSER) to take action at a polling level to service this LAM. (Appropriate calls to CLNK must be made to enter specific LAM service routines in LAMSER's tables if this is to occur. The routine NEWLAT called from IUSER in response to a CAMAC.DEF LAMLATCH directive is an example of such.) 4.0 NEW PROCESSES RELATED TO LINK APPLICATIONS In order to have all considerations related to these applications in one place, we list the processes identified in V. White's informal note. New processes are pieces of code with an identifiable function. The routines to which they correspond are given later. 1. designates code which does the initialization. V. White's note proposes having a variable which, if set true, indicates that the link hardware is present. If false, no link hardware is present and all code such as should immediately return without attempting to access the link driver or the device registers, etc. A similar proposal is made with regard to CAMAC. Implementation of this would require reading in some type of file in ISYSTM presumably before the existing IUSER is called. The CAMAC.DEF file might also be a place for putting such switches as it is called fairly early in the initialization. (We presumably generalize it to a sort of HRDWRE.DEF file then.) (I believe implementation of these features should be postponed--DJR.) 2. designates code which handles operator link commands. There are several sub-components of this code: 1. sets up the local MULTI to make requests over the link in order to obtain buffers of events to analyze. It speaks to a PTC=3 application in the remote system for this FOUR RTMULTI CONNECTED MACHINE APPLICATIONS DS-83 (DRAFT) Page 12 NEW PROCESSES RELATED TO LINK APPLICATIONS purpose. It listens as a PTC=4 application. It expects buffers of events as the result of such PTC=4 reads. 2. sets up the local MULTI to send events over the link in response to requests for events to log from the remote application. It speaks to a PTC=? application when it sends events (one at a time). It listens as a PTC=2 application for requests. It does a CDRCV specifying a completion routine in doing so. 3. sets up the local MULTI to send buffers of events over the link in response to requests for such buffers from the remote application. It speaks to a PTC=4 application when it sends buffers. It listens as a PTC=3 application for requests. 4. was specified in the V. White note as setting up the local MULTI to put all buffers onto the link analysis queue without limit. Later in this specification that situation is handled through the SET facility. 5. stops the local MULTI from sending buffers for remote analysis. This code presumably stops the local MULTI's speaking to PTC=4 applications although it may still listen as a PTC=3 entity. 6. was specified in the V. White note as setting up the local MULTI with a limit on the number of buffers on the link analysis queue. Later in this specification this situation is handled through the SET facility. 7. turns the link on so that the above applications can use it. This is effectively serving notice upon the software that the hardware now exists and that it may use it. It changes the state of the switch discussed under such as might be settable in the CAMAC.DEF file. (I propose implementation of this feature be postponed--DJR.) 8. flushes the link analysis queue. 3. administers a number of small buffers to listen for PTC=3 requests for buffers to remotely analyze. Whenever there is no read to listen for such a request, does a CDRCV specifying one of its buffers as the place to put a message. It keeps a list of the message block numbers of the CDRCV's which are active. FOUR RTMULTI CONNECTED MACHINE APPLICATIONS DS-83 (DRAFT) Page 13 NEW PROCESSES RELATED TO LINK APPLICATIONS 4. checks whether LKRCV has had its listening for PTC=3 satisfied. If it has (indicating that a message requesting a buffer for remote analysis has been received), it checks for errors. If there are none and an buffer is found on the LINKQ, it removes the buffer from the LINKQ and keeps it locally while it sends it with CDXMT to a PTC=4 application in the remote machine. It saves the buffer descriptor in association with the message block number for later use by . 5. , a completion level routine, is executed whenever the local machine's listening for a PTC=2 request is satisfied by the arrival of a remote logger's request for another event to log. This routine gets the next event to log, removing a buffer from MTAPEQ if necessary. If there are no events available, it sets a flag LOGREQ. Otherwise, it calls to send the next event using CDXMT and specifying as a completion routine. It sends to the PTC=? application which is listening in the remote machine. Errors set flags which are noticed by a routine called from MULPOL and which can therefore bring in an overlay to report the errors. 6. , a completion level code, is executed when the send to PTC=? application completes. If the event was the last one in the buffer, it transfers control (calls?, jumps?) to FULDUN in the same way as FULCOM does. Again, error handling is by flags and MULPOL as above. 7. , a completion level code, together with FULDUN, decides to which queue a buffer should be attached after being transferred over the link for remote logging. The possible queues are: LINKQ, ANALQ, and MTQ. The choice depends on the settings of SCHEME, LIMIT, LKANA, and LKANLM. 8. checks if the last CDXMT of an analysis buffer to a PTC=4 listener has completed. Any errors are reported by LKERR. If it has completed, the buffer is returned to the MTQ. If it has completed, it also calls LKSEND to possibly respond to additional PTC=3 completed listenings (by sending to a PTC=4 listener). 9. handles receipt of buffers of events to analyze. It has been listening for PTC=4 transmissions. It also sends additional requests (to a PTC=3 listener). It checks if the previous read has completed. If so, it checks for errors and reports them. It handles the busy count-down if the error given is that the remote sender is busy. If no errors, it puts the buffer just filled on ANALQ. When the read is completed, it gets the next buffer from the MTQ and does a CDRCV into it (listening for PTC=4). It then does a CDXMT (talking to a PTC=3) to request buffers for analysis. It does error checking FOUR RTMULTI CONNECTED MACHINE APPLICATIONS DS-83 (DRAFT) Page 14 NEW PROCESSES RELATED TO LINK APPLICATIONS on these CDPACK operations and saves the message block numbers in conjunction with the buffer descriptors for later use. 10. is called in an exit phase to tidy up the link. 11. is called to report errors detected at completion level as well as other errors detected for which it is desirable to put off the reporting to an overlay (possibly). 12. is a user-written routine to allow the current event selected by user-written analysis code (e.g., EUSERA) to be put on a link queue (presumably LINKQ) for later transmission to a remote analysis machine upon its request. Note that the event is put into a buffer and it is the buffer which is put on the queue. 5.0 ORGANIZATION OF PROCESSES INTO MODULES We want to organize the above processes in such a way that the four applications can be handled with a minimum amount of re-compilation. Rather, we would prefer replacing objects with dummys and take the inefficiencies that dummy calls bring in order to simplify the use of these systems in various ways. (We will have to measure the time consumed by these dummy calls; there is some indication that FORTRAN calls to FORTRAN dummy routines take an inordinate amount of time.) The following calls to routines will be present in the code of all applications. 1. LKINIT -- to initialize the link. 2. LIPROC -- to handle all LINK commands. 3. LKDUN -- called in the same area as FULDUN if buffers are being sent for remote analysis. 4. LKEVNT -- called in MULPOL to handle most non-time critical link send/receive functions. 5. LKEXIT -- called from STUSER if link was in use. The following module will be present. It will contain most routines referenced by the above, including completion routines. FOUR RTMULTI CONNECTED MACHINE APPLICATIONS DS-83 (DRAFT) Page 15 ORGANIZATION OF PROCESSES INTO MODULES 1. LKCRTN -- completion routines The following module will be conditionally assembled. 1. FULBUF -- this must be changed so that if logging is occurring over the link, FULBUF must call LKNXT instead of FULNXT. This should only be done if LOGREQ is set. Also, FULBUF needs to put buffers on MTAPEQ if other tape logging or LNKLOG flags are set (???). 6.0 MODULES FOR EACH APPLICATION The following lists the modules need for each application. 6.1 No Link System LKINIT, LKEXIT, LIPROC, LKDUN, LKEVNT, and LKCRTN are all needed as dummy modules as well as dummy CDPACK routines. 6.2 "Sending Events For Analysis" Application 1. LKINIT -- 2. LKEXIT -- 3. LKDUN -- 4. LIPROC -- ,..., 5. LKEVNT -- + + + 6. LKCRTN -- dummy FOUR RTMULTI CONNECTED MACHINE APPLICATIONS DS-83 (DRAFT) Page 16 MODULES FOR EACH APPLICATION 6.3 "Sending Selected Events For Analysis" Application 1. Same as previous section, but different LKSEND. 6.4 "Sending Events To Remote Logger" Application 1. LKINIT -- 2. LKEXIT -- 3. LKDUN -- dummy 4. LIPROC -- + 5. LKEVNT -- 6. LKCRTN -- + + 6.5 "Receiving Events To Analyze" Application 1. LKINIT -- 2. LKEXIT -- 3. LKDUN -- dummy 4. LIPROC -- + 5. LKEVNT -- + 6. LKCRTN -- dummy 6.6 Combination - "Remote Logger + Sending Events For Analysis" Application FOUR RTMULTI CONNECTED MACHINE APPLICATIONS DS-83 (DRAFT) Page 17 MODULES FOR EACH APPLICATION 1. LKINIT -- 2. LKEXIT -- 3. LKDUN -- 4. LIPROC -- ,..., 5. LKEVNT -- + + + 6. LKCRTN -- + + 7.0 THE "SENDING UNSELECTED EVENTS FOR ANALYSIS" APPLICATION Figure 2 shows the buffer and data flow diagram for this case and the following discussed application of sending selected events for analysis. We discuss the parts of the diagram special to this particular application. 7.1 Initialization Phase Additions are made to IUSER to call invoke the process. This will likely be simply a call to a subroutine LKINIT. This will do things, such as: 1. Fetch the CD driver. 2. Do a lookup for the channels required. 3. Establish RT-11 queue elements out of the FORTRAN Free Space using IQSET. Much of this sort of initialization can likely be done via the CDPACK routines, such as CDASGN. 7.2 Experimenter Interface There are several aspects to the experimenter interface: new variables which may be SET and a new command LINK ,,... FOUR RTMULTI CONNECTED MACHINE APPLICATIONS DS-83 (DRAFT) Page 18 THE "SENDING UNSELECTED EVENTS FOR ANALYSIS" APPLICATION 7.2.1 New Variables - We plan to have variables analogous to those related to magnetic tape for controlling the link. Thus: 1. SET LKANA=ALL and SET LKANA=SAMPLE respectively condition the link software to send all or a sample of buffers to the remote machine for analysis. 2. SET LKANLM= specifies a limit on the number of buffers that may be on the LINKQ at any time. This variable throttles this queue in the sample case (i.e., it keeps the system from placing all the buffers on the LINKQ, leaving none for acquisition and logging). In the ALL case, it has no effect. LKANA is implemented as a system name (see PN-97.5, page 54). LKANLM is implemented as a system variable. The objects ALL and SAMPLE are implemented as system values. If we pick 0 as signifying ALL and -1 as signifying SAMPLE, then the resulting system will allow SET SCHEME=ALL and SET SCHEME=SAMPLE to make sense as well, correcting a previous oversight in not making these references symbolic. (We should also include system values KCBUSY and KMBUSY with literal values -1 and +1, respectively, in order to allow symbolic setting of all the values of SCHEME. KCBUSY and KMBUSY stand for "Keep Camac Busy" and "Keep MULTI Busy." Better mnemonics should be suggested.) Implementing the system values is simple. We just add the names and their literal values (0 and -1 above) to the initial dictionary. LKANA and LKANLM are most easily implemented by extending the COMMON /SCOMN/ to include the two variables as was done with SCHEME and LIMIT. This allows them to be easily included in the "DI STATUS" display. (It is not as "orthogonal" to the existing system as one might like, but its display advantages outweigh this, I believe--DJR.) Routines which need to know these values include the *SCOMN in their code. We may also wish to include in another *XXXX area DATA statements defining ALL and SAMPLE so that the checking on the value of LKANA may be done completely symbolically (a la' the MULTI style). (References to the existing LIMIT and other variables are done slightly more indirectly: the addresses of the variables are communicated to the routines rather than including the COMMON blocks directly. This is done to ease the maintenance problems associated with making MACRO PSECT's and FORTRAN COMMON blocks coincide completely. See BERUN and the FULBUF, for example. We may wish to consider this also.) FOUR RTMULTI CONNECTED MACHINE APPLICATIONS DS-83 (DRAFT) Page 19 THE "SENDING UNSELECTED EVENTS FOR ANALYSIS" APPLICATION 7.2.2 LINK Command - The LINK command is implemented by the LIPROC routine. This routine (not shown in Figure 2) starts up the process of sending buffers over the link for remote analysis. The command may be given either directly or indirectly through the usual MULTI command file capability. The LINK command will indicate this machine's willingness to receive requests for buffers to be sent by setting a flag in an appropriate COMMON. We expect to have the following commands: (I am not necessarily happy with these particular names, etc., but I think it is important to put something down before we start the coding so we know what we are trying to do. Please feel free to come up with your own ideas.) LINK ANAL,SEND,YES ==> start sending buffers for analysis over the link. LINK ANAL,SEND,NO ==> stop sending buffers for analysis over the link. LINK ANAL,SEND ==> Similar to REC case (see LINK ANAL,REC below). Note that ...,SEND,YES/NO... commands are effective only in a macroscopic sort of way. They condition the local copy of MULTI as to how it should respond to a request from the other machine. At a microscopic level, the receiver of the application-level object (i.e., event, buffer, etc.) still is the one making the requests per our protocol. If ...SEND,NO... has been done, the SEND code just responds in the first round of code development by issuing another read for a possible request without satisfying this one. In a second round of code development, we may choose to send a signal back to indicate "try again later", etc. 7.3 Buffer Transmission Phase In this application, FULDUN is expanded as the starting point for sending buffers over the link. Two additional keyboard SETtable MULTI variables are present to aid in controlling how much sent, etc. These are termed LKANLM and LKANA in the diagram analogous to LIMIT and SCHEME for the local analysis case (which controls how much is sent on to ANALQ). Buffers which are selected to be sent over the link are placed on the queue LINKQ by a process denoted as . In the polling mode, LIPROC's setting of a flag will be picked up by MULPOL which will call LKEVNT. A section of LKEVNT will implement the process. To do this it will issue a read to the CD driver to obtain a request buffer for analysis packet (defined to be a read for FOUR RTMULTI CONNECTED MACHINE APPLICATIONS DS-83 (DRAFT) Page 20 THE "SENDING UNSELECTED EVENTS FOR ANALYSIS" APPLICATION packets with PTC=3). Once this read is outstanding, requests by the remote analysis machine for buffers to be analyzed will be placed in the buffer specified by the process. The process will also be implemented as a section of LKEVNT. This process will check whether a message requesting an buffer for analysis has been received (i.e., whether the read issued by has completed). It will check for errors and use CDPACK's error message output facilities to report them. It will invoke to establish a new CD driver read (???). If no errors have occurred in the receipt of a request for an analysis buffer, it will then remove a buffer from LINKQ, keeping its buffer descriptor address locally. It will send the buffer with CDXMT, saving the CDXMT message block number as well. The process , which will also be implemented as a section of LKEVNT, will check if the last CDXMT of an analysis buffer has completed. It will report errors via , also a section of LKEVNT. In any case, when the CDXMT has completed, it will return the buffer to MTQ by means of the saved buffer descriptor. 7.4 Exit Phase The process will be called from STUSER to wind down the link connection gracefully just before the MULTI program is to exit. This process should call CDCLOS for all PTC's with open sessions as the experimenter may have forgotten or simply chosen not to invoke the approriate LINK commands before typing STOP to MULTI. Some miscellaneous notes are: 1. It is important to not get in the user's way. The user must be allowed to type STOP to exit the program without going through any other gyrations, like having to type LINK END or something. If the complexities of winding down open sessions properly are too much to do in STUSER, then it is better to let a hardware RESET be done to turn off interrupts and let the user take the consequences rather than to get in the way. 2. An abnormal exit does do a hardware RESET. This action is setup via MULTI's IUSER call to USEREX.--DJR) FOUR RTMULTI CONNECTED MACHINE APPLICATIONS DS-83 (DRAFT) Page 21 THE "SENDING SELECTED EVENTS FOR ANALYSIS" APPLICATION 8.0 THE "SENDING SELECTED EVENTS FOR ANALYSIS" APPLICATION The addition to the previous structure to allow the sending of events selected by the local MULTI code to another machine for remote analysis involves a routine callable by the user from EUSERA, EUSERB, etc. See Figure 2 for this application also. This will put the event into a buffer and put the buffer on a queue for transmission to the remote machine. 8.1 One Event/Buffer If the parameters of the CAMAC.DEF file are such that there is only one event per buffer, then we implement this capability by constructing a routine which conditions GETEV upon passing this event on, to place it on the LINKQ rather than the MTQ. The flag which conditions GETEV is reset after this operation. The requests for analysis events from the remote machine will eventually obtain the queued event. 8.2 Many Events/Buffer If there are many events per buffer, then we must obtain an empty buffer, copy the current event into it, and following the one event/buffer operation. We postpone detailed consideration of this case. 9.0 THE "SENDING EVENTS TO A REMOTE CONCATENATOR/LOGGER" APPLICATION Figure 3 shows the buffer and data flow diagram for this case. 9.1 LINK Command The LINK command starts up the process of sending events over the link to be logged. We expect to have the following commands: LINK LOG,SEND,YES ==> Start sending events over the link for remote logging, etc. LINK LOG,SEND,NO ==> Stop sending such events. FOUR RTMULTI CONNECTED MACHINE APPLICATIONS DS-83 (DRAFT) Page 22 THE "SENDING EVENTS TO A REMOTE CONCATENATOR/LOGGER" APPLICATION LINK LOG,SEND ==> As with ANAL,REC case. Note that the other case of LINK LOG,REC,... is not present as our current assumptions are that receipt of events for logging is done by an RSX machine. 10.0 THE "RECEIVING EVENTS FOR ANALYSIS FROM A REMOTE SOURCE" APPLICATION Figure 4 shows the buffer and data flow diagram for this case. 10.1 LINK Command The LINK command starts up the process of requesting buffers to be sent from the remote machine to this one for local analysis. We expect to have the following commands: LINK ANAL,REC,YES ==> start receiving buffers for analysis over the link. LINK ANAL,REC,NO ==> stop this activity (not necessarily immediately but possibly after some kind of wind-down period). LINK ANAL,REC ==> Defaults to start if currently stopped and stop if currently started. FOUR RTMULTI CONNECTED MACHINE APPLICATIONS DS-83 (DRAFT) Page 23 Figures Figure 1 Process, Buffer, and Data Relationships in Existing RTMULTI System FOUR RTMULTI CONNECTED MACHINE APPLICATIONS DS-83 (DRAFT) Page 24 Figures Figure 2 Process, Data, Buffer Relationships (Send Events for Analysis Case) FOUR RTMULTI CONNECTED MACHINE APPLICATIONS DS-83 (DRAFT) Page 25 Figures Figure 3 Process, Data, Buffer Relationships (Send Events for Logging Case) FOUR RTMULTI CONNECTED MACHINE APPLICATIONS DS-83 (DRAFT) Page 26 Figures Figure 4 Process, Data, Buffer Relationships (Receive Events for Analysis Case)