Mp 16-August-1980 Mp MP - Macro Processor 1. Introduction 2. What Mp does 2.1 The pass 2.2 The macros 2.3 Treatment of non macros 2.4 Conditionals 2.5 Other internal statements 2.6 Switches 2.7 Macro definitions and arguments 2.8 The string stack 2.9 Macro argument operators 3. Operating instructions 4. Built in default macros 5. Macro summary 6. Alternate default macro librarys 7. Examples -1- Mp 16-August-1980 Mp 1. Introduction Mp is a macro expander. It came into existance after experience with macro-11 macro libraries failed to produce an efficient higher order (higher than assembler, not `high order') language compatible with the supported systems programming language of Rsx11 (which is regetably macro-11). This, along with an interest in macro processors, has resulted in a flexible medium level language translator. It is missing many of the niceties of true high order languages (such as true expressions, free format input, and high level data structures) but it does allow the construction of efficient `structured macro languages' without giving up intimate processor contact and full macro-11 compatibility. It is an order of magnitude faster than macro-11 alone (E.g. Dec's supermac) although it is not as fast as I might like. While its not Bliss or C, it is convenient and simple to use. The format of input is the compatible with the assembler on the PdP-11 (Macro-11). However, the arguments inside of macro definitions are referenced by number rather than symbolically. Labels and comments are compatible (i.e. the `:' and `;' delimeters for labels and comment fields). Parameters passed to macros have the same nesting rules (i.e. <>'s etc.) but there are no Keyword macros. The ^z...z syntax is supported and the form "..." is additionally allowed (although this means that "xx must be specified as <"xx>). Arguments have selector operators which permit extracting portions of a single argument or referencing multiple arguments (e.g. arguments 2 thru arg n, where n is the number of arguments passed to the macro). In addition, there is a string stack which may be accessed globally with operators for referencing absolute elements and elements relative to the top of the stack. The stack may be divided into two stacks with two stack pointers if desired. The elements on the stack may be assigned to the local parameters of the macro. A method is available for generating labels of the form xxxnnn where nnn is a decimal number and xxx is user definable (0-3 chars). Various statements exist for the conditional expansion of code either from within macros or at the statement level. Target assembler directives may be placed in line and will be output unmodified. Macro construction is very convenient. One can simply edit a file with the macro definitions, envoke the processor with both input and output -2- 1.0 Mp 12-October-80 Mp 1.0 directed to the terminal and proceed to test various expansions. Various dump options exist to interactively see all intermediate results. If a good text editor exists, one can spawn Mp without ever leaving the editor, test a macro, exit Mp, continue editing the macros, etc. In this way, one has almost all the capability of a good interpreter. Many of the routines I have used were originally written by David Conroy (of C compiler fame). Reading David's code is in many ways more enjoyable than a novel. I found many great ideas there and recomend his work to anyone with an interest in good technique. 2. What Mp does 2.1 The pass Mp reads the entire source file one time and acts as a filter copying from its input to its output without modification unless it detects an opcode that is either an internal code or a user defined macro. Macros are recursively expanded until a non macro and non internal opcode are detected, at which time the statement is output unmodified. Having been designed primarily for Macro-11 preprocessing, the method of handling listings is tied to the Macro-11 .print statement and how it works in conjunction with a list suppress option. Each input line is immediatly copyied to the output after being modified as follows: ".print ;statement" When the Mp output is run through Macro-11 with listing suppressed, only the .print statements are output. The assembler address of the beginning of the statement is output with the .print statement thereby allowing easy use of the assembler debuggers, while suppressing the actual expansion of the code generated by the macros themselves. If the input is a ".MACRO" (upper case) directive, then the macro-11 macro definition is simply copied from input to the output until a subsequent ".ENDM" is encountered. No .print statements are added here. -3- 2.1 Mp 12-October-80 Mp 2.1 2.2 The macros Mp accepts macros from various places. The order in which they are entered into the macro name table determines which definition takes precedence when the macro is multiply defined. The following lists the the search order for opcodes. a. The first set of names are the conditionals (these are hard- coded). b. The file which contains the source to be processed, may have macro definitions (if the /lm switch is specified - see later). They must precede all other source code and be separated from the source by a line with a slash `/' alone in column 1. c. One macro library may be specified and is a file of macros whose file-name is determined by Mp according the following rule. The default file-name is "[1,1]mplib.mac". The /ml switch changes the default name to "[your,uic]mplib.mac". A command-line option allows the setting of any file name in place of the defaults. The /ml switch then is meaningless. d. Built into Mp is a default set of macro definitions (loaded from an overlay makes processing faster). The loading of these may be suppressed by the /ni switch. 2.3 Treatment of non macros Any statement not recognized by Mp is copied directly to the output. These statements may be any Target assembler directives provided the label: and ;comment syntax aggrees. Note that the opcode is what determines if the statement is a macro and that opcodes are not limited to the rad50 subset. Also, string matching is case sensitive; so all macro definitions and later invocation must agree totally in case. -4- 2.3 Mp 12-October-80 Mp 2.3 2.4 Conditionals A variety of conditional statements exist, they are: a. I%B This expands statement if string is blank. b. I%NB Expands statement if string is nonblank. c. I%IDN Expands statement if both strings are identical. d. I%DF Expands statement if strings are different. e. I%IN Compares key to each of Sn until a match is found, then En is expanded. If no match is found, then the else statement is expanded. Else may be omitted. Null strings may be present where desirable. f. I%SW switch# Expands statement1 or the else statement depending on whether the switch is set. See below for explanation of switches. -5- 2.4 Mp 12-October-80 Mp 2.5 2.5 Other internal statements The following lists the rest of the internal statements. g. I%DOUT This will output statement directly without further recursive expansion. This is useful when one desires to supplement code to an existing target assembler directive. One example might be to output something additional to the .END statement by defining a macro called ".END" and having it output some code and end up with an output of the actual .END directive. Comments may be output in this fassion as well. h. I%ERR This outputs a .error statement to the ti: device as well as to the file. The format is: .error ;message i. S%SW switch# This sets the switch specified. The switch is turned on if switch# is positive and off if preceded with a minus sign. j. E%S n Enters symbol into a symbol table with the integer value of n associated with it. The symbol may be subsequently looked up, see below. k. T%YP function symbol index This directive requires that function be `T'. Later enhancements may add additional function codes here. The `T' functions causes a search of the symbol table for the specified symbol. If it is found, the numeric value assigned is placed into the global string stack at location index. If it is not found, the value placed is 0. -6- 2.5 Mp 12-October-80 Mp 2.5 l. D%SYM This directive dumps the symbol table as a series of macro calls. The format is as follows: $$$$$$4 n $$$$$$5 sym val $$$$$$6 n where n is the number of symbols and the $$$$$5 is expanded once for each symbol in the symbol table. The user may wish to define these macros as there is no default definition. m. ** macro parm1 parm2 ... This opcode expands `macro' once for each non-blank parameters. Note that `macro' may be null to execute multiple commands on a single line. 2.6 Switches Mp reads the command line and extracts switches without using the CSI modules. Thus switches can appear anywhere on the line. They are always 2 letters and have no negation or optional data capablilies. Each switch is assigned a number (from 1 to 31) and it is this number which may be referenced in the I%SW and S%SW statements described above. There are 16 switches which are reserved for Mp and the rest can be used by the user. The switches are: 1. /nl No .print statements generated. 2. /da Not implemented. 3. /is This will print on ti: internal macros as they are read. This aids in locating a faulty macro that bombs the preprocessor. 4. /me Expands macros one level deep only (for debugging -7- 2.6 Mp 12-October-80 Mp 2.6 macros). 5. /ml Use local uic for standard macro library name, not [1,1] 6. /op Allows opcodes that are hardcoded to not take precedence over macros with the same name. (Not supported). 7. /tr Implements the trace feature. This causes Mp to internally invoke two macro calls for every statement input. The two macro calls are passed the entire input line as parameters. One call occurs before expansion of the statement and one after. Example: input line: LET a = b affect: $$$$$$1 LET a = b (expansion of LET a = b) $$$$$$2 LET a = b The two macros $$$$$$1 and $$$$$$2 must be user defined. 8. /ni No internal macro definitions. 9. /md Expands macros and outputs (as a comment) all intermediate steps during expansion of the macro. 10. /pp Not implemented. 11. /un Outputs to the ti: device, amounts of storage used after macro libraries are read in (to determine how large table sizes should be in the task image). 12. /x2 Turns on unary operator recognition for first operator. It is not recomended that this switch be applied until all macros are read in, or any detection of the operators during macro load time will cause expansion which is not usually desirable. Rather they should be turned on by program control as needed. See see section -8- 2.6 Mp 12-October-80 Mp 2.6 on Macros for further information. 13. /x3 Turns on unary operator recognition for second operator. 14. /x4 Turns on unary operator recognition for third operator. 15. /x5 Turns on unary operator recognition for quote operator. 31. /lm Used to signal that local macros are included in the source file. The remaining switches are unassigned and are as follows: switch 16 as /x6 switch 17 as /x7 switch 18 as /x8 switch 19 as /x9 switch 20 as /x0 switch 21 as /y1 switch 22 as /y2 switch 23 as /y3 switch 24 as /y4 switch 25 as /y5 switch 26 as /y6 switch 27 as /y7 switch 28 as /y8 switch 30 as /y0 -9- 2.6 Mp 12-October-80 Mp 2.7 2.7 Macro definitions and arguments Macros have the form: $MACRO name macro text $ENDM Additional information on the $MACRO and $ENDM lines is optional and is treated as comments. Arguments are referenced as #@n where n is a hexidecimal (upper case) digit. For example, the following is a definition of a push macro. $MACRO push mov #@1,-(sp) $ENDM This macro would be called (for example): push this And it would generate the following code: mov this,-(sp) Note that if the definition is in lower case then the call must also be lower case (resp. for upper case). One additional syntax item (not found in macro-11) is the following: An argument of the form A[B] is translated into . Macro arguments are substituted into macro text as follows: A left to right scan is performed copying text into a secondary buffer character for character until an argument reference occurs. Then the argument text from the command line is copied replacing the argument reference. When copying the argument text itself, no interpretation of argument references is made and no second pass on the line is made. Thus recursion within macro argument replacement is disallowed. -10- 2.7 Mp 12-October-80 Mp 2.8 2.8 The string stack Mp has a string stack which is accessable to macros only. When a macro is expanded, the expander only has access to the current macro's argument strings. Because of this, a special macro construct for use only inside macros is used to transmit strings between this string stack and the parameters of the macro. The stack appears to be a 60 element (assembly parameter) string array (like those found in basic-11). The first 24 are generally used for stack 1, 25-30 as a scratch area, and 31-60 for stack 2. Item 29 contains the string used to generate labels. Stack operations are coded as statements which begin with a `?' character and start in column 1. There are 2 forms of this construct as follows: 1. ?? p1,p2,...,pn This format does not reference the string stack directly but generates automatic symbols of the form $$$nnn and places them into the list of parameters specified. Example: ?? 1,2 would generate (assuming no other labels have yet been gen'd) two symbols $$$1 which would replace argument 1 $$$2 which would replace argument 2 If the arguments had been defined (i.e. not null) then the previous contents are lost. The `$$$' is the default and by user control can be changed to some other string. See the absolute string placement below. 2. ?- relative absolute switch and ?+ argument relative absolute switch The above are pushes (-) and pops (+) of the string stack. The -11- 2.8 Mp 12-October-80 Mp 2.8 `?-' expects a literal string to be supplied as . This is usually done via some reference to an argument. Example: ?- #@1 This will push argument 1 onto the stack. The second parameter (if supplied) specifies a position relative to the stack pointer and ALSO denotes that the operation is NOT actually a push, rather it is a move. Stack relative placement does not modify the stack pointer. Example: ?- #@1,2 This will place argument 1 into the stack at position 2. The top grows downward and begins at location 1 of the string stack. The third parameter (if not null) specifies that the push (actually move) is not relative to anything but is absolute. Example: ?- abc,,29 This will place the string `abc' into absolute position 29 of the string stack, thereby setting the label generation string described above. The fourth argument is to conditionalize the entire operation on the status of a particular switch. Note that these special directives cannot be the expansion clause of other directives (E.g. I%B). For this reason the user needs to make a test somewhere else and set a switch. Example: ?- #@5,,,20 This specifies that if switch 20 is set then push argument 5 onto the stack. To specify that the condition is based on the switch NOT being set then supply the switch number as negative. The ?+ operation is analgous to the ?- operation with respect to the relative, absolute, and switch parameters. The first parameter is however the argument number in which to move to. This number is a decimal number. -12- 2.8 Mp 12-October-80 Mp 2.8 Example: Suppose we redefine the push macro such that if the argument supplied is reg0 then we will change it to R0. $MACRO push parm1 ?- R0,,25 ;put an R0 into string element 25 S%SW -26 ;clear switch 26 I%IDN reg0,<#@1> ;set switch 26 if arg1 = `reg0' ?+ 1,,25,26 ;if sw 26 set, move str 25 to arg1 mov #@1,-(sp) ;now do the actual push $ENDM The above example demonstrates some of the stack options in a macro but is not very efficient in terms of statements (the overiding concern for speed). The following shows how to do the same thing in one statement. Note the use of the I%IN to specify an else clause. $MACRO push I%IN <#@1> reg0 <> $ENDM The second stack pointer is referenced as ++ or --. Example: ?-- abc ;pushes "abc" onto stack 2. ?++ 3,2 ;pops next to top of stack 2 into parm 3 Macros may also be expanded by the use of certain unary operators. When the tokenizer encounters a token (each parameter of a macro and the opcode itself) it checks to see if the first character is one of 3 operators (defined in the tkb file for mp). If it is and switches 12, 13, or 14 respectively for operators 1, 2, and 3 is also set, then an expansion of the macro $x is performed (x is the unary operator). The token is stripped of the operator and passed onto the $x macro as parameters. The $x macro is user defined. It may generate code or just fiddle with the stack and switches etc. On exit however, it MUST push onto stack one a string which will replace the token which caused the expansion. Example: LET a = _function[a,b] where `_' is the unary operator would cause the following expansion to be invoked (before the LET is invoked): $_ function,a,b IF $_ returns (via a push on stack one) the string `replacement' then the let statement would be expanded as: LET a = replacement -13- 2.8 Mp 12-October-80 Mp 2.8 A suitable macro definition for `$_' could be: $MACRO $_ ?- replacement $ENDM The quote character (usually " but definable at mp build time) is a special case unary operator. Its interpretation depends on switch 15. If switch 15 is off, then a quoted string is treated as a macro argument in <>'s. For example: "Quoted string" is the same as: However, if switch 15 is on, then it is also treated as a unary operator but the entire argument is passed to the $" macro in one piece. For example: LET a = "Quoted string" would cause "Quoted string" to be passed to $" as: $" This allows an implementation of the $" macro to output a labeled ascii statement (possibly in a data psect) and replacing the "Quoted string" (in the LET statement) with the address of the ascii string. This is basically what the default macros do. See the file mcmd.mac. -14- 2.8 Mp 12-October-80 Mp 2.9 2.9 Macro argument operators Macro arguments may be either simple argument references or they may select portions of the argument. An argument reference which has the following format selects a portion of the argument selected. #@=n(f:l) In this format `n' is the argument number, `f' is the first character in the argument and `l' is the last. The `n', `f', and `l' parameters are all single hexidecimal (upper case) digits. However digits beyond 15 (`E') are permitted as no range check is performed upto the letter `Y'. Anything not a number or letter (or `Z') (or larger than the argument for the `l' parameter) is converted to the number of characters in the argument. For example: #@=1(2:Z) ;extracts chars 2-end of arg 1 #@=A(B:D) ;extracts chars 11-13 of arg 10 To reference multiple arguments the following form is used: #@*(f:l) which concatonates arguments `f' thru `l'. The same rules for digits as above apply here as well except a value larger than the number of arguments is converted to the number of arguments (not including locally defined labels). For example: #@*(2:Z) ;selects args 2 thru n-args #@*(3:B) ;selects args 3 thru 12 The `:' is actually a special case for the `separator' character which is output between each character (in extraction of characters) or argument (multiple argument selection). The following table describes the `separator' characters currently in use. ------------------------------------------------------------------- | " double quote separates with >< | | : colon no separation (null) | | _ underline character separates with a space | | x any other char separates with x | -------------------------------------------------------------------- -15- 2.9 Mp 12-October-80 Mp 2.9 For example, the macro call: TEST arg1 arg2 arg3 The following argument references: #@=1(1,Z) ---> a,r,g,1 #@=1(1:Z) arg1 #@=1(1"Z) a><1 <#@=1(1"Z)> <1> #@*(1,Z) arg1,arg2,arg3 #@*(1_Z) arg1 arg2 arg3 <#@*(1"2)> Note that arguments separated by <><> type pairs do not need a comma or space between them. The >< characters are the corresponding characters defined in the Mp task build file for left and right angle brackets. If these values are changed, the characters used as >< separators in the argument selector also are changed. One special case argument reference exists which permits conditionally appending (r5) to argument (the string appended is defined in stak.mac). An argument reference of the form: #@{n} where n is of the form used above to reference an argument number, causes the symbol table to be searched for the string defined by argument n. If it is present, then the string (r5) is appended to argument n and is output as the replacement. This was added to allow variables in a C-like macro language be declared and substituted efficiently. One useful technique to determine if a variable is in the symbol table is: I%IDN <#@n> <#@{n}> -16- 2.9 Mp 12-October-80 Mp 2.9 3. Operating instructions Mp uses the get-command-line routine with 1 level of indirection. It does not make direct use of the CSI parsing routines; rather it parses the command line itself. The format is: mp>OUT=IN[+MACROLIB][/sw's] or mp>file[+macrolib][/sw's] IN is the input file (def ext=.mac) OUT is the output file (def ext = .mpp) MACROLIB is an optional library of MP macros (def ext = .mac) If there is no `=' present, then "file" is converted into "file=file". In this case the extensions must be defaulted. The choice of .mpp for the intermediate file extension is to allow easy deletion of temporary files. If no command line is entered, repetitive mode is enabled. In this mode a file is translated until an eof on input. Then command mode is re-entered (like pip or tkb etc.). Thus multiple translations are permitted (usually from an indirect file). However, macros are only defined for the first command entered. This allows a set of modules to be translated with out the reloading of macro libraries (and the internal macro definitions). If new macros need to be defined the processor must be reloaded via a new invocation. One useful technique for debugging macros is the following: mcr>mp mp>ti:+lib/sw's -input and output to tty interactively test macros control-z mp>ti:/sw's more interactive tests with different switch settings control-z mp>control-z -to exit mp. This technique allows one to test a macro and switch on the macro expansion if a problem results. It is also a good way to see how a macro expands before actually writing code that uses a particular construct. -17- 3.0 Mp 12-October-80 Mp 3.0 Error reporting is poor in the expander. If messages come out out on the terminal that say things like string buffer overflow or string index out of range, you have exceeded some internal limit. Check the last thing you wrote. Usually the error is caused by some butchered statement or one with unterminated <> pairs. The expander will write the offending line on the terminal. In some cases, a .error statement is generated. Some times it may be necessary to rebuild the task with a larger stack and if many local symbols are generated there is a parameter which can be increased in the tkb file. Stak.mac also has some values which can be increased if you run out of space for macros. -18- 3.0 Mp 12-October-80 Mp 3.0 4. Built in default macros The following is a description of a macro language easily constructed using Mp. It is supplied with Mp as the default macro library, that is, the one built into the Mp task. The library's control structures were loosely copied from the C and Basic languages. The few statements that deal with data are meant to provide C compatibility (a far better choice than the ancient fortran) which provides reentrancy with a simpler calling sequence. C routines can use registers r0-r4 since only r5 is needed to point to both the local variables as well as the parameters. Passing of register contents and immediate mode constants are also much simpler as no temporaries are required. Even though two processors must run, it is 7-10 times faster than supermac because of the ease with which it can deal with stacks. It also need only expand macros once whereas macro-11 does the expansion twice. A short description will be given here with comments on how to set up an alternate built in set. In the following description: * repeated 0 or more times + repeated 1 or more times [optional] 0 or 1 time a syntactic item i.e. not literal OTHER STUFF litteral string if upper case other stuff example of thing if lower case `thing' denotes thing is not a meta-character Macro description: o PROGRAM Initializes macro library and sets to be the prefix for label generation. It should be <= 3 chars in length. This macro must be specified and should occur before all other macro usage. It enables lower case and sets switch 15 (for strings). o AUTO