.c;SPELL .TITLE;Spelling Checker .AP .HL1;Summary SPELL is a system of checking for spelling errors in text files on the RSX-11M operating system. The program SPELL does the checking and optionally may make changes, while DICTMK formats a file for SPELL to use. DCTNRY is a word looker upper that uses sends/receives with SPELL. They are written in FORTRAN-77 except for certain file access routines and alphabetic manipulation routines that are written in Macro. The program has been optimized as much as possible for speed. In non-interactive mode a file will be produced which contains all the words from the document that SPELL could not find in its dictionary. In interactive mode words can be replaced, added to the dictionary (DICTMK will need to be run again), or simply ignored. For an in-depth analysis of the operation of both programs, see the source code as it contains much comments. Automated extractions of comments (C+/C-) will tell the workings of all the routines. .hl1;Background The programs were written in FORTRAN-77 and were developed on a PDP-11/t55 with 6-RK05, running RSX-11M V3.2. Development has continued with our addition of 4 RL02 and upgrade to RSX-11M V4.1. The programs came from a desire on my part to have a system that could be used to check my submissions for the DECUS North Texas Local Users Group Newsletter. The only spelling checker I could find was the one included in the Software Tools section of the Spring 81 RSXSIG tape. As I didn't want to build software tools, I used the data base as a starting point for program development. As sequential reads were too slow for the upwards of 42,000 words in the data base, I used Marge Knox's analysis of the file access speeds of FORTRAN-IV+ (now FORTRAN-77). Random access - block i/o was decided on to minimize the time for disk i/o. Using Ralph Stammerjohn's Index to the past RSXSIG tapes, I was able to come up with Phil Cannon's routines for FCS file access to the FDB. This enabled me to build the dictionary (using DICTMK) using random access block i/o, set the size of the file, and to truncate to the last block used on exit. To increase speed and add information to the user a couple of routines were added to SPELL and DICTMK. These routines (GETVER and GETSIZ) come from the Copious library (FALL81 [361,212] GR Johnson of Battelle Northwest). New files are opened the same size as the old files by using GETSIZ. The version of an open file can be determined with GETVER. The prototype of the spelling checker would read the document to be checked and put each unique word in a list to be checked by the dictionary and those misspelled written out to a file. This worked somewhat all right for small files, but the larger the file the more time that was spent checking against the list for each word to insure the word was not already included. Even small files yielded 500 different words to be checked. This was before the index for each letter was included in the file itself and therefore there were on the average of 30 reads for each word. Due to the increase in the size of the OTS with F77 v5.0 the SPELL task was made overlaid. This should not effect the execution speed of the task since the initialazation and wrap up portions are the only sections broken out. With the added space other features are easier to add. .hl1;Use To build SPELL, copy DICT.SEQ to uic [3,3] or any UIC you want if changes are made to the source code. @DICTMK will produce the DICTMK.TSK, @SPELL will produce the SPELL.TSK, and @DCTNRY will produce DCTNRY.TSK. Run DICTMK to produce DICT.RAF (the random access file). Our system takes about 10 minutes to produce DICT.RAF. Install SPELL or set up CCL to do it and pass command lines. The installation of DCTNRY is mandatory so that SPELL can send to the correct task. To check a document for spelling, >SPE filename[/sw] will do it. If you are not using a ANSI VT100 or VT52 type terminal, please make changes to SUBROUTINE DISPLA for the right escape sequences for clear screen. .hl1;DICTMK DICTMK is the dictionary maker for the spelling system. It is used to provide a random access block i/o dictionary with internal indexes for all the letters. Access to the File Descriptor Block is possible by the use of Phil Cannon's routines that use an internal FDB in a byte array. Thus truncate on close is possible using block i/o. DICTMK will also provide statistics on the words in the dictionary after it has built the dictionary. .hl2;Dictionary The dictionary that goes along with this program comes from the software tools section of the Spring 81 RSXSIG tape. It was DICT.;1. It contains more than 42,000 words. The shortest word is 'a' and the longest has 22 letters. From looking through it, there are some proper names and places. This may or may not be advantageous. Proper names take up less than about 3% of the total space (as determined by checking words that start with 'a' or 'b'), so deleting them may not buy you much space. During field test of SPELL with Rockwell it was noticed certain words were not found even though they were in the sequential file. The common feature among them was that they had to do with an apostrophe. It was determined that the dictionary was not in true alphabetical order as I had assumed. SPELL assumes ASCII collating order. A FORTRAN routine was used to sort the sequential file into the correct order. After running for 12.5 hours everything is in correct order. .hl2;Adding words When spell runs and produces a list of words that you want to add to the dictionary, run DICTMK from where SPELL ran (that's where the file of words is). DICTMK will take the file and merge it into a new DICT.SEQ then produce another DICT.RAF. .hl1;SPELL SPELL is the spelling checker program. It uses the dictionary file and its own internal list, and checks the text for words not found in either. Several switches are available for the command line, and several commands are available in the interactive mode. .ts20 .HL2;Switches When invoking SPELL several switches are available. To invoke SPELL: .literal >SPE filename[switches] or >SPE SPE>filename[switches] .end literal The switches for the command line are as follows: .literal /Q process in quick mode (non interactive) /F file is FORTRAN source, process comments only /T terse comments, on interactive, display line and word only /N numbers are valid symbols in tokens /L[:filename] preload the add list with words that are correct if filename exists use this list, if not use 'SPELL.DCT' /S:x ignore words with 'x' or less letters /V switch to VT52 type escape sequences /R word requires a vowel to be considered for check /D:x skip down x lines into document (usefull for continued check of a long document after quit) .end literal If the default value is changed in SUBROUTINE GETFIL then /Q, /F, /T, /N, /R, and /V can be negated (/-V). .hl2;Modes There are two modes of operation for the spelling checker. Depending on the availability of time and disk space the proper mode can be chosen. The interactive is the non-quick mode, so named since it takes longer to process a document in this mode. .hl3;Quick The quick mode provides for no interactive processing. It will scan the document and make a list of words that it didn't find in its dictionary or in its internal list. This is the mode used for a first hack at long documents and batch checking. .hl3;Non-quick The non-quick mode (default) is interactive with the terminal and uses screen clearing. For adjustments to the escape sequences see SUBROUTINE DISPLA. In this mode the document is scanned for number of lines, to provide the operator with a scale of how far he has to go. If the verbose command has been used (default), the full display line and the given word is displayed. .hl4;Display The terminal display in the verbose mode will display the spelling header line which holds the title and version number. Next is a short display of the commands available. Then a short list of the statistics of the spelling checker so far this session. Thus the operator can see how many lines have been processed, how many words, and the number of words added to each of the lists. Next is the line containing the word in question and then the word itself with a prompt. .hl4;Commands The commands available are: .literal _# (where _# is a number between 1-9) ignore words _# or less in length +n skip down n lines (good for getting to where you quit A this word is correct, add to dictionary C this word is correct, do not add to dictionary D check variations on this word in dictionary E exchange words, but do not add to list L replace current line N next word, do not add to list Q quit, copy rest of document without changes (then use +n) R require words require vowel to be considered -R do not require vowels for consideration T terse display -T verbose display W wrong spelling, make correction to document X exit, as if end of document was here ^Z exit, as if we hadn't started redraw screen after erase .end literal .hl2;Adjustments Several adjustments can be made to this program by the interested user. Words can be added to the internal list (at the sacrifice of space). Adjustments to the display section are possible for other than ANSI VT100 or VT52 type terminals. The location of the dictionary can be modified to increase the speed of reads. .hl3;Add words There is a little space (4000 bytes) left in the program to allow for the expansion of the internal list it checks first. The internal list used is the first 90 entries found in the list in "Computational Analysis of Present-Day American English" by H. Kucera, Brown University Press, 1967. .hl3;Display Adjustments can be made to SUBROUTINE DISPLA to provide the escape sequences for other than those given. Even fancier terminal i/o can be done. VT100s can be set up for a variable scroll region for instance. .hl3;Disk A great increase in speed can be made if the dictionary is kept on a solid state disk. See past RSXSIG tapes to get part of your excess memory in 22 bit machines to be a solid state disk. (A luxury I don't have.) .hl1;DCTNRY This program has been addded to the collection to provide the capability to look up variations of the word in the dictionary if the spelling is only a little off. .hl2;Variation Generation There are four methods used by which the word is modified and checked with the dictionary. .hl3;Too Many Letters For each letter in the word, the letter is removed and the resultant "word" is checked with the dictionary. If the word is found it is saved in a list for later use. .hl3;Transposed Letters For each adjacent pair of letters in the word a transposition is made and the resultant word is looked up in the dictionary. If the word is found it is saved in a list for later use. .hl3;Not Enough Letters At the beginning, between each letter, and at the end letters are added to the word. In each position every letter is added a to z. If the SPELL says that numbers are possible in the words 0-9 are also added. Each of these words is looked up in the dictionary and if it is found it is saved in a list for later use. .hl3;Wrong Letter For each letter in the word, every letter in the alphabet is substituted and the resultant word is checked in the dictionary. If it is found it is saved in a list for later use. .hl2;Display The list of found words is displayed to the user and the choice made by the user is transfered back to SPELL for replacement in the line. (User uses command N for next after display). .hl2;Start/Stop If DCTNRY is started by MCR as >DCT word then the MCR buffer is scanned and the resultant word is used as if it had been sent by SPELL. In this case the list of words is displayed and DCT exits. If DCTNRY is active, SPELL will first try to unstop it. This is desirable since the RAF file will not have to be opened since it is already open. If DCTNRY is not stopped but active, SPELL will wait till it is stopped. If DCTNRY is not active then SPELL will request it be run, in which case DCTNRY will open the RAF file and do certain initializations. Then DCTNRY will do a receive of packet. The word it uses is located in bytes 1-25. Byte 26 is a status byte where certain conditions of SPELL are sent. These are if the terminal is a VT52, VT100 or some other. Also whether in SPELL the Terse condition is present and whether numbers are allowed in the word. Four other true/false conditions are available for the users if necessary. After getting the choice from the user the word is sent back to who ever sent the word in the first place. Thus several copies of SPELL could run the DCTNRY. After the word is sent DCTNRY stops itself. If ever there is a hang up with the spelling checker system, DCTNRY will exit with a warning of no send packet if it is unstopped from the terminal. This should not cause any problems since it will be requested from SPELL on the next dictionary look-up. At the end of a spelling session DCTNRY will be awoken and caused to exit by sending it a null packet. If DCTNRY is not installed, a warning message will be displayed in the SPELL program and no further trys will be made to it.