Subject: PDP-8 binaries

From: [email-masked]
Date: Mon, 22 Mar 1993 20:42:26 -0500

A while ago, some mention was made of someone (Mr. Jones at U of Iowa,
perhaps?  sorry, don't recall) going to find out from DEC if they could
distribute the binaries to various PDP-8 utilities so that people could
run them on their emulators.

I had a news outage for a few weeks :-( and was wondering if any progress
had been made on that front.

Thanks!

     _      _   _  _           ([email-masked]) 278 Equestrian Dr.
__\ |_) ^  |_) (  |_ SOFTWARE  Kanata, ON K2M 1C5 +1 613 599 8316 (fax 8317)
  / |  /-\ | \ _) |_ DEVICES   UNIX/VMS/QNX C/ASM86 Contract Services

---

From: [email-masked] (Charles Lasner)
Date: Tue, 23 Mar 1993 20:02:31 GMT

>had been made on that front.

In point of reality, DEC doesn't actually care what we do with the binaries.
Doug Jones has successfully gotten DEC to agree to his excellent reprints of
various manuals, to which he has added the phrase "reprinted with permission
19xx" in the appropriate place, etc.  Note that DEC doesn't do any of the
work, so they are pre-supposing that the user community has the manuals in
the first place.

So it is with the binaries as well.  Thus, while we don't have any express
"permission" at this time, there is no resistance from DEC in our doing
whatever we want with the programs.

In any case, the logistics of the distribution of the files is where I come
into the picture.  (Incidentally, I am also a source of some of the binary
files as well.)

PDP-8 binaries come in several flavors.  The most common is expressable as a
stream of 8-bit paper-tape frames, and is known as BIN format.  It consists
of a leader of 200 octal code frames of indefinite length, followed by binary
data, which includes unique codes for data, origin settings and field settings,
followed by a checksum and a trailer identical to the leader.  Origins are
never defaulted; the tape always includes at least an initial origin, usually
at 0200.  Data words may be present without (many) origin settings, but the
checksum just before the trailer is mandatory.  Additionally, binary tapes
may be "spliced" together where the trailer of one becomes the leader for
the next section.  This is usually done only for related binaries, where the
loading of the follow-on sections is possibly optional, etc. (or an overlay).

A cousin of BIN format is RIM format.  The only practical difference is that,
to shorten the loader code length, certain features were deleted.  Each data
word can only be loaded at the current origin setting; there is no automatic
incrementing of the load point.  Thus, RIM tapes consist of origin settings
followed by data.  These pairs are the bulk of the tape.  Often the origin
settings are in fact sequential, but of no consequence in RIM format.  (P?S/8
includes utilities to convert RIM tapes to BIN tapes by optimizing out all of
the origin settings that prove to be sequential; the resultant file is of
course then a BIN file, not a RIM file!  Also, P?S/8 can convert BIN into RIM
inserting sequential origin settings, etc.)

Additionally, RIM format has no checksum.  It is possible to make a tape
loadable by both BIN and RIM by using sequential origin settings at the
source code level.  Thus:

	*200

	CLA
	TAD	377
	HLT

becomes:

	*200

	CLA

	*.

	TAD	377

	*.

	HLT

	*.

The reason the final origin setting is added is to ensure that there is a
sacrificial location.  BIN formatting of this tape after assembly causes a
checksum to appear at the end of the tape.  If the RIM loader is used for
this purpose, then were there not an additional origin setting, the checksum
would be loaded (as garbage) over the last loaded location.  Using this method
it gets loaded *after* the loaded data addresses.  (P?S/8's utilities that
convert BIN to RIM automatically do this for the same reason.)

It would appear that the most likely format we want BIN and RIM binaries
stored in is a stream of 8-bit bytes in the exact order they were originally
in.  This also is preferred by the PDP-8 emulator authors I have spoken to,
etc.

Towards this end, I have written certain OS/8 utilities that can help
distribute binary files.

OS/8 stores files such as the above in a format that is equivalent to the
stated format, just packed into twelve-bit words.  The exact format is:

Word[0]bits[4-11]	First Byte
Word[1]bits[4-11]	Second Byte
Word[0]bits[0-3]	High-order Nybble of Third Byte
Word[1]bits[0-3]	Low-order Nybble of Third Byte

This sequence is repeated 128 times thus storing 384 bytes in one logical
record.  At the driver call level, all OS/8 devices look alike in this manner.
It is the responsibility of the caller to pack/unpack the data in the
appropriate manner, etc.  Other aspects of the system control overall file
length in records, etc.

ASCII and Binary files are stored this way; core-image files are stored in
memory-image preceded by a header block that describes the memory allocation
of the file, etc.

Thus, any utility that can accurately unpack files into the 8-bit quantities
described can transfer the files in their 8-bit representations according to
the requirements of other systems.

In the MS-DOS world (in fact almost anywhere that C exists) there exists a
set of utilities to pack/unpack .BOO format files.  This is a printably
encoded binary format with some run-length compression capabilities, etc.  It
was developed originally to distribute MS-KERMIT through potential "hostile"
links such as e-mail, etc.

I have written PDP-8 ENBOO and DEBOO for the OS/8 family of systems.  The
file format is totally compatible with the MS-DOS equivalents, and since the
underlying unpacking format is obeyed, ASCII and BIN and RIM files are
translated correctly into byte streams as required by MS-DOS, etc.  It is
the user's responsibility to ensure that any CR/LF convention be obeyed
correctly.  This means that CR/LF of OS/8 ASCII files be handled correctly,
and that incidental occurences of characters that appear to be CR or LF are
not modified in binary files, etc.

Thus, any PDP-8 binary paper-tape can be read into OS/8 and then converted to
an OS/8 file.  The OS/8 file can be encoded into .BOO format, and then sent
via e-mail, etc. to another site where it can be decoded back into a byte
stream.  Thus we have a viable method of converting ASCII, BIN and RIM files
between unlike systems, etc.

For other OS/8 files, they can essentially be treated as 12-bit files with
no regard for the underlying format.  This includes other binary files such
as .RL files, or save images (.SV), etc.  To handle these files, I have
written a set of utilities called ENCODE and DECODE which work with a file
format intended as a replacement for uuencode or .BOO, but PDP-8 oriented.
The file format includes an overall checksum, and run-length compression for
12-bit values, which often defy compression on systems where the run-length
compression is only considered on 8-bit quantities.  (ENCODE can compress
PDP-8 binaries containing regions of HLT codes while unix LZW-compress and
ZIP and .BOO will not compress the file because the objects compressed have
to be multiples of 8-bits.)

The intermediary file format is more robust than either .BOO or uuencode, and
can accurately pass through more hostile environments, including EBCDIC
sites.  Unlike .BOO (and certain uuencode implementations) the file includes
descriptive phrases to help define the file's purpose.  User commentary is
easily added and actually encouraged.  The file format is immune to white-space
distortions and can also pass through WPS environments.  (Neither uuencode or
.BOO are capable of getting through WPS because of the dependence on an overly
large character set.  Some EBCDIC systems are also problematic due to the
presence of non-standard translation tables at some sites.  ENCODE format does
not depend on any characters that might be mis-translated, etc.)

The file also includes both its own creation date and the file date of the
original file to help in the recreation of the original file.  (The file dates
of the original file may have an 8-year anomaly caused by a limitation of OS/8
itself; some systems, notably MS-DOS, are incapable of expressing file dates
preceding 1980 while many of these files date back to 1965.  OS/8 itself can't
express a date outside of the range 1970-2001.  The actual date reckoned by
OS/8 can be wrong by exactly 8, 16, or 24 years precisely because only the
low-order 3 bits of that 32-year span are stored in the file.  Other systems
may have additional date resolution problems, so it's best to store the date
info within the file itself.)

Thus, any OS/8-expressable file may in turn be encoded for e-mail, etc. in a
reliable and robust manner, complete with an internal self-description that
will eliminate file confusion, etc.

This should handle all of the present requirements of PDP-8 emulator authors,
archivers, etc.

Additionally, the ENCODE/DECODE programs have the ability to handle image
files of entire devices, either whole or split into 2 roughly equal parts.
Thus, it is possible to store the entire image of a bootable RX01/02 diskette
as a compressed encoded file, etc. (or two smaller files).  This is helpful
for dealing with other systems such as bootable POLY BASIC DECtapes or P?S/8 
RX01 diskettes, etc. to enable e-mail/net transfers, etc.  As more device
handlers become available, the ENCODE program will automatically correctly
deal with them.

At the present time, it is only possible to transmit a subset of an RX50
diskette via this method.  The reason is that the distributed handlers for
OS/278 RX50 aren't capable of addressing the entire diskette.  (For each
two 8-bit bytes, only 12 bits are used; the other four are zeroed rather
than used for data.  Additionally, track 0 is inaccessible.)  Thus, usage
on OS/278 for image transfer is limited to OS/278 12-bit file structure
since this usage of the diskette is unique to current OS/278.  (Other RX50
software generally used 8-bit mode and thus can address all bits of every byte.
Some systems disallow easy access to track 0, or tracks 0, 78, 79.)

I am also working on replacement handlers for OS/278 which will be full
8-bit mode.  This will represent a 35% improvement in device size for all
purposes, and additionally will allow the ENCODE/DECODE utility to address
the entire RX50 diskette, etc.

cjl