USE OF MS-KERMIT ON THE IBM PC FAMILY WITH THE HEBREW CHARACTER SET

Following is some unedited correspondence between Baruch Cochavy of
IIT, Technion (Israel) and Joe Doupnik of Utah State University
regarding the implementation of the support Hebrew (and Arabic and
other right-to-left languages) in MS-DOS Kermit on the IBM PC.

------------------------------

Date: Sat, 26 Nov 88 21:54:37 +0200
From: Baruch Cochavy <baruchc%TECHUNIX.BITNET@cuvmb.cc.columbia.edu>
Comments:  Domain style address is "baruchc@techunix.technion.ac.il"
To: fdc@cunixc.cc.columbia.edu, jrd@usu.bitnet
Subject: Hebrew Kermit

1) Hebrew character set

        the Hebrew character  set  is  with  the  display  adapter
     character  generator.  While up until  version 3.30 of DOS no
     formal  definition of national characters for DOS existed (as
     far  as I know), characters in the 80H and above were used to
     store national  character fonts.   This  way, in  Israel, IBM
     used the codes 80H-9AH for that purpose.  In other countries,
     the same codes were used for their national characters.

        The  space within the character generator selected for the
     national   characters,  does  not  agree  with  the  existing
     standards for  national  characters.    There  are  different
     ISO8859 for many languages,  Hebrew  included  (ISO8859-5,  I
     think).   ISO standard  defines the Hebrew  characters in the
     range  0E0H and above.   However, I have yet  to see a single
     application  that uses that code.  Usually, in Israel anyway,
     the  computer you buy come with Hebrew fonts in the character
     generator  already installed.   You should do  nothing if you
     like it.  If you don't, you still have it ...  .

        EGA is a  special case, and  there is a  program that will
     upload Hebrew fonts  into  the  character  generator  at  the
     correct place.  If  one  wishes  DOS  be  informed  of  being
     'Hebrew' (DOS still talks that  same  way  as  in  any  other
     English speaking), than  a  line  like  'COUNTRY  =  972'  is
     included in the  CONFIG.SYS file.   This  also sets  the date
     display to  the  way  it  is  costumed  in  Israel,  that  is
     DD-MM-YY.   No other change to the AUTOEXEC file is required,
     unless  you use an EGA.  If  so, then running HEBEGA would do
     the task of character set uploading.

        As  of version 3.30 of DOS, 'code page' is part of DOS, as
     to help support national character fonts and keyboards.  This
     only works  on a 3.30 version, that  has one of the following
     display adapters:

        - EGA
        - LCD (IBM PC convertible)
        - IBM PS/2 display (EGA)

        The code page re-defines 256 characters.  It also requires
     the  proper support files.  This national support is provided
     when  you purchase DOS.  In Israel, the Hebrew support needed
     is supplied.   Other  countries  and  languages  may  have  a
     special  support also, along with  the commonly supported DOS
     languages.     The  DOS  3.30   reference  manual,  (numbered
     94X9575), page B-2 lists national character codes.

2) Keyboard

        Hebrew typwriting have the  Hebrew  aleph-bet  ordered  in
     such a  way as to cause severe  headache to anybody who tries
     to  find any connection between  their representation and the
     ASCII code  of  the  corresponding  English  characters  that
     occupy  the same physical space on the keyboard.  The letters
     are arranged the  same way  every Hebrew  typewriting machine
     is.

        Usually, DOS versions  prior  to  3.30  provided  national
     keyboard  support   through  the  use   of  special  keyboard
     handlers, provided  with DOS.  KEYBHE  is the Hebrew keyboard
     support.   Switching Hebrew and English  is done but pressing
     CTRL-TAB.   You do not have to use any special key to produce
     Hebrew characters  from then  on, unless  you have  to switch
     back to English.  Programs that use Hebrew entry, and have to
     work through DOS, may do the switch by using the IAC area.  A
     special byte  there tells the software  what kind of keyboard
     handling  to do.  Most of the programs, however, choose to do
     a direct screen and keyboard handling, an thus, do not resort
     to this measure.

        There  is   also  Hebrew  screen   support,  that  enables
     right-to-left  writing and some other things.  This software,
     supplied by  IBM here in Israel, is  such an illogical use of
     the computer, that hardly anybody ever uses it.

3) SET TRANSLATION INPUT

        This  command does play some role in Hebrew representation
     of characters for Kermit.    Hebrew  characters  are  usually
     represented in  the computer in a way  that has nothing to do
     with the  character generator used in the  PC.  that way, for
     example, aleph is represented in code 80H in the PC, while it
     can be represented as 60H  in  the  computer,  or  any  other
     imaginable  way.    Using  the  input  translation  for  that
     purpose, is essential.   Setting  the translation  table off,
     re-establishes the link between host character representation
     and the PC character  display.   This is  the usual  state of
     things.   However, it cannot be  utilized by the software, as
     there is  no direct means of setting in  off and on.  This is
     now   changes  with  the  addition   of  the  \TERMINALS  and
     \TERMIANLR macros.

4) Hebrew software

        Hebrew software, and mixed Hebrew/English software exists.
     The whole  issue of a Hebrew Kermit  has arisen form the need
     to   provide  a  general  way  of  doing  remote  editing  of
     Hebrew/English  text, without resorting to a specially abated
     terminal that  supports Hebrew  screen functions,  but rather
     use an existing PC with the proper handling.  Also, Hebrew is
     implanted into  the terminals (in  most cases) in  such a way
     that  almost   every  screen  function   ceases  to  function
     correctly  in the right-to-left direction.   Those that still
     work,  do work because the Hebrew firmware change has nothing
     to do with those function.

        VIH is the Hebrew adaptation of the wide spread VI editor,
     used  with the UNIX(tm) operating system.   On the DEC, there
     is  a TPU/HEBREW that enables Hebrew editing, but I am unsure
     as to the amount of Hebrew terminal capabilities used.  ALEPH
     is a borrowing system for the universities that also works in
     Hebrew.  This software is connects nation-wide through DECNET
     links.   Other software exists, and are usually an adaptation
     of existing programs for Hebrew.  It should not be to hard to
     modify  TPU to use  \TERMINALR and \TERMINALS  macros.  Other
     software might be tedious  to  change,  some  might  even  be
     impossible.  Only a well defined extension might catch enough
     interest as to prompt future use of it.

5) Other languages, generalization, and some personal thoughts

        When I  first became  involved with  this project,  is was
     intended to  be a practical  solution to a  special problem -
     that of using VIH  on a  PC terminal  emulator.   The initial
     approach was to  use the  supplied DOS  modifications (KEYBHE
     and  HEBREW, the  Hebrew screen  handler), and  some existing
     communication program, and have the hack work.

        Being impractical,  I was  thinking of  an elegant  way of
     doing it.  What I ended  with  was  an  approach  that  would
     enable not only  Hebrew, but  other languages  as well.   The
     screen functions were  extended  in  a  logical  way  to  the
     current  VT102 ones.   This, as far  as I know,  is the first
     time this  extension was made  within that logical  frame.  I
     also devised the concept of two translation tables, as to not
     cause any change to the current Kermit functionality.

        My way of doing it is not the only one possible, and there
     are may others.   The end  result should be  the same: Kermit
     should be able  to handle  national character  sets, keyboard
     and  displays, as efficiently as  possible, causing as little
     burden to the user as possible.


        Baruch Cochavy
        IIT, Technion, ISRAEL

------------------------------

Date: Sat, 26 Nov 88 16:37 MDT
From: <JRD%USU.BITNET@cuvmb.cc.columbia.edu>
Subject: Some thoughts on Baruch's discussion.
To: Baruchc@TECHUNIX, fdc@cunixc.cc.columbia.edu, JRD

Baruch and Frank,
        I've reproduced and annotated your (Baruch's) interesting comments
today. Mine are in square brackets. The summary is: Baruch feels the screen
blink of terminalr/s activation is (almost) a disaster and I feel there is no
apparent alternative. Additionally, the host side is probably a mess.
        Regards,
        Joe D.
-------------------------------------------------------------------------

From:   IN%"baruchc@TECHUNIX.BITNET"  "Baruch Cochavy" 26-NOV-1988 12:54
To:     fdc@cunixc.cc.columbia.edu, jrd@cc.usu.edu
Subj:   Hebrew Kermit - answers

Date: Sat, 26 Nov 88 21:54:37 +0200
From: Baruch Cochavy <baruchc@TECHUNIX.BITNET>
Subject: Hebrew Kermit - answers
To: fdc@cunixc.cc.columbia.edu, jrd@cc.usu.edu

1) Hebrew character set

        the Hebrew character  set  is  with  the  display  adapter
     character  generator.  While up until  version 3.30 of DOS no
     formal  definition of national characters for DOS existed (as
     far  as I know), characters in the 80H and above were used to
     store national  character fonts.   This  way, in  Israel, IBM
     used the codes 80H-9AH for that purpose.  In other countries,
     the same codes were used for their national characters.

[from jrd - Hmmm. Curious, and perhaps convenient if uniformly done.]

        The  space within the character generator selected for the
     national   characters,  does  not  agree  with  the  existing
     standards for  national  characters.    There  are  different
     ISO8859 for many languages,  Hebrew  included  (ISO8859-5,  I
     think).   ISO standard  defines the Hebrew  characters in the
     range  0E0H and above.   However, I have yet  to see a single
     application  that uses that code.  Usually, in Israel anyway,
     the  computer you buy come with Hebrew fonts in the character
     generator  already installed.   You should do  nothing if you
     like it.  If you don't, you still have it ...  .

        EGA is a  special case, and  there is a  program that will
     upload Hebrew fonts  into  the  character  generator  at  the
     correct place.  If  one  wishes  DOS  be  informed  of  being
     'Hebrew' (DOS still talks that  same  way  as  in  any  other
     English speaking), than  a  line  like  'COUNTRY  =  972'  is
     included in the  CONFIG.SYS file.   This  also sets  the date
     display to  the  way  it  is  costumed  in  Israel,  that  is
     DD-MM-YY.   No other change to the AUTOEXEC file is required,
     unless  you use an EGA.  If  so, then running HEBEGA would do
     the task of character set uploading.

------------------------
[from jrd - Here there might be a problem when doing Tektronix displays which
normally use all of display memory, including the part assigned to hold
replacement fonts. Additionally, not all EGA boards properly support the EGA
dynamic save/restore functions (Kermit thus avoids them). As we know, the
MS Kermit terminal emulator does not use DOS for screen i/o so Code Pages,
and other filters of DOS device CON, are bypassed. SET TERMINAL NONE allows
Code Pages, ANSI.SYS, et al. to be used.]
-------------------------

        As  of version 3.30 of DOS, 'code page' is part of DOS, as
     to help support national character fonts and keyboards.  This
     only works  on a 3.30 version, that  has one of the following
     display adapters:

        - EGA
        - LCD (IBM PC convertible)
        - IBM PS/2 display (EGA)

        The code page re-defines 256 characters.  It also requires
     the  proper support files.  This national support is provided
     when  you purchase DOS.  In Israel, the Hebrew support needed
     is supplied.   Other  countries  and  languages  may  have  a
     special  support also, along with  the commonly supported DOS
     languages.     The  DOS  3.30   reference  manual,  (numbered
     94X9575), page B-2 lists national character codes.

2) Keyboard

        Hebrew typwriting have the  Hebrew  aleph-bet  ordered  in
     such a  way as to cause severe  headache to anybody who tries
     to  find any connection between  their representation and the
     ASCII code  of  the  corresponding  English  characters  that
     occupy  the same physical space on the keyboard.  The letters
     are arranged the  same way  every Hebrew  typewriting machine
     is.

        Usually, DOS versions  prior  to  3.30  provided  national
     keyboard  support   through  the  use   of  special  keyboard
     handlers, provided  with DOS.  KEYBHE  is the Hebrew keyboard
     support.   Switching Hebrew and English  is done but pressing
     CTRL-TAB.   You do not have to use any special key to produce
     Hebrew characters  from then  on, unless  you have  to switch
     back to English.  Programs that use Hebrew entry, and have to
     work through DOS, may do the switch by using the IAC area.  A
     special byte  there tells the software  what kind of keyboard
     handling  to do.  Most of the programs, however, choose to do
     a direct screen and keyboard handling, an thus, do not resort
     to this measure.

-------------------------
[from jrd - KEYBHE appears to be a "hot-key" TSR program, to judge by the
Control-Tab toggle. This means it operates between the keyboard hardware
and the Bios in such a way that regular calls on the Bios by a program or
by DOS produce national characters. Since Kermit obtains keystrokes at the
Bios or DOS level, but not below, KEYBHE could provide the translation
without Kermit's assistance.
        KEYBHE, however, requires the user to select the translation state
by means of the Control-Tab toggle. This means the host cannot do the same
through Kermit.
        Given Baruch's situation of changing translations perhaps every few
words a toggle becomes a menace. Avoiding the toggle syndrome is the primary
reason for having not one but two host-reachable macros: terminalr and
terminals.

        To summarize the keyboard area, let's define some specifications:

1. We are concerned with translations occurring only during CONNECT mode.
   DOS level translations might be beneficial but occur without knowledge of
   Kermit.
2. The translation, or language, to be used can be chosen by the user at
   any time. This suggests controls at Kermit command level and during a
   Connect session. Both Kermit and KEYBHE can do this job.
3. The host can also choose the translation, by sending an appropriate escape
   sequence or similar during a Connect session. Only Kermit can let this
   happen; it implies a coupling between terminal emulator (recognition of
   the incoming message) and the otherwise independent keyboard handler.
   Curiously, Code Pages provide no known change mechanism from a host.
4. It would be convenient to know which translation is in effect at any moment
   but the mechanics need thought (what to display as an indicator). Only
   Kermit would be satisfactory here. Without an indicator the user needs to
   be able to force a particular translation (item 2, but only via Kermit
   since KEYBHE is a toggled mechanism).
5. At most two translations would be available this way: Kermit native and
   "translated."  Additional translations might require exiting Connect mode
   and loading a new table for "translated".
6. Keyboard translations may or may not be coupled with other operating
   conditions, such as screen writing direction, depending on the ultimate
   use of the translations. Macros terminalr/s permit this flexibility.
7. What is missing here? So far MS Kermit/IBM v 2.32 does all these things.
   The only objection to date is a screen blink while terminalr/s operate,
   and that has to occur because these macros can do any non-Connect mode
   operation.]
-------------------------

        There  is   also  Hebrew  screen   support,  that  enables
     right-to-left  writing and some other things.  This software,
     supplied by  IBM here in Israel, is  such an illogical use of
     the computer, that hardly anybody ever uses it.

3) SET TRANSLATION INPUT

        This  command does play some role in Hebrew representation
     of characters for Kermit.    Hebrew  characters  are  usually
     represented in  the computer in a way  that has nothing to do
     with the  character generator used in the  PC.  that way, for
     example, aleph is represented in code 80H in the PC, while it
     can be represented as 60H  in  the  computer,  or  any  other
     imaginable  way.    Using  the  input  translation  for  that
     purpose, is essential.   Setting  the translation  table off,
     re-establishes the link between host character representation
     and the PC character  display.   This is  the usual  state of
     things.   However, it cannot be  utilized by the software, as
     there is  no direct means of setting in  off and on.  This is
     now   changes  with  the  addition   of  the  \TERMINALS  and
     \TERMIANLR macros.

-------------------------
[from jrd - Kermit now has two methods of mapping characters destined for the
screen: SET TRANSLATION INPUT and the VT102 built-in character sets. The
latter is under both user and host control via SET TERM CHARACTER-SET and
escape sequence/Control-O/-N, respectively. Host control of SET TRANS requires
the terminalr/s macros. SET TRANS is the more flexible approach by mapping
all 256 patterns of input characters; it does require an exit from Connect
mode to change its on/off state; an escape sequence pair could be added to
change it from the host side.]
-------------------------

4) Hebrew software

        Hebrew software, and mixed Hebrew/English software exists.
     The whole  issue of a Hebrew Kermit  has arisen form the need
     to   provide  a  general  way  of  doing  remote  editing  of
     Hebrew/English  text, without resorting to a specially abated
     terminal that  supports Hebrew  screen functions,  but rather
     use an existing PC with the proper handling.  Also, Hebrew is
     implanted into  the terminals (in  most cases) in  such a way
     that  almost   every  screen  function   ceases  to  function
     correctly  in the right-to-left direction.   Those that still
     work,  do work because the Hebrew firmware change has nothing
     to do with those function.

        VIH is the Hebrew adaptation of the wide spread VI editor,
     used  with the UNIX(tm) operating system.   On the DEC, there
     is  a TPU/HEBREW that enables Hebrew editing, but I am unsure
     as to the amount of Hebrew terminal capabilities used.  ALEPH
     is a borrowing system for the universities that also works in
     Hebrew.  This software is connects nation-wide through DECNET
     links.   Other software exists, and are usually an adaptation
     of existing programs for Hebrew.  It should not be to hard to
     modify  TPU to use  \TERMINALR and \TERMINALS  macros.  Other
     software might be tedious  to  change,  some  might  even  be
     impossible.  Only a well defined extension might catch enough
     interest as to prompt future use of it.

-------------------------
[from jrd - "Here there be tigers!"  One of the outstanding questions is
what the host side does and expects of the "terminal." I surmize that there
seems to be really no firm convention and programs probably follow one another
with a time lag and add local embellishments. I would really like to know the
answer(s) to that question before inventing yet another convention.

        The elements of the questions are, clearly, which symbol to display
for arriving bytes, control of the cursor (including the automatic
displacement implied by writing a symbol), and which byte codes to send to
the host when keys are pressed. In principle at least, the host could do
all of this work if the communications channel allowed the likely 8-bit char
codes to pass unmolested and the host knew what symbol would appear for
each such character code. The host could translate arriving (English ascii?)
characters to appropriate national ones for storage/echoing, and even sense
commands to change translations. Baruch's discussion leads me to believe the
host software is in disarray (polite word meaning an awful mess). If disarray
is an accurate description then flexibility in MS Kermit could be a great
service to many people, provided we are very careful.]
-------------------------

5) Other languages, generalization, and some personal thoughts

        When I  first became  involved with  this project,  is was
     intended to  be a practical  solution to a  special problem -
     that of using VIH  on a  PC terminal  emulator.   The initial
     approach was to  use the  supplied DOS  modifications (KEYBHE
     and  HEBREW, the  Hebrew screen  handler), and  some existing
     communication program, and have the hack work.

        Being impractical,  I was  thinking of  an elegant  way of
     doing it.  What I ended  with  was  an  approach  that  would
     enable not only  Hebrew, but  other languages  as well.   The
     screen functions were  extended  in  a  logical  way  to  the
     current  VT102 ones.   This, as far  as I know,  is the first
     time this  extension was made  within that logical  frame.  I
     also devised the concept of two translation tables, as to not
     cause any change to the current Kermit functionality.

        My way of doing it is not the only one possible, and there
     are may others.   The end  result should be  the same: Kermit
     should be able  to handle  national character  sets, keyboard
     and  displays, as efficiently as  possible, causing as little
     burden to the user as possible.


        Baruch Cochavy
        IIT, Technion, ISRAEL
----------------------------------------------------------------------------
From:   IN%"baruchc@TECHUNIX.BITNET"  "Baruch Cochavy" 26-NOV-1988 13:08
To:     jrd@cc.usu.edu
Subj:   Hebrew, Kermit - more remarks

Date: Sat, 26 Nov 88 22:09:45 +0200
From: Baruch Cochavy <baruchc@TECHUNIX.BITNET>
Subject: Hebrew, Kermit - more remarks
To: jrd@cc.usu.edu

Hello Joe,

I understand your point of veiw, as implemented in the test version of
Kermit  you sent me. While that way of doing the national  translation
does answer some of the questions, I still feel my original implementation
is more well adapted to handling the case in hand. Here are some further
remarks:

        While  the state I described is  not most likely to happen
     for  each and every line, it does serve as an example to what
     might happen.

        There  is no such thing as a 'Hebrew computer'.  But there
     *are* Hebrew programs, that use Hebrew or Hebrew/English mix.
     Adapting Kermit for  these  program  demands  that  no  prior
     assumptions  be made  as to  the nature  of screen  access it
     does.  Doing so,  might  result  in  one  program  performing
     acceptably (from screen behavior  point  of  view),  and  the
     other  one unacceptable for anyone to use.  One cannot impose
     a  certain solution on an existing program, but rather supply
     the  required tools  for it  to work,  along with  a 'proper'
     mechanism for doing things the right way.  This way, existing
     programs would  work   flawlessly,   and   new   ones   would
     (hopefully) be written  as  to  make  good  use  of  the  new
     features.

        A good example  for that is  VIH.  VIH  originated for the
     standard UNIX(tm) editor, being changed to support Hebrew and
     English text editing.    Hebrew  can  be  mixed  freely  with
     English.   Any line can have any possible mix.  To accomplish
     this task, a special adaptation  of  VI  took  place.    This
     included definitions  for Hebrew  capable terminals,  that do
     exist,  and also a  special TERMCAP entry  for that terminal.
     Some  of the changes were using some of the Hebrew terminal's
     special functions, some  utilized  software  emulation  ofred
     functions to  accomplish the task, as  the Hebrew terminal is
     NOT a natural extension  of  VT102  function  set  of  screen
     handling commands.

        VIH if not the only program  written  to  make  use  of  a
     Hebrew  terminal.  There are many others, though the names do
     not strike me  right now.   One of the  others is TPU/HEBREW,
     and adaptation of TPU for Hebrew.  This program runs on a DEC
     computer.  Also,  the  central  library  has  a  computerized
     borrowing  service, called ALEPH, that uses Hebrew terminals.
     This system works through the national DECNET networks, and a
     student can phone  to the  nearest node,  and have  the books
     borrowing period extended.

        Again,  as no  assumption could  be made  as to  the exact
     nature of use a program  may  exercise,  no  such  assumption
     should  be made upon the Kermit  implementation.  As I stated
     before,  I feel that forcing Kermit out of CONNECT mode to do
     national  language switch is unacceptable  in terms of speed.
     Execution  from memory IS  fast, but not  nearly fast enough.
     Imposing such  a burden on the  user's shoulders would result
     in  a waist of time, since no user will ever use the features
     we are  so delicately trying  to trim, and  after all, user's
     benefit should be the measure.

        Stated  again, I repeat that Hebrew  is not the only issue
     on  hand.  One must consider other  languages as well.  It is
     wrong to assume  that  the  exact  way  the  Hebrew  language
     character  translation is accomplished, the same way would be
     any other language.   Being  in  the  range  80H-9AH  on  the
     character generator is  pure accident.   Other  languages can
     have  their   characters  spread   all  over   the  character
     generator.   Using the  range a-z  for Hebrew  characters (or
     national  characters) is also just one  example of how it can
     be  done.  That way  (IBM's) is not even  the CORRECT one, as
     there  exists an ISO8859-5 that  defines the Hebrew character
     set standard.   It is  a pity that  no one ever  used it, but
     this is  unlikely to happen,  if one considers  the amount of
     time that will  be  required  to  do  the  changes  to  every
     existing Hebrew program.

        I do feel that using two translation tables, and providing
     the  mechanisms I provided in my  Hebrew Kermit are more then
     enough to  answer the problems.  Now,  that a macro table has
     been   tested,  I  would  suggest  that  the  case  of  other
     mechanisms be considered.

-------------------------
[from jrd - On the item of two keyboard translation tables. There would need to
be two complete sets of keyboard translations, including the strings and
"verbs", rather than your hardwired mapping of raw ascii key codes 60h-7ah to
80h-9ah plus some punctuation revisions. We would also need to permit the Scan
Code class of keys to participate. The particular translations would vary
from country to country and user to user. Further, there would need to be a
switching mechanism attached to both user and host controls. The host control
would need to be wired into the terminal emulator to avoid leaving CONNECT
mode. And the standard SET/SHOW KEY system would then need a table selector.
Needless to say, the size of the keyboard translator, file MSUIBM, would grow
greatly from much extra code and the duplicate set of data structures.
        Finally, just how would one decide which host controls result in
changes to keyboard translation and must other items (e.g., writing direction)
change at the same time? We could add other escape sequences for the host
to use, even though that makes Kermit more specialized. In short, the coupling
problem is too diverse to hard code.
        My opinion is the duplicated data structures plus management code
become unwieldly, the selection process by either host or user becomes
complicated, and foremost, the joint action/coupling business has no fixed
solution.
       To belabor a previously stated point, the terminalr/s macros have the
capabilities to modify the keyboard translation jointly with other system
changes, but there is no easy way to restrict them to only CONNECT mode
related activities. In fact, another person wants to use them as a way of
exiting Kermit completely when the host sends its logout text. Given that any
Kermit command can appear in the macros we simply have to exit CONNECT mode,
and that in turn results in the screen save/restore blinking. An alternative
is to run all MS Kermit screen output through the terminal emulator and thus
never leave it. That entails a major rewrite of all the MS Kermit source code
for all variations of machines. I think there are at least some things the
host software can do to aid multi-lingual work without invoking terminalr/s
and hence avoiding the blinking; that depends on the writers of host programs.]
-------------------------

                    It were not best that we should all think
                 alike; it  is  difference  of  opinion  that
                 makes horse races.
                                                   Mark Twain

------------------------------

Date:  Sat, 26 Nov 88 18:33 MDT
From: <JRD%USU.BITNET@cuvmb.cc.columbia.edu>
Subject:  More on the translation stuff
To: Baruchc@TECHUNIX, fdc@cunixc.cc.columbia.edu, JRD

Baruch and Frank,

        An addendum to my remarks earlier today: a personal view.

        I am still very concerned about constructing special mechanisms in
MS Kermit, such as extended escape sequences and terminalr/s, which invent
a separate way of doing business. The reason is that the MS Kermit to host
connection is a two sided affair, with the host needing some information
about the local side's capabilities. The way we are proceeding now invents
a new terminal, using the standard DEC terminal identification strings, with
screen and keyboard modifications/changes which are unknown to the host.
I've stated many times that almost all the new bells and whistles in v2.32
can and probably should be accomplished by the host, with an assumption of
a standard VT102/VT52/Heath-19/Tek4010 terminal at the microcomputer end.
Those standards are widely accepted and are of long standing.

        Needless to say, any host software attempting to employ these special
mechanisms will cause adverse reactions from all persons not using MS Kermit
2.32 and later (and that includes properly setting up MSK 2.32 with tables,
etc). Since the host software needs to be written to use the special features
I'd much rather those writers assume a conventional terminal and accomplish
the goals within their own program(s). Thus a regular terminal or many other
communications programs could operate smoothly in the intended fashion.

        I know that this attitude appears to be reactionary. However, I think
the general user is ill served by specialized fun and games which deviate
widely from standards now accepted or likely will be accepted in the near
future. Thus, while MS Kermit will undoubtedly continue to accumulate features
to solve some local problems I think we should be exceptionally cautious about
modifying the very adequate terminal models and about encouraging general use
host software to be written for those modifications.

        Regards,
        Joe D.

------------------------------

Date: Sun, 27 Nov 88 13:28 MDT
From: Joe Doupnik <JRD@cc.usu.edu>
Subject: Multilingual Word Processing
To: fdc@cunixc.cc.columbia.edu, baruchc@TECHUNIX.BITNET

Frank and Baruch,
	Frank - thanks for the reprint of the Scientific American article
"Multilingual Word Processing." That's an excellent introduction to the
business. One thing I spotted right away is what needs to occur when languages
are mixed on the same line but they use different directions of writing. As I 
understand the matter it goes like this.
	Suppose we are starting in Arabic and the cursor moves right to left
as characters are entered. In the middle we wish to insert English words. The
article says that the cursor should then stay fixed on the screen and as new
English characters are entered they cause text to flow away from the cursor
to the left. Kermit will do this if INSERT mode is active, ESC [ 4 h.
	Now the interesting part is what happens to text pushed off the left
margin? The article says it should flow to the next line, assuming that line
wrap is active. Kermit does not do this. It also says to do the job properly
in English for a basically Arabic document (hmmm, more context) when the
initial line fills instead of bumping overflowed text to the next line the
cursor itself should jump down to the next line (right margin) and stay put.
	Let's see if I can reproduce part of the strategy according to the
picture in the article. Let Arabic be lower case here and English be Upper
Case.
starting off with Arabic:
								     dcba
caret is the cursor                                                 ^
now add some ENGLISH:
							   THE UNITED dcba
                                                                     ^
					            THE UNITED STATES dcba
                                                                     ^
Then, eventually we reach a full line status. Adding more words should do

						    THE UNITED STATES dcba
								OF AMERICA
								 	 ^

rather than
							    OF AMERICA dcba
								      ^
							  THE UNITED STATES	


	Pretty complicated to think through unless one remembers to scan for
the native starting point of a language and read from there. Clearly, we have
broadened the concept of a line to include multiple lines in some systematic
fashion. Kermit could, but does not, wrap DISPLACED material to a new line
because that gets involved with subsequent line overflows and joining down
the whole document (whatever "whole" means). Thus, the simple line wrap and
character insert concepts of a VT102 are obviously insufficient to do word
processing. With a VT102, inserting material into an already full line
results in loss of material pushed beyond the terminating margin, wrap or no
wrap. Under MS Kermit 2.32 this is still true, for both directions of writing.
Since a VT102 is really a character oriented (oops, wrong word here) device
true multilingual support needs programs organized to consider words (word
wrap) and lines (line folding in two languages) and so on for hundreds of KB
of code, with both ends reasonably cognizant of the capabilities of the other.
	A lesson for me is to use the "terminal" as a character device and
let the host end manage the screen intelligently with its own massive set of
software. Attempting to do the job at both ends becomes complicated and adds
unknown bulk on the micro side.
	A second lesson for me is the eventual requirement for Kermit, among
other programs, to support text characters with multiple byte codes. Goodness,
my Lattice C compiler does this already for certain Asian languages.
	Third lesson for my end is: before launching bright ideas in code, get
a firm set of specifications and think about them.
	Thanks again for the article,
	Joe D.

------------------------------

Date: Tue, 29 Nov 88 00:16:36 +0200
From: Baruch Cochavy <baruchc@TECHUNIX.BITNET>
Subject: More remarks: Kermit, national characters, keyboard handling
To: jrd@cc.usu.edu

Hello Joe,
     
        Thank you for your remarks. I would like to read that article
myself, but I do not have the exact pointer to it. Can you please supply
me with the vol/number of the issue ?
        I feel you are right about considering national languages from
host viewpoint as well as terminal's. However, as you noted, one must be
very careful about it. If we are to bring some progress, we must face
the existing limitations, that is, no one will ever change his software
to help us make Kermit simpler. Supporting existing programs, and
providing a logical growing path are the only way we can hope to achieve
better programs and better national support.
        Modifying Kermit to support right-to-left direction is a good
example. You can say 'let the host do right-to-left', but then, no one
will ever use Kermit for Hebrew (Arabic) software, even if national
character sets are supported. Also, the current way VT102 emulation was
extended might be argued. Again, this extension was made in accordance
with the limited extension already done to support Hebrew. Extending
Kermit to support other concepts of screen handling for national
languages and right-to-left direction will most likely result in waste
of effort, since no one will bother trying to modify existing programs.
        What comes is that the extension has to be made in light of
previous work. This is the exact way I did it. Every existing program
that support Hebrew on a VT102 terminal will work with the proposed
extension, while new ones can be written to take full advantage of the
new features.
        There is a trend toward multi-byte characters. I do not feel that
issue should be addressed for the time being, since it will probably
require Kermit be re-written, not just modified. The case is, however,
supporting national keyboards and character sets. Here, again, we must
be practical, and face current limitations. Supporting national
keyboards including alt-(national), ctrl-(national) and the like is
nice, but would impose such burden on the software as to prohibit its
implementation. So, considering what we can do, I suggest a second
translation table should be put before the current keyboard translation
table, so that normal national keyboard can be supported, at least.
        I do not oppose the TERMINALR - TERMINALS feature. I do feel
that I will have hell of a time persuading a user to use Kermit on a
daily basis facing such a behavior. It is not the cost one has to pay
for such flexibility that would bring one mind to peace on that matter.
Given the current state, and knowing the intended audience, I fear no one
will ever use those features.
        What I suggest is that: is the cost of having a second 256-byte
translation table to high ? I do like the TERMINALS/R concept, and like
to keep it, however.

        Regards,
        Baruch.
     
------------------------------
                                                                                                                                                                                                                                                                                                                                                         