
Simlis v2.0 -- Similarity sort.

Usage:
  Simlis [-dpvrfm] [-snn] [-cx] [a[nnnn] [-o...;...] [-lx..] [-wddd[,DDD]]
         [gddd,ddd[,pp]]                          < infile [{>|>>} outfile]

Options: 

 -d   :displays similarity score (0..1000..) as output line header.
       This value may be higher than 1000 if options -a.. or  -f are activated.

 -snn :skips assumed (fixed length xx) header data in every line.

 -p   :makes process item position sensitive:
      Simlis takes into account the relative item position within the lines.
      Assume 2 lines:
      "dog bites man"   and
      "man bites dog"
      only the -p option would see a difference (lower grade similarity).

 -cx  :everything before first character x (if present) within a line
      will be ignored.

 -o....;...;...  :strings given here are ignored in evaluation,
      max. 10 strings, max 14 chars. each.

 -lx.. :uses given character(s) x as line delimiter(s) generating 'phrases' 
      from it, ignoring physical separators CR LF on input and on 
      output. 

 -v   :operates case sensitive. Default is insensitive.

 -wddd[,DDD] :range of byte values to process, given as decimals,
      possible maximum is -w2,255. Characters outside will be ignored.
      Default range is -w65,255 (65 is character 'A'). 
      Exception: linefeed control characters (-w10) will be omitted because
      of their special meaning on input.
      For full binary operation set -v too.
      Hint: setting values below 33 may reduce selectivity.

 -a   :applies a combined pattern length correlation algorithm, best to 
       explain by 2 example lines:
       "all the day everything is ..." and
       "every day all the thing is ..."
	
       "everything" is rated higher than "every...thing..",
	what would not be the case without that option.

 -r   :shows input byte address (position in input stream) on every sorted
       output line, enclosed in '#' chars.  Using this, data can be better
	retrieved within the original data set.

 -a9999:sort based on the longest common substrings.
        Outputs the longest substrings that are occuring multiple times
	on input. As these substrings are showed without their
	context it is recommended to set the -r option too to get
	the appropriate byte position from the input stream.
	Every string shown is contained at least in each previous line,
	given there with its byte position.
	
 -f   :activate line length correlation,
       a substring within a short line will be rated higher than that same
       string in a longer line.
 
 -m   :applies the Metaphone algorithm on the input lines prior to further
       Simlis processing. This algorithm, created by L.Philips, as published in
       "Computer Language" Dec.'90, returns a rough approximation of how 
       English words sound.
        
 -gddd,DDD[,pp] :generates pseudo records of variable lengths from input
       stream.  Lengths will be choosen randomly in the range ddd..DDD.
       The optional value pp (1..99) gives the percentage of stream data 
       to be processed whereas these are selected from the whole data set
       randomly without overlapping.
       Given unstructured input data or data with unknown properties so far, 
       using this option it may be possible to subject the results to further
       statistical examination for recognizing any hidden regularities.
       The percentage value allows to take samples from huge input files.
       

Executable:
   simlis:   LINUX version
 
Notes:
   Simlis v2.0 is a command line utility that sorts input data depending
   on the similarity of the items contained (words, character/byte sequences,
   etc). The program works independently of item positions, element length
   and sequence (if not changed by options). It accepts physical lines (CRLF
   separated), unstructured data stream, or 'phrases'  (e.g. sentences,
   records) automatically generated based on selected delimiters (in the
   following simply called "lines").
   It takes every line from standard input, evaluates the similarity in 
   relation to all other lines and inserts it in the appropriate line 
   position of standard output.  Simlis is well suited for sorting address 
   lists, directories (identifying identical or similar entries in a way not 
   available from "normal" sort filters), and even for cluster-, literature -,
   linguistic -  and genome - analyses.  
   
   In any case output goes to the device stderr (usually the display) too,
   with no regard as to where standard output is redirected.

   Simlis sacrifices speed to precision, not using any temporary work file:
   We have kind of in-place sorting here, don't expect the fastest 
   execution imaginable. 
   

Restrictions: 
   Note maximum values for input lines and line length displayed 
   on startup screen.
   

Examples:

 1. To see the effects of options -d -r -p -m -f -v in their different 
    combinations provide the file simpat.txt as input.

 2. For special language analysis capabilities  see example simex.txt.
 
 3. For demonstrating genom data processing  see simgen.txt,
    where options -g -a -a9999 -v etc are used.
     
 4. Some basic examples:
 
 Simlis -d <readme.txt
     readme.txt ist sorted and as header of each output line the
     internal calculated 'similarity score' (range 0...1000) is given.
     This score means degree of similarity to the respective previous 
     and following line.

 Simlis  -pr -w48 <readme.txt >readme.srt
     readme.txt ist sorted, including the contained numerics starting
     with 48 = digit '0'; output goes to file readme.srt.
     -p produces a more logical result (if input is structured accordingly).
     -r outputs the original line position (byte position) from the
        input stream, leading every sorted line, enclosed in # characters.
       
 Simlis -s33 -c: -ofree;freeware;win;windows <00_index.txt       
     If 00_index.txt is a Simtel formatted file list, the lines will be kind
     of keyword sorted, ignoring the (here irrelevant) leading 33 bytes 
     headers. The first colon ':' (if present) within each line  starts
     operation, everything before is ignored. 
     The words "free", "freeware", "win", "windows" are ignored too,
     assuming these word should not have any special relevance for distinction.
     
 Simlis -l.;:!? <ebook.txt
     physical ebook.txt lines are converted to phrases according to
     the 5 stop characters given with -l option.
     All operations work on these phrases thereafter, even output.

     
Status of the program and distribution:
   Simlis v2.0 is Freeware.
   It can be freely distributed in its unmodified form and 
   be included in any software collection such as CD-ROM's 
   but may NOT be sold.  


Installation:
   As you can read this you are ready to run it right here,  
   no need for further installation procedures.


History:
   vers. 2.0:
   New options -f -m -g -a9999,
   progress indicator.
   
   vers. 1.3:
   New option -r
   
   vers. 1.2: 
   New options -l, -w, -a, -v
   16-bit DOS, 32-bit Windows version.
   
   vers. 1.1: for MS-DOS
   New options -c and -o,
   faster operation and other minor improvements.
   
   vers. 1.0: for MS-DOS
   First published version.


Compatibility:
   The same Simlis version should be available for different platforms

Comments, suggestions, requests for information to:
   joda@sdf.lonestar.org
File access:
   ftp://sdf.lonestar.org/pub/users/joda/simlis/20/linux/simlis20.zip


Legal Stuff:
   Copyright 2001, Joachim Dathe ("The author")

The software is provided "as-is" and without warranty of any kind, expressed,
implied or otherwise, including and without limitation, any warranty of
merchantability or fitness for a particular purpose.  In no event shall the
author be liable for any special, incidental, indirect or consequential damages
whatsoever (including, without limitation, damages for loss of profits,
business interruption, loss of information, or any other loss) , whether or not
advised of the possibility of damages, and on any theory of liability, arising
out of or in connection with the use or inability to use this software.





