Subject: Bug#3253: Pine over-encodes files (came from Bug#932)

From: Ian Jackson <[email-masked]>
Date: Sun, 9 Jun 96 14:29 BST

Package: pine
Version: 3.93-?

Some time ago I submitted a bug report which included the complaint
that Pine base64-encodes text attachments, so that Pine users
frequently send mail which the recipient's non-MIME-aware mailer can't
understand - despite the fact that MIME was carefully designed to
allow you to send messages which can be processed both automatically
by a MIME-aware mailer and manually with a simple cut-and-paste.

The package maintainer complained that this bug (#932, now
long-closed) contained many separate bug reports; so I closed it and
submitted (or so I thought) an number of replacement reports.

This particular one, however, got sent to debian-devel rather than
debian-bugs by mistake.  I've just noticed this because I wanted to
refer someone to the bug report.

When the bug system and debian-devel and everything are working again
I'll attach to the logs for this report a huge digest of all the
correspondence that has been had on this issue, so that it can be
perfectly clear to everyone what's going on.

This bug has been reported numerous times to the Pine developers by
practically everyone who has had to deal with the bad consequences of
this behaviour of Pine feature.  The Pine developers think it is a
feature.  I shall stop this message now before I become very rude
indeed about them.

Ian.

---

From: Dale Scheetz <[email-masked]>
Date: Wed, 12 Jun 1996 15:23:02 -0400 (EDT)

Ian, Ian, Ian ... As you indicated in your post, this topic has been
discussed at great length.
I know your position on the subject backwards and forwards.
It doesn't matter!
The Pine Developers have made themselves equally clear on this subject.
They are not going to change this "feature".

My advice is that you could use your energy to more effect by teaching
everyone you contact on the subject that they should not use the
attachment feature of pine to send plain text. Control R is the "correct"
method for including plain text in an e-mail. The more users who
understand this issue, the fewer "broken" email messages you will recieve.

Unless you can show some compelling argument for leaving an unfixable bug
in the bug reporting system, I will mark this bug done (well, at least
dead).

Later,

Dwarf

------------                                          --------------

aka   Dale Scheetz                   Phone:   1 (904) 877-0257
      Flexible Software              Fax:     NONE 
      Black Creek Critters           e-mail:  [email-masked]

------------ If you don't see what you want, just ask --------------

---

From: Juergen Menden <[email-masked]>
Date: Thu, 13 Jun 1996 08:18:58 +0200 (MET DST)

> They are not going to change this "feature".

well, before you close this bug, can you please tell me what
stopps us from changeing this behaviour in our diff file?

i thought we all agree that this behaviour, although the 
upstream developers call it a feature, should be considered
a bug?!

jjm

ps: if this is question is already discussed in length too,
i apologize. sometimes i'm not able to read every single
msg on debian-devel... ;-((

-- 
Juergen Menden                   | Disclaimer: The opinions expressed by me, 
tel:    +49 (89) 289 - 22387     +-----------+ are (usually) not the opinions 
e-mail: [email-masked]     | of anyone else on this planet.

Hi! I'm a .signature virus!  Add me to your .signature and join in the fun!

---

From: [email-masked] (Erick Branderhorst)
Date: Thu, 13 Jun 1996 07:44:24 +0100

> understand this issue, the fewer "broken" email messages you will recieve.
Is it possible to configure smail, or some mail engine so that it returns
such kind of messages with an appropriate explanation how it should be done.
Or will this increase net usage enormously?

Erick

---

From: Dale Scheetz <[email-masked]>
Date: Thu, 13 Jun 1996 07:18:11 -0400 (EDT)

> stopps us from changeing this behaviour in our diff file?

I assume you mean modify the source to change the behavior? There are two
reasons for not doing this. First, it would create a version of Pine that
was vastly different than all others versions in existance. This would not
fix the problem of recieving such mail, as all the other versions of Pine
would still have this behavior. Second, just how would the feature be
crippled? Remove the attachement facility entirely? If not, how does one
determine whether mime encoding should occur? Must the message be 7 bit
clean?

> upstream developers call it a feature, should be considered
Well ... some (not all) do agree with you, but others don't. If everyone
agreed with you, there would be no use of this feature and it would not be
a problem

> msg on debian-devel... ;-((
This has been discussed at length, but a quick rehash for new maintainers
is probably a good idea ;-)

Luck,

Dwarf

------------ If you don't see what you want, just ask --------------

---

From: "Brian C. White" <[email-masked]>
Date: Thu, 13 Jun 1996 10:58:47 -0400

> stopps us from changeing this behaviour in our diff file?

I think having different behavior on our system compared to other
distributions is a "bad idea" (tm).

> upstream developers call it a feature, should be considered

No, not everyone.  I agree with how Pine does things.  If you
want to attach something then it goes through mail encoded such
that absolutely _nothing_ can be changed.  Plain-text can be
altered.  If you want to send plain text, then simply include
the file instead.  Makes perfect sense to me.

                                        Brian
                               ( [email-masked] )

-------------------------------------------------------------------------------
    In theory, theory and practice are the same.  In practice, they're not.

---

From: Lars Wirzenius <[email-masked]>
Date: Thu, 13 Jun 1996 17:34:19 +0300

> > Control R is the "correct" method for including plain text in an e-mail. 

That depends.  Attachments are easier to handle for people with mime'd
mailers.

> Or will this increase net usage enormously?

It's a bad idea.  It will _really_ annoy people.

(The single most important problem with MIME is that people fight so much
about it.  MIME may be big, it may be complicated, but it's a standard,
and it does make things better in the long run.)

---

From: Dale Scheetz <[email-masked]>
Date: Thu, 13 Jun 1996 13:25:06 -0400 (EDT)

> That depends.  Attachments are easier to handle for people with mime'd

I strongly disagree. Pine is a mime'd mailer. I use Pine. I hate getting
mail with attachments! Attachments are not easier to handle! Any other
mail I just press enter off the index list (or 'N'ext) and I can read the
posting. But with attachments you first have to ask to view the file, and,
most often, then must save the file before you can view it (outside the
mail agent) because many text files are improperly identified.

I must point out that there are several Debian maintainers who submit
their upload announcements as attachments. I find this so obnoxious that I
just delete such mail, out of hand.

> and it does make things better in the long run.)
Just because it is a standard, doesn't mean you should use it in a
universal fashion. The RS232 standard does not apply to parallel ports.
Neither should MIME encoding be used on plain text files, unless the file
is a like a diff file and must get through filters, character intact.
It is only that task that mime was designed to solve. It's indescriminate
use has caused more problems than it was supposed to solve.

DON'T MIME ENCODE PLAIN TEXT DATA! Specialy if your intent is to
communicate with a human at the other end. (Or a reasonable approximation
thereof ;-)

Later,

Dwarf

------------ If you don't see what you want, just ask --------------

---

From: [email-masked]
Date: Thu, 13 Jun 96 13:44 EDT

Dale> I must point out that there are several Debian maintainers who submit
  Dale> their upload announcements as attachments. I find this so obnoxious
  Dale> that I just delete such mail, out of hand.

You will get flamed for this, and rightly so. 

All that these maintainer, your's truly included, do is to use mailcrypt to
PGP sign a posting. Emacs doesn't even know what an attachment is. Too bad
for pine, or rather the pine user, if this encoding isn't recognised
properly. 

We won't get anywhere with this discussion. We heard it all before. Pine
exists, and some users will want it. Such is life. Let's not unload our
frustation with an upstream design issue on the Debian maintainer. We all
have things to do.

-- 
Dirk Eddelb"uttel                             http://qed.econ.queensu.ca/~edd

---

From: Guy Maor <[email-masked]>
Date: Thu, 13 Jun 1996 12:47:18 -0500 (CDT)

> mail agent) because many text files are improperly identified.

Tell pine to pipe the file to cat.

Guy

---

From: Richard Kettlewell <[email-masked]>
Date: Thu, 13 Jun 96 18:55:12 +0100 (BST)

>posting.

I would argue that this is poor user interface design in Pine, not an
inherent feature of attachments.

>communicate with a human at the other end. (Or a reasonable
>approximation thereof ;-)

It looks, from reading this, as if you've seriously confused MIME and
BASE64.  Please tell me this isn't true...

I can see why Pine might lead one to believe that MIME and BASE64 were
the same, given that it is so resistant to doing anything with MIME
without also BASE64ing everything in sight.

- Richard (who thinks Pine should MD5 and then BASE64 its message ids
           - they'd be shorter...)

-- 
http://www.elmail.co.uk/staff/richard/
GCS d- s+:- a-- C++ ULVS+++$ P+++ L++ E++ W(++,--) N(++,+) o? K w---
O? M- V? PS(+,+++) PE Y+ PGP+ t- 5++ X+@ R tv--- b++>++++ DI+ D+ G e++
h r% y++

---

From: Dale Scheetz <[email-masked]>
Date: Thu, 13 Jun 1996 14:53:33 -0400 (EDT)

> for pine, or rather the pine user, if this encoding isn't recognised

Well, Dirk, you have misunderstood. I have no problem with pgp signed
posting, they pop right up, no problem. We were talking about attachments
and mime encoding. 
There is no need to mime encode the changes file as the check sums are
already composed of a tamper resistant character set and will come through
fine.

> frustation with an upstream design issue on the Debian maintainer. We all
Thank you for your productive position.

Later,

Dwarf

------------ If you don't see what you want, just ask --------------

---

From: Buddha Buck <[email-masked]>
Date: Thu, 13 Jun 1996 16:01:56 -0400

> inherent feature of attachments.

I concur.  My mail agent (exmh) is also a MIME complient mailer.  
MIME-encoded mail is displayed, inplace, or piped to the appropriate 
program silently.  It will (properly) encode text files as text/plain, 
and offer me the choice of encoding methods, including unencoded.  I do 
not see properly labeled quoted-binary encoded parts as mangled, and 
even alternate fonts are handled transparantly (as I discovered when 
reading a list that had someone's name, in hebrew, in their .sig file).
> >use has caused more problems than it was supposed to solve.

MIME was designed to solve a number of problems.  One was the 
transmission of binary media.  Another was the transmission of text in 
different character sets.  And so forth.  MIME was designed from the 
start with the ability to transmit safely text files.  That is why 
text/plain and text/rfc822 are -required- MIME types.  That is why no 
encoding is a permitted option.  

Please note that if you send a file with a binary attachment in Pine, 
the entire mail message -is- MIME encapsulated, as multipart/mixed, the 
first part of which is MIME encapsulated as text/plain with no 
encoding, the second part of which has BASE64 encoding.  So my question 
is:  If Pine can generate the first part as text/plain with no 
encoding, why can't it the second?

I don't use Pine myself.  It has been suggested that ^R is a reasonable 
alternative to attaching a text file.  Does ^R automatically protect 
the included text from having leading or trailing blank lines deleted?  
This was recently important to me when I had to send a bug report to 
Ian Jackson that included a file with a trailing blank line.  I sent 
it, MIMEd, text/plain, unencoded.  If I used the analogous "insert text 
file" feature of my mailer, I would have had to place "cut here" lines 
before and after the file to make certain that the blank line (which 
might have been important in this case) was preserved.
> >approximation thereof ;-)

As I said above, if I attach a binary file, I have absolutely no choice 
BUT to MIME encapsulate plain text data, since one part of my multipart 
message is plain text, sent as text/plain with no encoding.  There is 
nothing wrong with -properly- MIME encapsulating plain text data.  In 
some cases, it is vitally important for proper communication (imagine 
if I were using the character set iso-2022-jp, which is a  7-bit 
Japanese encoding designed for MIME.  The RFC (1468) that defines it 
states that Base64 or Quoted-Printable encoding would render it 
unreadable.  Without MIME encoding, many readers would not be able to 
properly identify it as non-ASCII.  I hope you don't claim that 
Japanese language documents are -not- plain text.)
> without also BASE64ing everything in sight.

Strangely enough, the loudest complaints I hear about MIME come from 
either a) people without MIME-compliant mailers (who could be 
considered to have a semi-reasonable gripe), and b) Pine users or 
people dealing with Pine users.  

-- 
     Buddha Buck                      [email-masked]
"She was infatuated with their male prostitutes, whose members were
like those of donkeys and whose seed came in floods like that of
stallions."  -- Ezekiel 23:20

---

From: Dale Scheetz <[email-masked]>
Date: Thu, 13 Jun 1996 16:28:27 -0400 (EDT)

> without also BASE64ing everything in sight.
Well, Base64 IS the encoding scheme used by Mime, right? The confusion is
over when an attachment (I read MIME here, right?) should Base64 encode a
file to be attached.
The Pine developers say: Gee, they want this included as an attachment,
they must need to protect it, so, let's encode it! If they didn't want it
encoded they would just have inserted it into the message!
There are, obviously, those in the group who object to this point of view
by the developers.
My primary advice is, if it hurts when you do that, then don't do that!
Seriously, use control R unless you REALLY want Base64 encoding. Then and
only then can you make it an attachment.

For those of you who can't deal with this solution, I can only say:

You and the Pine Developers go to neutral corners, and only come out
fighting after I have left the room. (I don't like religios wars)

Later,

Dwarf

------------ If you don't see what you want, just ask --------------

---

From: Lars Wirzenius <[email-masked]>
Date: Thu, 13 Jun 1996 23:00:39 +0300

[ Note: I read this mailing list.  There is no need to CC me on replies,
  unless it is _really_ urgent.  I pay for my PPP connections.  Thanks. ]

> mail agent) because many text files are improperly identified.

Pine is a fair first attempt at a mime'd mailer.  It's not a really good
one.  It's inability to show uninterpreted mime'd letters can be a major
pain in the ass when dealing with MIME types it doesn't know about it.
However, at least the versions of Pine I've used or tried (3.91-3.93),
do show attachments as well.  Even base64 encoded ones -- I just tried
it.  That is, if the attachment is marked "text/plain" -- if it is of
type "application/octet-stream" (i.e., binary data), then Pine doesn't
show it.

Anybody sending plain text attachments as "application/octet-stream" are
doing something wrong.

FWIW, I use exmh.  It handles MIME somewhat better than Pine.

> their upload announcements as attachments. 

I've never even noticed. :)

> Neither should MIME encoding be used on plain text files, 

I assume you mean base64 encoding.  I agree with that (and the great
thing with exmh is that asks which type want to make things).  I don't
agree with not making it an attachment, if it is a separate part.  If it's
something separate, not part of the message, I don't want to have to hack
and slash at the mail file to extract it into a file.  I prefer to just
click once or twice with the mouse.  _That_ is the reason why MIME is a
Good Thing (all the flames about base64 vs uuencode and quoted-printable
notwithstanding).

(I'll quit defending MIME now.  It's not really interesting to anyone.
Besides, it's better to spend the energy on getting 1.1 out.)

---

From: Buddha Buck <[email-masked]>
Date: Thu, 13 Jun 1996 17:11:24 -0400

> Well, Base64 IS the encoding scheme used by Mime, right? 

Not really.  MIME currently defines at least three encoding styles, 
BASE64, Quoted-Printable, and unencoded.  Base64 is AN encoding scheme 
used by MIME.

>The confusion is

MIME doesn't encode attachments per se, MIME ecapsulates whole mail 
messages.  When an attachment is made, MIME encapsulates the entire 
message into a singe MIME message of type "multipart/mixed", which 
allows for multiple, relatively independant, embedded MIME encapsulated 
messages.  From experience I have with my mailer, if I were to attach a 
GIF file to this message (which I won't), the entire message would be a 
"multipart/mixed" message, the first part of which would be 
"text/plain", with no encoding (that you would be reading right now), 
the second part would be "image/gif", BASE64 encoded, and the last part 
would be "text/plain", unencoded, and would consist of my .signature 
file (and would -not- be attached as a downloadable file).

> The way attachemnts

Sometimes there are times when that is also unacceptable.  MIME allows 
different parts of a multipart message to have different character 
sets.  Perhaps I want to discuss two text files, one in Hebrew (or with 
embedded Hebrew characters), and one in Japanese (or with embedded 
Japanese characters), with interspersed commentary on the two 
selections in English (say I'm corresponding with a colleage, 
discussing relevant linguistic or political issues).  MIME allows me to 
do that, WITHOUT Base64 encoding the two files in alternate 
charactersets.  (Although I will admit that my mailer doesn't display 
ISO-2022-JP encoded text files properly.  I can embed them, however).
> fighting after I have left the room. (I don't like religios wars)

Some of us have needs that were forseen (or at least dealt with reasonably) by the MIME designers that are ignored by Pine.  Some assumptions made by the Pine developers (like all attached files are binary files, regardless of if they are or not) don't meet our needs.

For the most part, I have sat back and ignored the MIME flamewars because MY mailer (mostly) does the Right Thing with MIME.

---

From: [email-masked] (Kai Henningsen)
Date: 14 Jun 1996 22:01:00 +0200

> No, not everyone.  I agree with how Pine does things.  If you

Not everyone, but nearly everyone who knows much about Internet mail - I'd  
say close to 90%. Maybe more. (It's not the only severe design bug in  
pine. Another one is the  handling of mail with newsgroups: headers.)

> the file instead.  Makes perfect sense to me.

Makes no sense at all to me. If that made sense, why not encode  
_everything_ as base64? What is it about attaching that necessarily makes  
possible changes so disastrous that you are willing to ignore  
interoperability?

IMHO, the _only_ sane way to do this is somewhat similar to the way I've  
seen it with pmail.

This is how pmail does attachments:

1. Locate the file.
2. Ask user about file type (include at least choices of "text", "binary"  
and "unknown").
3. Ask user about possible encodings (include at least choices of "none",  
"base64" and some sort of type-dependant default; pmail also offers stuff  
like uuencode and mac-specific encodings)
4.Attach with content-type: made from 2 and content-encoding: made from 3.

Simply _assuming_ answers to 2 and 3 is just plain stupid. Note that I  
would consider it just as stupid to attach everything as text/plain with  
no encoding at all.

MfG Kai

---

From: [email-masked] (Kai Henningsen)
Date: 14 Jun 1996 22:55:00 +0200

> Well, Base64 IS the encoding scheme used by Mime, right? The confusion is

Wrong.

Base64 is _one_ encoding scheme used by MIME. There are three: base64,  
quoted-printable, and none.

Furthermore, those encoding schemes are about the least interesting part  
of MIME.

The most important thing there is in MIME is content labelling. Then,  
IMHO, comes multipart messages. And only then encoding.

> encoded they would just have inserted it into the message!

And this is, as I have already pointed out in another message, just plain  
stupid.

> only then can you make it an attachment.

Understand about MIME. There is absolutely no reason why an attachment is  
any different from your "main message".

In fact, MIME doesn't even _have_ the concept of "attachments". That's an  
envention of systems like MS-Mail.

MIME has multipart messages. All those parts are equal. Each one can be  
plain text, each one can be a GIF, each one can be an included mail, and  
so on - and each one could as well be different.

The mail user agent should not presume to know what you want to do.

> fighting after I have left the room. (I don't like religios wars)

This is just as much a religious war as when someone insists the correct  
spelling for a certain continent is "Ameerika" - i.e., NOT.

MfG Kai