Subject: Re: Source packaging - alternatives

From: Juergen Menden <[email-masked]>
Date: Mon, 1 Jul 1996 10:22:19 +0200 (MET DST)

On Mon, 1 Jul 1996, Ian Jackson wrote:
> 
> Here's the alternative I've dreamed up:
> 
> * The source package is three files:
> 
>  - hello_1.4-3.dsc
>   DSC= Debian Source Control (suggestions for better extensions
>   welcome).  This contains a dchanges/debian.control-like format, for
>   example:
>     Source: hello
>     Version: 1.4-3
>     Architecture: any
>     Depends: gcc         { standard dpkg control file syntax }

it just as well depends on make, libc, libc-dev, bash, patch, binutils, 
dpkg, tar, gzip, dchanges and probably a real bunch of other things. 
this is simply far to much to reliably keep track of. note that in 
most packages configure is already used, which will find the 
apropriate tools much more reliable. (keep the souce of information 
in one single point :-)

on the other hand, leaving out this information does not have
too much drawbacks. the build will not succeed, and people which
try to rebuild any packages should be able to see what tools they
need.

third: think about the depends/recomends/suggest problems we 
already have. and that is a far more easier issue than a source
file depends.

>   The `Architecture: any' means
>   that the source is arch-independent, but the binary isn't.
>   `Architecture: all' would mean it was a truly arch-independent
>   package (documentation, for example).

this doesn't have to be distinguished. the source description file
should _only_ describe the source, not the binary. whether the
binary is arch-independend or not is in the realm of the debian.rules
file.

> * Issues:
>  - Perhaps we need to think again about these hyphens in the context
>   of source packages, so that we can distribute GNU source tarfiles
>   unchanged.

please do not do this! the file naming convention has changes to
often lately, nearly every one is simply confused about this. do
not start another discussion about this issue right now.

we have some guidelines concerning file name conventions, so
lets consolidate on this issue for at least one year, getting some
experience with it. another discussion about file names should be
delayed up until june next year.

>  - We need a dpkg-source script to unpack the tarfile and apply the
>   diff safely (and it can automatically change debian.rules to be
>   exectutable), but people can do it themselves if they want.  Also,
>   we need a dpkg-source script which runs diff 

supported. we do not really need this, but it might be handy for some
people. 

>   and checks that there
>   are no differences that diff can't handle (deleted files, retargeted
>   links, &c).

a repair: target in debian.rules might handle that things, if
it's really needed and not already done in the configure: target.

>  - We need to store the .dsc somewhere 

well, do we really need this file? md5sums might be handy, but as
all files are gziped a gzip -t might do. Architecture might be 
interresting, but it doesn't hurt if the information isn't there.
the same applies for the Generates: field. 

Depends fields are simply impossible to fill. building a really 
complete list and keeping it up to date and reliable is too 
expensive if you compare it to the improvements one gains of.

and for package-name/version scanning: this is, as far as i can
see, only needed on master, and there is already additional 
information in the overwrite files on master. why don't use them.

about signing: this is only needed to make uploads secure. this
is provided by signing the .changes file. no need to do this again.

still, i do not see the need for this file. :-)

> * As a reminder, the `one file' format I favour at the moment is an `ar'
> archive with members for the control file, diff, original source
> tarfile and optionally signature.

only makes sense if we need a .dsc file.

jjm

-- 
Juergen Menden                   | Disclaimer: The opinions expressed by me, 
tel:    +49 (89) 289 - 22387     +-----------+ are (usually) not the opinions 
e-mail: [email-masked]     | of anyone else on this planet.

Hi! I'm a .signature virus!  Add me to your .signature and join in the fun!

---

From: [email-masked] (Erick Branderhorst)
Date: Mon, 1 Jul 1996 11:07:09 +0100

> advocating this were merely advocating that we should be able to
> extract the original source files somehow from our source archive,
> perhaps in a differently named directory or some such).

I recommend to include the original source file in the archive but let it
keep the directory is this a good one. Let the dpkg-source script / program
move the sources around if needed.  With gnu sources this will probably not
required.
 
>     Generates: hello     { names of binary packages, comma-separated }
>     Data-MD5sums:
>      faa56f7d564b1972f66a2d17ddf97413  hello_1.3-4.diff.gz
>      d2cb670eee141fc08eaa4a794b8b68fe  hello_1.3.orig.tar.gz
>   This would optionally be PGP-signed.  The `Architecture: any' means
>  - hello_1.3-4.diff.gz
>   The Debianisation diff, as we have now.

Ok but I wouldn't use the underscore.

> 
>  - hello_1.3.orig.tar.gz
>   The original source code, reorganised if necessary into a tarfile
>   that unpacks into a subdirectory called hello-1.3 or perhaps
>   hello_1.3.

Unpacking in another directory requires repacking of the original sources
all the times.  If we simply inpack in the <package>-<mainstreamversion>
directory we don't need to do this in a lot of cases.

>   of source packages, so that we can distribute GNU source tarfiles

I'm in favour of thinking about it, discussing and whatever but make the
decission part of a script and the rest variable.  I mean provide the
<packagename> and <version> and <revision> info somewhere in the control
part and let the dpkg-source script manage how files are named if this is
needed at all.  By this we only need to change the script if we feel that
the current names aren't appropriate.

> tarfile and optionally signature.

I think this is the most extensible approach we can think of. 

* The `ar' archive can have a md5sum member (with appropriate md5 in it). 

* It can also be used to store more than one source package, for packages
which are build from more source trees. This archive would have one debian
diff file but we can choose for seperate debian diff files.

* An option might be that patches are appended to the `ar' archive
containing updates, like the updates fed into the bug tracking system on the
m68k compiles.  The updates will not be merged into the debian.diff
automatically but can optionally be applied by dpkg-source on the sources.
This provides a stable (debian-maintainer certified) and a very up to date
(non-original-debian-maintainer certified) source. Patches and up dates will
become available sooner in this way.  The patches can be uploaded only on
course, their external names (not in archive names) are made by dpkg-source
and when they are appended to the original source an their names will be
parsed and the appropriate package archive will get them.

* Symbolic links can be used to provide a consistent non version name for
the source archive.  This is a bit similar to the dosnamed binary packages.
In this way the www interface doesn't need to be updated for every version
change of a package.  One can always point to the most recent sources for
example: dpkg.dsc or dpkg.dsp  (whatever the extension will be).

Erick

---

From: [email-masked] (Kai Henningsen)
Date: 01 Jul 1996 09:45:00 +0200

> Here's the alternative I've dreamed up:

Looks good so far.

>   of source packages, so that we can distribute GNU source tarfiles

We might make it optional and include some field in the .dsc to say what  
the maintainer did.

>   we need a dpkg-source script which runs diff and checks that there

Maybe we shoul look at having an optional pre-diff and post-diff script of  
some form in the .dsc?

>  - We need to store the .dsc somewhere when the source is `opened up',
>   ie, made into a filesystem tree.  Either this should be put
>   somewhere when dpkg-source unpacks an archive, or perhaps it should
>   have some canonical name in the Debianized source archive
>   (debian.sourcecontrol perhaps).  The dpkg-source script can produce
>   the Data-MD5sums and Version fields, and check the Source field, so
>   only Soource, Architecture, Depends and Generates would be needed
>   here.

It might help to sketch the resulting directory tree in something like the  
FSSTND style.

MfG Kai

---

From: Mark Eichin <[email-masked]>
Date: 01 Jul 1996 15:58:34 -0400

> dpkg, tar, gzip, dchanges and probably a real bunch of other things. 

Perhaps there should just be a package-set alias (not a virtual
package, but a group of packages) and that perhaps "basedev" be
everything that hello depends on?

Of course, finding them all is a little tricky, but using "logwrites"
(extending it to log exec's and open's instead?) would be a start
(similar to what clearcase does, actually.)

---

From: Steve Greenland <[email-masked]>
Date: Mon, 1 Jul 1996 16:10:18 -0500 (CDT)

Kai Henningsen wrote:
> [email-masked] (Ian Jackson)  wrote 
> >  - hello_1.3.orig.tar.gz

The whole point of changing the file name format was so tools
could reliably parse filenames to obtain package name, version, etc.
If you make it optional, then there is no point. 

Someone else said that we shouldn't even *discuss* file naming until
June, 1997. I, for one, agree whole heartedly. Unless a truly serious
flaw appears. Except that I would like to see *exactly* what
the standard now is, for .debs, sources, and diffs. Ian?

Steve Greenland

-- 
The Mole - I think, therefore I scream 

			"...so, like everybody was rooted to the spot.	He
			 picked up a little girl, and like, we'd all seen
			 FRANKENSTEIN, right?  It was scary.  Then he sits
			 her on his shoulder, an' everybody laughs and claps.
			 Man...

			 Man, that was the *best*.  Best moment of my life."
[Chester, a social and cultural rarity: a good man.  From SWAMP THING]

---

From: Juergen Menden <[email-masked]>
Date: Tue, 2 Jul 1996 08:21:39 +0200 (MET DST)

> everything that hello depends on?

what i wanted to make clear is. that this thing is entirely impossible.
makin some list of exclude packages is not helpful since we still have
to get all the needed packages together.

> (similar to what clearcase does, actually.)

no, this again doesn't help. if you don't believe me, try to track
down which packages your favorite configure depends on. if you get 
this done, try the configure of dpkg, emacs, binutils, perl, ppp...
(hello is not representable, don't forget this!)

these script are only semi-automated created. it's not uncommon to 
have some specialized code integrated, which might depend on whatever 
the upstream author likes. so that 'tricky' work has to be done every
time the upstream sources change.

well, we all know that there are regularily binary packages which 
have wrong dependencies set. well, the binary dependencies are a 
lot easier to handle then the source dependencies. there will be
more problems with the source dependencies than they are worth it.

what's the improvement you want to gain from the source 
dependencies? 

jjm

ps: i'm no source maintainer, but as part of the m68k port i have 
to recompile more packages then most of you. in my honest opinion
we will only get a real bunch of work by that 'feature'. we will
not gain anything from it, as for all non-trivial packages we will
not be able to build that dependencies reliable enough to be usefull
to anyone.

---

From: Mark Eichin <[email-masked]>
Date: 02 Jul 1996 03:39:07 -0400

You may be right to conclude that complete source dependencies are
impractical. (I'll note, however, that we do kerberos v5 builds with a
stripped down path so that they can be convincingly reproduced -- the
path has one directory, containing only the files we allow the build
to run [or rather, symlinks to them] -- and it *does* work, so the
problem is bounded...) 

I'll note that my logwrites example, where I said clearcase does
this... I wasn't kidding, clearcase will keep track of *everything*
you reference during the build, unless you tell it not to. Header
files, compilers, if some script prints the motd it'll catch it :-)

However -- even *incomplete* dependencies will have some value. It
might be useful to note that a package *needs* g++, or gnat, or some
html tool, that isn't common, a hint to that effect will at least
indicate "oh, got to port/download that tool first" to someone trying
to do a quick build who isn't going to read the INSTALL docs but
expect the source package, like All Things Debian, to Just Work :-)

---

From: Lars Wirzenius <[email-masked]>
Date: Tue, 02 Jul 1996 22:11:14 +0300

[ NB: I read the list.  Don't CC replies to me.  I pay for my PPP.  Thanks ]

Mark Eichin:
> However -- even *incomplete* dependencies will have some value. 

Indeed.  May I suggest that we assume that the "normal" UNIX development
and other tools are available -- cc, make, sed, awk, perl, X libs, and so
on -- but that we still note any unusual dependencies -- python, gawk
(not generic awk), g++, specific versions of some tools, and so on?
Actually, I doubt anyone is going to oppose this.  It should be encoded
in the guidelines, of course.

If we start to get complaints that, for example, X libs aren't part of
the normal UNIX developent environment, then we start adding dependencies
for X libs.  A bit annoying to the source customers, but they aren't
as many as the binary customers, so it's not that big a deal.

Complete dependencies may come later.  I hope they will; I'd love to
have dpkg-source tell me "you need g++ to compile this, but it isn't
installed" or "this package depends the Tk 4.1 sources on-line".

BTW, does the format allow encoding source depedencies as well?  I've
installed one or two Tcl/Tk programs (only to remove them later -- they
were awful) that required the Tk sources online.  A stupid way to do
it, I think, but that's life.

---

From: Juergen Menden <[email-masked]>
Date: Wed, 3 Jul 1996 16:04:31 +0200 (MET DST)

On Tue, 2 Jul 1996, Lars Wirzenius wrote:
> for X libs.  

complain! ;-)
xlibs package isn't that easy to port to m68k so i don't think we have
a xlibs package available for some time.
(even though i have the x libs installed! :-)

NB: even if you have an exclude list, you need to have the full 
dependencies first, otherwise you'll likely forget some.

> the source customers, but they aren't
> as many as the binary customers

really? why is this so? does anyone really needs to recompile packages
on intels?

> Complete dependencies may come later. 

i won't say any more to that. anyone would like that, but no one
ever has told me how to compile one. one person said he has a script
running, which can do that. could that person please try this script
with perl and demonstrate the results here?

note also that on different architectures the dependencies might be
different (i think of lilo and syslinux).

> I'd love to

no doubt, anyone loves to have it, that's not the question.
but noone will want the problems they get if they have to build
them. but voila, that's not my problem as i will only create the
bug-reports on that issue and do not have to compile a complete
dependencies list.

jjm

ps: has anyone looked at the examples i've given? please do so,
before you require something like that. it _is_ much more difficult
than it might seem at first glance.

---

From: Dale Scheetz <[email-masked]>
Date: Thu, 4 Jul 1996 13:06:47 -0400 (EDT)

> really? why is this so? does anyone really needs to recompile packages

Yes. Consider Pine. This package has many features that can only be
enabled/disabled in the os.h file, before the build. Many users of pine
build their own custom configured binaries. Being on an intel platform has
nothing to do with it.

Later,

Dwarf

------------                                          --------------

aka   Dale Scheetz                   Phone:   1 (904) 877-0257
      Flexible Software              Fax:     NONE 
      Black Creek Critters           e-mail:  [email-masked]

------------ If you don't see what you want, just ask --------------

---

From: [email-masked] (Kai Henningsen)
Date: 06 Jul 1996 13:57:00 +0200

> If you make it optional, then there is no point.

I guess what I mean is that our archive filenames should follow our  
convention, but the names of directories in the source archive need not;  
however, to enable automatic handling, this should be noted - machine- 
parsable - in the .dsc file.

MfG Kai

---

From: [email-masked] (Kai Henningsen)
Date: 06 Jul 1996 14:04:00 +0200

> expect the source package, like All Things Debian, to Just Work :-)

So, how about changing the field as follows:

Needs: some freeform text here that tells about "unusual" requirements

MfG Kai

---

From: Ian Jackson <[email-masked]>
Date: Tue, 9 Jul 96 23:26 BST

> the same applies for the Generates: field. 

No, those fields (and the file) are necessary (a) for the bug tracking
system - we need to be able to associate maintainers with source
packages, even when there are no binary packages with the same name -
and (b) for automatic generation of new binary releases for other
architectures.

Ian.

---

From: Ian Jackson <[email-masked]>
Date: Tue, 9 Jul 96 23:32 BST

Kai Henningsen writes ("Re: Source packaging - alternatives"):
> [email-masked] (Ian Jackson)  wrote on 01.07.96 [...]:
...
> Maybe we shoul look at having an optional pre-diff and post-diff script of  

I'm getting really really really tired of people suggesting that the
source format ought to include scripts.

Read my lips: NO UNPACK SCRIPTS IN SOURCE PACKAGES.

Ian.

---

From: Ian Jackson <[email-masked]>
Date: Wed, 10 Jul 96 01:35 BST

> Needs: some freeform text here that tells about "unusual" requirements

This is pointless.  We should either do the job properly, or not spend
effort on it.

Ian.

---

From: [email-masked] (Kai Henningsen)
Date: 11 Jul 1996 19:11:00 +0200

> source format ought to include scripts.

Well, they *do* seem to be useful. And I have yet to see a proposal that  
would handle otherwise what these scripts would be used for.

> Read my lips: NO UNPACK SCRIPTS IN SOURCE PACKAGES.

Shouting doesn't improve the quality of your argument, you know. (And it  
doesn't make people more willing to listen to you, either.)

However, instead of commanding people to listen to you, maybe you should  
instead read more carefully what those other people write.

Look again.

It says "script of some form".

I certainly wouldn't use this wording if I meant "shell script".

In fact, I used it exactly because I *do not* think it should be a  
"general" script. Certainly not an "unpack script".

Again, look what's written. It's a pre-diff / post-diff script, *not* an  
unpack script.

You should certainly be able to unpack everything without trusting the  
package. Applying the diff, however, is not the same as unpacking the  
package.

In fact, a diff file is itself a script of some form, and patch is the  
interpreter for this script. It's just missing some features we could use.  
And we certainly have to trust that script if we "apply the diffs".

In fact, we either have to trust the tar.gz archive not to contain any  
filenames outside the current directory, or else see if it does, or tell  
tar to do something about it. It already does for leading slashes; is  
there an option for filenames containing /../ sequences?

                                  ******

On rethinking this, there seems to be a way we can have general scripts  
without trust problems.

We would need to define a way that unpacking definitely *doesn't* include  
applying the diffs; instead, this would be a separate step invoked after  
unpacking. In that case, I see no problem with that step applying  
arbitrary commands; this isn't any different from debian.rules applying  
these commands with respect to trust.

In fact, this enables us to apply this second step after substituting  
another version of the original source, which seems to be useful.

MfG Kai

---

From: [email-masked] (Kai Henningsen)
Date: 11 Jul 1996 20:02:00 +0200

[email-masked] (Ian Jackson)  wrote on 10.07.96 in <[email-masked]>:

> > [email-masked] (Mark Eichin)  wrote on 02.07.96 in
> > > <[email-masked]>: However -- even *incomplete*
> > > dependencies will have some value. It might be useful to note that a
> > > package *needs* g++, or gnat, or some html tool, that isn't common, a
> > > hint to that effect will at least indicate "oh, got to port/download
> > > that tool first" to someone trying to do a quick build who isn't going
> > > to read the INSTALL docs but expect the source package, like All Things
> > > Debian, to Just Work :-)
> >

Either perfect, or not at all?

This is not a useful attitude.

Noting unusual requirements is obviously helpful, even if it is  
incomplete; it is far from "pointless".

MfG Kai

---

From: Ian Jackson <[email-masked]>
Date: Tue, 16 Jul 96 14:05 BST

...
> Either perfect, or not at all?

Either we say that we aim to make it accurate and useable, or there's
little point in putting in the effort to write anything at all.

> incomplete; it is far from "pointless".

There are already plenty of places to put freeform waffle.  I think
this particular bit belongs in debian/README.

Ian.

---

From: Ian Jackson <[email-masked]>
Date: Tue, 16 Jul 96 14:12 BST

> package. Applying the diff, however, is not the same as unpacking the  

Yes, it is.  Unpacking the *Debian* source package involves unpacking
the upstream source and then applying the diff.

You're just moving the goalposts.

It is a requirement that one can unpack the Debian source tree, to
look at it, without having to execute anything.

> And we certainly have to trust that script if we "apply the diffs".

It is easy to secure against `patch' (or indeed `tar') doing something
bad.  I intend to do this.

Look, all of you people who want to include scripts in this might as
well give up.

I'm not interested in continuing this discussion; my mind is made up.
I *will not* implement or accept any scheme that involves executing
(trusting) parts of Debian source packages when they are unpacked (and
that means the process of producing the Debianised source tree from
the Debian source files).

If you have a problem with this you will have to take it up with Bruce
and get him to overrule me.

Thanks,
Ian.

---

From: Dale Scheetz <[email-masked]>
Date: Tue, 16 Jul 1996 19:56:45 -0400 (EDT)

> the Debian source files).
Unless I have misunderstood, the purpose of this exercise is, among other
other things, to produce Debian source from the upstream source. (I assume
the above is a typo)
The problem that is causing this discussion comes from the fact that
diff/patch are not up to the task we wish to assign them. Are we going to
deal with this by re-writing these programs to fill this need?
How do you propose to create the Debian Source Tree from the pristine
source without the option of scripts that repair these inadequacies.
I don't have any problem with your dictating this issue, as long as it
doesn't leave us with a source packaging scheme that doesn't work for all
packages.

Later,

Dwarf

------------ If you don't see what you want, just ask --------------

---

From: Juergen Menden <[email-masked]>
Date: Wed, 17 Jul 1996 12:14:22 +0200 (MET DST)

> the Debian source files).

and if anyone thinks (s)he _really_ needs that, why not putting that
stuff in the debian.rules file? 

jjm

---

From: Ian Jackson <[email-masked]>
Date: Fri, 19 Jul 96 21:22 BST

Dale Scheetz writes ("Re: Source packaging - alternatives"):
...
> doesn't leave us with a source packaging scheme that doesn't work for all

Right, now I see what you're getting at.

I propose to support at least the following differences between the
debianised and original source:
 * Change to contents of an ordinary file.   } using diff and a bit
 * Creation of an ordinary file.             }  of glue script
 * Making the debian/rules executable (special case).

Symlinks will be ignored entirely, so that you can't change any
symlinks during the `debianisation', but likewise they won't cause any
problems.  This applies to named pipes and other similar things too.

Other permission changes, changes to the targets of symlinks, renaming
of files and directories and so forth will probably not be supported.

Is this sufficient to meet your needs ?

It's hard to do anything more without requiring people who want the
debianised source to download our special tool.  As it is they can
download our source package and apply the diff and the only thing that
won't be right will be the permissions on debian/rules.  You will need
the special tool to create a source package, because (for example) you
need to avoid trying to dereference symlinks, &c.

Ian.

---

From: Ian Jackson <[email-masked]>
Date: Fri, 19 Jul 96 21:29 BST

> stuff in the debian.rules file? 

No, because the debian/rules file is not looked at during the unpack
process.

Ian.

---

From: Juergen Menden <[email-masked]>
Date: Wed, 24 Jul 1996 20:38:44 +0200 (MET DST)

> No, because the debian/rules file is not looked at during the unpack

right. 
but such things could very well be run in front of the build
target. eg. the first actions taken when building a package might 
be to correct permissions and set symlinks. 

at least this is what configuration is meant about. :-)

jjm