Subject: Data flow during package building.

From: [email-masked] (Ian Jackson)
Date: Wed, 10 Jul 96 18:55 BST

I've done some thinking about how the various bits of control
information flow around when you build packages, and the list below is
the result.

First comes a list of the fields in the various files we want to
generate - the control files of the binary packages, the .changes
file, and the .dsc files for the source package.

Then comes a proposed scheme for the sources of these pieces of
information.  I propose to use information from a new mandatory
changelog file, a new-format unified control file parsed directly by
the tools (unlike debian.control, which is at the moment just a
convention and only accessed by debian.rules) giving various
version-independent details, and a couple of other sources.

Each source and destination has a letter - see the list below - and
next to each individual field I've put a short note saying where I
think the information is coming from (fields in destination files) or
going to (pieces of source information).

Finally, I propose briefly what new tools will be required to do this
manipulation.

Ian.

Destinations of information
---------------------------

Binary package control file fields (B):
  Package                                       P
  Version (binary)                              L (unless in R)
  Maintainer (usual)                            G
  Architecture (`all'|specific)                 P (calculated)
  Source (includes version, has defaults)       L and perhaps G
  Description                                   P
  Priority, Section (optional here)             P, depending on R
  Depends et al, Essential                      P

Changes (.changes) file fields (C):
  Format                                        A
  Date                                          L, or otherwise A
  Source                                        L and/or G
  Version (source)                              L
  Binary (list)                                 P
  Distribution (list)                           L
  Architecture ([`source'] `all'|specific)      P (calculated)
  Maintainer (this revision)                    L
  Description (summary list)                    P (calculated)
  Urgency                                       L
  Changes                                       L
  Files (`standard' files) } one                calc. from PLFRA
  Files (`extra' files)    } field              G (and perhaps R)

Source control (.dsc) fields (S):
  Source                                        L and/or G
  Binary (list)                                 P (calculated)
  Maintainer (usual)                            G
  Architecture (list|`all'|`any')               P (calculated)
  Version (source)                              L
(Of course no .dsc can be generated if doing a binary-only build.)

Proposed sources of information
-------------------------------

Log (debian/changelog) in source package (L):
  Source (checked against G if in both)         SBC
  Version (source)                              SC, B (if not from R)
  Maintainer (this revision)                    C
  Changes                                       C
  Urgency                                       C
  Distribution (list)                           C
  Date (?)                                      C or A
(These are not fields in an 822-style header; a script determines the
format of the changelog and parses it appropriately.)

Control file (debian/control) in source package -
General, binary-package-independent fields (G):
  Source (checked against L if in both)         SBC
  Maintainer                                    SB
  Files (`extra files')                         C
A blank line separates general fields from package-specific ones, and
each binary package from the next.

Control file (debian/control) in source package -
Per-binary-package fields (P):
  Package (perhaps make this optional?)         SBC
  Architecture (list|`all'|`any')               SBC
  Description                                   BC
  Priority/Section                              C, and B if R says so
  Depends et al, Essential                      B

Coded for in debian.rules (R):
  Version (binary, defaults to same as source)  B, C (Files field)
  Include priority in control file ? (flag)     B
  Versions and stuff in names of extra files    C (Files field)

>From the person or script doing the build (F)
  Full release or binary-only ? (flag)          C
  Changes (mostly for binary-only builds)       C
Of course if we're doing a binary-only release there are no source
packages and so no S.

Automatically generated (A):
  Date (822 format)                             C
  Architecture (specific, from dpkg)            BC
  Format (of .changes file)                     C

New tools
---------

To keep the interface reasonably simple, it makes sense to have one
tool to generate each kind of destination file.  So:

* dpkg-gencontrol

Generates a binary package control file, ready for building a package
using dpkg-deb.  This reads the information in debian/control and
debian/changelog.  There are options to override and/or modify the
Version field in the generated file, to include the Section and/or
Priority in the output control file (they won't be by default), and
for use in emergencies to add, remove and change fields in the output.

This command will be called by debian/rules's binary target as part of
its preparations for calling dpkg-deb --build.

* dpkg-source

In this context, generates a .dsc (Debian source control) file.  This
again reads debian/control and debian/changelog, and again has options
to override aspects of the output.  It also creates the .diff.gz and
.orig.tar.gz, though there will have to be options for creating a .dsc
using already-prepared files &c.

It will be called by debian/rules's source target.  The diff target
will disappear.

This script will also be the one that unpacks a source package.  It
should probably check that the .dsc agrees with the contents inside
the package in debian/control and debian/changelog when it unpacks.

This wouldn't sign the .dsc, since signing is very
environment-specific.

* dpkg-genchanges

Generates a .changes file.  It reads debian/control and
debian/changelog to find out the source package name and version,
changes since release, &c.

It's not clear to me how it should find out the versions of the binary
packages (whose names it will already know by now).  These could
either be supplied on the command line, or be accumulated somehow
during the build process when each package is built, or perhaps the
script could search the directory for binary packages with the correct
source version in their Source field.

This script would *not* be called by debian/rules.  This is so that
the caller can give it options like `don't try to find source
package' for a binary-only build or `use this changes info instead,
don't use the stuff from the changelog'.

Like dpkg-source, this wouldn't sign its output.

* dpkg-parsechangelog

This parses a changelog file, turning it into a series of fields and
values.  This is an internal routine, but probably wants to be shared
by the others and exported too.

-- 
Ian Jackson   [email-masked]        These opinions are my own.
Cambridge University Computer Laboratory, New Museums Site. + 44 1223 3 34676
Home: [email-masked]   Churchill College, CB3 0DS. + 44 1223 3 31579
PGP2 public key available.               http://www.cl.cam.ac.uk/users/iwj10/

---

From: [email-masked] (Erick Branderhorst)
Date: Fri, 12 Jul 1996 10:02:09 +0100

> (Of course no .dsc can be generated if doing a binary-only build.)

I miss the url reference to the original sources here. If this url is
included, and perhaps some additional descriptive information, this can
be the input for an autogenerated copyright file which comes with the
package.

>   Format (of .changes file)                     C

The copyright file might be a autogenerated file as well.

> source version in their Source field.

What about calling debian.rules (or debian/rules) with the target
<package>.<version>.  
$ make -f debian/rules textutils.version
1.18e-1

---
debian/rules

# configure.in contains package and version info in gnu packages
# for other packages it might be hardcoded in debianrules or found
# in some other way
p:=$(shell sed -n -e 's/^PACKAGE=\(.*\)$$/\1/p' configure.in)
v:=$(shell sed -n -e 's/^VERSION=\(.*\)$$/\1/p' configure.in)

$(p).version:
	@echo $(v)-$(d)

$(p)-dev.version:
	@echo another version
---

The rest seems very well thought to me.

Erick

---

From: Ian Jackson <[email-masked]>
Date: Tue, 16 Jul 96 03:10 BST

Erick Branderhorst writes ("Re: Data flow during package building."):
...
> be the input for an autogenerated copyright file which comes with the

Urgh, autogenerated copyright file ?  It's full of random waffle, how
can you automatically generate it ?!

...
> $ make -f debian/rules textutils.version

make doesn't work like that; it prints lots of messages (which we
don't want to suppress).

I'm leaning towards the `search the directory for binary packages with
the correct source version' approach.

Ian.

---

From: Christian Hudon <[email-masked]>
Date: Tue, 16 Jul 1996 08:03:23 -0700 (PDT)

> can you automatically generate it ?!

Indeed! Let's not try to be too clever.

Erick, take a look at the copyright files for pgp and termcap-compat, for
instance. The first one has stuff from Ian and PRZ about export
restrictions, etc. The second one is a combination of the ncurses termcap
database copyright and of the Linux termcap library copyright... and the
latter was derived from the GNU termcap library, which has its own
copyright notice.  You can't autogenerate these. 

  Christian

---

From: [email-masked]
Date: Wed, 17 Jul 1996 10:04:36 +0200 (MET DST)

> copyright notice.  You can't autogenerate these. 

I can take a look but it isn't needed.
I propose a four part /usr/doc/copyright/<package>

part 1:  A few sentences automatically generated from source control
information telling, who did what with what from where under gpl.

part 2:  The most recent debian/Changelog information (only the current
revision).

part 3:  cat debian/README >>

part 4:  optionally cat some other stuff generated by debian/rules.

Erick

---

From: Ian Jackson <[email-masked]>
Date: Fri, 19 Jul 96 21:27 BST

[email-masked] writes ("Re: Data flow during package building."):
...
> information telling, who did what with what from where under gpl.

The source control file doesn't have this information and in any case
it is free-form.

> part 2:  The most recent debian/Changelog information (only the current

This belongs in a separate file.

> part 3:  cat debian/README >>

This is the same as 1.

> part 4:  optionally cat some other stuff generated by debian/rules.

This is unnecessary.

I plan to mandate in the new source format that debian/README or
perhaps debian/copyright be copied to /usr/doc/copyright/<package>
verbatim.

Ian.

---

From: [email-masked] (Erick Branderhorst)
Date: Mon, 22 Jul 1996 09:59:05 +0100

> The source control file doesn't have this information and in any case

debian/rules has this information during the package building (I meant
package building control instead of source control)

> This belongs in a separate file.

I mean the changes in the most recent revision, sort of similar to the
information which is included in the .changes messages announcing a new
revision of a package.  This can be skipped as well.

> > part 3:  cat debian/README >>

Same as 1.
 
> > part 4:  optionally cat some other stuff generated by debian/rules.

IYHO perhaps, but I think that it makes sense.  I include all the lines
containing a copyright message in tex packages to this file.  In this way
I'm almost sure that I don't forget to mention anyone who should be
mentioned.
 
> perhaps debian/copyright be copied to /usr/doc/copyright/<package>

And I would like to see some of the above added as well.  In this way
we can uniform at least part of the copyright message and reduce 
general developer load.

Erick