Linux Certif

Toute la documentation sur la certification Linux LPI

emtargetcmp

Name

emtargetcmp - Compare the Emdebian target repository with Debian unstable

Copyright and Licence

  Copyright (C) 2007-2008  Neil Williams <codehelp@debian.org>
 
  This package is free software; you can redistribute it and/or modify
  it under the terms of the GNU General Public License as published by
  the Free Software Foundation; either version 3 of the License, or
  (at your option) any later version.
 
  This program is distributed in the hope that it will be useful,
  but WITHOUT ANY WARRANTY; without even the implied warranty of
  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
  GNU General Public License for more details.
 
  You should have received a copy of the GNU General Public License
  along with this program.  If not, see <http://www.gnu.org/licenses/>.
 
 

Bugs

A long standing bug in apt is that when switching architecture or when switching between repository sources, apt has a habit of purging the lists directory of whatever cache is being used. Despite apt-cross using -o APT::Get::List-Cleanup=off in all calls, this behaviour has returned in recent versions of apt.

To get around this, emtargetcmp now creates a temporary directory, copies the system cache, queries that and then removes the copy. This allows apt to use lock files and status files in the temporary directory without requiring sudo. (The temporary directory used is $apt_cache_dir/host/.)

Description

emtargetcmp prepares cache data from the emdebian/emdebian target cache at buildd.emdebian.org against the Debian unstable cache at ftp.uk.debian.org for arm with support for mapping Emdebian version strings to Debian versions.

Uses ftk.uk.debian.org or ftp.de.debian.org due to location of buildd.emdebian.org

One of the remaining issues with this approach is the apparent lack of support for creating a sources.list in memory without needing lots of little files. Therefore emtargetcmp creates a special sources list in the apt-cross directory (e.g. ~/.apt-cross/) and switches the content of the file as necessary.

This small utility also serves as an example of what can be done with Cache::Apt::* from libcache-apt-perl.

Currently, emtargetcmp only supports ARM because Emdebian only has a root filesystem for ARM.

Output uses ANSIColor (where supported) so that packages which are older in Emdebian than in Debian are listed in red and packages which are the same version in Emdebian as in Debian (accounting for the emdebian version suffix) are shown in green. If, for whatever reason, Emdebian contains packages that are newer than in Debian, these are shown in blue.

Commands

  -m|--migration-check
 Special mode along the lines of Britney (and in fact, reliant on a
 working Britney in Debian). Downloads the data for Debian and Emdebian
 testing and unstable, tries to calculate if the correct package versions
 exist in Emdebian unstable to bring Emdebian testing into line with
 Debian testing.
 Overrides the -c and -s options to force a run for unstable and testing.
 
 

Options

  -a|--arch
 Override the dpkg-cross default architecture.
 
  -c|--complete
 Add a complete check of the Emdebian archive at the end of the run.
 
  -v|--verbose
 Make the output more verbose.
 
  -q|--quiet
 Make the output quieter.
 
  -s|--suite
 Override the default suite (unstable) - experimental.
 
 

migration-check, aka Britney

The idea for this function is:
 1. Compare Emdebian testing with Debian testing.
 2. Where a package is old in Emdebian, check Emdebian unstable.
 3. If the correct version exists, advise on which packages
 reprepro should pull from unstable but do not actually try to
 run reprepro. If the correct version does not exist, raise an
 error so that the correct version can be built and uploaded.
 4. Run edos-debcheck against testing in the same way as with
 the '--suite testing' option.

Trying to automate this function any further is likely to cause errors and dropped packages. This is an intensely manual task in Debian and Emdebian and needs careful management to ensure that testing is always installable. In particular, Emdebian lacks support for ``release-critical'' bug detection/blocks - RC bugs in Debian will block Debian migration and therefore Emdebian migration but bugs against buildd.emdebian.org cannot be RC at this time as issues within the Emdebian packages cannot delay the main Debian release.

The 'embritney' script can assist on the server-side but testing migrations still need manual oversight. To do this, the list of .debs needs to be identified which is hard for emtargetcmp at the time as the migration check might not take place on the buildd machine.

Migrations

Migrations can require several manual steps and involve some planning to ensure that the right versions are available locally even if the repository gets temporarily updated to a higher version.

This documentation only covers reprepro - if you use a different repository manager, migrations may need different solutions.

Pulling packages from unstable

Packages that need to be pulled from Emdebian unstable can be migrated using reprepro support but this tends to be all-or-nothing. Selectively pulling the version from unstable into testing requires individual commands to reprepro, specifying the actual location of the .deb, e.g.:
 $ reprepro -b /path/to/repo include testing /path/to/repo/pool/main/.../foo_$arch.deb

For large migrations, a reprepro pull should be followed by running 'emtargetcmp --migration-check' again and then fixing the repository as described below.

Or use copysrc in recent versions of reprepro:

  reprepro list testing $src_package
  reprepro list unstable $src_package
  reprepro copysrc testing unstable $src_package $version_in_unstable
 
 

Avoid migrating too many packages in one go - re-test with emtargetcmp -m and comparing the dependencies (and versions) using the Emdebian website:

  http://www.emdebian.org/packages/search.php?distro=$distro&package=$package
 
 

(In time, emdebcheck will gain this functionality.)

For small migrations, select the .deb for the package and version in unstable and include it into testing manually, as above. Ensure that the source package and all binaries are migrated - once the last file is migrated, reprepro will drop the previous files. Check the pool directory to see if a file still needs to be migrated and use:
 $ reprepro -b /path/to/repo dumpunreferenced to see which files might have been missed and
 $ reprepro -b /path/to/repo deleteunreferenced to tidy up the repository.

Manually fixing packages

In certain circumstances, the autobuilder can proceed to building the next version of a package before the previous version has migrated into testing. This can be due to delays in fixing the cross-build, for example. If, in the meantime, the package has migrated into Debian testing at the previous version, Emdebian has a problem.

If the previous version can be built (it was just delayed), the locally built package can be manually added to testing using reprepro.

If the previous version was actually broken and needed a bug fix in Debian to be able to cross-build, Emdebian testing might have to be ahead of Debian testing for this specific package.

The version included into Emdebian testing should be as close as possible to the version in Debian testing, otherwise other dependencies may need to also be modified.

Note that you may need to specify the '--ignore=wrongdistribution' option to reprepro if you include a .changes file directly into testing when manually fixing a package that is too new in Emdebian testing and it is usually necessary to upload each file listed in the .changes separately, rather than using dput (unless you create a bespoke dput target that does not run the usual post_upload scripts).

Packages that need to be built

Listings of packages that have yet to be built for Emdebian unstable should only arise during times when the autobuilder is paused for some other reason.

It is particularly important that all packages needing a manual fix are resolved before the autobuilder is restarted, or the problems in trying to manually fix those packages will just get worse.

If a package in Debian testing no longer builds certain binaries, e.g. after a SONAME transition or similar, the package may show up as needing to be built when what is actually needed is for the old binaries in Emdebian testing to be removed when the version in unstable migrates into testing. In this situation, a reprepro pull is advisable.