Debian package management

Links

Wajig

Summary

The wajig package manager, renamed "just", makes it easier to update and upgrade software packages in Debian.  It's now working together with apt-move, the program that maintains the local mirror (see below). Packages should be installed manually on mantaray, then use the round-robin install script for all the others.

Guides

  • Wajig guide (external)
  • just doc | less
  • just help
  • just commands

Commands

  • See a list of commands in the Linux Commands file -- the normal command to install a package is

    • just install mozilla-firebird

  • This can become tiresome on a dozen computers, so here's a trick for installing a package on a sequence of remote computers in quick succession:

    • ssh starfish just install mozilla-firebird 1>&log

    With this command you don't have to wait for one installation to complete before you move on to the next.  You need to test first that there are no problems of course.

  • If you want to automate even further, use the scripts install, purge, and remove in mantaray:/root/scripts. They are round-robin scripts that install, purge, or remove packages from all the machines on the Baja subnet.

    • It's generally a good idea to use these scripts, as it's easy to forget one of the machines when you update or remove packages. The scripts also save you hundreds of keystrokes. To run a script, issue this command:

    • ./install

    and give the package names when prompted.

  • If you're on a slow connection and have downloaded a lot of packages, you can create a repository out of them:

    dpkg-scanpackages . /dev/null | gzip -9c > Packages.gz
    dpkg-scansources . /dev/null | gzip -9c > Sources.gz

    Add the path to /etc/apt/sources.list and run update.

Tweaking

To allow non-root users to acces wajig, configure the sudoers file -- this has now been done.

Local Debian Mirror

Summary

A local Debian mirror has been established so that only bottlenose now downloads new packages, and all the other machines on the Baja network to access them from there. The mirror is also available through the web.

Note that a significant advantage of this setup is that individual packages are now not cached on each machine (/var/cache/apt/archives), saving huge amounts of space and leaving ample room for more programs.

[For an alternative solution suitable for smaller networks, see apt-cacher.]

Install a new package

  1. Let's say you want to add a new package, iproute, to all machines
  2. First log on to bottlenose and run "just update" and install the package
  3. Next issue "apt-move update >& log &" (takes a couple of minutes)
  4. Run the /scripts/install script to install iproute on all nodes

It's not worth using the alternative "apt-move movefile <package>" (which anyway often fails), as  "apt-move update" only takes a few minutes and you'd need to run "apt-move packages" afterwards to rebuild the packages list, or the new packages won't show up to the other nodes.

Don't use "just move", since this performs "apt-move update" and "apt-get clean" -- cleaning out mantaray's repository. We want to keep it in case we need to downgrade a package.

Nightly mirror updates

Bottlenose runs install-less nightly downloads; as of 31 August 2003, this is now combined with running "apt-move update" to update the local mirror:

just update && just auto-download && apt-move update

That should do the trick -- I should have an updated "installed-packages only" mirror at all times.

The /etc/apt-move.conf file has the option "delete" enabled, so non-current packages will automatically be deleted from the mirror (though not from mantaray's own archives).

This has the effect, afaics, of moving all generations of packages into the mirror each night and then deleting them. It would make more sense to set it up the other way: obsolete packages are removed from mantaray's archives, but remain in the mirror in case we need to revert an upgrade.

A list of files that have been ignored, moved, and deleted is listed in the daily e-mail to user kfa on mantaray; it can be consulted to see the list of daily updates available to the system.

Note that if you put a package on hold, it won't get updated in the apt-move mirror.

Commands

apt-move update >& log & get the latest files from the cache
apt-move delete
 delete non-current packages (run
apt-move packages afterwards!)
apt-move sync >& log & get the latest files by downloading
apt-move movefile <filename>
 moves an individual package (often fails)
apt-move packages
 regenerate the packages file
apt-move listbin sync create a list of files to be moved
apt-move mirrorbin <file list> move the list of files specified
apt-move -t <any option>
simulation mode (test only)

"Running apt-move periodically will assist in managing the resulting partial mirror by optionally removing obsolete packages, and creating valid local Packages.gz files.  It can also build a partial or complete local mirror of a Debian binary distribution (including an 'installed-packages only' mirror)."

Reinstalling all packages

There may be a simple way of doing this -- see the procedure for duplicating another system's installation under bulk install.

What follows is a crude procedure for reinstalling all packages on a system, in case you've deleted some random part of the operating system and can't reconstruct exactly what needs to be reinstalled.

  1. just list-installed > sigillo-installed-2003-09-22
  2. just -t file-install sigillo-installed-2003-09-22
    This produces a list of all packages without line shifts at the start, and possible a list of errors at the end -- clean up the errors and repeat before proceeding
  3. Copy the list of packages and paste into this (the -w means no wrapping):
    nano -w reinstall
  4. Divide the single huge line into smaller bits as separate files (e.g. four letters at a time)
  5. Copy the text produced by "cat reinstall-a-d" and paste it into this:
    just reinstall <pasted list>
  6. Repeat till done

Installation history

On 16 January 2004, the /mantaray partition was 100% full, so I moved the local mirror to bottlenose, which has a larger drive. I installed apt-move, copied over the /etc/apt-move.conf file from mantaray, and copied over the dists and pool directories from /mantaray/debian to /bottlenose/debian and ran apt-move update. Note that you have to copy over the whole /debian mirror, which includes a whole machinery of hidden /debian/.apt-move files -- without these files, you won't be able to include all the old files in the new packages files.

I changed crontab on mantaray by commenting out apt-move and added it to bottlenose's crontab. Note that when cron runs, you get e-mail for user kfa, not root (this is defined in exim, the mail client). I changed /etc/apt/sources.list on all the other machines to point to the new mirror. Briefly tested; seems fine.

On 1 October 2003 I added a link from graywhale's web server directory to mantaray's local Debian mirror:

root@graywhale:/graywhale/www# ln -s /mantaray/debian debian
You can now add the following lines to /etc/apt/sources list to use this mirror:

deb http://67.116.162.14/debian/ unstable main contrib non-free
deb http://67.116.162.14/debian/ unstable/non-US main #contrib non-free

On a local system, you can substitute graywhale's local IP address, 187.26.16.74, or its name, for the external IP address:
deb http://graywhale/debian/ unstable main contrib non-free
deb http://graywhale/debian/ unstable/non-US main #contrib non-free
Note that if you get packages this way, local copies will be stored in your /var/cache/apt/archives directory, which you avoid by using the mirror on a mounted drive.

This way of using the mirror is particularly useful for new installations that lack full networking capabilities.

On 25 August 2003 I issued this on mantaray:

just install apt-move
I edited the configuration file, /etc/apt-move.conf using these values:
APTSITES="marillat.free.fr linux.csua.berkeley.edu_debian ftp.debian.org_debian linux.csua.berkeley.edu_debian-non-US"
LOCALDIR=/mantaray/debian/
DIST=unstable
DELETE=no
COPYONLY=no
CONTENTS=yes
The rest of the values I left alone. You'll need to add more sites to APTSITES if you're going to use the "apt-move sync" command, which downloads the current version of all installed packages. In ATPSITES, replace slashes with underscores or you'll get errors.

I created a new /mantaray/mirror directory for the new local mirror, and then then ran this command to establish the master files for the new mirror:
apt-move get
This generated lists of all available packages in the mirror (see man apt-move). I then moved the current packages from the /var/cache/apt/archives into the mirror with this command:
apt-move update
This implies "delete obsolete packages" -- if you want to move the obsolete ones too, just issue
apt-move move
The apt-move update too a long time -- if you're doing it remotely over the wireless network, use "apt-move update >& log &" so the process would be cut off -- in which case it has to start all over again.

I then added these lines to mantaray:/etc/apt/sources.list:
# Local mirror on /mantaray
deb file:/mantaray/debian/ unstable main non-free contrib
deb file:/mantaray/debian/ unstable/non-US main
For all the other machines, I substituted the previous content of sources.list for just these two lines -- they will now get their packages only from the local mirror.

  

 

top