new package pkg_rolling-replace

[Greg Troxel]

pkg_rolling-replace (in pkgtools) was written to provide a middle way
between the missing-packages problems of "make update" and the
inconsistencies caused by "make replace". It does a "make replace" on
packages that need update in dependency-sorted order, where needs
update means

unsafe_depends flag set on package

rebuild flag set on package

[only if -u flag given] pkg_chk says the installed package doesn't
match pkgsrc

As 'make replace' is done, unsafe_depends is set on depending packages
(by pkgsrc; this is a feature added because of pkg_rolling-replace,
but is not part of pkg_rolling-replace). Thus, one can issue 'make
replace' as one pleases, and invoke pkg_rolling-replace to do a
cleanup pass walking the dependency tree.

One needs pkgsrc-2006Q3 or newer (head is of course fine) to use
pkg_rolling-replace; 2006Q2 and earlier don't have the unsafe_depends
change.

pkg_rolling-replace is beta quality software. I routinely run it on
my workstation, my notebook, my personal web/mail server, and on
production machines (we're a research group, not a bank) at work.

The first time you try to run this, especially on a machine which is
quite out of date, you'll likely have a lot of problems, which are
fixed in the normal fixing-pkgsrc-build sort of way. I have found
that once the machine is pretty recent, then updates mostly just work,
even if half the packages are rebuilt. This is really an impressive
statement about pkgsrc quality.

Nick Goffee is the primary author. I thought of this plan several
years ago but never got enough round tuits. The effort was funded by
the US Defense Advanced Rerseach Projects Agency under the ACERT
program.

The man page is in pretty good shape, but just in case I'll utter the
magic words:
# kill -1 `cat /var/run/wizd.pid`

Greg

[Steven M. Bellovin]

Very nice!

One question -- can it ever end up in a state where something doesn't
build, making it stop and lose state about what else needs to be
rebuilt?

[Greg Troxel]

Very nice!

Thanks. Bug reports, fixes welcome.

One question -- can it ever end up in a state where something doesn't
build, making it stop and lose state about what else needs to be
rebuilt?

It can easily end up stopped, and does so if any package operation
fails. This happens to me all the time, when I have a bad distfile
due to download problem, or more often when a package has been renamed
or split into foo and foo-lib, and then foo wants to build foo-lib
which can't install because it conflicts with foo.

But, pkg_rolling-replace doesn't have a lot of internal state, and
rerunning it picks up where it left off. There are several kinds of
state:

which packages are "dirty", and that's a "pkg_admin set" variable
'unsafe_depends'. So not only does pkg_rolling-replace not need to
store this, but you can do a manual make replace without breaking
the invariant that a package is marked unsafe_depends if a
dependency has been make replaced out from under it.

manual rebuild requested, which is a "pkg_admin set" variable
rebuild.

packages that are 'mismatched' according to pkg_chk, which will just
get recomputed if you do 'pkg_rolling-replace -u'

dependencies. These get observed from the installed packages, and
then additional dependencies from each package to be built are
pulled in via 'make show-depends' just before a package is 'make
replaced'. If any new ones appear, the dependency list is
re-tsorted and a new package to replace is chosen. This catches new
dependencies in the source package, and build-only dependencies.
Arguably pkg_rolling-replace should just do 'make show-depends' on
each source package before starting out, but I don't think it
matters in practice and in theory I'm not quite sure.

That's about it. I sometimes need to run it 5 times to do a rebuild
of everything after a few months, but I just fix the reason it stopped
and rm all the work dirs and then run it again.


If you are comfortable doing make replace, and picking up the pieces
when you have a missing shlib, and working around build breaks, then
I'd say go right ahead and try pkg_rolling-replace. I think you'll
find it easier than doing make replaces by hand, but it's still more
error prone than building everything you need in a chroot because it
doesn't solve the renamed/restructured package problem.

[Blair Sadewitz]

This sounds very similar to wip/pkgmanager, which is impressive,
especially given the speed at which it calculates dependencies.
Currently the author is rewriting it in Chicken to make it more
portable. If anyone here knows LISP, you could possibly modify it to
use this framework and have the best of both worlds ...


Regards,

--Blair

[Peter Schuller]

Post by Blair Sadewitz
This sounds very similar to wip/pkgmanager, which is impressive,
especially given the speed at which it calculates dependencies.
Currently the author is rewriting it in Chicken to make it more
portable. If anyone here knows LISP, you could possibly modify it to
use this framework and have the best of both worlds ...

The current clisp version of pkgmanager doesn't really have much of a
framework as such; it's all intermingled.

For the re-write, I am however trying to "libify" the package database /
package origin handling stuff, along with generalizing it to support e.g.
FreeBSD ports aswell. That is, if I get off my butt and finish it.