[[!meta title=“DEP-17: Improve situation around aliasing effects from
Title: Improve situation around aliasing effects from `/usr`-merge
Drivers: Helmut Grohne <email@example.com>
This document summarizes the problems arising from our current `/usr`-merge
deployment strategy. It goes on to analyze proposed solutions and analyzes
their effects and finally proposes a strategy to be implemented.
Debian has chosen to implement merged
/usr by introducing symbolic links such as
/bin pointing to
In the presence of such links, two distinct filenames may refer to the same file on disk.
We say that a filename aliases another when this happens.
The filename that contains a symlink is called the aliased location and the filename that does not is called a canonical location.
At its core,
dpkg assumes that every filename uniquely refers to a file on disk.
This assumption is violated when aliasing happens.
As a result, we exercise undefined behavior in
This is known to cause problems such as unexpected file loss and is currently mitigated by a file move moratorium.
We currently prohibit most situations that may provoke problematic behavior using policy. This mitigation is not without cost and we want to eliminate it. Shipping files in their canonical locations tends to simplify packaging. Once files are moved to their canonical locations, a number of aliasing problems are effectively mitigated. The goal of this work is to reduce the impact of these matters to the typical package maintainer. It aims for removing the cognitive load of having to keep in mind which files must be installed to aliased locations and which files must be installed to canonical locations.
Regardless of what strategy we end up choosing here, we will likely keep some of the temporary changes even in the
forky release to accommodate external repositories and derivatives.
When moving a file from its aliased location to a canonical location in the
data.tar of a binary package and moving this file from one binary package to another,
dpkg may unexpectedly delete the file in question during an upgrade procedure.
If the replacing package is unpacked first, the affected file is installed in its canonical location before the replaced package is upgraded or removed.
dpkg may then delete the affected file by removing the aliased location - not realizing that it is deleting a file that still is needed.
This problem was originally observed in #974552 and is the one that motivated the issuance of the moratorium.
Since the moratorium came into effect and file moves have been prevented, no new cases surfaced.
Had the moratorium been lifted for the bookworm release, we know that problems would have been caused in a small two-digit number of cases.
/lib/systemd/system/dbus.socket could have been canonicalized while it has been moved from
There is an artificial test
case1.sh demonstrating the problem.
When packages declare a
dpkg file trigger interest in a location that is subject to aliasing without also declaring interest in the other location, a trigger may not be invoked even though that was expected behavior.
No issue arises when a file trigger is declared on a canonical location and all packages are shipping their files in that canonical location.
However, when the trigger is declared for an aliased location and packages move their files to the canonical location, triggers can be missed.
This problem is also currently being prevented by the moratorium.
Had the moratorium been lifted for the bookworm release, we know that problems would have been caused in two cases.
udev packages declare an interest to aliased locations and would start missing trigger invocations when canonicalizing files in other packages.
When a package uses
dpkg-divert to displace a file from another package, the diverted location may have become aliased due to the
If a package whose files are being diverted were to canonicalize its files, such a diversion were to become ineffective.
As a result, the content of the affected file were to be dependent on the order of unpacks.
This problem is also currently being prevented by the moratorium.
Had the moratorium been lifted for the bookworm release, we know that problems would have been caused in a small two-digit number of cases.
zutils diverts files from
/bin and a number of packages such as
molly-guard divert power management tools such as
Beyond diversions issued by packages, local diversions added by an administrator may also become ineffective.
When packages use
update-alternatives, the alternative location or one of its providers may refer to an aliased location.
As packages move files to their canonical locations, they might want to move their provided alternatives as well.
Just replacing the location in the
update-alternatives --install invocation is actively harmful in this case as the aliased location would not be removed.
If it were to be removed, a user configuration may inadvertently be deleted unless care is taken to preserve it.
Similarly, we may want to canonicalize the location of the alternative itself.
If it were just moved,
update-alternatives would have two seemingly distinct alternatives that conflict with one another.
If such a move is desired, it must be carefully coordinated among all alternative providers.
Last but not least, the choice of alternative provider is usually referred to using an absolute path.
Therefore, this path is part of the interface and is often scripted via automation tools such as
Changing this path would break such automation.
These problems affect a small two-digit amount of cases. They are also mitigated by the moratorium.
When packages move their files to canonical locations, a
dpkg-statoverride may still refer to the aliased location and thus become ineffective.
This could happen if files were moved without updating the corresponding maintainer scripts accordingly.
Usually, statoverrides are issued in the same package that contains the files being modified.
This affects a one-digit amount of cases and the moratorium is effective as well.
A statoverride may also be configured as an administrative change. As files are canonicalized, such overrides become ineffective without any warning.
Andreas Beckmann discovered that an empty directory may unexpectedly disappear when a package containing an aliased location is being removed as part of a package upgrade or removal.
The first instance of this problem is with
systemd, but it is a generic problem.
The generic problem affects a two-digit number of situations.
This problem is not mitigated by the moratorium and really affects upgrades from
bookworm and package removal on
As the moratorium is lifted, empty directories moved from
/usr also will go absent.
Like directories can be shared between different packages (as in P6), regular files with identical content can be shared between multiple instances of a
Multi-Arch: same package.
When upgrading one instance such that a location is canonicalized and removing another instance, the canonicalized file may be lost in the transaction if the removal happens after the upgrade.
This scenario has been reproduced in an artificial case.
It is not observed in practice, because it is also mitigated by the moratorium.
Had the moratorium been lifted for the bookworm release, we know that a two-digit number of case would have been affected by this.
In most cases, the files affected by loss are
udev rules and
hwdb files contained in shared library packages.
Filesystem bootstrap implementations have diverged due to
debootstrap now installs the aliasing symbolic links prior to the initial package extraction.
Whether it performs the merge is dependent on the chosen
Other tools such as
multistrap rely on the
usrmerge package for merging.
As we canonicalize files in packages, that latter strategy will fail once essential files such as
/bin/sh or the dynamic linker become canonicalized, because running
usrmerge.postinst becomes impossible.
Conversely, the former strategy fails if a package such as
base-files were to actually contain the aliasing symbolic links.
The moratorium also prevents these effects from happening in practice.
There is a mail with more detail.
As more and more packages release aliased locations, eventually one package is the last package that contains a location referring to a top-level symbolic link.
When upgrading or removing that package,
dpkg sees that the location is released and deletes it - in effect deleting an important symbolic link.
libc6:amd64 is the only package that contains
Canonicalizing its files would cause
/lib64 to be deleted and make the dynamic linker unavailable.
This is prevented by the moratorium for now.
As of this writing and in Debian bookworm, the root filesystem used by the Debian installer, which is constructed from
udeb packages, is not merged.
If packages were to move their files to canonical locations and thereby affect the contents of
udeb packages, that could make the installer dysfunctional.
When diverting a location that is provided as a symbolic link, the link may become relative due to policy 10.5. If the destination of a diversion is located in a different directory than the diverted path, a relative symbolic link may become broken.
Some proposed mitigations (M7, M18) rely on declared
Conflicts to prevent aliasing effects from taking effect during concurrent unpacks.
However, concurrent unpack is possible in some situations despite declared
It can be experienced by invoking dpkg directly and policy is being clarified.
Since experiencing it with apt seems unlikely (unless mutual conflicts are involved), so it is not clear yet whether this needs to be mitigated.
Many but not all problems relate to tools contained in the
dpkg to improve this situation is a natural thought.
We classify modifications to
dpkg in three different categories:
dpkg without adding to its API.
dpkg’s API (such as adding an option or a new
control.tar member) is changed.
If changes to
dpkg are part of the solution, we have to further spread the transition.
dpkg usually picks up new
glibc symbols and therefore gains a
Pre-Depends on the new
Therefore we cannot assume a fixed
dpkg for unpacking
Other than that, a number of mitigations rely on implementation-defined behaviour of
In particular, the various mitigations that employ diversions, use
dpkg features in ways that they were not meant to be used (e.g. diverting directories).
dpkg about aliases
One approach is to explicitly tell
dpkg about the relevant symbolic links such that it can identify canonical and aliased locations.
This strategy can resolve (depending on how many tools use this information) most problems, except for the bootstrapping aspects (P8).
It is considered an explicit change and can take multiple forms.
An earlier version of this document proposed the addition of an
--add-alias option for
dpkg to record aliasing symbolic links after their introduction.
Simon Richter proposed adding a
control.tar member to record aliases.
A recurring suggestion is to hard code the aliases used by Debian into
That latter approach has the downside of affecting unmerged installations such as old releases when using the
--root option as well as non-Debian users of
dpkg and is effectively ruled out for that reason.
In any case, this is a new feature in
dpkg with a fairly involved implementation as it touches on core data structures.
Since it adds to the API in non-trivial ways, it is not something we can remove anytime soon, so it adds to the permanent maintenance cost of
It is plausible that this feature interferes badly with other developments in
dpkg such as filesystem metadata tracking.
Any package that wants to rely on the new behavior requires a versioned
Doing this to
libc6 would introduce a dependency cycle.
While moving files to their canonical location causes most of the problems, the final state of having all of them moved frees us from the majority of aliasing related problems as well. Once everything is moved and a new policy prohibits adding files in aliased locations, problems P1, P2, partially P3 (for package authored diversions), partially P5 (for package authored statoverrides), P6, P7 and P9 become irrelevant for the future. For this reason, moving files also is a mitigation strategy if done exhaustively. The process of doing this requires a combination of other mitigations in order to not break existing use cases, such as smooth upgrades or bootstrapping.
dpkg use the filesystem as source of truth
Problems P1, P6, P7 and P9 are concerned with unexpected deletion of filesystem resources that are subject to aliasing.
As such, a targeted behavior change to the code dealing with deletion of unused filesystem resources in
dpkg is plausible.
dpkg could be updated to perform a new check prior to deleting filesystem objects.
dpkg normally considers its internal database as the sole source of truth, it can be changed to determine the canonicalized location according to the actual filesystem before performing its deletion.
If that resolved location happens to be known in the internal database, a warning can be emitted instead of deleting the file.
A similar mitigation already implemented in
dpkg converts the attempt to delete a theoretically empty directory that happens to not actually be empty into a warning.
This is a significant deviation from the principles
dpkg is built upon, but it is a fairly isolated change that is only useful during the move of filesystem resources to their canonical locations.
For systems that do not exercise aliasing, the only downside is a (hopefully minor) degradation in performance of upgrades and removals.
Once all files have been canonicalized (M2), this change can be reverted in
dpkg given that it has warned about relying on this.
As such, this change does not impose a permanent cost to maintaining
It also requires versioned
dpkg, which again is impossible for
This mitigation needs to be part of two stable releases to accommodate derivatives and external repositories.
For each of the aliasing symbolic links, we can introduce a diversion that redirects it to some unimportant location.
Since diversions are not intended to be used with directories,
dpkg only applies a diversion to the exact filename that is being diverted.
When adding a diversion for one of the aliasing symbolic links, files that are installed below that directory component are unaffected by the diversion.
Any attempt to remove a diverted symbolic link will instead remove the corresponding unimportant location.
In order to avoid a
Pre-Depends loop, the diversions are created by a dependency-less package (e.g. a new
libc6 as the sole owner of
/lib64 needs a
Pre-Depends and can do so without introducing a loop.
base-files is a prominent owner of many other directories that have become symlinks and also needs such a
With these in place, P9 is addressed.
The diversions can be removed if the symlinks are installed into some
data.tar or after two stable releases to accommodate external packages and derivatives.
/sbin as directories and as long as it contains them,
dpkg will not delete the symbolic links that are now placed there.
base-files were to also ship
/lib64 and all other multilib directories, that would effectively prevent them from being deleted, thus mitigating P9.
postinst script could still convert them to symbolic links unless already converted.
base-files to ensure the right unpack order, as doing so would create a loop.
We can spread this part to two releases or additionally ship the directories in a dependency-less package (e.g.
dpkg-divert can be wrapped by a script that modifies its behavior using the diversion mechanism on itself.
Whenever a diversion is added for a location that is subject to aliasing, the diversion is duplicated to both affected locations by the wrapper.
Similarly, removal of diversions is also duplicated.
Upon installation of the wrapper, all existing diversions are also duplicated.
If combined with M4, these diversions need to be ignored.
Packages that use diversions and packages that canonicalize files affected by diversions need to issue a
Pre-Depends on the package that installs this wrapper (e.g. a new
The affected packages can be determined in a mechanical way.
The wrapper would be required for at least two stable releases.
This way P3 can be mitigated.
The wrapper can also duplicate local diversions. However, for removing the wrapper (and thus removing the aliased diversions), users must update their scripts and automation to canonicalize the locations to be diverted.
When files are moved between packages, these are usually accompanied with
A concurrent canonicalization may render this
Replaces ineffective and cause the file loss described in P1.
This scenario cannot happen when the replacing package is unpacked after the replaced package has been upgraded or removed.
As such, changing
Conflicts makes this scenario impossible.
The mitigation can be applied in an as-needed manner such that moving files without canonicalizing remains unaffected as does canonicalizing without moving.
The situation can be detected using automated tools and reported mechanically.
If packages had their content canonicalized in bookworm, a low two-digit number of packages would have their
Replaces changed to
This strategy mostly mitigates P1, but it can become impossible to apply when essential packages are involved (as those must not be temporarily deinstalled) or the upgrade becomes too complex for
apt due to an excess of
These cases can be complemented with the next mitigation M8.
Also consider the limited effectiveness of this mitigation due to P12.
A package that is at risk of loosing files as in P1 can set up a protective diversion for each affected location in the aliased form.
preinst script has to set up these temporary diversions.
When the replacing
postinst is run, the replaced package is already upgraded or removed (due to associated
Breaks) and it can therefore remove the protective diversions.
These diversions only exist during an upgrade, but writing the maintainer scripts can be difficult to get right.
Therefore, M7 should be preferred when applicable.
In a similar vein to M8, empty directories can be saved in a P6 scenario.
If there is one and only one package shipping the empty directory, it can set up a protective diversion in a permanent way.
dpkg errors out when creating a diversion for an existing directory as it is not meant to be used that way, but this can be worked around by temporarily moving the directory away.
As long as the diversion exists, the package owning the diversion must ensure that the diverted location actually is a directory in the filesystem.
Otherwise, unpacking other packages may fail.
These diversions probably need to stay for two stable releases.
A directory can also be empty in multiple packages.
In most such cases, this seems to be accidental and the directory can be deleted from the
In other cases, the empty directory can be migrated to a common package, but this is expected to not be needed in practice.
In a similar vein to M8 and M9, shared files in
Multi-Arch: same instances can be saved in a P7 scenario.
preinst can divert the aliased locations contained in the old version, but it must assign these diversions to some other package (e.g.
usrmerge-support) in order to become effective for itself.
Likewise the new
postinst can remove these diversions, because all other instances must have been removed or unpacked by this time.
As with M8, such diversions only exist during an upgrade.
In order to fix bootstrapping tools, the aliasing symbolic links can be shipped in some
data.tar of a binary package (e.g.
Doing so requires that all packages participating in filesystem bootstrap have canonicalized their paths first, because we would otherwise get a directory vs symbolic link conflict.
In the presence of such a conflict, the behavior of
dpkg depends on the order of unpacks.
Instead of avoiding the conflict, all bootstrapping tools can be updated to unpack the symbolic link package before all other packages .
It also requires changing
debootstrap to no longer create the links prior to extraction as it presently passes
tar, which would result in an unpack failure even without any conflict between the various
With this, P8 is addressed.
In order to address the missing trigger invocations from P2, the one-digit number of trigger interests can be manually duplicated. A relation of triggering packages on the triggered package is not required, because configuration of the triggered package is equivalent to a trigger invocation and from that point on triggers work as expected. This mitigation also needs to last for two stable releases.
However, this does not at all mitigate trigger interest in external packages. External packages declaring an interest in aliased locations need to have this mitigation applied as well.
If we keep the alternatives and alternative providers at their aliased locations, we can still move the package contents in the
data.tar without otherwise impacting the use of alternatives.
The major downside is that we have to eternally remember that some locations in alternatives are expressed in a non-canonical way.
In a sense, we can skip P4 by not canonicalizing alternatives.
A wrapper to
update-alternatives can canonicalize all paths and ensure that paths may be referred to by either way.
All packages relying on this behavior must issue a
Depends on the package introducing the wrapper, thus solving P4.
For the trixie release, both aliased and canonical locations would be accepted such that external repositories and derivatives have a full release cycle to get updated.
The few packages that install statoverrides can migrate them on their own, mitigating P5.
This totally leaves local statoverrides and statoverrides from external packages unaddressed.
We can consider modifications to the bootstrap protocol to alleviate the problems. Thus far no concrete proposals to this end have emerged.
To avert the loss of empty directories, they could be duplicated to both the canonical and aliased location.
Since both locations are recorded in the
dpkg will not delete them.
Diversions of aliased locations can be duplicated to the corresponding canonical location to make them effective in both situations.
If the destination of those diversions equal up to aliasing and a diverted package moves a file from
dpkg’s usual protection against removing just unpacked files fails and we experience file loss.
Therefore, the diversion destinations must differ in more than just aliasing.
If the diverting package needs access to the original implementations, it has two options.
It can either take care to look up both diverted locations or declare
Breaks for all providers that have not moved their files to
Packages that are being diverted must take care that they are not unpacked in the presence of a diverter that has not been updated yet.
In principle, this is possible by declaring
Conflicts for diverters, but due to P12, this approach is not always sufficient.
In particular, when the diverter declares
Breaks with the provider, this is known to cause file loss.
The alternative is to detect the diversions in the divertee’s
preinst and duplicate them on behalf of the diverting package.
When this is done, the diverting package has to be modified to consider the case where the divertee has duplicated the diversions.
Also the divertee has to handle the case where the diverter has been removed after its
preinst has been run and delete the duplicated diversion in such a case.
There is an example implementation of this approach for cryptsetup and cryptsetup-nuke-password.
We recognize that effectively,
debootstrap has changed the bootstrap protocol, but other implementations have not changed.
Adding the aliasing symlinks to any package (M11) would therefore break
This can be mitigated by performing the merge after the initial unpack before the initial configuration.
Doing so partially reverts the change in bootstrap protocol implemented in
debootstrap such that this change becomes a no-op for
trixie and beyond.
When an empty directory is lost as part of an upgrade, the
postinst script runs after the unpack phase loosing the directory and can manually restore the directory.
When an empty directory is lost due to a different package being removed, this removal can activate a trigger and similarly invoke a maintainer script of the package owning the empty directory to restore it.
Since a directory is kept when it is non-empty, adding any file to it will prevent its loss.
In order to avoid breaking the Debian installer, we can perform the same aliasing that we are doing in the main archive. This has been implemented.
Files lost in an a Multi-Arch upgrade scenario are definitely lost by the time the upgraded
Shared files that used to be aliased can be augmented with a hard link in the upgraded package.
This hard link will persist in the installation even though it is not otherwise needed.
postinst script can check for the existence of possibly lost files and restore them from the other link name.
This approach yields to a window during which affected files actually are missing.
For instance, reloading
udev at such a time may lead to rules not being applied.
In the following table, we map mitigations to their properties.
The most fundamental property is which problems they fully (✓) or partially (*) address.
We also record whether explicit (E), implicit (I) or no ( ) changes are required to
The order of affected packages is judged as “significantly changed packages + mechanically changed packages”.
A mitigation is considered temporary if the relevant changes can be reverted after two stable releases.
A prototype is linked when available.
When a mitigation is incompatible with another, this is noted.
Also indicate which mitigations happen to work when trivially (✓) backported to
bookworm and which can be made to work with effort (*).
In effect, the most fundamental decision becomes how much change we want in
On one end, we can make it aware of aliasing and move files to their canonical location only as a measure to simplify packaging (M1).
As a middle ground, we can accept a temporary behavior difference (M3) to ease a complete move (M2) with additional mitigations (to be selected).
On the other end, we can work around the problematic behavior (to be selected) while moving the files (M2) without applying any changes to the
dpkg source code.
The other fundamental decision becomes how to deal with architecture bootstrap. On one end is an underspecified idea of changing the bootstrap protocol somehow (M16). The other end is shipping the aliasing symlinks in some package (M11), but that implies (M2).
firstname.lastname@example.org indicates that the project prefers to finish the transition without relying on changes to
dpkg as primary mechanism.
This amounts to rejecting M1 and selecting M2.
While there is agreement on this, there is disagreement on the reasons for this choice.
Regarding the bootstrapping matter, a small amount of developers that previously worked on
debootstrap reached consensus on partially reverting the bootstrap protocol changes added to
Proposed combination of mitigations:
This combination resolves the majority of problems. For P3 (ineffective diversions), we may either choose a central mitigation (M6) or a decentral mitigation (M18). For P6 (empty directory loss), the majority of instances are being deleted and the remaining ones may choose between M20 (maintainer scripts and triggers) and M21 (placeholder file) and maybe also M9 (protective diversions) on a case-by-case way in cooperation with relevant package maintainers.
We continuously monitor the archive for problematic situations.
When problems are detected, bugs (including RC bugs) are automatically filed (continuous MBF) for the affected packages.
Concurrently, mitigations that are compatible with the moratorium are implemented (e.g. deleting unused empty directories, M12, M19).
debhelper is being extended with a new
dh_usrmerge helper that performs the moving of files in package contents.
This helper can also be enabled using the new
usrmerge addon or a sufficiently high compatibility level.
A MBF at minor severity informs maintainers about possible problems when canonicalizing paths inside their packages.
Once the automatic bug filing works and buildds are converted to merged chroots, the file move moratorium is lifted.
A small set of essential packages (including
util-linux) still must keep some of their files in aliased locations.
Maintainers are asked to always upload changes that move files between packages and changes in canonicalization to
experimental first and only proceed to
unstable if they do not receive an automatic bug report within three days.
To speed up the process, patches and NMUs are used to convert packages.
Once most of the transitively essential packages are converted, a concurrent upload of remaining packages is coordinated with the affected maintainers and NMUed.
This ensures that no transitively essential package ships aliased paths and
base-files contains the aliasing links.