So, sometimes people act before they think, sometimes that leads to problems that appear to the uninitiated to be catastrophic. But then the great thing about Linux, BSD and Solaris is that generally nothing is totally catastrophic, well almost nothing.
In this case the problem occurred during an upgrade, the identities of the guilty will be obscured, but the lesson may be useful regardless.
The problem started with something quite innocuous. A problem with an upgrade.
Unpacking replacement coreutils
dpkg: error processing /var/cache/apt/archives/coreutils_7.4-2_sparc.deb (--unpack):
trying to overwrite `/usr/share/man/man1/arch.1.gz', which is also in package util-linux
dpkg-deb: subprocess paste killed by signal (Broken pipe)
Errors were encountered while processing:
/var/cache/apt/archives/coreutils_7.4-2_sparc.deb
E: Sub-process /usr/bin/dpkg returned an error code (1)
server-002:/home/tech-003#
So, the installation of the latest version of coreutils cannot overwrite a man page also provided by util-linux. There are a number of ways to get around it, however the one chosen wasn’t one of the greatest ideas…
The tech in question decided that the best approach would be to remove both offending packages and reinstall them…
server-002:/home/tech-003# dpkg -r coreutils
dpkg: error processing coreutils (--remove):
This is an essential package - it should not be removed.
Errors were encountered while processing:
coreutils
Dpkg quite sensibly refuses.
Not to be thwarted however, our diligent tech decides that a little force will solve the problem…
server-002:/home/tech-003# dpkg --force-all -r coreutils
dpkg: warning: overriding problem because --force enabled:
This is an essential package - it should not be removed.
dpkg: coreutils: dependency problems, but removing anyway as you requested:
dpkg depends on coreutils (>= 5.93-1).
linux-image-2.6.18-6-sparc64-smp depends on coreutils (>= 5.96).
ucf depends on coreutils (>= 5.91).
debianutils depends on coreutils (>= 4.5.8-1).
(Reading database ... 35142 files and directories currently installed.)
Removing coreutils
So coreutils is removed, with dpgk screaming about it not being a good idea. Turns out dpkg was right…
server-002:/home/tech-003# apt-get install coreutils
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following NEW packages will be installed
coreutils
0 upgraded, 1 newly installed, 0 to remove and 352 not upgraded.
Need to get 0B/3814kB of archives.
After this operation, 11.2MB of additional disk space will be used.
WARNING: The following packages cannot be authenticated!
coreutils
Install these packages without verification [y/N]? y
Selecting previously deselected package coreutils.
(Reading database ... 34831 files and directories currently installed.)
Unpacking coreutils (from .../coreutils_7.4-2_sparc.deb) ...
dpkg: error processing /var/cache/apt/archives/coreutils_7.4-2_sparc.deb (--unpack):
trying to overwrite `/usr/share/man/man1/arch.1.gz', which is also in package util-linux
dpkg-deb: subprocess paste killed by signal (Broken pipe)
dpkg (subprocess): failed to exec rm for cleanup: No such file or directory
dpkg: error while cleaning up:
subprocess rm cleanup returned error exit status 2
Errors were encountered while processing:
/var/cache/apt/archives/coreutils_7.4-2_sparc.deb
E: Sub-process /usr/bin/dpkg returned an error code (1)
The key here of course is not that arch.1.gz cannot be overwritten, but rather that ‘rm’ cannot be executed (it is provided by coreutils after all….
Rather than giving up, or calling for help, our diligent tech decides that removing util-linux will solve the problem…
Server-002:/home/tech-003# dpkg --force-all -r util-linux
dpkg: warning: overriding problem because --force enabled:
This is an essential package - it should not be removed.
dpkg: util-linux: dependency problems, but removing anyway as you requested:
util-linux-locales depends on util-linux (>= 2.12r-0).
util-linux-locales depends on util-linux (< < 2.12r.0-0).
util-linux-locales depends on util-linux (>= 2.12r-0).
util-linux-locales depends on util-linux (< < 2.12r.0-0).
(Reading database ... 34830 files and directories currently installed.)
Removing util-linux ...
install-info(ipc): could not backup /usr/share/info/dir in /var/backups/infodir.bak: No such file or directory
Hmm, so no coreutils and no util-linux. The next step, would have been to install the pair again...
server-002:/home/tech-003# apt-get install coreutils util-linux
Reading package lists... Done
Building dependency tree
Reading state information... Done
You might want to run 'apt-get -f install' to correct these:
The following packages have unmet dependencies.
util-linux-locales: Depends: util-linux (< 2.12r.0-0) but 2.15.1~rc1-1 is to be installed
E: Unmet dependencies. Try 'apt-get -f install' with no packages (or specify a solution).
server-002:/home/tech-003#
Of course now we have a rather broken system so a repair is needed to fix the varios dependency issues...
Server-002:/home/tech-003# apt-get -f install
Reading package lists... Done
Building dependency tree
Reading state information... Done
Correcting dependencies...Done
The following packages were automatically installed and are no longer required:
sensible-utils
Use 'apt-get autoremove' to remove them.
The following extra packages will be installed:
coreutils sensible-utils util-linux util-linux-locales
Suggested packages:
dosfstools
The following NEW packages will be installed
coreutils sensible-utils util-linux
The following packages will be upgraded:
util-linux-locales
1 upgraded, 3 newly installed, 0 to remove and 350 not upgraded.
Need to get 4442B/5701kB of archives.
After this operation, 13.9MB of additional disk space will be used.
Do you want to continue [Y/n]? y
WARNING: The following packages cannot be authenticated!
coreutils util-linux-locales util-linux sensible-utils
Install these packages without verification [y/N]? y
Get: 1 http://debian.virginmedia.com unstable/main sensible-utils 0.0.1 [4442B]
Fetched 4442B in 0s (25.0kB/s)
dpkg (subprocess): failed to exec rm for cleanup: No such file or directory
dpkg: error processing /var/cache/apt/archives/coreutils_7.4-2_sparc.deb (--unpack):
subprocess rm cleanup returned error exit status 2
Errors were encountered while processing:
/var/cache/apt/archives/coreutils_7.4-2_sparc.deb
E: Sub-process /usr/bin/dpkg returned an error code (1)
server-002:/home/tech-003#
Ah, still no ‘rm’… so a failure on clean up.
Confused our diligent tech decides to look for ‘rm’…
server-002:/home/tech-003# rm
rmail rmmod rmt rmt-tar
Which isn’t there…
server-002:/home/tech-003# ls
bash: ls: command not found
Actually there are quite a few things missing…
server-002:/home/tech-003# mv
bash: mv: command not found
server-002:/home/tech-003# cp
bash: cp: command not found
server-002:/home/tech-003# touch
bash: touch: command not found
Time to start panicking…
Now, happily this server was also an NFS server, happily up and running doing it’s NFS thing. Our diligent technician finally went for help. So first things first, it was rather necessary to get some of those core userland tools cp, mv, ls and rm back in place. The easiest way it seemed was to grab (from another system) the relevant binaries, for sparc, and put them somewhere accessible to the still logged in root user.
The relevant package was downloaded from the local apt repository and unpacked onto the NFS share.
Then we could start copying the tools we needed back to somewhere useful…
server-002:/srv/disk1/bin# ./mv ./rm /bin/
We tried first with ‘rm’
server-002:/srv/disk1/bin# rm
rm rmail rmmod rmt rmt-tar
Which worked nicely and as expected…
So on with everything else:
server-002:/srv/disk1/bin# ./mv ./* /bin/
Now we have a more functional system back…
server-002:/srv/disk1/bin# cd /root/
server-002:~# ls
172.db printers.conf dhcpd.conf
All that was left to do was fix the dependency issues:
server-002:~# apt-get install -f
Reading package lists... Done
Building dependency tree
Reading state information... Done
Correcting dependencies...Done
The following packages were automatically installed and are no longer required:
sensible-utils
Use 'apt-get autoremove' to remove them.
The following extra packages will be installed:
coreutils sensible-utils util-linux util-linux-locales
Suggested packages:
dosfstools
The following NEW packages will be installed
coreutils sensible-utils util-linux
The following packages will be upgraded:
util-linux-locales
1 upgraded, 3 newly installed, 0 to remove and 350 not upgraded.
Need to get 0B/5701kB of archives.
After this operation, 13.9MB of additional disk space will be used.
Do you want to continue [Y/n]? y
WARNING: The following packages cannot be authenticated!
coreutils util-linux-locales util-linux sensible-utils
Install these packages without verification [y/N]? y
(Reading database ... 34756 files and directories currently installed.)
Unpacking coreutils (from .../coreutils_7.4-2_sparc.deb) ...
Replacing files in old package mktemp ...
Setting up coreutils (7.4-2) ...
(Reading database ... 35072 files and directories currently installed.)
Preparing to replace util-linux-locales 2.12r-19etch1 (using .../util-linux-locales_2.15.1~rc1-1_all.deb) ...
Unpacking replacement util-linux-locales ...
Selecting previously deselected package util-linux.
Unpacking util-linux (from .../util-linux_2.15.1~rc1-1_sparc.deb) ...
Setting up util-linux (2.15.1~rc1-1) ...
Installing new version of config file /etc/init.d/hwclock.sh ...
Selecting previously deselected package sensible-utils.
(Reading database ... 35216 files and directories currently installed.)
Unpacking sensible-utils (from .../sensible-utils_0.0.1_all.deb) ...
Replacing files in old package debianutils ...
Setting up util-linux-locales (2.15.1~rc1-1) ...
Setting up sensible-utils (0.0.1) ...
server-002:~#
Then we could go back to doing the upgrade…
What a fun 15 minutes.
