after apt upgrade to kernel 6.6.31 pi still boots into 6.6.20

[mgrouch]

Describe the bug

I did

sudo apt update
sudo apt upgrade

it installed kernel 6.6.31

however old kernel 6.6.20 is still not removed and takes lots of space

Worse, pi5 still boots into old 6.6.20 kernel

Steps to reproduce the behaviour

sudo apt update
sudo apt upgrade

it installed kernel 6.6.31

however old kernel 6.6.20 is still not removed and takes lots of space

Worse, pi5 still boots into old 6.6.20 kernel

Device (s)

Raspberry Pi 5

System

cat /etc/rpi-issue
Raspberry Pi reference 2024-03-15
Generated using pi-gen, https://github.com/RPi-Distro/pi-gen, f19ee211ddafcae300827f953d143de92a5c6624, stage2

vcgencmd version
2024/04/20 11:53:30 
Copyright (c) 2012 Broadcom
version d1744d21 (release) (embedded)

uname -a
Linux lysmarine 6.6.20+rpt-rpi-2712 #1 SMP PREEMPT Debian 1:6.6.20-1+rpt1 (2024-03-07) aarch64 GNU/Linux

Logs

No response

Additional context

No response

[mgrouch]

Unfortunately same doesn't work for me.

I got some warning during install about MD subsystem is not loaded. Might be relevant

Setting up linux-image-6.6.31+rpt-rpi-2712 (1:6.6.31-1+rpt1) ...
/etc/kernel/postinst.d/initramfs-tools:
update-initramfs: Generating /boot/initrd.img-6.6.31+rpt-rpi-2712
W: mkconf: MD subsystem is not loaded, thus I cannot scan for arrays.
W: mdadm: failed to auto-generate temporary mdadm.conf file.
'/boot/initrd.img-6.6.31+rpt-rpi-2712' -> '/boot/firmware/initramfs_2712'
/etc/kernel/postinst.d/z50-raspi-firmware:
'/boot/vmlinuz-6.6.31+rpt-rpi-2712' -> '/boot/firmware/kernel_2712.img'

Processing triggers for shared-mime-info (2.2-1) ...
Processing triggers for desktop-file-utils (0.26-1) ...
Processing triggers for initramfs-tools (0.142) ...
update-initramfs: Generating /boot/initrd.img-6.6.31+rpt-rpi-v8
W: mkconf: MD subsystem is not loaded, thus I cannot scan for arrays.
W: mdadm: failed to auto-generate temporary mdadm.conf file.
'/boot/initrd.img-6.6.31+rpt-rpi-v8' -> '/boot/firmware/initramfs8'
update-initramfs: Generating /boot/initrd.img-6.6.31+rpt-rpi-2712
W: mkconf: MD subsystem is not loaded, thus I cannot scan for arrays.
W: mdadm: failed to auto-generate temporary mdadm.conf file.
'/boot/initrd.img-6.6.31+rpt-rpi-2712' -> '/boot/firmware/initramfs_2712'
update-initramfs: Generating /boot/initrd.img-6.6.20+rpt-rpi-v8
W: mkconf: MD subsystem is not loaded, thus I cannot scan for arrays.
W: mdadm: failed to auto-generate temporary mdadm.conf file.
update-initramfs: Generating /boot/initrd.img-6.6.20+rpt-rpi-2712
W: mkconf: MD subsystem is not loaded, thus I cannot scan for arrays.
W: mdadm: failed to auto-generate temporary mdadm.conf file.
Processing triggers for hicolor-icon-theme (0.17-2) ...
Processing triggers for libc-bin (2.36-9+rpt2+deb12u7) ...
Processing triggers for man-db (2.11.2-2) ...

Did old kernels got deleted for you or they are still installed?

[mgrouch]

cat  /etc/apt/apt.conf.d/01autoremove 
APT
{
  NeverAutoRemove
  {
	"^firmware-linux.*";
	"^linux-firmware$";
	"^linux-image-[a-z0-9]*$";
	"^linux-image-[a-z0-9]*-[a-z0-9]*$";
  };

  VersionedKernelPackages
  {
	# kernels
	"linux-.*";
	"kfreebsd-.*";
	"gnumach-.*";
	# (out-of-tree) modules
	".*-modules";
	".*-kernel";
  };

  Never-MarkAuto-Sections
  {
	"metapackages";
	"tasks";
  };

  Move-Autobit-Sections
  {
	"oldlibs";
  };
};

so old kernels are not removed by

apt-get autoremove -y --purge

What would be the way to properly remove them?

Thanks

[mgrouch]

sudo apt reinstall raspberrypi-kernel

installed even older kernel 6.1.21+

but on re-boot pi is booted into 6.6.20 kernel.

[pelwell]

Have you attempted to update from Bullseye to Bookworm, or similar? Nobody else is (or should be) having these problems.

[XECDesign]

sudo apt reinstall raspberrypi-kernel

That's definitely the wrong thing to do.

Your logs earlier shows the right version getting installed. If it's not booting the right one, then your partition mountpoints may be set up incorrectly.

Apt will keep the currently running kernel and the previous kernel available just in case something goes wrong and you need to copy the old one.

There's a proposal to add an option to raspi-config to remove unused kernels here,

[mgrouch]

I’m building an image on CircleCi.
I’m downloading 2024-03-15 lite image,
I mount all partitions and run apt update/upgrade.
Then unmount it and compress with xz.
Apt installs new kernels, doesn’t remove old one.
And on newly created image it boots with old kernel instead of new one.

[XECDesign]

I mount all partitions

Can you show how you do that?

[mgrouch]

I mount all partitions

Can you show how you do that?

Via these scripts:

https://github.com/bareboat-necessities/lysmarine_gen/blob/bookworm/cross-build-release/raspbian.sh
https://github.com/bareboat-necessities/lysmarine_gen/blob/bookworm/cross-build-release/lib.sh

And whole circleci starting point is here:

https://github.com/bareboat-necessities/lysmarine_gen/blob/bookworm/.circleci/config.yml

[XECDesign]

I think this should be /boot/firmware rather than just /boot.

[pelwell]

(with a corresponding change to the line above as well)

[jhmcclure]

I'm literally showing up in the middle of this, and I have no idea what the surrounding discussion is about, but I saw something about a RasPi not loading the newest installed kernel. I actually have one that's doing that.. It's a RasPi 3 (one of the two varieties). The discussion is VERY old, and I don't know how I can help, but I want to. One thing that's a bit unique about this box is that it has ONLY run headless. I think some upgrades may not have happened due to that fact. Make of that what you will. I don't have a reason to come to this Github very often, but you can "ping" me at "[email protected]" if I need to come look at something. Please tell me what info I can collect from this box that may help. This is an annoying problem, and I want to help.

[mgrouch]

I’m original reporter of the issue.
With proper handling of /boot/firmware
(In bookworm it’s now /boot/firmware instead of /boot) the issue was resolved for me.

[popcornmix]

I'll close this - it's not a linux kernel issue.