OVH Custom Kernel problems after upgrading Docker

I recently encountered a weird problem when updating my Debian stretch Docker server at OVH. During the upgrade process I noticed that both service Containerd and the Docker daemon (which depends on
containerd) failed to start, so I started investigating and was looking for suspicious things in my syslog logfile and found the following:

1
2
Nov  8 16:25:03 ns3085344 modprobe[2234]: modprobe: FATAL: Module overlay not found in directory /lib/modules/4.9.33-mod-std-ipv6-64
Nov 8 16:30:01 ns3085344 modprobe[2893]: modprobe: ERROR: ../libkmod/libkmod.c:514 lookup_builtin_file() could not open builtin file '/lib/modules/4.9.33-mod-std-ipv6-64/modules.builtin.bin'

I got confused, because the file was right there.. nothing out of the ordinary.

1
2
3
4
5
6
7
8
9
[email protected]:~# stat /lib/modules/4.9.33-mod-std-ipv6-64/modules.builtin.bin
File: /lib/modules/4.9.33-mod-std-ipv6-64/modules.builtin.bin
Size: 0 Blocks: 0 IO Block: 4096 regular empty file
Device: 903h/2307d Inode: 4718609 Links: 1
Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root)
Access: 2018-11-08 16:17:26.556194318 +0000
Modify: 2018-11-08 16:13:13.579640187 +0000
Change: 2018-11-08 16:13:13.579640187 +0000
Birth: -

So I started looking through some forums, but couldn’t find anything useful. I contacted a good friend of mine, Shyim. After digging around he found some interesting resources regard
the OVH kernel. It seems like OVHs custom linux kernel is the cause of this issue.

So we started investigating and tried to remove the kernel from the server and replace it with the official one from Debian. Installing the new kernel was fairly easy: apt-get install linux-image-amd64

After installing, I connected to my server using OVHs IPMI web console and chosed the new installed kernel during the boot process. Everything was working again, yay!

But I wanted to get rid of the nasty kernel and remove the boot option. I simply moved the configuration file and updated grub:

1
2
3
4
5
6
7
[email protected]:~# mv /etc/grub.d/06_OVHKernel ~/06_OVHKernel.backup
[email protected]:~# update-grub
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-4.9.0-8-amd64
Found initrd image: /boot/initrd.img-4.9.0-8-amd64
Adding boot menu entry for EFI firmware configuration
done

One more reboot aaaand YES. The old kernel is gone, all my services are working again and my bootloader config has been updated.

I still don’t know what caused this issue, but as far as I can tell I’m not the only one having this issue. We also noticed they probably just copied their kernel onto my server, because we
couldn’t find it as installed package

1
2
[email protected]:~# dpkg -l | grep ovh
[email protected]:~#

I hope this post helps you in case you got the same problem.