Meltdown & Spectre vulnerabilities: What we're doing about them


#1

Last night a set of vulnerabilities in computer processors was announced that affects the security of almost all computer hardware, including Bytemark’s Cloud Servers. The vulnerabilities affect the CPU at the heart of any computer, opening a channel allowing security boundaries between different users of a computer to be broken. The vulnerabilities have been named Meltdown and Spectre. Meltdown affects Intel processors only, which are used to run Bytemark’s Cloud Servers.

Unfortunately these vulnerabilities surfaced a few days before a planned embargo end date, so we’re not certain that all aspects of the solution are publicly available yet.

So far we have decided on two actions: 1) rebuilding the Linux kernels that host our customers’ Cloud Servers, and 2) updating the microcode for our Intel CPUs. This will mitigate the Meltdown vulnerability. It will also be useful for starting to address Spectre.

As with every other security update to our Cloud Server platform, we’ll apply it using live migration. So customers should not see any interruption to their service as we refresh our software and reboot our own systems. We will start on this work by the end of the day, and are aiming to have the work finished by Tuesday 9th January (we’ll confirm this when it’s done).

However, information on the bugs is still emerging, and we may have to repeat this operation with newer software in the coming weeks.

We’re particularly concerned that the Spectre vulnerability is still being understood, and we’d expect that there will be knock-on effects.

Advice for server administrators

As this vulnerability announcement has broken an embargo date, most OS vendors do not have complete patches or advice available yet.

We would therefore advise (as always) that you continue to apply vendor supplied security updates as they emerge, but to expect some particularly important ones in the next few days.

If we have particular advice that is specific to Cloud Server administrators, or have to enforce a reboot to apply patches, we’ll let you know here.


#2

All of our Cloud Server hosts have now been patched for Meltdown and everything appears to be running ok.


#3

I found with my BigV Debian Jessie instance, the jessie-updates repo was not enabled by default, only Debian security. However the package linux-image-3.16.0-4-amd64 update recommended to address Meltdown is available in jessie-updates, not Debian security.

Therefore doing an apt-get upgrade against the security repo could lead to a false sense of security :slight_smile:

After upgrading against jessie-updates, I got the new kernel and saw

Found kernel: /vmlinuz-3.16.0-5-amd64
Found kernel: /vmlinuz-3.16.0-4-amd64
Found kernel: /vmlinuz-3.2.0-4-amd64
Updating /boot/grub/menu.lst … Updating the default booting kernel

Then rebooted and thought the job was done. However uname -a reported that I was still running 3.2.0-4.

For some reason /boot/grub/menu.lst had the default kernel to boot set to 4 rather than the usual zero (top of the list).


#4

Looks like being stuck on kernel 3.2.x may be a consequence of this particular instance starting life on Debian Wheezy and being upgraded to Jessie later.


#5

I suspect it’s more likely that you’ve got ‘savedefault’ in your grub config file or something similar (depending on grub variant).