Last week, it came to our attention that some Bytemark servers which were provisioned with certain, recent Linux distributions share some of the same SSH host keys.
During the server imaging process, SSH host keys were routinely being removed and regenerated but certain keys from the imager remained in place.
This by itself is not a major cause for concern, but it is possible that affected servers are more vulnerable to man-in-the-middle attacks using these keys. However this would require additional steps to exploit, for example DNS spoofing of network addresses. It would not allow casual drive-by decryption of traffic since SSH uses disposable session keys for encryption.
Once we discovered this issue, we took the following steps:[ulist]
sent a list of SSH host keys present in the image for all distributions to our managed services team, so that they could proactively replace any keys that might not be unique.notified all affected customers and provided instructions on how to fix this for themselves.
removed all existing SSH host keys in our imager, for all distributions.
our image management tools now remove any host keys left over after creating or updating an image.
new ECDSA host keys will not be generated for images that had them previously. This will be re-instated in the future.
the imaging process makes sure all host keys are removed, before generating new ones.[/ulist]
We also committed to publishing this forum post today.
So far, we’re unaware of any servers that have been exploited as a result of this vulnerability.
However, as soon as this flaw was identified, we took proactive steps to inform customers and minimise any additional risk.
How to check if your keys are non-unique
Your machine may have three or four types of keys of which only RSA, DSA, and ECDSA keys need to be checked, if your machine has them. These will usually be in files in /etc/ssh named ssh_host_rsa_key, ssh_host_dsa_key or ssh_host_ecdsa_key. For each of these files run:
$ ssh-keygen -l -f /etc/ssh/ssh_host_rsa_key
$ ssh-keygen -l -f /etc/ssh/ssh_host_dsa_key
$ ssh-keygen -l -f /etc/ssh/ssh_host_ecdsa_key
Each command will give you a fingerprint which can check against the published list of known-bad fingerprints shown below. You only need regenerate keys that have fingerprints that appear in the list below.
How to generate fresh host keys
In order to regenerate the affected keys, you should run the following comands on the affected machine, as root. Remember, you only need regenerate the keys with fingerprints that appear in the list below.
[code] # rm /etc/ssh/ssh_host_ecdsa_key /etc/ssh/ssh_host_ecdsa_key.pub
# /usr/bin/ssh-keygen -t ecdsa -N ‘’ -f /etc/ssh/ssh_host_ecdsa_key
# rm /etc/ssh/ssh_host_dsa_key /etc/ssh/ssh_host_dsa_key.pub # /usr/bin/ssh-keygen -t dsa -N '' -f /etc/ssh/ssh_host_dsa_key # rm /etc/ssh/ssh_host_rsa_key /etc/ssh/ssh_host_rsa_key.pub # /usr/bin/ssh-keygen -t rsa -N '' -f /etc/ssh/ssh_host_rsa_key # /etc/init.d/ssh restart[/code]
Please be aware that the next time you log in to your machine over SSH you may be greeted with a large warning message stating that the remote host identification has changed. This is expected since you’ll have changed the host keys.
While all affected customers have been contacted, please get in touch if you have any questions or concerns - we’re at email@example.com, or drop a comment on this forum post.