If you’ve been following this blog (my colleague, David Busby, posted about it yesterday) or any tech news outlet in the past few days, you’ve probably seen some mention of the “Heartbleed” vulnerability in certain versions of the OpenSSL library.
So what is ‘Heartbleed’, really?
In short, Heartbleed is an information-leak issue. An attacker can exploit this bug to retrieve the contents of a server’s memory without any need for local access. According to the researchers that discovered it, this can be done without leaving any trace of compromise on the system. In other words, if you’re vulnerable, they can steal your keys and you won’t even notice that they’ve gone missing. I use the word “keys” literally here; by being able to access the contents of the impacted service’s memory, the attacker is able to retrieve, among other things, private encryption keys for SSL/TLS-based services, which means that the attacker could be able to decrypt communications, impersonate other users (see here, for example, for a session hijacking attack based on this bug), and generally gain access to data which is otherwise believed to be secure. This is a big deal. It isn’t often that bugs have their own dedicated websites and domain names, but this one does: http://www.heartbleed.com
Why is it such a big deal?
One, because it has apparently been in existence since at least 2012. Two, because SSL encryption is widespread across the Internet. And three, because there’s no way to know if your keys have been compromised. The best detection that currently exists for this are some Snort rules, but if you’re not using Snort or some other IDS, then you’re basically in the dark.
What kinds of services are impacted?
Any software that uses the SSL/TLS features of a vulnerable version of OpenSSL. This means Apache, NGINX, Percona Server, MariaDB, the commercial version of MySQL 5.6.6+, Dovecot, Postfix, SSL/TLS VPN software (OpenVPN, for example), instant-messaging clients, and many more. Also, software packages that bundle their own vulnerable version of SSL rather than relying on the system’s version, which might be patched. In other words, it’s probably easier to explain what isn’t affected.
What’s NOT impacted?
SSH does not use SSL/TLS, so you’re OK there. If you downloaded a binary installation of MySQL community from Oracle, you’re also OK, because the community builds use yaSSL, which is not known to be vulnerable to this bug. In fact, anything that’s been built against yaSSL (or some other SSL library that isn’t OpenSSL) should be fine. Obviously, any service which doesn’t use SSL/TLS is not going to be vulnerable, either, since the salient code paths aren’t going to be executed. So, for example, if you don’t use SSL for your MySQL connections, then this bug isn’t going to affect your database server, although it probably still impacts you in other ways (e.g., your web servers).
What about Amazon cloud services?
According to Amazon’s security bulletin on the issue, all vulnerable services have been patched, but they still recommend that you rotate your SSL certificates.
Do I need to upgrade Percona Server, MySQL, NGINX, Apache, or other server software?
No, not unless you built any of these and statically-linked them with a vulnerable version of OpenSSL. This is not common. 99% of affected users can fix this issue by upgrading their OpenSSL libraries and cycling their keys/certificates.
What about client-level tools, like Percona Toolkit or XtraBackup?
Again, no. The client sides of Percona Toolkit, Percona XtraBackup, and Percona Cloud Tools (PCT) are not impacted. Moreover, the PCT website has already been patched. Encrypted backups are still secure.
There are some conflicting reports out there about exactly how much information leakage this bug allows. What’s the deal?
Some of the news articles and blogs claim that with this bug, any piece of the server’s memory can be accessed. Others have stated that the disclosure is limited to memory space owned by processes using a vulnerable version of OpenSSL. As far as we are aware, and as reported in CERT’s Vulnerability Notes Database, the impact of the bug is the latter; i.e., it is NOT possible for an attacker to use this exploit to retrieve arbitrary bits of your server’s memory, only bits of memory from your vulnerable services. That said, your vulnerable services could still leak information that attackers could exploit in other ways.
How do I know if I’m affected?
You can test your web server at http://filippo.io/Heartbleed/ – enter your site’s URL and see what it says. If you have Go installed, you can also get a command-line tool from Github and test from the privacy of your own workstation. There’s also a Python implementation. You can also check the version of OpenSSL that’s installed on your servers. If you’re running OpenSSL 1.0.1 through 1.0.1f or 1.0.2-beta, you’re vulnerable. (Side note here: some distributions, such as RHEL/CentOS, have patched the issue without actually updating the OpenSSL version number; on RPM-based systems you can view the changelog to see if it’s been fixed, for example):
1 2 3 | rpm -q --changelog openssl | head -2 * Mon Apr 07 2014 Tomáš Mráz <tmraz@redhat.com> 1.0.1e-16.7 - fix CVE-2014-0160 - information disclosure in TLS heartbeat extension |
Also, note that versions of OpenSSL prior to 1.0.1 are not known to be vulnerable. If you’re still unsure, we would be happy to assist you in determining the extent of the issue in your environment and taking any required corrective action. Just give us a call.
How can I know if I’ve been compromised?
If you’ve already been compromised, you have no way of knowing. However, if you use Snort as an IDS, you can use some rules developed by Fox-IT to detect and defer new attacks; other IDS/IPS vendors may have similar rule updates available. Without some sort of IDS in place, however, attacks can and will go unnoticed.
Are there any exploits for this currently out there?
Currently, yes, there are some proofs-of-concept floating around out there, although before this week, that was uncertain. But given that this is likely a 2-year-old bug, I would be quite surprised if someone, somewhere (you came to my talk at Percona Live last week, didn’t you? Remember what I said about assuming that you’re already owned? There are No Secrets Anymore.) didn’t have a solid, viable exploit.
So, then, what should I do?
Ubuntu, RedHat/CentOS, Amazon, and Fedora have already released patched versions of the OpenSSL library. Upgrade now. Do it now, as in right now. If you’ve compiled your own version of OpenSSL from source, upgrade to 1.0.1g or rebuild your existing source with the -DOPENSSL_NO_HEARTBEATS flag.
Once that’s been done, stop any certificate-using services, regenerate the private keys for any services that use SSL (Apache, MySQL, whatever), and then obtain a new SSL certificate from whatever certificate authority you’re using. Once the new certificates are installed, restart your services. You can also, of course, do the private key regeneration and certificate cycling on a separate machine, and only bring the service down long enough to install the new keys and certificates. Yes, you will need to restart MySQL. Or Apache. Or whatever. Full stop, full start, no SIGHUP (or equivalent).
Unfortunately, that’s still not all. Keeping in mind the nature of this bug, you should also change / reset any user passwords and/or user sessions that were in effect prior to patching your system and recycling your keys. See, for example, the session hijacking exploit referenced above. Also note that Google services were, prior to their patching of the bug, vulnerable to this issue, which means that it’s entirely possible that your Gmail login (or any other Google-related login) could have been compromised.
Can I get away with just upgrading OpenSSL?
NO. At a bare minimum, you will need to restart your services, but in order to really be sure you’ve fixed the problem, you also need to cycle your keys and certificates (and revoke your old ones, if possible). This is the time-consuming part, but since you have no way of knowing whether or not someone has previously compromised your private keys (and you can bet that now that the bug is public, there will be a lot of would-be miscreants out there looking for servers to abuse), the only safe thing to do is cycle them. You wouldn’t leave your locks unchanged after being burgled, would you?
Also note that once you do upgrade OpenSSL, you can get a quick list of the services that need to be restarted by running the following:
1 | sudo lsof | grep ssl | grep DEL |
Where can I get help and/or more information?
In addition to the assorted links already mentioned, you can read up on the nuts and bolts of this bug, or various news articles such as this or this. There are a lot of articles out there right now, most of which are characterizing this as a major issue. It is. Test your vulnerability, upgrade your OpenSSL and rotate your private keys and certificates, and then change your passwords.
Another attack vector that might be worth mentioning: the exploit does not only affect the server, both ends of the secure communication channel can be compromised. In theory, a malicious server could obtain information from a client application that’s linked against a vulnerable version of OpenSSL as well.
Lenz
“Any software that uses the SSL/TLS features of a vulnerable version of OpenSSH.”
I think that’s a typo.
@Dan, yeah, typo indeed. Fixed. Thanks.
Lenz,
Very interesting so you’re saying if we have hacked MySQL Server it can use this to request any data from the address space of the client, which can potentially disclose some confidential data ? But of course the issue is much more with general web clients which are much commonly connecting to random web sites
Peter: yes, that how I understood the description of the issue at http://heartbleed.com/ – “Bug is in the OpenSSL’s implementation of the TLS/DTLS (transport layer security protocols) heartbeat extension (RFC6520). When it is exploited it leads to the leak of memory contents from the server to the client *and from the client to the server.*”
I think this issue is more serious for web clients connecting to a malicious web site, just think of a web browser with multiple tabs open to other sensitive web sites (or your intranet sites).
In the case of MySQL, a hacked SSL-enabled MySQL Server could be used to exploit any application (e.g. a PHP web app) that connects to this server. In a multi-user scenario (e.g. web hosting environments), this could potentially affect hundreds of web sites.
Lenz
FYI, here’s a Python-based tool that attempts to exploit this vulnerability on the client side:
https://github.com/Lekensteyn/pacemaker
On RHEL6.5 we have used the following command:
lsof | grep -P “DEL.+libssl.so.1.0.1e$”
In some cases servers were not rebooted after an update to RHEL6.5, and not all services were restarted after the update either. So we did have some services using the older openssl.so.1.0.0 that was not vulnerable.
Generally, my recommendation is to perform a reboot after patching servers with important security updates (especially if these include general libraries or the kernel), but in this case you might have been slightly better of if you never did 😉
Hmm, this form seems to remove escape characters used in regular expressions (like \b or \w).
Not sure how I can prevent this:
Dag,
Thank you. Indeed very neat trick to check if some apps using old library are still running. Rebooting server is of course the most simple solution
Hi!
Something you might want to consider is that not all versions of MySQL are affected by Heartbleed. Any version compiled with yaSSL are safe.
Cheers,
-Brian
I get internal confirmation that MariaDB tar.gz and MSI binaries are safe but all the rest should be affected.
Heartbleed Advisory for Percona Users and Customers
https://www.percona.com/ceo-customer-advisory-heartbleed
Well, nice article on Heartbleed.