Massive Failures of Internet PKI

The Internet's Public-Key Infrastructure is Critical but Flawed

The Internet relies on a distributed system of Public-Key Infrastructure. We users want network security, obviously for encryption to protect the confidentiality of sensitive data in transit, but even more importantly for authentication of the servers to which we are sending our data. Unfortunately, the issuers of digital certificates trusted by our browsers, and therefore by us, have frequently violated that trust. The problem is not in the underlying technology. The problem is that not all issuers of digital certificates apply their security policies carefully and consistently at all times.

The technical side of setting up Public-Key Infrastructure or PKI is not that difficult. The OpenSSL package contains everything needed to establish your own PKI. My detailed description of how to set up secure RSyslog over TLS shows you how to do most of it.

We have the technology, we just fail to use it properly.

First, a few terms and concepts

A certificate authority or CA is an issuer of digital credentials. In human terms, they vouch for identity, they don't really vouch for character. As a physical-world example, I have both a U.S. passport and an Indiana (U.S.A.) driver's license. These objects represent the U.S. Department of State and the Indiana Bureau of Motor Vehicles vouching for my identity, that I somehow proved my identity to them according to their requirements (which are analogous to a CA's CPS or Certification Practices Statements).

Public-Key Infrastructure or PKI is based on asymmetric or public-key cryptography. Click here for "Just Enough Cryptography" and a light explanation of how public-key cryptography works. The extremely short version is:

• A key is a string of bits used to control the operation of a cipher used to encrypt and decrypt data.
• In public-key cryptography, keys are generated as pairs. The two keys of the pair are labeled as the public and private key of the pair.
• Those names are meaningful — the private key must be kept secret (or all security is lost) and the public key must be made public (or everything is pointless).
• There is no risk in making the public key truly public. Security is based on a certain category of math problem being so hard to solve that you don't worry about your adversary doing so fast enough to matter. If you're still worried, use a harder problem (this generally means longer key pairs).
• Anything encrypted with one key can only be decrypted with its mate.

Someone wanting to run a secure web site generates a key pair and then takes the public key to some CA. Whoever does this convinces the CA (as required by their CPS) that they really are acting on behalf of the private key owner.

The CA creates a digital certificate which is a data structure including (among other things) an assertion of identity (the domain or domains to which the certificate belongs), inception and expiration dates, the identity of the CA, a serial number assigned by the CA, and, most significantly, the public key of the certificate's owner. All of that is wrapped inside a digital signature by the CA. Click here for an explanation of how digital signatures work, but the point is that a simple mathematical check by anyone can verify whether or not the enclosed data or message is precisely the original transmitted or signed by the sender. X509.v3 is the format used for digital signatures.

Validity is your confidence that you really have the public key for the CA. Saying "This is a valid key for Yoyodyne" means that you believe that you really have the public key of the pair used for signing by that company, whether you trust what they would say or not..

Trust is your confidence that some person or organization is a trustworthy introducer or provider of credentials. Saying "I trust Yoyodyne" means that good enough for Yoyodyne is good enough for you, at least in terms of identity — they vouch only for identity, not for character.

Web browsers and other software have public keys of CAs which they believe to be valid (keys) and trust (CAs) on your behalf.

So, here is the sequence of assumptions whether you are really conscious of them or not. You are confident that:

• Whoever wrote your software (the Mozilla project in the case of Firefox, Google for Chrome, Apple for Safari, Microsoft for Explorer, and so on) is interested only in helping you.
• The software and associated data files on your computer are really the original programs and data from that trusted provider with no changes or additions.
• Your software (primarily browsers, but possibly other client software) already has some public keys for CAs embedded in them. Your software provider is making good decisions about trust on your behalf. When they decided to trust a company, when they said "Good enough for CA Yoyodyne is good enough for us, the producers of the XYZ browser", that was also appropriate for you. You, too, also trust every one of the CAs in the sense that you are willing to just take their word about what the public keys are for various web sites.
• Your software provider was correct about the validity of the trusted CA public keys stored in your software.
• Going deeper yet, it is appropriate to trust those CAs because none of them could be hacked, misled, confused, or subverted.
• The CAs involved would never inappropriately give away control or trust.

Well, that's a deep pile of assumptions. You can imagine how this falls apart, especially in the last two items listed.

Wait, it gets worse. Something called a subordinate CA or sub-CA certificate, or the closely related intermediate CA certificate, means that things are even fuzzier. A root CA is one of the CA organizations generally considered world-wide to be trustworthy. A website's digital certificate signed by a root CA will be considered as valid by any browser (unless that root CA's public key has been removed from the browser). If a root CA issues a sub-CA or intermediate CA certificate to another organization, that secondary organization can create certificates that browsers will trust just as if they came from the root CA. It's a mechanism for generating all the digital certificates for all the (possibly faked) domains you would like.

The theory is that the following sequence of events will secure a connection in terms of the server authenticating to the client (and, optionally, also the reverse) and then the pair agreeing upon a unique shared secret with which to encrypt the two-way data stream in this one connection.

1. Client establishes a TCP connection to the server.
2. Server sends its digital certificate.
3. Client trusts the certificate signer (CA) and has a valid copy of its public key.
4. Client verifies the digital signature, and so the digital certificate should be believed. Client is convinced of the validity of the public key for the web site under consideration. However, it might have connected to an imposter.
5. Client sends some unique challenge to the server. Something like the number of milliseconds elapsed since some past epoch (e.g., 00:00:00 UTC 01 Jan 1970) would be appropriate.
6. Server encrypts the challenge with the private key and sends the resulting response to the client.
7. Client decrypts the response with the known valid public key, verifies that the result is equal to the challenge. Server is authenticated.
8. Client and server now agree on a shared secret key used encrypt the following communication with symmetric cryptography.
   • For example, the client generates some random 256-bit string and then uses the server's public key to asymmetrically encrypt this message:
     "Let's use AES-256-CBC with key 0x1fea424d365a81736d80c437b7ea8072c5245694641c8fe9bca5f37f76e3ca66."
   • The client sends that message, the server decrypts it, and everything from here forward is handled that way. Symmetric ciphers using shared secrets are far more computationally efficient and therefore appropriate for the main data transfer work.

CAs Behaving Badly

The math works just fine.

The problem is that the CAs get confused, hacked, misled, bribed, ...

The browser producers trust a collection of CAs, therefore all the users of those browsers find themselves implicitly trusting CAs who are not necessarily worthy of that trust.

Let's look at these in reverse order, going back in time by the dates these fell apart:

Comodo, October 2016

Comodo was using OCR (or Optical Character Recognition) to read the content of email messages in an automated process. That allowed two security researchers to obtain a server authentication certificate from Comodo for a domain they did not own or control.

Incident report More details Report from researchers

GlobalSign, October 2016

GlobalSign inadvertently revoked its intermediary certificates, breaking the chain of trust and invalidating its customers' certificates. They maintain several root certificates, and have cross-certificates between those roots.

This took out sites including Wikipedia, Dropbox, Logmein, Financial Times, Guardian, and many more.

They revoked a cross-certificate linking two roots. This led to browsers concluding that the intermediate certificates signed by those root certificates were invalid.

They fixed the problem quickly, but by then the revokations were stored in browser caches across the Internet. Unless the users knew about this problem, and were tech-savvy enough to find and follow directions to clear their browser's CRL cache, they would be unable to connect for one to four days, depending on their browser's cache update timing. The Register and Dark Reading described the problem.

National Informatics Centre of India, July 2014

On 8 July 2014 a Google blog reported that on the 2nd they became aware of unauthorized digital certificates for several Google domains, plus one Yahoo domain, plus unknown others. The bogus certificates were issued by the National Informatics Centre (NIC) of India, which held several intermediate CA certificates trusted by the Indian Controller of Certifying Authorities (India CCA).

Microsoft includes the India CCA certificates in the Windows Root Store, so applications running on Windows (including Google's Chrome browser) would trust those certificates.

Google pushed out a CRL (Certificate Revocation List) update. They later learned that "NIC's [certificate] issuance process was compromised" and they learned of other misissued certificates in addition to those reported by India CCA. Google then decided and announced:

ANSSI, December 2013

On 7 December 2013 a Google blog reported that on the 3rd they became aware of unauthorized digital certificates for several Google domains. They had been issued by an intermediate certificate authority linking back to ANSSI, a French CA. An intermediate CA carries the full authority of the issuing CA itself, and so with ANSSI being generally trusted, this intermediate CA can create a certificate for any domain they or a conspirator wish to impersonate.

The explanation was as follows, where "their" must mean "ANSSI's":

Within the week, Google decided to confine ANSSI to the Francophone corner of the Internet:

However, regardless of whether you consider this an appropriate response or not, this decision by one browser creator only matters if you are using their browser, Google's Chrome in this case.

Google had a stake in this, it was their *.google.com domains being hoaxed by certificates based on the authority of ANSSI. Did Mozilla (Firefox), Apple (Safari), or Microsoft (Explorer) care? Should they have? What, if anything, happened for the users of those other browsers? What should have happened?

Dark Reading had a story about this one.

Mozilla's reaction to the problem trend, February 2013

On 15 February 2013 Mozilla announced a change to their CA Certificate Policy, requiring that subordinate CA certificates be constrained through technical measures defined in RFC 5280, or else be subject to the same requirements imposed on root CA certificates.

Trustwave, February 2012

On 4 February 2012 Trustwave admitted that they had issued a subordinate CA certificate to a company so it could inspect SSL traffic passing through its internal networks as part of a DLP or data loss prevention system. This issuing of a subordinate CA certificate brought negative attention and ominous announcements from Mozilla that this likely violated Mozilla's CA Certificate Policy regardless of the precautions taken by Trustwave. Trustwave announced that they would no longer issue this type of certificate.

Türktrust, August 2011 — January 2013

Türktrust, or TÜRKTRUST Bilgi İletişim ve Bilişim Güvenliği Hizmetleri A.Ş. according to the record embedded within Firefox, is a Turkish CA.

Türktrust issued a subsidiary CA to EGO Genel Müdürlüğü, the transportation directorate of the city government of Ankara in August, 2011.

The certificate and key were installed on a Checkpoint firewall with reverse proxy capabilities allowing it to decrypt and inspect TLS/SSL traffic. The Checkpoint firewall automatically created man-in-the-middle certificates when it got these certificates, and those certificates, apparently valid but not really valid for Google, were spotted by Chrome browsers and automatically reported to Google. On January 3rd 2013 Google reported:

According to a ArsTechnica report of 4 January, this was due to a simple mistake made by Türktrust in 2011 during a test of their certificate production process.

However, according to a Reuters report of 3 January, this was likely done by the public transit agency EGO in order to spy on its own employees' use of Gmail and Google searches.

Türktrust officials said there was no evidence of certificate use for "dishonest purposes" nor was their any break of Türktrust system security.

Google notified Mozilla and Microsoft, with the result that Chrome, Firefox, and Explorer now blocked *.ego.gov.tr sites authenticated by Ankara transit authority EGO, and also *.e-islam.kktcmerkezbankasi.org sites authenticated by another Türktrust customer.

DigiNotar, June–September 2011

DigiNotar was founded in 1998 by a Dutch notary and evolved into a CA initially focused on notaries and related professionals. They issued certificates under their own name, as "DigiNotar Root CA", and on behalf of the Dutch government's PKIoverheid or "PKIgovernment" program. This continued after DigiNotar was purchased by Chicago-based VASCO in January, 2011.

DigiNotar first detected an incident on their systems on 19 June 2011, the following day VASCO issued a press release quoting Chief Operating Officer Jan Valcke: "We believe that DigiNotar's certificates are among the most reliable in the field."

On 10 July 2011, an attacker with access into DigiNotar's systems created a wildcard certificate for *.google.com, and that was used for man-in-the-middle attacks against Iranian users of Google services including Gmail and Google Docs. 300,000 Iranian users of Gmail seem to have been the targets, bringing the Iranian government under suspicion as the perpetrator. On 28 August 2011, multiple Iranian ISPs detected these faulty certificates. The bogus *.google.com certificate was posted on Pastebin.com and verified.

DigiNotar revealed that they had detected an intrusion into their CA infrastructure back in mid July, and subsequently admitted that fraudulent certificates had also been issued for *.android.com, login.live.com, *.microsoft.com, *.mozilla.org, *.skype.com, *.torproject.org, twitter.com login.yahoo.com, several root CAs, and many more totaling at least 531.

Further investigation showed that the DigiNotar website had been compromised and defaced by Turkish and Iranian hackers in 2009. Mozilla, Apple, Microsoft and Opera quickly removed the DigiNotar root certificate.

This broke the trust chain and disabled access to many Dutch government web sites (not all, DigiNotar was not the only CA in the PKIoverheid program, but it caused serious problems). The Dutch government stepped in and took over Fox-IT was contracted to analyze the sequence of events, which they called "Operation Black Tulip".

On 20 September 2011, VASCO announced that its subsidiary DigiNotar was filing bankruptcy. The Fox-IT report was suppressed due to fears that it would lead to further claims against the now bankrupt DigiNotar.

The "ComodoHacker", supposedly an Iranian college student, claimed that DigiNotar was one of five CAs he hacked. F-Secure published a report saying that they found the claim "plausible", although it was not obvious how that would lead to the misdirection and interception of 300,000 Gmail connections, 99% of them from Iran.

Slate published a nice description of the DigiNotar hack and its impact.

Comodo, March 2011

Comodo is a prominent CA based in New Jersey, USA. On 15 March 2011 Comodo reported that a user account at InstantSSL.it, a reseller affiliate in Italy, had been compromised. That was used to create a new user account, which then created nine certificates for mail.google.com, www.google.com, login.live.com. addons.mozilla.org, login.skype.com, and login.yahoo.com. The attack came from IP address 212.95.136.18, allocated to ISP Pishgaman TOSE Ertebatat Tehran Network, in Iran. Comodo described the attack as well-planned, tightly focused, and rapidly executed.

Someone calling themself ComodoHacker appeared on Pastebin providing some of the details of this attack plus claims the following September of being behind the attacks on DigiNotar, GlobalSign, and two other CAs. Wired ran an article in late March analyzing the hack and Pastebin postings as things were known at the time.

Comodo was seen as handling the situation well, quickly detecting the breach and revoking the certificates, and describing the situation completely and openly. Comodo is still in business as the second most used CA as of October 2013 into early 2014. DigiNotar quickly ceased to exist after they were hacked.