Cloud security vendor Wiz announced yesterday that it found a vulnerability in Microsoft Azure’s managed database service, Cosmos DB, that granted read/write access for every database on the service to any attacker who found and exploited the bug.
Although Wiz only found the vulnerability—which it named “Chaos DB”—two weeks ago, the company says that the vulnerability has been lurking in the system for “at least several months, possibly years.”
A slingshot around Jupyter
-
Jupyter notebook functionality in CosmosDB enables many advanced data visualization techniques with relatively little coding experience or effort.
-
A privilege escalation vulnerability allowed anyone with a Cosmos DB account to filch the private key for any other Cosmos DB account, by way of the Jupyter notebook functionality.
-
Once an attacker has the victim’s primary key, it’s game over—full read/write/delete access is granted permanently, and cannot be revoked without replacing the affected keys.
In 2019, Microsoft added the open-source Jupyter Notebook functionality to Cosmos DB. Jupyter Notebooks are a particularly user-friendly way to implement machine learning algorithms; Microsoft promoted Notebooks specifically as a useful tool for advanced visualization of data stored in Cosmos DB.
Jupyter Notebook functionality was enabled automatically for all Cosmos DB instances in February 2021, but Wiz believes the bug in question likely goes back further—possibly all the way back to Cosmos DB’s first introduction of the feature in 2019.
Wiz isn’t giving away all the technical details yet, but the short version is that misconfiguration in the Jupyter feature opens up a privilege escalation exploit. That exploit could be abused to gain access to other Cosmos DB customers’ primary keys—according to Wiz, any other Cosmos DB customer’s primary key, along with other secrets.
Access to a Cosmos DB instance’s primary key is “game over.” It allows full read, write, and delete permissions to the entire database belonging to that key. Wiz’s Chief Technology Officer Ami Luttwak describes this as “the worst cloud vulnerability you can imagine,” adding, “This is the central database of Azure, and we were able to get access to any customer database that we wanted.”
Long-lived secrets
Unlike ephemeral secrets and tokens, a Cosmos DB’s primary key does not expire—if it has already been leaked and is not changed, an attacker could still use that key to exfiltrate, manipulate, or destroy the database years from now.
According to Wiz, Microsoft only emailed 30 percent or so of its Cosmos DB customers about the vulnerability. The email warned those users to rotate their primary key manually, in order to make certain that any leaked keys are no longer useful to attackers. Those Cosmos DB customers are the ones which had Jupyter Notebook functionality enabled during the week or so in which Wiz explored the vulnerability.
Since February 2021, when all new Cosmos DB instances were created with Jupyter Notebook functions enabled, the Cosmos DB service automatically disabled Notebook functionality if it wasn’t used within the first three days. This is why the number of Cosmos DB customers notified was so low—the 70 percent or so of customers not notified by Microsoft had either manually disabled Jupyter or had it disabled automatically due to lack of use.
Unfortunately, this doesn’t really cover the full scope of the vulnerability. Because any Cosmos DB instance with Jupyter enabled was vulnerable, and because the primary key is not an ephemeral secret, it is impossible to know for certain who has the keys to which instances. An attacker with a specific target could have quietly harvested that target’s primary key but not done anything obnoxious enough to be noticed (yet).
We also can’t rule out a broader impact scenario, with a hypothetical attacker who scraped the primary key from each new Cosmos DB instance during its initial three-day vulnerability window, then saved those keys for potential later use. We agree with Wiz here—if your Cosmos DB instance might ever have had Jupyter notebook functionality enabled, you should rotate its keys immediately to ensure security going forward.
Microsoft’s response
Microsoft disabled the Chaos DB vulnerability two weeks ago—less than 48 hours after Wiz privately reported it. Unfortunately, Microsoft cannot change its customers’ primary keys itself; the onus is on Cosmos DB customers to rotate their keys.
According to Microsoft, there’s no evidence that any malicious actors found and exploited Chaos DB prior to the Wiz discovery. An emailed statement from Microsoft to Bloomberg said, “We are not aware of any customer data being accessed because of this vulnerability.” In addition to warning 3,000+ customers of the vulnerability and providing mitigation instructions, Microsoft paid Wiz a $40,000 bounty.