Hey, tech: we need a better long-lived secrets manager

Photo by Tim Evans on Unsplash.

Here is a free product or startup idea for anyone interested: one of the biggest unsolved problems in software is the total absence AFAIK of a secure, usable, cloud-native, batteries-included, and scalable long-lived secrets manager. The simplest metaphor is something like a Hashicorp Vault that comes with plugins out-of-the-box for all the secrets that TruffleHog can detect. A tagline might be: TruffleHog — but the inverse — in Vault.

When I was looking into this problem for my current employer, I was surprised that such a product did not already exist. “But, Trishank, LULZ, WDYM: don’t we already have ABC or XYZ?” No, at least not AFAICT. To clarify, let me address each of these necessary properties in turn:

• Secure. A secure lifecycle (bootstrapping, creation, reads, updates/rotation, expiration, deletion, cryptographically-verifiable audit logging, etc.) of secrets that Sir David Attenborough himself would be proud to narrate. You want to go beyond securely generating secrets: you want to safely use them, too (i.e., secrets should be infeasible-to-exfiltrate). Who cares if all your secrets are encrypted at rest if all you are ultimately going to do is to use them in plaintext? Now that is security theatre at its pinnacle. Obviously, authentication and authorisation must also be first-class citizens as a result.

This is how you should be able to get to secrets. (Thanks to CDC for the vivid picture.)

• Usable. At the same time, it must be dead-simple yet secure-by-default for developers to use. Obviously, it can’t be as scary and tedious as the picture above suggests (for one thing, secrets are not contagious). That partly means speaking in languages or formats they are familiar with (even if that means accepting — ugh — YAML configurations). Think about it: why is it that, after the warp-speed development of mRNA vaccines for a novel coronavirus, official documentation is still asking developers to create and use iOS and Android signing keys on disk, even if encrypted?
• Cloud-native. Everyone is moving to the cloud, even the financial industry and governments who are famous for having run their own data centres. This means that the threat model must be accordingly practical (hint: most users don’t need to worry about the Mossad carrying out in-person side-channel attacks). A corollary is that any such product should also be cloud-agnostic for obvious reasons.
• Batteries-included. Symmetric and asymmetric keys (more importantly, the latter must include high-level, application-specific asymmetric keys such those for SSH, or Apple/GPG/Google/Microsoft code signing). API keys. Passwords. TOTP seeds. Tokens. You name it: only the imagination of corporate tax accountants should be the limit. Oh, BTW, batteries for these many different types of secrets should be included, or else both software and security engineers will say, “fuhgedaboutit.” Alas, this requirement alone rules out existing secrets managers such as those from Hashicorp, AWS, GCP, or Azure. Think CoreOS Fero, Mozilla Autograph, or Bytedance Keyhouse, but on steroids. My peer Andrés Vega had a similar idea with the (unannounced) Open Secrets project.
• Scalable. Must be able to handle, for example, a large number of secrets, or number of operations (which means that sometimes high-latency network roundtrip operations are not an option, so you might need to resort to secure fallbacks such as operating on locally cached secrets safely copied using key wrapping and trusted hardware). Must also obviously handle replication and backups (like AWS CloudHSM).

“But, Trishank, isn’t this exactly why we should just be using short-lived secrets everywhere instead?” Indeed — there is a growing trend in the industry towards moving to systems such as OAuth, OIDC, SPIFFE, AWS ephemeral credentials, Kubernetes ephemeral service account tokens, Fulcio, and so on. All of these systems share the philosophy that secrets should naturally expire and, thus, rotate. This is indeed a great idea, and I am not at all against it. However, a moment’s reflection will show that these short-lived secrets are still backed by long-lived secrets that need to be protected as discussed above. This is not to mention that short-lived secrets are not always ideal, and have their own tradeoffs; never mind that legacy applications that will continue to be used for years, if not decades, to come may not have this option in the first place. And so, in an ideal world, there will be more short-lived than long-lived secrets, but we still have a long way to get there, and in the meantime, we need practical solutions that will bridge the gap.

So, why don’t we already have such a product despite this shared pain across the tech industry? I don’t know. Part of the answer may be that the security industry tends to focus on scanning for leaked secrets after the fact rather than also making them infeasible-to-exfiltrate and easy-to-rotate in the first place. Another answer is that it is difficult and time-consuming to write and maintain plugins or engines for hundreds, if not more, of types of secrets, which is why this product must be commercial or at least have an open core. Anyway, the reason for writing this blog post is precisely to serve as a call to arms to our industry to get out there and build something, perhaps together.

Acknowledgements: Thanks to Andrés Vega, Eli Nesterov, Dan Lorenc, and Aditya Sirish for their feedback.

Disclaimers: The opinions expressed here do not necessarily reflect those of my current employer.