Resilience in Supply Chain Security

What is the work that you are doing today?

I am the engineering lead for Google Open Source security team. Our role is to make it easy for Google to both consume and produce open source software and software based on open source securely.

What's the security part in this?

Google is well-known for using a monorepo for most of our internal development practices, where all the code, including all of the transitive dependencies, are checked into one place. This falls apart when we move into the cloud and open source environments where we're using the tooling that is familiar by upstream and shipping open source projects to customers who want to use standard open source tooling. We need guarantees around source code, provenance, supply chain security, vulnerability management, secure built systems and all those things for open source, and doing this in open source as well. 

And this is for Google's open source projects, or do you provide this for others, too?

We want to make this as free and easy to use for everybody as possible. We consume open source produced by other people. By making it easy for them to use these tools, it improves the full supply chain for everyone, including us. 

Are you also involved in the various fuzzing services that Google offers?

Yes. This is one of the big parts of our team as well, that's been running for years, finding thousands and thousands of critical security vulnerabilities.

How do you run the fuzzers? Do you just point them at the source code or do you customize them for individual projects? Because fuzzing can be a bit noisy.

We have a couple of different strategies for fuzzing. There is the oss-fuzz, where there's a big open source repo where people can check in configurations for their projects that they want to be fuzzed. And when they do this, we fuzz it for free using our huge compute infrastructure. As part of the fuzzing integration, people have to tell us how to report problems that we find. We do a pretty good job here trying to figure out exactly where things were introduced and looking for false positives. As part of that reporting, we can't just file these things publicly on open GitHub because a lot of them can be critical security issues. So we report this privately to the maintainers and let them deal with it privately at first. Because we do this, we can actually track things from when stuff was actually introduced. When we find a fuzzing crash, all the way to getting the bug fixed and we maintain a high rate, in the 95 percent rate, of things that we find that actually end up getting fixed.

What are your goals for your talk?

The topic is supply chain security in general and then aligning it with the resilience topic for the whole track. I want to give people an overview of what supply chain security means for open source and why it's such a big problem. Then talk about the different ways that people can improve their supply chain security by designing for resiliency in particular. It's actually a pretty good fit for this talk. A lot of approaches to supply chain security assume that no bugs will be found, and you'll do a perfect job preventing malicious code from being introduced and your keys will never be leaked, and your developers won't ever lose their hardware tokens and all this stuff and everything will be signed. Really, that creates a pretty brittle system because as we know, mistakes always happen, and being able to plan for key revocation and mitigation when bad things do happen is just as important as securing the overall supply chain. 

So it's largely about dealing with predicting the things that could go wrong and putting in mitigations. 

Yes, exactly. Trying to encourage people to step back and plan for when things go wrong instead of just trying to prevent things from going wrong. 

In your view, what are the biggest issues of supply chain security nowadays, Is it the NPM, package managers, is it that people bribing their way into open source projects or anything like that? 

I like to say there are two classes of problems. The two problems, the way I like to describe it, with open source security and open source supply chain security are, one, that it's open source, which means anybody can see it, anybody can contribute to it, and not everybody is nice. So you get a lot of attacks like typosquatting and malicious commits, and people attacking build systems because everything is done in the open and all the stuff is exposed. So that's one class of the supply chain attacks. That's pretty serious. 

Dan: The other one is just that the open source software is still software and it has lots of bugs in it, just like all software, not more or less, but there are a ton of them out there. And people have this perception that just because it's a widely used package on NPM, it's bug free and people have hit all the issues so far. Even if you have the most secure supply chain, these bugs are still going to come in. So we need to do fuzzing in programs to fix those bugs, as well as stop the leaks in the supply chain, but also fix everything that is coming in. 

What about new threats? Do you think something will come up that we haven't seen yet? 

I don't think they are really new threats. It's a great question, though. I think we've known about this class of attacks since 1984. Ken Thompson, sometime in the early 80s, wrote the Reflections on Trusting Trust paper. This can go back to a single compiler. Now, everything was built from that compiler and everything from those compilers, all the way down. Now you have no trust. And for some reason that just hasn't been a big worry. It hasn't been a big threat. People have not been taking it seriously until now. I think the real reason it's finally rising now because somehow it got easier. I think it's a good enough job at locking down all the other doors and all the other ways in, things like let's encrypt. SSL finally got deployed everywhere else. Password managers are being used, 2-factor is now a thing. Supply chain attacks haven't really gotten any easier. They've just become the easiest way in and one of the easiest attacks to carry out now because everything else has gotten so much harder. I think it's going to get worse. People are going to do it more and more. Luckily, a lot of it is being done by researchers and testers, and internal red teams. But some of these things can have very, very far-reaching consequences for everybody.