BLOG
The OpenSourceMalware Show #29
Jared Wray, maintainer of Keyv and Cachable, talks about what it was like to be at the center of the ChainDrop software supply chain attack.
By cb482791-4ef1-4762-96ad-b0ca4bdd538e ยท
The OpenSourceMalware Show is available on YouTube, LinkedIn, and as a podcast.
This week we had a special guest on the show: Jared Wray, who is the maintainer of several popular open source projects, including Keyv and Cacheable. In August 2026, he was the victim of an account takeover. A threat actor used a GitHub token he thought he'd retired to push malware into his Kevy and Cachable, and his own release pipeline shipped it to npm with valid provenance. It contained a worm that propagated the malware into 400+ npm packages. The campaign became known as ChainDrop, and the worm code was based on TeamPCP's open-sourced Mini Shai-Hulud worm. We talked about:
How the attack happened: A single old GitHub token on a compromised desktop let the attacker publish straight from GitHub without ever touching his npm account.
Getting his accounts back: Losing GitHub meant losing every tool that used it as his login, the only way to reach GitHub and npm was email, and his restored repos still had the malware branches in them.
Rebuilding his setup: He turned his recovery into a defense-in-depth agent skill and a "director of security" bot, built on the idea that every pull request needs an isolated environment.
Security controls that are harder than they sound: Code signing, branch rulesets, Dependabot, install-time scanners, and release gating each took hours of research, and he still can't get signing to work for commits made by agents.
Vetting new dependencies: Before he adds a package, his skill checks how often it publishes, whether it uses OIDC, how many dependencies it brings in, whether people actually engage in pull requests, and whether it's in OpenSourceMalware.
Staying on GitHub Actions: None of the other CI/CD platforms he tried worked end to end, so he went all in on locking down GitHub Actions.
Resources
[00:00:00] Jenn Gile: Hello. It is Thursday, November 12th. I'm Jenn Gile. I have Paul McCarty with me. We're the hosts of the OpenSourceMalware Show, and we're excited to have a special guest with us today. Jared Wray is the, uh, maintainer of, I did some math before, over 60, uh, open source packages totaling, we think, if the agents are right, uh, a billion weekly downloads.
[00:00:29] Jenn Gile: And he has the, uh, I guess, unfortunate pleasure to have been the victim of a software supply chain attack earlier this year, um, that resulted in a whole bunch of packages getting infected, uh, on NPM, which we'll talk about in just a moment. But Jared, uh, why don't you give a quick intro for yourself?
[00:00:51] Jared Wray: Uh, great.
[00:00:51] Jared Wray: Yeah, thanks for having me on. Uh, Jared Wray, uh, entrepreneur. I've done multiple companies. Uh, also, uh, started doing open source just about a decade ago, uh, really just giving back to the community and things like that. Um, I'm the CEO of Hyphen, which is a, uh, AI-based agent for DevOps
[00:01:15] Jenn Gile: And you, uh, are no stranger to engineering.
[00:01:18] Jenn Gile: You're a recovered CTO yourself.
[00:01:21] Jared Wray: Yeah. Yep. Yeah. I, I've been CTO in many companies, uh, from streaming media companies, uh, doing live streaming, to energy, uh, companies like Palmetto, um, all the way through.
[00:01:35] Jenn Gile: And you told us, uh, that when you retired from your last company, um, you decided to pick up maintaining as a hobby.
[00:01:46] Jared Wray: Yeah, yeah. So in 2013, my first startup, uh, got acquired, I think a couple years before that, like 2011, uh, by a company called CenturyLink. Um, and so I was there for two years, and then I was like, "Okay, I'm gonna retire." Um, found out that basically all my friends and... uh, basically worked at the company that I retired from, and also they had jobs, and I have no other hobbies other than work.
[00:02:13] Jared Wray: So I decided to give back, and I was like, "Hey, I really like open source." And I actually started contributing to a co- couple open source projects. Uh, eventually, I ended up, uh, taking over keyv, uh, which is one of the ones that, uh, basically got... had the supply chain attack on it. And at the time, it was probably, I don't know, about 40 million downloads or something like that.
[00:02:37] Jared Wray: Since then, it's kind of grown quite a bit, and I think last e- estimate, it's, like, over 600 million downloads a month.
[00:02:45] Jenn Gile: Yeah, crazy. So let's rewind the clock to April... Or sorry, Au- August 4th. Um- Yeah ... and this sticks in my mind in particular, 'cause Paul and I, had we just gotten to Las Vegas, or were we just going?
[00:03:00] Jenn Gile: Yeah. It was somewhere in there. We were headed for Hacker Summer Camp, and we were like, "Cool." The, the hackers know that, uh, the security industry is gonna be at conferences all week, and so of course they picked this week to attack. And, um, it kicked off, Jared, with you as the initial victim. They, uh, gained access to your publishing accounts, uh, published trojanized versions of several of your packages, and it resulted in an npm worm that infected 444 packages.
[00:03:35] Jenn Gile: Um, the largest impact that we saw as we were analyzing it in real time was to a organization called ServiceTitan. Mm. Um, but you know, we're gonna focus more on your experience rather than the malware itself. Um, I think listeners of this podcast know that Paul and I are Team Open Source, and, uh, we'd like to see a bit more understanding in the security industry of what open source maintainers go through, um, what it's like to be the victim of a supply chain attack, and, uh, maybe give people some inspiration for what they could be doing better.
[00:04:16] Jared Wray: Yeah. Uh, do you want me to just kind of start off with, you know, date, the day of?
[00:04:21] Jenn Gile: Yeah, but first, uh, I think I didn't actually ask you this, so I'm gonna ask you now. What made you say yes to coming on this podcast, and to talking to Paul and I in general?
[00:04:31] Jared Wray: Um, I think there's a couple things. Uh, one, the... I, I was actually really surprised how much of the community actually reached out and has helped me along the way.
[00:04:42] Jared Wray: Mm-hmm. And that, that really changed my thoughts. I've always wanted to help people, and contribute, and things like that. But on the security aspect, I know nothing about security, like, to be honest, and it's actually quite hard to actually get into security and understand what's going on. And I had so many people reach out and actually help me.
[00:05:04] Jared Wray: Uh, they were reaching out on Twitter. They were like... I had people actually get on a video call with me to kinda explain, like, and start looking at a couple things, 'cause I was like, "I have no idea how this works." Um, and so after that, you know, I, I, I was talking with, uh, another person that I know that does open source, and I was like, "You know what?
[00:05:24] Jared Wray: I am... I'm gonna start giving back on this," because, like, it's, it's going to keep on happening, supply chain attacks. And a lot of the open source maintainers, they don't always maintain their code. Like, if you look across the NPM re- registries, s- part of the reason I ended up taking over a lot of these packages is because they just weren't being maintained.
[00:05:47] Jared Wray: Mm-hmm. And it's hard to do that, because it's free, right?
[00:05:50] Paul McCarty: Mm-hmm.
[00:05:50] Jared Wray: And so a lot- Mm ... of maintainers are like, "I don't get paid for this. I might get a $50, you know, donation once a year." And so that's why I started doing it.
[00:06:00] Jenn Gile: Yeah. I mean, people get burned out, you know, cycle off it at some point. We saw with the xz utils case a couple years ago- Mm-hmm
[00:06:09] Jenn Gile: that that's how that package ended up getting taken over by a threat actor, was the maintainer was burned out. Mm-hmm. So I, I don't know. I'm glad to hear that you had a good experience with the security community. Uh, we think the community's pretty special, so you know, glad to bring this all together.
[00:06:25] Jared Wray: Yeah. And now- Yeah ... it's like it would be even better. Like, so now I'm starting to reach out to other open source maintainers, and I'm like, "Hey," like, "you should start locking this down. If you need some help, let me know," like, "I'm happy to help
[00:06:38] Jenn Gile: It's super
[00:06:38] Paul McCarty: cool. I, um, I think, I just wanted to say personally, thanks for coming on the show and thanks for being open to talking about this.
[00:06:47] Paul McCarty: You know, I cheekily wore my Got Mailware shirt today. Yeah. But I mean, I also wanna be honest and say, listen, I get that August 4th and that whole week would've been one of the crappiest par- days or weeks of your life and, and I, and I don't want to be flippant about that, right? That's- Oh, yeah ... I have done a lot of incident response with open source maintainers and with, with software engineering teams, and I get that it's a terrible thing to go through, and so I don't want you to think that I'm being flippant about that.
[00:07:15] Paul McCarty: Um, so I, I just personally wanna say thank you for coming on the show.
[00:07:18] Jared Wray: Oh, no problem. And be as flippant as you want 'cause at the end of the day, you kinda just have to laugh about it because you go through something like that and it really changes, like, how, how you think about things and what you're doing.
[00:07:31] Jared Wray: I know before we were even talking about, like, trusting people who reach out to you. Like, I used to always trust people, no problem, and now I'm like, "I don't know if you're actually the person that you're s- you say you are," which is- I'm
[00:07:44] Paul McCarty: from Detroit. Yeah. I'm from Detroit. I don't trust anybody. Yeah. I don't trust nobody.
One old GitHub token published straight to npm
[00:07:51] Jenn Gile: Well, why don't we start with, like, August 4th. How did your day start? How did this, you know, how did you find out you were compromised? What was it like? What did you
[00:08:01] Jared Wray: go
[00:08:01] Jenn Gile: through?
[00:08:01] Jared Wray: Yeah, yeah. So, um, I usually go to sleep pretty late. Uh, I'll start August 3rd, because most likely that's when it happened.
[00:08:10] Jenn Gile: Mm.
[00:08:10] Jared Wray: So August 3rd, uh, I was working on my desktop, and just to let everybody know, quick background on it, I, I have a desktop and a laptop that I usually work out of and I'm constantly working My desktop was d- uh, compromised.
[00:08:24] Jared Wray: On that desktop, just like every other developer, you have a lot of stuff over time that accumulates, right? And I've had this desktop for five-plus years, so it had secrets on it, it had ENV files, it had, um, you know, uh, GitHub PATs. Uh, and that's actually how it happened. So we, we could not figure out, and I've asked many security experts, like how did this happen?
[00:08:52] Jared Wray: Like, was it a, you know, piece of malware that got installed? Because I think something happened like that. Uh, but a lot of them delete themselves and things like that. Or was it a phishing thing? Nobody has been able to figure out how it got compromised, but they were, like, definitely sure that this desktop got compromised.
[00:09:11] Jared Wray: So that was the night before. The... By the morning, which I usually get up around 7:00, 8:00. I had probably over 140 text messages, uh, 40 to 50 Insta, like, or Twitter messages, and I literally, like, was downstairs pouring coffee and I just freaked out 'cause everybody was like, "Call me, call me, call me." Like, literally, "You've been hacked."
[00:09:40] Jared Wray: And of course, I came home to my desktop- So talk to us, Micah ... which was the compromised machine. I log on I, and I couldn't even see my GitHub account. Like, it literally was, like, pushing, trying to... You know, I think they were trying to take it down, uh, and it was basically, like, I had no access to GitHub. I couldn't even figure out what was going on.
[00:09:59] Jared Wray: Um, I went to npm, no access on npm, and then I started looking on Twitter and, like, literally, like, everybody was like... It basically, my entire feed was full of this, which was
[00:10:11] Jenn Gile: kind of crazy. Yeah, we saw your, um, messages while, like, during the incident itself, kind of trying to get access back to your accounts so that you could- Uh-huh
[00:10:22] Jenn Gile: you know, incident respond, take down packages. Um, so what, what can you tell us about the experience of trying to regain control of your, your maintaining accounts? I mean, again, we said at the top of the episode, it's not just keyv and cacheable. You know, your accounts have access to possibly- 60 plus ... you know-
[00:10:45] Jared Wray: Yeah
[00:10:45] Jenn Gile: yeah, billions of downloads a week kind of thing.
[00:10:48] Jared Wray: Yeah, yeah. Uh, it w- it was actually quite scary because a lot of the people that I, I personally know who have been, like, uh, Charlie Robbins, he was, like, one of the first people for Node.js, right? He was one of the founding members. He reached out, we've known each other for a long time, and he said, "You might never get your account back."
[00:11:08] Jared Wray: And I was like, "That's 10 years of work," plus, like, you know, corporate work and things like that. Right. So it was quite scary to kinda figure it out, and I ended up... Luckily, like, I knew a lot of people at Microsoft and at GitHub, and I was reaching out to them and being like, "I need you guys to help me." And one of the things you probably saw posted is I offered, like, $5,000 to, to talk to a security expert, and, uh, no one basically would respond.
[00:11:38] Jared Wray: So all I could do is just apologize on Twitter and wait for them to slowly engage with me, and the only way they engage, which is really scary honestly, is through email
[00:11:52] Paul McCarty: Yeah
[00:11:53] Jenn Gile: Given how things got compromised, I mean- Yeah ... we know your npm and GitHub accounts were compromised for all- Mm-hmm ... you know. For all they know, they could've been talking to the threat actor through your email.
[00:12:05] Jared Wray: Exactly. Yeah. It, and it's kinda crazy, like, I, I couldn't get any responses from them, and I didn't know if they were working on it or what was going on, so I was kinda clueless. And luckily, the security community was able to, like, reach out to a couple people they knew still at, like, npm or GitHub and be like, "Hey, you know, how can we help?"
[00:12:26] Jared Wray: Um, eventually that got to the point that they started responding to emails and being like, "Hey, we're gonna activate your accounts, but you need to, like, do a couple things and start cleaning up." But it
[00:12:38] Jenn Gile: was- So did they take down the malware or did you take down the malware?
[00:12:42] Jared Wray: So they took down, uh... So, so n- uh, to their credit, like, npm, uh, took down the malware, uh, so it would, would stop publishing, which was awesome of them, and they went in and basically, like, removed all my, uh, OIDC- Mm
[00:13:00] Jared Wray: publishing. Which if you've noticed the improvements that they've done since then have actually been pretty remarkable. At the time, I thought I was doing all the best practices, right? I was being secure. I actually were mo- was moving all my packages to trusted publishing. Uh, but they didn't have at the time when I started moving it, which I think was, like, three months, uh, before the August event, it basically was, like, trusted publishing to fully publish.
[00:13:29] Jared Wray: They didn't have staged packages or anything like that. Mm. Right? A lot of those best practices just barely started coming out and rolling out for people. And a lot... So all the person had to do, or whoever hacked me, is I had one GitHub PAT that was really old, and it was still being used by a small application that I had, and it was back in the day when PATs didn't have limited access controls.
[00:13:59] Jared Wray: And all it was doing was reading my repos. That, that, that's what the app was doing. It would read my repos and, uh, do some work for me. But that one PAT that was on my desktop, that's what they pulled- It had so much access ... and it had access to everything. And so once they had access to that, they c- they literally were able to change, uh, some of the settings and start publishing.
[00:14:21] Jared Wray: So they didn't even access npm, my accounts on there. They were able to just fully publish right from GitHub.
Account recovery by email, with malware left behind
[00:14:29] Paul McCarty: This is really, really fascinating and in- interesting to me because this has, this has connections to other things that Jenn and I deal with all the time right now. Um, so I, I wanna double-click on the npm and the GitHub account.
[00:14:42] Paul McCarty: So you, y- you know, you, you saw that ... Did you see that the bad guys were trying to push something to GitHub, or you just noticed that I can't do anything to GitHub because it's down, my account is effectively deleted?
[00:14:54] Jared Wray: Yeah, so at the time I saw another, like they were trying to push another release So you saw them, like, trying to bump it and actually push another release.
[00:15:06] Jared Wray: Right. So they already had one, I think it was, like, earlier in the morning, right? Yeah. I think it was, like, 3:00 AM when the first one went out or s- or 3:00 AM my time. Um, and then w- by the time I got up and I'm panicking, like, I couldn't log into any of my accounts or anything like that, and you could see basically the CICD pipeline still trying to do a release, and they even moved to another package of mine, which is much smaller, um, called Ecto, and they were trying to publish through there also.
[00:15:34] Jared Wray: Mm-hmm. Uh, but just by doing keyv, which is, you know, about 600 million downloads, cacheable is a whole bunch of caching libraries that keyv basically is the backend storage adapter for, and that's like ESLint uses that, uh, Got uses that. Um, it has a whole bunch of different caching libraries, uh, that I basically maintain and support.
[00:15:58] Jared Wray: And so- Right ... by getting both of those out the door, that's what they were trying to do again, and it looks like GitHub, uh, security team really stepped in at that point and stopped all the jobs once they figured out what was going on. Right.
[00:16:12] Jenn Gile: So the attacker presumably had changed your GitHub password, or w- did you still have access in any way?
[00:16:20] Jared Wray: Yeah, I couldn't log in. So I don't know if they changed it, most likely they did- Yeah ... or if GitHub basically, uh, just turned off the user, and then eventually they took down the entire, my entire personal account for a little bit.
[00:16:34] Paul McCarty: Yeah, so I think that this, getting back to this observation I had about how Jenn and I see this, when we see that, that right now there's thousands of GitHub users that are being affected by, like, PolinRider or GhostAction or- Mm-hmm
[00:16:47] Paul McCarty: just this new one that's happened over the last couple days called WFX. And the problem there is s- those people that have that bad stuff in their packa- uh, sorry, in their repos, are victims. Now the problem- Right ... is when I, when I s- tell GitHub about that, what they do is they actually delete that person's account, and I think, and this is kind of, this is what happened to you.
[00:17:07] Paul McCarty: I mean, you got your account back-
[00:17:09] Jared Wray: I mean,
[00:17:09] Paul McCarty: it's security- ...
[00:17:09] Jared Wray: but my point being- ... like, the, it could be deleted, right?
[00:17:13] Paul McCarty: 100, 100%, because not only did you have all your repos, you had your connections to enterprises and organizations for your GitHub account. Mm-hmm. And here's the other thing is GitHub acts as an identity provider for all kinds of stuff.
[00:17:27] Paul McCarty: Oh, everything. You can even use, you can use GitHub as your identity provider to log into Microsoft, Microsoft Live, and Microsoft 365. Yeah. So I think the reason I wanted to kind of double click on this is just to say this default behavior, which I get it, so I think we get it on some level, let's, let's kill the user temporarily so figure out what's going on.
[00:17:48] Paul McCarty: Uh-huh. We can fix it and then we can re-enable. But this thing where they're, like, deleting accounts, you know, permanently in some cases now, that's very destructive because that... A user, that might affect other things too as well.
[00:18:00] Jared Wray: Yeah. It... And I actually had that experience, uh, because- Mm ... the first thing, I was using Aikido.
[00:18:06] Jared Wray: I actually installed Aikido security scanning on my PRs, and I wanted to see what they saw before it happened, and I couldn't log in because I used my GitHub access token.
[00:18:18] Jenn Gile: Really?
[00:18:19] Jared Wray: And when I had that experience, I was like, "I really have to change the way I think about this moving forward."
[00:18:25] Paul McCarty: Right.
[00:18:25] Jared Wray: Um, because you're right.
[00:18:26] Jared Wray: Like, I u- I was using it for identity everywhere, right? And to have that access taken away was pretty crazy.
[00:18:34] Paul McCarty: So Jenn, side note, we use GitHub as our
[00:18:37] Jenn Gile: IDP- I know, I'm thinking about
[00:18:38] Paul McCarty: it ... for some.
[00:18:39] Jared Wray: Yeah. Well, a lot of people do. I mean, it- Yeah ... it's just so good to have that. And a lot of d- all developers have a GitHub account.
[00:18:47] Jenn Gile: Yeah. I mean- Mm-hmm ... I think this goes back to something I talk about a lot when I'm trying to help people understand, like, GitHub started as one thing- Mm-hmm ... and it has become another thing. And in the time in between, it hasn't necessarily matured in ways to support that other thing. So, you know, we haven't even talked- Right
[00:19:08] Jenn Gile: about GitHub as an IDP, but, you know, this whole security, um, gap that's with it where- Massive amounts of malware can be uploaded and persist on GitHub for days, weeks, months, and the only real solution is to kill accounts. That's certainly not supporting what I would assume is GitHub, like probably does wanna be your IDP.
[00:19:36] Jenn Gile: Yeah. So there's gotta be some layer of, like, better trust, better account validation. Like to your earlier point about, uh, the only way to talk to them was over email. Like, that's concerning for a whole host of reasons, never mind the whole account takeover part. But in the, you know, need to validate who you are, to have quick communication.
[00:20:00] Jenn Gile: Um, uh, Paul, I don't remember the last time I sent you an email. Like- Yeah ... trying to communicate- Well- ... through email is not exactly a rapid, rapid route.
[00:20:11] Paul McCarty: Yeah. And, and I'm assuming that Jared was paying at least for a pro account, if not for enterprise in some of his accounts, right? Or at least he was connected- Oh, yeah
[00:20:19] Paul McCarty: to paying enterprise, right?
[00:20:20] Jared Wray: Yeah, I-
[00:20:20] Paul McCarty: So- ... I
[00:20:21] Jared Wray: paid for the pro account.
[00:20:22] Paul McCarty: Yeah. I mean, like, I'm a paying customer. Jared's a paying customer. We're a paying customer, Jenn. I... This is part of my frustration with GitHub, and, and this is where I wanted to go with this next, is, you know, in your postmortem, you thanked GitHub and npm security teams, which I thought was lovely.
[00:20:40] Paul McCarty: Um, I also wonder how helpful they really were. So I'd like to double-click on this here, like if, if you're okay with that. Yeah. And say, like, I mean, what were they really... How were they helping in their emails, and like how, how responsive were they? Um, and by the way too, um, your npm account also got killed temporarily.
[00:21:00] Paul McCarty: Is that correct, or it did not? Yeah, it...
[00:21:01] Jared Wray: Yeah, my npm account, uh, was taken down while they were do- doing the cleanup and removing stuff. Um, so we'll, we'll double-click a little bit on this. And just to let everybody know, like, I know a lot of people at Microsoft and GitHub. You know, I've, I've been in the Seattle area for 25 years, right?
[00:21:19] Jared Wray: Or over 25 years. And so I, I thanked them because it's a lot of work to basically, like, clean up somebody's mess, um, and figure out what to do and build those policies. I would probably say, like, the communication thing is the one thing that if I had the chance or opportunity, I would really want to help out with.
[00:21:46] Jared Wray: Um, not, you gotta think about it, an incident like this where you're affecting hundreds, if not millions of packages that can be affected, can be installed, secrets can basically be released. If you take my, that update, like, in your, you know, building out your Docker container and you take that update, it's gonna scan those secrets right away, right?
[00:22:10] Jared Wray: And so that's the one thing, like communication-wise, I really wish I was, I would be able to call a number and get someone on the phone- Mm-hmm ... to just talk through it. Like, what do they want me to do? Interesting. What's their plan? Like, how can I help? And from my understanding, the current policy is you can't do that as a security person at Microsoft.
[00:22:34] Jared Wray: You can't basically, like, red team an incident with the person who's affected, but that would've really helped 'cause I would've known the next steps that I needed to do instead of just wildly asking the entire community, like, "This is what's happening."
[00:22:51] Jenn Gile: Yeah, to your, your point, your projects have such an enormous blast radius.
[00:22:57] Jenn Gile: It's not just the download statistics, but also they're used as a transitive dependency in- Mm-hmm ... tons of projects. You know, I wouldn't necessarily expect Microsoft or GitHub to help everyone, you know, directly over the phone, but for a compromise of this scale-
[00:23:16] Jared Wray: Yeah ...
[00:23:16] Jenn Gile: I would wanna see more.
[00:23:17] Jared Wray: Yeah. And, and by the way, like, you know, we've talked about this before, I offered $5,000 to talk to a security expert, like any donation.
[00:23:25] Jared Wray: At this point, like, and I've floated it to other people at Microsoft, I was like, "Look, if I could talk to the person who's over that policy and a security person for one hour about just changing it, I'm happy to give"- Mm-hmm ... $50,000 to any donation. Like, I'm serious about it. Because, like- Well- ... it is so critical that you have to, like, get everybody in the room to solve a problem.
[00:23:50] Jared Wray: And I've been CTO for many companies. It is really hard to have a security incident without everybody in the room figuring something out.
[00:23:59] Paul McCarty: I feel like GitHub, when Microsoft bought GitHub, they had the strategy for why they did it in 2018, and I think they didn't realize that they were buying plutonium- Yeah
[00:24:11] Paul McCarty: which is great if you know how to use plutonium.
[00:24:13] Jared Wray: Yeah.
[00:24:14] Paul McCarty: But if you don't know how to use plutonium, it's radioactive and it will kill you, and I think that that's kind of where Microsoft is. And I, and I wanna briefly, before we move on, 'cause I know Jenn wants to, uh, we both wanna move on, get to m- more cool stuff.
[00:24:27] Paul McCarty: But I think Amazon, for example, you can pay them for support, and even if you don't pay them for support- Mm-hmm ... there's an ability to quickly escalate. And listen, I'm not saying Amazon's amazing, but I'm saying Microsoft, Amazon, they're both really big companies. They both have a lot of money. Yeah. One's doing a good job here, and one's doing a, just a really, really poor job of addressing this.
[00:24:47] Paul McCarty: And I would make the argument that the stuff that happens in GitHub has, to your point earlier, has a larger kind of attack, you know, structure... Sorry, not structure- Yeah ... it has a, it has a bigger impact, sorry, uh, uh, you know, internationally and globally than perhaps somebody's A- Amazon account does. So there's that, too.
[00:25:05] Jared Wray: Yeah. I mean, here's a great example. Like, I have been using GitHub for a really long time. When I got my repos back, I did not realize that they still had the, uh, branches with the malware on them. Mm-hmm. Mm-hmm.
[00:25:19] Jenn Gile: And
[00:25:20] Jared Wray: I didn't know what my next steps were.
[00:25:21] Jenn Gile: They didn't tell you that they didn't clean them out.
[00:25:23] Jenn Gile: You could've-
[00:25:23] Jared Wray: Yeah, I mean- Yeah ... they, they're like, "Hey, we're gonna restore your access," you know? And I... Remember, I'm, I'm using, like, two-factor authentication, MFAs for, like, I've been doing that for a long time. So it's like, okay, we can restore access, but luckily I had a couple people on this, like, in the security ecosy- system that's like, "Hey, just to let you know, you gotta figure out how to clean this."
[00:25:45] Jared Wray: And then I'm like, how do I actually clean this? Like- Yeah ... I'm not a GitHub or a Git expert, right? Right. I use it for revision tracking, but I'm not that, right? So I had to research basically for about two to four hours to figure out how to clean this properly without infecting myself or anybody else, and I was still getting pinged by the, like, everybody out there being like, "You're still infected
[00:26:14] Paul McCarty: Well, and, and you're probably worried at this point, is GitHub gonna take me down again and, like, break this whole thing?
[00:26:18] Paul McCarty: I'm like, "I'm 80% of the way to, like, addressing it, and now you take away my access again."
[00:26:23] Jared Wray: Yeah.
[00:26:23] Paul McCarty: Um, I, I think that's a great point too as well, which is GitHub chose to build their platform on top of Git, which was, was written a long time ago in what I describe as a time- Mm ... before security, and it doesn't have any security built into it.
[00:26:39] Paul McCarty: There's no authentication. There's no authorization. No. Nothing.
[00:26:42] Jared Wray: Yeah.
[00:26:42] Paul McCarty: And a lot of these fixes, to your point, have to happen in the Git repo and in the Git branches and in the index itself, not in GitHub. Yeah. So they just handed you the plutonium right back. They said, "Oh, here's your account. Here." And you're like, "Oh- Yeah
[00:26:56] Paul McCarty: what am I gonna do with this?"
[00:26:57] Jared Wray: And I, I, I honestly was... I wanna thank everybody again because people were like, "You just... It's back online. Do not download and run anything." You gotta figure out how to clean these 'cause it's still infected. I was like, "How is it infected?" Like, I didn't even know how it was infected.
Recovery became a defense-in-depth agent skill
[00:27:15] Jenn Gile: So how did you figure it out? I could only imagine the anxiety of- Was this, you know, diving down Reddit holes or- ... Stack Overflow, or how did you, how'd you teach yourself?
[00:27:24] Jared Wray: I mean, that... So after that incident and I got back my repos, the first thing I did is disable all publishing. I went through every single one of my packages, all my GitHub actions.
[00:27:35] Jared Wray: I deleted all the actions. Like, basically I went to every single one- So the workflow files
[00:27:40] Paul McCarty: themselves?
[00:27:41] Jared Wray: Yeah. I, well, I j- well, not the files, but, like-
[00:27:44] Paul McCarty: You
[00:27:44] Jared Wray: just disabled them ... the GitHub runner, I disabled every single one of those. All my publishing, all my OIDCs, I went through every single package and got rid of them.
[00:27:54] Jared Wray: I was like, "Okay, we're just not gonna publish for a while," right? And I have a lot of repos, so, like, usually I'm publishing multiple times a week, things like that. And then I basically, uh, it, it's kinda funny because my family was heading to the beach that day.
[00:28:11] Jenn Gile: Oh, no.
[00:28:13] Jared Wray: So, uh, luckily my, uh, they... I was supposed to actually drive and pick up someone in Portland, and then, uh, head to Manzanita.
[00:28:20] Jared Wray: So they all left, and, uh, my brother-in-law came and picked me up after the incident was over. And o- basically I was on my phone, like, basically talking to people about, like, "Okay, what should be the next steps?" And this is how I ended up with this whole defense in depth, uh, skill
[00:28:39] Jenn Gile: that I have. Yeah. Walk us through what you sent me to take a look at, you know, a couple of months ago.
[00:28:44] Jenn Gile: What you've written, I don't wanna call it a manifesto- Yeah ... because I feel like manifestos are not actually practical. It's just like a rant on a piece of paper.
[00:28:52] Jared Wray: Yeah.
[00:28:53] Jenn Gile: You built, you know, an architecture for yourself that you've- Mm-hmm ... been, you know, using and tweaking and writing about and telling other maintainers about.
[00:29:04] Jenn Gile: So how did you approach that? 'Cause like you said, you didn't have a background in security, secure coding- Yeah, yeah ... other than I'm gonna use all the widgets that GitHub has here.
[00:29:13] Jared Wray: I mean, yeah. And the... Yeah. So- I was talking on the phone for basically, like, four hours calling friends, uh, people who I thought were more secure.
[00:29:24] Jared Wray: I talked to, uh, security experts, and one thing that I learned in that, that four-hour drive is everybody has a different opinion, and no one can tell me how to lock down... What, what the best practice is to lock down an entire flow. So like, how do you lock down your desktop and create isolation and reduce your attack surface?
[00:29:47] Jared Wray: Most people just have opinions that are like, "You should try this. What's it gonna work?" Especially for someone that's, you know, needs a really good developer experience. Like, there's days that I do 100, almost 200 pull requests in one day. Like, I can't just be like, "Oh, I'm gonna spin up, like, a brand-new Parallels desktop-" Right
[00:30:07] Jared Wray: wait two minutes to basically pull down some code. Like, you gotta figure out how to rapidly do it. But after those four hours, I was like, okay, no one knows, like, really h- the- They're like, "Here's the best practice to lock down GitHub. Here's how NPM staging should work. Here's a tool that you should look at.
[00:30:25] Jared Wray: Here's this other tool that you should look at that does something similar but a little bit different and is not really maintained, but we all like it. Here's a, a scanner that you should look at." And so basically, I decided that I was going to, maybe we can call it a manifesto. But I w- I was gonna build an agent skill that basically does full defense in depth.
[00:30:48] Jared Wray: Because after thinking about it, I was like, this is gonna take me three to four months if I did this manually to lock down everything and set all the right permissions. And, you know, e- even GitHub, I couldn't figure out how to lock down certain things in GitHub because it's so complex, right, over time.
[00:31:09] Jared Wray: Like, how do you require certain runners to run? How do you make it so that nobody can add a runner by accident? How do you scan the code? How do you, you know, have release gates? I didn't know any of that, and honestly, I didn't have that type of stuff. And so I basically spent, instead of being like, "I'm gonna lock down everything," I got to the beach, I figured out how to clean the repos so the malware wasn't there, which took me about six, seven hours to figure that out and do it properly in an isolated environment.
[00:31:43] Jared Wray: I spun up basically, um, a Parallels desktop and basically ran the commands, working with, uh, ChatGPT and Fable to tell me what to do right. 'Cause it, it was actually kinda scary, like I, I was double-checking those Git commands, 'cause it's basically you're deleting, like, branches.
[00:32:03] Jenn Gile: Right. You wanna be extra sure.
[00:32:05] Jared Wray: Yeah. And then I, once I got that done, I was like, "Okay, I'm gonna spend the next couple days really figuring out how this needs to work," and I had to try things out, right? Like, I first was like, the first recommended was use, um, uh, local containers, right? Like, uh, spin up local containers and use Visual Studio Code, uh, with those containers, and they're all isolated and just delete them after every pull request.
[00:32:34] Jared Wray: And I actually tried that, and I was like, "This is a horrible developer experience." It's really, really hard to do.
[00:32:40] Paul McCarty: Mm-hmm.
[00:32:40] Jared Wray: Because you're, you're constantly just churning on those containers. And then I started realizing, like, I'm gonna change it a lot more. Uh, there was two truths, and I actually blogged about this, that I realized I was gonna have to do.
[00:32:55] Jared Wray: One is in the age of AI, most likely a lot of my code's gonna be written by AI. And the second one is I need to be in isolated environments for almost every pull request. And so that's how I started building it out. So I- Took those fundamental values and I was like, "Okay, where is gonna be the attack surface?
[00:33:17] Jared Wray: Where is gonna be the bottleneck moving forward by moving to isolated environments, to configuring things correctly?" And it's literally this checkbox that I run on every single repo, and it's, uh, it's a really good skill. So it locks down all of GitHub now. GitHub is basically my release gate system.
[00:33:37] Jared Wray: That's probably the biggest attack surface at this point. But all AI, all, all coding is done in isolated, iso- isolated environments. Uh, usually they're, they only run once and then they get deleted. Uh, only time that it's a little bit different is Codespaces from GitHub, so, uh, and that basically only lasts, I think, seven days before it gets deleted.
[00:34:01] Jared Wray: Um, so that's kinda what I did. And it took-
[00:34:05] Jenn Gile: Yeah, you said that you've... One of the components is an agent that's, like, your security manager. Yeah. Yeah. Uh, talk us through, like, how did you decide for that agent to, you know... W- what was it gonna do? How did you, uh, make sure it had the right context, given you're not a security manager?
[00:34:22] Jenn Gile: How did you, you know, teach it to be the thing that you're not?
[00:34:26] Jared Wray: Yeah, so, uh, to give you an idea, I... Once I got defense in depth done and I started applying it to smaller repos, I was learning a lot as I was going on what needs to be there and what doesn't need to be there for AI to actually apply those features.
[00:34:42] Jared Wray: Even to the point there's actually a script called Lockdown, um, that's a .sh file. Uh, and it basically, like, you log in via terminal to get, uh, your GH, your GitHub CLI, and then you run the lockdown on a specific repo, and it kinda figured out how to lock everything down. 'Cause I honestly couldn't figure out how to find everything in GitHub to lock down- Like,
[00:35:08] Jenn Gile: if you needed to do this again-
[00:35:10] Jared Wray: Yeah, there's no way
[00:35:11] Jared Wray: where's the break glass- So I had AI write the script for me, that I could run it over and over again. Um, but getting back to this, like, what I ended up doing is, after I got all the repos basically with the, the full defense in depth and it can validate it, I was like, "Okay, what's the next step?" And the next step is constant looping on it, 'cause I'm g- I know I'm gonna improve defense in depth, and I've already...
[00:35:38] Jared Wray: It's probably gone through, like, three or four iterations. And I, I've even had some people be like, "Hey, we really like what you've done with dependency management skill that you have," which I run every single month uh, on, on my repos to basically update the packages and things like that. But they're like, "Can you do it for other ecosystems, like Python- Mm-hmm
[00:36:00] Jared Wray: Go, stuff like that?" So I'm already working on that for Defense in Depth, basically having that all in there. You
[00:36:05] Jenn Gile: got another project to maintain.
[00:36:07] Jared Wray: Yeah. So I was like, how do I loop this? How do I basically... Because the security stuff is coming much faster now. Like, wasn't it two weeks ago that, like, we had, like, four zero-days in, like, a 72-hour window?
[00:36:22] Jenn Gile: Bad week.
[00:36:23] Jared Wray: Yeah, yeah. Bad week. But that's gonna keep on going, so I decided the better thing to do is to have a, a bot that is my director of security, and what it does is it actually looks at a couple sources. Uh, I can't remember whether... But anyway, looks at the sources, also looks at Twitter, and then it's like, hey, if something posts, I basically had to train it to verify it, validate it, and then look at all my repos and see if I'm affected.
[00:36:55] Jared Wray: And if it does, it automatically opens up a PR for me to actually, uh, look at.
[00:37:01] Jenn Gile: Mm. '
[00:37:01] Jared Wray: Cause I want that loop always going. The other thing it does is it makes sure that Defense in Depth is actually still on my repos.
[00:37:09] Paul McCarty: Mm.
[00:37:11] Jared Wray: So it checks that every single day. Uh, usually every two hours it's checking all my repos and things like that.
[00:37:17] Jared Wray: And then the final thing is the dependency management skill that I have, now it runs every single month, and it picks up the changes for Defense in Depth. So all I have to do is update that one skill with a change that I really want, and then next time I run dependency management, it's now updating with Defense in Depth.
Security controls that took hours to set up
[00:37:39] Paul McCarty: I, I think there's like, there's so much that I like about this, but at the same time, my security hat is like... 'Cause I've gone through a very similar process, Jared. Like, I, you know, I moved to, like, isolated environments for, for, uh, coding. But then the problem is that you have to- And we were talking about this before we went live.
[00:37:57] Paul McCarty: You know, you have to- Yeah ... you have to seed that environment with access to GitHub to be able to get things, especially if these are private-
[00:38:04] Jared Wray: Uh-huh ...
[00:38:04] Paul McCarty: repos. And that means using the GitHub CLI or using PATs or using some other way to give that machine or that instanti- instantiation access to those things.
[00:38:15] Paul McCarty: Right. But then even worse, you have to allow it to then write, and, like, the GitHub CLI cannot do pushes, right? It can- Oh, yeah ... it can clone, but it can't push. So that means that you need to, if you're using SSH keys or if you're using, you know, GPG signing and all these things, you have to get all that into that environment, orchestrate it, and, and make it work and do that again and again and again, and it's hard.
[00:38:36] Paul McCarty: It is genuinely difficult. I- Oh, I totally agree ... struggle with it. Yeah. I mean, I struggle with it personally, and I've been doing this a long, long, long time, and that is very, um It just reminds me how difficult this is for developers that haven't had 30 years of security experience- Yeah ... to do it as well.
[00:38:55] Paul McCarty: And so, so e- everybody listening, Jared asked me the other day, he asked me very pointedly, he's like, "What, what can I do to make..." And I was like, "Sorry, mate, there is not one thing." This is like, you know, I can't tell you to put your seatbelt on or the ABS brakes are gonna fix you or the sensors, right? Like, it's all of it together, and a bunch of other stuff that we have to constantly be iterating on.
[00:39:15] Paul McCarty: And so like you, Jared, I think it's about, you know, putting stuff in place and then iterating on it and testing it, and not assuming that you've solved the problem and moving on.
[00:39:26] Jared Wray: Yeah,
[00:39:26] Paul McCarty: I don't think it's sol- It's not. Uh, the attack that we're seeing right now in GitHub, I showed it to a friend of mine and he went and took a look at it, and he's like, "Oh, all they're doing is writing markdown."
[00:39:35] Paul McCarty: And I'm like, "No, mate. If you call the GitHub API, you'll see that they wrote markdown, then they pushed an action, the action ran, they did all these things, and then they deleted it all, and all you can see that's left over is one line in a markdown file. You missed the point." They just looked at it and thought, oh, there's nothing going on there, right?
[00:39:51] Paul McCarty: The reality is that a lot of bad things went on there, and you just didn't understand that you had to go and look at the event API to get that data.
[00:39:59] Jared Wray: Yeah.
[00:39:59] Paul McCarty: Um, and I, we were talking about CLI, sorry.
[00:40:02] Jared Wray: I, I totally
[00:40:03] Paul McCarty: agree. We were talking-
[00:40:03] Jared Wray: Like, it is- Go ahead ... also crazy, like th- even the SSH, 'cause you need a good developer experience, and sometimes what...
[00:40:10] Jared Wray: Like, there's things I... There's things on my attack surface that, like, I'm still trying to figure out. Like, for instance, what you were talking about, you have to give it access to GitHub, usually- Mm-hmm ... through OAuth, right? If you try and do- Yeah ... a PAT or something like that, it gets really cumbersome when you have this many repos going- 100%
[00:40:29] Jared Wray: and this many agents, right? Like, SSH is a great example. Everybody was like, "Hey, you should start signing your code." And so I dug into that for a week trying to figure out how I can use agents, provi- do SSH signing, and sign my code, and still have it published pretty seamlessly. I could not get it working-
[00:40:50] Paul McCarty: Yeah
[00:40:51] Jared Wray: without, like, a huge amount of hassle. Like, waiting for an agent to ask for a prompt so I can give it an SSH signing envelope, and then it's still on that desktop, and then sometimes it would mess up trying to push something. It was a nightmare. So that's why when- So to this day, I can't do SSH signing.
[00:41:08] Paul McCarty: Y- so that's why when you go and look at people's Git history in, in a particular repo, in a particular project, you'll see they've signed a commit because they did it themselves manually.
[00:41:18] Paul McCarty: But then the next one- Yep ... which is pushed by Claude or their agent, it's not verified because- It's verified,
[00:41:22] Jenn Gile: yeah.
[00:41:22] Paul McCarty: Because it's- Yeah ... really painful. And I wish there was, were things, you know, going back to the GitHub CLI, the GitHub CLI is a particularly pernicious problem. Wow. Look at my alliteration. Of course.
[00:41:36] Paul McCarty: I'm crushing it today. My kids would be so impressed. Um, didn't even plan that. Uh, but you know, it... There's no, there's no MFA, there's no two-factor built into it. It saves your credentials in the config file in clear text. A lot of people don't realize that, regardless of whether or not- I didn't realize that until you told me.
[00:41:52] Paul McCarty: Yeah. And, and even it... So if you hand GitHub a PAT, it's gonna store the PAT in the config directory in clear text. If you auth in over OAuth, like, it's gonna save that session, which is the GHSA_S- in that same directory in clear text. Come on, GitHub. Like, millions of people are em- empowering their, their agents with the GitHub CLI.
[00:42:16] Paul McCarty: I mean, as the default behavior now is to use the GitHub CLI if it sees that it's authenticated. It often doesn't even ask if you're in auto mode. It just says, "Oh, sweet, the, the GitHub CLI's installed and I got a auth status yes. Let me go ahead and use it." W- we need to be addressing these things individually.
[00:42:32] Paul McCarty: And I think there's only so much that you and I, Jared, can do. And honestly, getting back to Paul kicking GitHub, they need to do more with a lot of these tools they put out there. They need to be doing more.
[00:42:44] Jared Wray: Yeah. Yeah, I, I think they're trying to do the best they can. And to be honest, once I learned about some of the security stuff that you can do, especially that, that they've added probably over the last five-plus years, like branch u- uh, rule sets on branches, like that's an amazing achievement to add that in.
[00:43:03] Jared Wray: Like, that's really great. Uh, and but I could not figure out myself how to actually set this up without making it really painful, right? I didn't even know about a code owner's file. Like, none of these best practices, like why would a developer even look that up? They're, they're trying to- Right ... ship as fast as possible, especially in th- these days, this day and age.
[00:43:24] Jared Wray: And it was really hard to figure things out and be like, like you get a suggestion, "Hey, you need to add a code owner's file and lock down certain things that need basically, like someone to actually look at and approve."
[00:43:38] Jenn Gile: Yeah, it might... I mean, I would be interested to hear about your list of things that you thought would be easy that were hard.
[00:43:47] Jenn Gile: Because as we were talking about as we were prepping, there's definitely a lot of narrative when a developer gets compromised. "Well, why didn't you have code signing turned on? Or why didn't you have this?" Or, you know, as we were, you know, talking about, uh, you know, should we just stop, uh, using open source and, you know, maintain all that code in-house?
[00:44:08] Jenn Gile: Uh, these are all things that are easier said than done, but they're also like- Particularly pernicious myths. Sorry, I couldn't- Yeah ... come up with a third P.
[00:44:19] Jared Wray: Um-
[00:44:19] Jenn Gile: PPM.
[00:44:19] Jared Wray: You're close.
[00:44:21] Jenn Gile: I, I worked on it. It didn't go far. Uh, but there, there are these narratives that float around the security community that I also do hear from developers of, "Oh, well, this feature is there.
[00:44:33] Jenn Gile: You know, this wouldn't be a problem if you just turned it on." Yeah. So I'm hearing you say code signing, not as easy as it sounds.
[00:44:40] Jared Wray: Still can't get it to work.
[00:44:41] Jenn Gile: What else?
[00:44:42] Jared Wray: Uh, so code signing was one of them. I was like, this is actually way harder, especially in the age of AI, 'cause things have shifted, right?
[00:44:51] Jared Wray: It's not... If, if I had a local desktop always running, I could have that key always there, right? But you don't wanna give AI that key, and so how do you publish, right? It's, it's hard that way. So code signing was one of them. Uh, branch rule sets. When you go into that UI, like, y- it is actually really hard to understand what you should and should not turn on or off.
[00:45:14] Jared Wray: And it, it, it is not descriptive in any way for someone like me. Like, I had to go through, like, every single one. At one point I took a screenshot and I told Claude to describe what each feature did and h- how it would change things.
[00:45:31] Jenn Gile: And was it able to find documentation for those features?
[00:45:36] Jared Wray: Yeah. Yeah. But even the documentation- At least it was there
[00:45:39] Jared Wray: is really confusing. Mm. Like, it is, it is written by a security expert without, like, hey, a common developer should have these default things set. Like, it is really hard to do. So I had to go through almost every single one of them and be like, "Okay, if I turn this on and turn this off, and this is off, what would happen?"
[00:46:04] Jared Wray: And I had to do it for every single one on the rule set page. I did it for, uh, a lot of the security settings. Uh, one thing I, I turned on and instantly turned off is Dependabot. It was horrible. Like, I can't manage it because it- Was
[00:46:19] Jenn Gile: it just too noisy, or was it
[00:46:21] Jared Wray: not- Super noisy ... yeah. And it would basically kept on, keep on pop- popping up, like, alerts.
[00:46:26] Jared Wray: But you go into the alerts- Yeah ... and to be honest, like, those CVE alerts are unbelievably hard for a developer like me to figure out, like, do I need to care or do I not?
[00:46:39] Jenn Gile: Mm.
[00:46:40] Paul McCarty: Well, and also too, like, y- like, let's just assume that what Dependabot is telling you is legit. Mm. It is difficult in many of these cases to actually do the thing that Dependabot says, because it immediately, you test it on your own machine and it breaks the app.
[00:46:55] Paul McCarty: It breaks. Yeah. And you're like, "Oh, that's because- Yeah ... that's because it's now using the new version of keyv or whatever," you know, whatever the case may be.
[00:47:01] Jared Wray: Mm-hmm.
[00:47:01] Paul McCarty: Like, this, this overly simplistic idea that, and this is another one of those InfoSec myths, right? Where people, InfoSec people, me, like, people like me-
[00:47:10] Jared Wray: Yeah
[00:47:10] Paul McCarty: you know, who never came to the software engineering space to understand how it works, saying these simplistic things like, "Well, just upgrade to the newest version of that." I'm sorry, mate. That doesn't, that doesn't work, you know, 80% of the time or whatever- Yeah ... because of N reasons, right? Yeah. Uh,
[00:47:26] Jared Wray: and that- So-
[00:47:26] Jared Wray: that's one of the things I had to do with my, uh, director of security. Like-
[00:47:31] Paul McCarty: Mm ...
[00:47:31] Jared Wray: basically, like, when it finds something and it needs to do an override, luckily, like, I have end-to-end testing in GitHub actions, but it also, like, has to verify everything. If it's breaking, I tell it to upgrade the model and figure out what all the changes have to be.
[00:47:47] Paul McCarty: Dependabot is the single most useless feature they have ever pushed because people think it's a-
[00:47:53] Jared Wray: It's just, it's really noisy. I, I end up by default turning it off and people are like, "Well, you're not secure." And I was like, "Well, I also have like security scanners running on every PR." Right. I do regular dependency management, which a lot of people don't do.
[00:48:06] Jared Wray: Uh, the other thing, Jenn, that like was actually, it took me a while to figure out, is how to lock down containers and- Hmm ... add things like, you know, a, a scanner, like, you know, a security scanner. I had a look at a whole bunch of them that, like PMG got recommended and I really like that project because they're moving in a direction that I, I like.
[00:48:28] Jared Wray: I ended up with Aikido's supply, uh, their Aikido shim. Um- S- so chain ... but to set that, yeah, to set that up I had to figure out like, oh, I need a, a .dev container folder, and what is the config for that, and how do I verify that it a- is always there? How do I tell an agent that they can't run anything until everything's installed, right?
[00:48:51] Jenn Gile: Yeah. So to... I think what you're saying is another thing to add to that sounds easier than it really is list is turning on code scanning of some kind.
[00:49:00] Jared Wray: Yeah. I mean, these are like two to three hours of discovery that you have to do as a non-security expert for every single little piece of flow. The last one, which took me the longest, is how to basically do release gating in GitHub actions.
[00:49:16] Jared Wray: And I- Mm ... like scan things beforehand before you publish it, and what is a staged publish on NPM now because it's a new technology. And, you know, how do we verify that that stage publish is really good? Well, you have to use something like Drydock or something like that. And it's really interesting going through that process, adding all these tools and then trying to make them actually work together.
[00:49:40] Paul McCarty: And if you're, if you're, whatever it is that's trying to ascertain whether or not that staged thing is good or not has to have to ac- sorry, has to have access to NPM to be able to see it. 'Cause- Yeah ... if you're looking from the outside, that staged artifact has nothing in it. It's a package.json and a, and a README, and that's it, right?
[00:49:59] Paul McCarty: Yeah. So you need to have... So you have the same problem. When you're building these Docker containers, right, to do it all the right way, you have to get your code in there, and you have to get your code in there in the, in a way that's not gonna, you know, blow up inside the container now because you gave it access to the code and gave it the ability to do things that you needed it to.
[00:50:15] Paul McCarty: Oh,
[00:50:15] Jared Wray: yeah. Yeah, and I made a policy that, um, and, uh, even on my blog, 'cause a lot of people ask, like, "What is my desktop set up now?" I use very little, um, apps anymore on my desktop. Like, I have, like, my calendar, my password manager, uh, you know, my music app, but pretty much, like, I do not install, other than Visual Studio Code, which I do not install any plugins except for the AI that I'm using, which is usually, uh, Codex for that, but that's running in a dev container also.
[00:50:49] Jared Wray: That's the only dev app that I'm running because, to be honest, like Cursor, uh, Claude desktop, uh, even ChatGPT desktop, I don't trust them anymore 'cause I think that's gonna be the next attack vector, so I run them all in a browser
[00:51:07] Paul McCarty: Uh, that attack, we deal with that attack factor all the time, right? And it's really hard when you open up your VS Code or your cursor to understand what plugins and extensions are already there existing when you open it up, right?
[00:51:19] Jared Wray: Yeah.
[00:51:20] Paul McCarty: Yeah.
[00:51:20] Jared Wray: Yeah. Yeah. So I had to move away from that, and it's kinda crazy. So you- people like us i- across all open source, like, we're gonna have to really think about this, and I think a lot of developers are gonna have to really think about the entire process. What happens at your desktop, what happens in the cloud, what happens when a pull request lands, what happens when you're ready to publish, what that looks like.
[00:51:45] Jared Wray: Like I did not know that you could actually set releases. Well, actually, one of my friends told me. He was like, "Hey." He was reviewing the defense in depth, and he goes, "Hey, by the way, you should make your published releases non-editable." And I was like, "I didn't even know GitHub had that feature."
[00:52:02] Paul McCarty: Right. I have...
[00:52:04] Paul McCarty: Like, there's so many other things I wanna say here, but I... Like, I wanna... And looking at time, being respectful of people's- I
Jared's skill scores dependencies before he adds them
[00:52:09] Jenn Gile: have one last question that I wanna talk through that I feel like is a little bit of a wrap-up. Um, we're all still team open source in this room right now. For sure. Uh, so presumably, like, you're still in circumstances where you need to use a new project from time to time, or your agent suggests it.
[00:52:27] Jenn Gile: So, like, how do you decide whether you're going to bring a new project in, outside of, you know, does this thing do the thing that I need it to do? You know, how do you decide this is not going to, uh, increase your attack surface unnecessarily?
[00:52:46] Jared Wray: Yeah. Uh, I've been having a lot of conversations with other developers around this.
[00:52:51] Jared Wray: Um, some people have said, hey, like, open source is gonna be dead. I know people have mentioned that. I think that's actually the wrong approach, um, just because a lot of these have a lot of eyes on these projects, and usually if they're being maintained well, they're, they're actually having a lot better security fixes and best practices.
[00:53:10] Jared Wray: When I look at dependency management, um, I usually try to keep it as little as dependency tr- uh, you know, chain or tree as, as possible, especially, uh, with Node.js. But when I'm looking at, like, something an A- AI's recommending to bring in, I I actually have a skill that I run on that, uh, that specific one, and it checks a couple big things.
[00:53:40] Jared Wray: One, it checks how often do they publish. Is it basically being actively maintained on the publisher? I now am checking if they actually are doing OIDC, if it's more modern. Uh, I also check how many dependencies it actually brings in And if it's, like, over, like, 20 or 30, I start really considering, like, do I not want it?
[00:54:03] Jared Wray: And then the final thing is it actually goes out to the GitHub repo where that is and sees how many contributors have been contributing to it, when was the last updates, are they actually doing security stuff, and then it does a quick code review of their process. And then it basically, it gives me a score, high, medium, low, like, should I do it?
[00:54:25] Jared Wray: Should I not? And there's been many times now that I actually decide not to bring in a dependency and have just for that one feature, have basically, like, AI code it.
[00:54:37] Jenn Gile: I'm curious, as you're looking at activity, this is something Paul and I talk about a lot because we see it faked a lot.
[00:54:46] Jared Wray: Mm-hmm.
[00:54:46] Jenn Gile: Are you looking at the activity in the web portal, or are you pulling, um, like the Git history?
[00:54:55] Jared Wray: It's pulling the Git history, but when I talk about, um... So it does pull the Git history, because I, I know in the, like, the web portal it's a little bit interesting. Uh, the activity that I'm more interested in is the pull request comments.
[00:55:11] Paul McCarty: Mm.
[00:55:12] Jared Wray: I wanna know if people are actually actively engaging- Are people
[00:55:14] Jenn Gile: engaging
[00:55:16] Jared Wray: and understanding what's going on as maintainers. And to be honest, like, a lot of open source, and we've talked about this before the show, it's really hard for maintainers to actually maintain because it's free, right? It's not their day job. Yeah. It's really hard to b- you know, keep up on things. Um, so yeah.
[00:55:37] Jenn Gile: Yeah, that's a really interesting signal to look for actual engagement rather than just-
[00:55:43] Jared Wray: Human
[00:55:44] Jenn Gile: comments ... appropriately as code shipping.
[00:55:45] Jared Wray: Yeah.
[00:55:46] Paul McCarty: There's a, there's a project from the OpenSSF called Scorecard, which I actually have done some open source work on myself-
[00:55:52] Jared Wray: Oh,
[00:55:52] Paul McCarty: that's cool ... which does a lot of that as a project.
[00:55:54] Paul McCarty: And, um, so if, if, if you're not familiar with it, Jed- I should check
[00:55:56] Jared Wray: it out ... '
[00:55:57] Paul McCarty: cause there might be some things there that you can steal or just start using Scorecard in parallel to your own stuff. But are you at the same... Uh, in that skill, are you also checking OpenSourceMalware to see if, if any of those packages are, are malicious?
[00:56:09] Jared Wray: Yeah, I do.
[00:56:11] Paul McCarty: Cool. That's super cool.
[00:56:12] Jared Wray: But-
[00:56:12] Paul McCarty: Uh, can we- ... to
[00:56:12] Jared Wray: start, like, it's, it's hard to detect that stuff.
[00:56:17] Paul McCarty: It is. It is, especially when we start talking about those transient dependencies to our point- Mm-hmm ... earlier, right? What's in the package manifest, that's one thing. Yeah. But when you start looking, when you start taking a look at the, at the, the lock file or trying to figure out yourself in real time, you know, what, what the, the, the actual dependency structure is-
[00:56:37] Jared Wray: Yeah
[00:56:38] Paul McCarty: it's, it, yeah.
[00:56:39] Jared Wray: And I think also, like, in the open source world, it- I think it's a very long tail of packages, but most of the time people, especially AI, they're gonna choose the bigger, more popular packages, uh, that are more maintained just because they've been trained on that. It's been used more. Um, and I think that's gonna continue, and people are...
[00:57:03] Jared Wray: You know, it used to be like every little thing that w- you needed, there was a package for, and I don't think- Mm ... that's gonna be the end state of open source. I think it's gonna be a lot bigger roll-up packages that do a lot more stuff.
[00:57:16] Jenn Gile: Yeah. You know, it may be that direction for the packages, but we're seeing the other direction for, uh, skills and, um- Yeah
[00:57:27] Jenn Gile: other agent-focused things. MCPs.
[00:57:30] Jared Wray: Yeah.
[00:57:31] Jenn Gile: MCP servers. You know, we're seeing a boom of that on GitHub. Many of it is malicious and designed to agent bait. Oh, wow. And, uh, there's, um, some really good research from a company called Island Security that they did over the summer where they showed, and I was able to recreate it, asking your, uh, agent for a recommendation, and it's serving up malicious, uh- Oh, wow
[00:57:58] Jenn Gile: GitHub repositories. That's crazy. So I was able to get it to do that. It gave me a malicious- Yeah ... MCP server that was pretending to be for Walmart.
[00:58:06] Paul McCarty: It's called agent baiting
[00:58:09] Jared Wray: That's amazing
[00:58:10] Jenn Gile: It is. Yeah. Amazing and scary. Never even heard of
[00:58:12] Jared Wray: that before.
Other CI platforms couldn't replace GitHub Actions end to end
[00:58:13] Paul McCarty: I, I wanna, I wanna ask the GitHub Actions question. I know we're going long now, but I don't care.
[00:58:17] Paul McCarty: Mm-hmm. Yeah. Um, uh, when we talked last week, Jared, you said at the end of this whole thing, and I thought when I heard this, I was just, I was really amazed, and I was struck by your optimism too as well- ... which I think is really cool. You said, "After all this happened, it made me really wanna understand what GitHub Actions does, and really focus and really centralize everything I do inside of GitHub Actions."
[00:58:38] Paul McCarty: And at the time, you know me, you know, uh, the way my brain thinks, I'm thinking, "I wanna run as far away from GitHub Actions as possible. I wanna get away from it as quickly as possible." And here's this guy that's just gone through this compromise that has happened in GitHub Actions, and he wants to do exactly the opposite.
[00:58:54] Paul McCarty: So first, I commend you on sticking around and wanting- Yeah ... to make the neighborhood better. But I guess, kinda can you talk to, like, what your thinking is? Like, what your thinking was about why you really wanted to do that?
[00:59:06] Jared Wray: Yeah, I mean, I, I first started being like, "Maybe I shouldn't use GitHub Actions anymore."
[00:59:13] Jared Wray: That was actually my first... You know, when I really started getting down to, like, basic principles, I was like This could, and I know that, that you have to, like, do GitHub actions a certain way to really secure them and lock them down, and it's an old technology. I mean, it's a YAML file. Like, it's kinda crazy that we think that way.
[00:59:34] Jared Wray: So I actually spent about a week trying out other CI/CD platforms and trying to see, like, how it all flows together and how you, you know, still use GitHub or use something else, and then use a CI/CD. I, I really tried to open my mind up to, like, h- how am I gonna do this? Just like I did with my desktop.
[00:59:57] Jared Wray: And what I basically realized is if you want a really good developer flow, you basically have to go all in on something to get it to work correctly, because when you try to add all these, like, currently... And I do think in the future that will not be the case.
[01:00:19] Paul McCarty: Hmm.
[01:00:20] Jared Wray: I think CI, there, there's a lot of work going on, and I saw the potential.
[01:00:25] Jared Wray: There's a lot of companies starting up that are, like, really fast CI/CD. They understand agents. This is an old school technology that was built for humans and slow code reviews.
[01:00:35] Paul McCarty: Right.
[01:00:36] Jared Wray: And so I tried a lot of those, but they weren't mature enough to actually have the flow end to end. Like, a lot of them you can't do OIDC publishing to NPM.
[01:00:46] Jared Wray: So I was like, "Oh, well, maybe I'll just have that one action," and then, like, all the tests on something else. But then I started digging in and I was like, well, now I have to have a defense in depth for them, which is another attack surface, and- Hmm ... basically GitHub also. And so you're getting down to this point where you're like, "Well, wait a second, now I'm opening up this attack surface," 'cause what if they get hacked, and how do I know?
[01:01:11] Jared Wray: And these are newer companies, right? building these out. So then I was like- Right ... okay, well I still have to figure out how to lock down actions.
[01:01:19] Paul McCarty: Right.
[01:01:19] Jared Wray: And secure those and get those working. So I kinda just went all in at that point. So that's kinda how I went to that. I do think it's an old school technology.
[01:01:30] Jared Wray: I, I think they're gonna improve it over time. Um, but-
[01:01:35] Jenn Gile: I mean, I've said this probably on repeat the last two years, it's gonna be an interesting 12 to 18 months.
[01:01:41] Jared Wray: Yeah.
[01:01:41] Jenn Gile: Like, let's see where this technology goes. Maybe if we talk to you in a year you will be totally off GitHub.
[01:01:48] Jared Wray: Yeah. And by the way, I even, uh, I looked at Origin, right, from Cursor, and I was like, "Oh, this is actually really cool."
[01:01:55] Jared Wray: But then you start having things fall apart. And I agree with you- Yeah ... like, it could be, like, you don't have to be on GitHub anymore, and there's a lot of development going on for that, and I'm excited about it, but it doesn't work end to end yet, and that's what I need to go fast or faster, right? To keep that developer experience going.
[01:02:14] Jenn Gile: Yeah. As a maintainer who's doing this as, uh, you know, a way to give back and not a full-time job, you don't have a platform person on staff and a security person on staff to be able to make sure these things are strung together, right?
[01:02:29] Jared Wray: Yeah. I, I think honestly, like, so I've been CTO for a lot of companies- To really lock things down, you need almost, like, an entire DevOps security team.
[01:02:40] Jared Wray: It can't be just even one person anymore 'cause it's so many different variances.
[01:02:45] Jenn Gile: I think a lot of security people, uh, will be really happy to hear you say that. Those are our- We've definitely seen a lot of jobs go away because of, uh- Yeah ... the perception that AI can do it better- Yeah ... faster, stronger.
[01:03:01] Paul McCarty: App- AppSec teams, particularly in the last two years, have really been hard hit, it's unfortunate- Yeah, they've been hit
[01:03:06] Jared Wray: because-
[01:03:06] Paul McCarty: That's interesting ... the assumption, the assumption is that AI would do it for us, which I think is ridiculous. I, I also think- Yeah ... to your point, I think you need A, you need A-team players, too. You need top-tier players, right? Mm-hmm. Like, you, like you can't... If your, if your AppSec person is just doing vulnerability management, which by the way, a lot of AppSec people are just stuck doing vulnerability management and vulnerability triage, right?
[01:03:28] Paul McCarty: Let's just be honest, right? They're not really doing AppSec, they're just managing- Yeah ... whatever it is your vulnerability engine is, right? Yeah. So, uh, you know, you need top-tier players to be able to really kind of tackle this stuff and be looking ahead to see, kinda like threat modeling- Mm-hmm ... you know, where are things gonna go, right?
[01:03:43] Paul McCarty: To your point earlier about how AI is changing, the speed and the pace, and the fact that you've got a bunch of different agents working on the exact same piece of code, and you have to, like, kind of figure out how to kind of bring this back together into a unified code base and do that securely. That is really difficult to do at velocity.
[01:04:02] Jared Wray: Yeah.
[01:04:02] Paul McCarty: And we live in the, we live in the cult of speed, right? Speed is, is most important above all things else, which I think is a problem. But in that cult of speed, then, the problem is you have to be able to do these secure things at that lion speed, and that's just... I mean, it's basically impossible to do.
[01:04:18] Jared Wray: Yeah. And I like that you said A, A players because, you know, I would probably say I've had this problem in the past w- w- working with security teams. Mm-hmm. To be honest, like, most, uh, many non-A players, and maybe it's a skill that, like, we need to really focus on, and I'll, I'll say this also for, uh, uh, SREs or DevOps engineers, they're so nerded out by, like, what they're doing in security or infrastructure management or anything like that, they forget the fact that who their customer is, is actually the people trying to produce something-
[01:04:56] Paul McCarty: Mm-hmm
[01:04:56] Jared Wray: right, for the company. And they either make it, the developer experience, so painful that nobody actually ever wants to work there, like, you know, locking down their desktop. No. Actually, this is really funny. At CenturyLink, we, we were a cloud computing company that they bought, right? So we had data centers, we had, you know...
[01:05:16] Jared Wray: All of our engineers had access through VPN to their infrastructure because if you build it, you ran it. And I remember the first day or week after we got r- acquired, they, uh- Gave us a new internet that had no access to VPN, so we couldn't actually get to our infrastructure, told us we could only do that.
[01:05:38] Jared Wray: Then they sent us new Mac laptops, and they were so locked down that an engineer, remember, an engineer making 250K a year would have to call the, uh, help desk- Right ... to get Visual Studio installed, and the help desk person would do the install.
[01:05:56] Jenn Gile: Mm.
[01:05:56] Jared Wray: I was like, "
[01:05:57] Jenn Gile: Yeah." That's antiquated.
[01:05:58] Jared Wray: Yeah. And, and you're like, uh, it's improved over time, but, like, you got, you gotta figure out how to help people, like, where they're at and make it still go fast or make it so they can produce.
[01:06:09] Jenn Gile: Yeah. You gotta make the right thing easy.
[01:06:12] Jared Wray: Mm-hmm.
[01:06:13] Paul McCarty: I mean, enterprise is building these, like, locked down golden image Windows images for company, for, for the developers. I just recently, I'm not gonna say who it is, but I recently did... Anyhow, and I just, the limitations in... that their engi- engineers, they basically can't use any agents at all except Microsoft Co-Pilot and, like, just, just the limitations.
[01:06:33] Paul McCarty: Yeah. I'm like, how can they perform at their top peak? P- and then that's the thing then, is that you can't expect them to operate at peak speed, peak performance, because you haven't given them the tools to do that. So you're gonna get B team and C team players, right? 'Cause that's what you're giving them.
[01:06:47] Paul McCarty: Yeah. I think on the other side, though, there's also an argument to be made that developers haven't, you know, really brought themselves towards the security thing because it was always, well, the security team was doing that, and they're idiots. They can't do it right. They don't know, they don't know how the software sausage is made.
[01:07:02] Jared Wray: Oh, I totally
[01:07:02] Paul McCarty: agree. They need... Yeah. Jenn's wrapping me up.
[01:07:06] Jared Wray: Yep.
[01:07:06] Paul McCarty: But, but it, but the reality is, the reality is that developers need to come, it, th- both teams need to come towards the middle, and that was what DevSecOps was supposed to deliver for us, but it just hasn't happened, right? Yeah. That's the reality.
[01:07:19] Jenn Gile: Well, ending on a positive note, I think this has been a really interesting conversation, you know, to listen to what you went through and how hard it was and how hard it continues to be. I think I'm trying to picture you trying to do this maybe two or three years ago without agents and how prohibitively expensive and difficult it would've been.
[01:07:43] Jenn Gile: So I think we're hearing generally a success story, but man, it's still hard.
[01:07:48] Jared Wray: Yeah, yeah. I, I did an estimate. It was, it would, to clean everything up and lock everything down, it would've taken me personally probably full-time three months.
[01:07:58] Jenn Gile: Wow.
[01:07:58] Paul McCarty: Yeah.
[01:07:59] Jenn Gile: And then what happens- I'm not, I'm not surprised by that
[01:08:00] Jenn Gile: to your open source projects at that point?
[01:08:02] Jared Wray: Yep, exactly, exactly.
[01:08:04] Paul McCarty: When your family and your, like, how you make money and all the other things that we have to do too as well that you're juggling as well.
[01:08:09] Jared Wray: Yeah.
[01:08:09] Paul McCarty: Hey, well, let us, let us know if OpenSourceMalware can help out in, in any way, and, you know, just, uh, you know, feel free to reach out anytime to me personally or to Jenn.
[01:08:17] Paul McCarty: Um, you know, we wanna be part of the solution.
[01:08:20] Jared Wray: Awesome.
[01:08:22] Jenn Gile: Cool.
[01:08:22] Jared Wray: Will do.
[01:08:22] Jenn Gile: Thank you.
[01:08:23] Jared Wray: Yeah, thanks.
[01:08:24] Paul McCarty: Cool. Thanks, buddy. Cheers, mate.