Mayhem Makers: Ryan Goulden, Engineer 

“Mayhem Makers” is an employee Q&A series dedicated to our growing company.

For this month’s profile, we talked with Ryan Goulden, an engineer on the Mayhem team, who joined the company in 2014 and is based out of Palo Alto, California.

‍

1. Tell our readers a little bit about what your role as an engineer on the Mayhem team entails. 

I previously worked a lot on the back end of Mayhem. So, that’s the fuzzing and the symbolic execution components, the actual analysis tools running in the background.

Recently, I've left those to other capable hands. Now, I’m working on longer-term research projects related to improving Mayhem. One of the things we're looking to do is bring our coverage-directed analysis approach to entire systems. So we're talking about things like embedded platforms, IoT, or routers. 

Historically, fuzzing has worked best with small, single program executables that you can run thousands of times per second. To fuzz something like a router image, you'd have to go in by hand, dissect the pieces, and fuzz them individually. We would like to be able to throw an entire router image under fuzzing. 

The idea of trying to fuzz a whole system like this isn’t new. There's many people, both in the industry and in academia, working on it. We're trying our own approaches, some of them quite novel.

‍

2. What did your career path look like before you came to work here?

I’ve worked here most of my career, but this has looked very different in the past from how it does today. I first worked under David Brumley when I was at university, as part of the Carnegie Mellon hacking team that ForAllSecure was born out of.

Back then, ForAllSecure was about trying to find buyers for the early version of Mayhem, which was not the full-featured platform that we have today. It was more just the analysis tools on the back end that we started developing as students. Continuing that work on Mayhem was what we did as ForAllSecure. 

Another thing we brought with us from CMU into the early days of ForAllSecure was running educational programs and events for people interested in cybersecurity. It was mainly the military academies that were our main clients, but we also ran programs for a few other groups. We ran some elaborate and engaging in-person events that, to this day, set a high standard for educational CTF-style events. 

We eventually moved out of a lot of those other spaces and transitioned into selling Mayhem commercially. We’ve changed a lot as a company, but everything came out of that passion and expertise we developed as a hacking club at CMU. 

‍

3. What is your typical day like at work?

My day usually consists of a bit of coding, a bunch of debugging, and a whole lot of theory work and experimentation to plan out how to tackle the next challenge in a research project. I do a lot of staring at code and reading papers to try to figure out how other people have solved a problem.

My days are actually more defined by the departures from my regular routine, because I also occasionally help out with customer support. Sometimes, I join calls with customers or prospects to help explain how Mayhem works and how it can help them, or, for existing customers, I help to solve any problems that they're having. I also do emergency IT work, like rescuing unstable servers or fixing broken networking.

‍

4. What has been your favorite project you’ve worked on as part of the Mayhem team?

There have been many fun projects over the years, but definitely a big highlight was working on the Cyber Grand Challenge. It was like launching a one-of-a-kind rocket, very much racing towards an end goal with a hard due date. Super stressful, but all the more satisfying when everything mostly worked, and we won. It's very different from software development, where the goal is more to maintain a large body of software indefinitely. 

‍

5. What have been your toughest challenges at work?

I'm not sure if challenge is the right word for it. It's more like an adventure. Our goals as a company and what we’re doing day to day have evolved so much over the years. There are challenges with that, with adapting to what the new thing is, but it's been a great experience. 

Originally we were kind of a contract company, doing security education, running CTF competitions, while also working on Mayhem. And then, as we've transformed Mayhem into an enterprise product, obviously, that's a very, very different process. And our approach to that process has changed dramatically over the years. 

When I put it like that, it sounds boring. But all of this is very new to us, who are a bunch of academics out of CMU. So every step of the way has been something new and has had its own challenges. 

‍

6. How have you grown professionally while working on the Mayhem team?

I think that ties in with the previous question. We started out very much doing what we already knew how to do as an academic hacking group. As we've transitioned into more of a traditional software engineering organization, there's definitely been a learning curve. 

I think it's been harder for us than the average engineer out of college, just because we were far more firmly academic in our pursuits. And it's not just academic, because the work we did was hacking research. Hackers and that community are generally not looked up to as reliable software engineering types. 

Coming out of the academic and hacker world, learning how to do software engineering was definitely growth that we've all experienced. We're all smart people. We code well. We've always coded well. It's just trying to juggle between all three worlds and all three different styles of doing things is sometimes a challenge. 

‍

7. What parts of our mission do you connect with?

The part of our mission that I think is one of the most interesting challenges is the goal of bringing new research technologies to a practical state where people can actually use them. There's a huge rift between what you read in a paper and something that is actually going to help the world and can be used by the wider public. 

Part of our company’s mission is helping bridge that gap. And that's not always a process that succeeds. Not every new research idea can make it to a state where it's approachable enough for people to use. What works in practice is a whole under-explored world.

It's a really interesting problem, trying to take what's great about the latest research, isolate what’s going to work for people, and then translate that from research into something that really works. The question of how to straddle that research and practice divide is something I work with daily.

‍

8. What advice can you offer to someone looking to get into a role like yours?

At the end of the day, no one's figured out how to teach software engineering super well. So to learn it, to get into it, you have to be self-motivated. You have to, you know, figure out how to learn things on your own, online. And the internet's a great resource, but you do have to be motivated to try stuff yourself and figure things out on your own, with the internet as a resource. 

Security is harder to learn online. It's weird, because despite the tech-based nature of computer security, the best way to learn and get into it is to find a person who can help mentor you. Try to find an organization that does some sort of security work. Or find an online community and work with other people. 

I got into security through the hacking club at Carnegie Mellon. It was the other people there that taught me things that I don't think in a million years I ever would have been able to figure out on my own. 

So it's not not a great answer for people looking to break into security, because not everyone has access to join Carnegie Mellon's world-leading CTF team. But if you've just started college and you're trying to get into this, you can get into it the way I did, which is through CTF. Capture the Flag, if you’re not familiar, is a competition where teams write security challenges for each other and weave them into competitive events.

‍

9. Tell us about yourself outside of work. What are your hobbies?

More and more, I like to get away from the computer when I’m not working. So I do a lot of things like backpacking, camping, and hiking. I end up doing a fair bit of international travel. I try to do at least one trip a year somewhere interesting, sometimes in the country, sometimes out of it. 

‍

10. Can you talk a little bit about being a part of the winning team at the DEF CON CTF this year?

This year, we played DEF CON as a joint team. The PPP team, which is the CTF team out of Carnegie Mellon that I’m a part of, joined with two other teams, "The Duck'' and "Maple Bacon". Together, we call ourselves MMM. 

Some of the other members who played on the Carnegie Mellon CTF team now have a very successful computer security related business in Korea, and have started a CTF team with all their employees. So that's one of the teams we joined with to play DEF CON. Also, someone who was on the hacking team at Carnegie Mellon is now a professor at UBC in Vancouver and has started up a CTF team out of there. 

So, those three teams all joined together to play in DEF CON this year, which is a much bigger group than we're used to playing with. Historically, the PPP team at DEF CON would be about twelve people, maybe twenty as we grew. This year, with the combined team, it was over sixty. 

A lot of us older folks actually worked more on logistics than participating in the CTF. We’ve played in the CTF for over a decade, so we’re fine with being less hands on. We still hack, we still solve problems, but we also work to make sure everyone's comfortable, doing things like finding the team food and accommodations in Vegas.

‍

11. How has your experience working on the Mayhem team contributed to your performance in the DEF CON CTF competitions?

Things we've developed over the years while working on Mayhem can sometimes be applied in the DEF CON competitions. The year of theCyber Grand Challenge, DEF CON was basically the same format as the Cyber Grand Challenge, but slightly rejiggered for humans. 

All of the tools that we had been developing over the last two years for Mayhem’s automated system, we used in the competition. We completely crushed all other teams. We’d had two years of preparation working on software tooling to help us play that game.

How much of what we’ve worked on to develop Mayhem that can actually be applied to the competition changes every year. But every once in a while, some of the tools we’ve developed can be brushed off and applied. 

For the most part, though, it's more the other way around, where playing in CTFs has helped me in the work we do on Mayhem. ForAllSecure and Mayhem were born out of a fusion of the academic research we were doing, but also what we were doing in CTFs. 

The idea of getting in there and actually hacking something is what separates us from a lot of other security companies, both from a technical standpoint of what our tools do, but also from a philosophy standpoint. We're not we're not a red team company. We're not a pen test company. But that's the sort of result. Mayhem is essentially a hacking software. We’re hackers. Let's show you the hack. That sort of philosophy is what drives our product and our team.

‍

12. Is it true that the PPP team has won more of the DEF CON CTF competitions than anyone else? What do you think contributes to the team’s success?

Yes, and by a long shot at this point. Every year that I've played since 2012, we've come in either first or second, and usually first.

Part of what helps us be the winningest is that we're still here to win. We've managed to have incredible longevity as a team. I still play DEFCON every year, but after playing CTFs for over a decade, I don’t put as much time into them anymore. Being a winning CTF team takes a lot of time and focus. 

So our advantage there is we've had a good pipeline of getting new members. Having new blood to keep the team going is definitely one of our not-so-secret advantages. It's hard for a lot of other teams to sustain that. Their members have been doing it for too long and just get less interested in playing CTF. They're still very good security professionals. And they could still do very well in CTFs if they wanted to, it's just that they don’t play as much anymore and might not have as many new members.

‍

13. Do you have a memorable moment or experience from DefCon this year to share? 

It's not totally unique to this year, but now as a part of the main competition, they do what they call the “LiveCTF” event. It's a head-to-head hacking race, where two competitors at a time face off in a double-elimination bracket. They broadcast their screens while they hack, with live commentary from announcers. LiveCTF has always been a highlight to watch. 

It's only one of your teammates at a time that gets to participate, but anyone who's not in the middle of something urgent in the main CTF gathers around the TV in the suite to watch our teammate hacking live. I'm sure that's not just my highlight. I'm sure it’s a highlight for many people.

We actually used to do the same thing ourselves for the CTFs we ran as ForAllSecure, so it's great to see that other people are continuing that.

‍

14. Can you share a memorable moment or experience from your time as part of the PPP CTF team?

It's hard to choose. This is less of a specific event, but especially going to the competitions where you compete in person, getting to interact with all of your competitors, and the organizers as well, is really rewarding. I really just like seeing the passion of all these security people and all the different things that they do. 

If you go to an in-person, CTF competitive event, and then have to hack against all these other people, seeing how good everybody is is something that sticks with you. There are other people out there doing what you do who have very different skill sets. Pitting yourself against them, with your related but very different experiences, and then getting to chat with them after the competition—it's a great social event between hackers. I guess that's the thing that really sticks with me. I like talking with people, getting a better idea about what hackers are out there and what they're hacking. 

‍

15. Looking ahead, what do you see as the next big challenge or opportunity in the field of cybersecurity?

I don't think that answer has changed much in the last five years. The biggest challenge is reducing the amount of expert work required for security testing. The promise of a solution like Mayhem is to help automate that away. So that's been the challenge the whole time. And that's where a lot of similar, and even dissimilar, security tools like static analysis tools, die—when they're unable to actually replace security experts. 

Because there's not going to be a security expert at every company who's able to apply the tool. That's out of reach for most companies. So to gain wider-spread adoption, we need to actually reduce the barrier of entry of applying these advanced techniques. 

With Mayhem, we've had great success. But there are always new challenges. Can we make it better? Can we apply it to more things? Can we make it even easier to use? We need to keep doing more, by pursuing things like what my current research project is trying to address. So that's the big challenge. It has been the big challenge. And for the foreseeable future, that will continue to be a challenge.

‍

{{code-cta}}

‍