Here is the second half of the Iron Chef interviews from Black Hat 2008. This interview is with Sean Fay, Fortify’s chief architect. I should point out that we interviewed Charlie and Sean separately, so they answered without being able to hear the other person’s answers first. The fact that they both talk about cycling and hamburger helper just adds to the mystery of the cosmos.

Charlie hates Apple. What do you hate?

I guess if Charlie hates Apple, I have to go with Microsoft. It’s nothing personal really. It’s just that I find it frustrating that no matter what my job title is, I seem to spend a fixed percentage of my time rebooting Windows machines.

You did Iron Chef last year too. Any advice for Charlie?

An hour goes a lot quicker than you might think. If you’ve invented some kind of machine to slow time down, now would be a good time to deploy it.

Can people build their own static analysis tools? Should they? Any advice for a tool builder who’s just getting started?

There are lots of cases out there in the software world where some custom static analysis can be really useful. But building a static analysis tool is a lot of work! As with most things, it’s better to build on top of someone else’s work to achieve your objectives. Both commercial and open source static analysis tools tend to be designed to support customization and extension.

Why aren’t there better open source static analysis tools?

Building a static analysis tool is a lot of work! There actually are some great open source static analysis tools out there - like FindBugs – but they tend to be domain and language specific.

Even a great static analysis tool by itself isn’t that useful. A tool needs a set of rules that describe the behavior of libraries and define the problems that the tool should look for. It’s a lot of stuff to pull together, and right now there’s no good standard for describing checks or characterizing libraries.

What’s wrong with fuzzing?

Fuzzing is a great technique for finding vulnerabilities, but it has two fatal flaws:

The first is that it’s skin-deep. The deeper and more involved the bug, the harder it is to uncover with fuzzing. The person fuzzing the app doesn’t have much of an advantage over an attacker. That means you have to hope that the attacker just isn’t going to try harder than you did.

The second is that it requires someone with very specialized skills and understanding. This makes it a hard technique to scale in an environment where lots of software is being produced.

People say static analysis produces too many false positives. Are they right?

No, it’s just that static analysis tools are complicated to use. Most people forget to turn on the switch to disable false positives before they run their tool.

Seriously though, false positives are a fact of life with static analysis, but a good toolset gives you easy ways to manage them and quickly sort out what is relevant and from what isn’t.

Charlie wrote a book about fuzzing. If static analysis is so cool, why haven’t you written a book about it?

I tried writing a book but I couldn’t get the first chapter to compile.

The static analysis concept has been around for a long time, but it seems like it’s gained popularity lately. Why?

I think that’s a combination of a lot of different factors. The tools are getting friendlier and more powerful. At the same time, a growing awareness about security is making their value more apparent.

When you meet women, how do you explain your job?

Well, before that Adam Sandler movie came out I used to explain that I was secretly a Mossad agent and this was just my cover. That hasn’t been working out so well for me lately.

What’s the biggest mistake people make when they use static analysis?

Lack of preparation. Static analysis is a means to an end. If that end is making your application secure, you need to spend time thinking about the architecture of your application and the assumptions that are being made about trusted and untrusted data.

A security-oriented static analysis tool will tell you a lot about where untrusted data is used within your application. But it’s very difficult to interpret that data unless you’ve already made some decisions about where validation is supposed to happen and where untrusted data is allowed to go.

What do you expect the fuzzing results will be like?

Hard to predict since I don’t know what the application is that we’ll be looking at. These guys are good at what they do, so I’ll bet that at the end they’re going to have something pretty interesting to show us. But with the limited time, we probably won’t see a very broad analysis of the application.

Why do you think you’re going to win?

I’ve been learning about competition from watching “Le Tour”. In preparation for Iron Chef, I’ve been doping with EPO for weeks. My red blood cell count is off the charts.

What's your favorite vulnerability or security incident from the last year?

That has to be Mark Dowd’s flash exploit.

What other disciplines do you take inspiration from?

Besides professional cycling? I guess online poker.

Where did you grow up?

I grew up in a small town called Ipswich in Massachusetts. It’s famous for clams and old houses.

If this were the original Iron Chef, what would you cook?

Well, I’ve been vegetarian for 7 years, but tofu just isn’t a crowd pleaser. I’m pretty sure I can still make a mean hamburger.

I’m cleaning up my machine, and I came across the interviews we did for the Iron Chef competition at Black Hat in August 2008. They’re too much fun for me to keep to myself. Our lead Chefs were Charlie Miller from ISE and Sean Fay from Fortify. Charlie fuzzed his way to victory, but Sean gave him a run for his money with static analysis. Here’s what Charlie had to say before the game. I’ll post Sean’s interview in a few days.

Charlie Miller is a principle analyst at Independent Security Evaluators, but he’s better known as the first guy to hack the iPhone, and he’s made a habit of walking away with a Mac in the annual CanSecWest Pwn2Own contest. We just think he’s an Apple hater. Charlie, why you gotta hate?

I’m not a hater. I just like pointing out how the “cool” company isn’t perfect. That and I like to irritate all the fanboys. Oh wait, I guess I am a hater.

How did you prepare for the competition today?

Actually, I did A LOT of preparation. More than I thought I’d need to do when I signed up. I was told the target program would speak Bonjour, HTTP, and DAAP. So I made sure I had some fuzzers that would work on those protocols.

For some fuzzers this meant getting packet captures or client programs (for mutation) or learning the protocol and writing a simple specification for generation-based fuzzers. Unfortunately for me, there aren’t a lot of DAAP or Bonjour fuzzers sitting around.

It’s easy to get started with fuzzing, but should people build their own fuzzing tools? What are the advantages/disadvantages of doing it yourself? Any advice for people just getting started?

The advantage of doing it yourself is the ability to customize it. However, just like most development, it’s usually best to use something that’s been around for a while and been tested rather than rolling your own version.

I’m using 3 fuzzers against 3 different protocols and didn’t write any of the fuzzers (although I made some small modifications to some of them.) It turns out that debugging a fuzzer is super difficult!

There are a lot of open source fuzzing tools out there. Should people consider commercial tools too? If you pay money, what do you get that doesn’t come for free with open source?

Yes, commercial fuzzers are an option. If you want the most thorough fuzzing, you need to build in a lot of protocol knowledge into the fuzzer. This takes a considerable amount of time and effort.

When you buy a fuzzer, you are buying the fact someone did all the research for the protocol and built the test cases for you. If you are fuzzing a proprietary format, you probably won’t be able to find a commercial fuzzer for it. It all depends on whether you have more time than money. For me, I have more time.

What’s wrong with static analysis?

Static analysis tools are often plagued by false positives. Instead of looking for bugs, users of static analysis tools end up spending all their time trying to find the one bug out of the 10,000 alerts generated by a tool. I also think static analysis fails to find extremely complex bugs.

Fuzzing is negative testing, so you can never be sure you’ve found all the bugs. How should people balance fuzzing with other techniques?

Fuzzing should be used as only one part of a complete and balanced diet of bug finding techniques. It can find bugs and point static/manual analysis to untested portions of code. However, it isn’t the end-all-be-all. For example, I probably wouldn’t waste my time fuzzing a program written in Java.

You’ve got a PhD in math. If fuzzing is so cool, why didn’t you study it in school?

Yea, I totally wasted those 5 years. When I got my PhD, I could barely program, much less understand fuzzing. However, I knew everything about nonlinear partial differential equations! (Sadly, that last statement is a gross exaggeration) Also, keep in mind; I’m an old dude, so fuzzing wasn’t exactly hip in 2000 when I graduated (much less 1995 when I started grad school). The OULU guys hadn’t blown the world away yet.

Barton Miller coined the term “fuzzing” in 1990. Why didn’t it become a popular technique sooner?

That’s a great question! I’d say that it was a combination of a couple of things:

First, it wasn’t exactly hard to find bugs in something like Microsoft RPC 10 years ago. You didn’t NEED fuzzers.

Another is that Miller was an academic so the guys who actually look for bugs weren’t aware of the work. It wasn’t until OULU broke the Internet with their ASN.1 fuzzer that everyone took notice.

After you fuzz a program, do you still respect it in the morning?

It matters how easy it gave it up to me. Apache is a total prude! I currently have a restraining order against QuickTime.

What do you expect the static analysis results will be like?

Lots of alerts about coding style. Lots of “lame” bugs that don’t get you a shell. Lots of “bugtraq”-esque bugs that say “if they configure it this way and run it as root and write their configuration file in UTF-8, there is a bug”. Maybe there will be a good bug thrown in by accident.

Why do you think you’re going to win?

Because my buddy from work picked the target program. No, wait, that’s why Sean is going to win. I’m going to win because I’ve got mad skillz…and I hand picked a couple of the judges.

What’s the biggest mistake people make when they fuzz?

Probably that they don’t check what they are doing. The biggest problem with fuzzing is if you do it wrong, you don’t find any bugs, which means either there weren’t any bugs or there were bugs and you missed them. Often, it’s hard to tell which.

The other common mistake is not fuzzing long enough. For me at least, I’m very impatient and want to turn off the fuzzer after a couple of hours when I should let it run for a week.

What's your favorite vulnerability or security incident from the last year?

I kinda liked the cross-site scripting attack that redirected Obama’s site to Clinton’s, even though in general I think XSS is lame.

What other disciplines do you take inspiration from?

They don’t have anything to do with computer hacking, but I like to race my bike (like Lance Armstrong except way slower) and I like watching hockey. I’m also possibly the oldest man to play dance dance revolution and I kicked ass at it.

Where did you grow up?

I’m a product of St. Louis County public schools.

If this were the original Iron Chef, what would you cook?

I cook a mean Hamburger Helper. Probably whatever the special ingredient was, I’d just throw it in with a box of the helper. Depending on what it was, I might choose Lasagne, or Three Cheese flavor, or maybe Cheeseburger Macaroni. I’d never use Beef Noodle which is horrible!