Life inside Facebook: how head of developers organises 500 people

Mike Schroepfer in October 2007. Photo by adactio on Flickr. Some rights reserved

Ever wondered what it’s like to be an engineer inside Facebook? We got an exclusive interview with Mike Schroepfer, vice president of engineering at the company, who has previously been named one of the 50 most important people on the web, to ask him how he gets things done in an organisation that’s not only growing fast – but also setting the pace when it comes to a lot of web implementations.

Schroepfer is in charge of a team of 500-odd developers who do everything on the site – and, as he explains, if a new recruit hasn’t changed something on the site within their first week, they begin to look at them askance, and wonder if something’s wrong.

But that doesn’t mean it’s a monolithic organisation: instead, teams tend to work in groups of three to five on small, short projects. As for management structure – well, there’s the developers, and there’s Mark Zuckerberg. And that’s about it…

Where were you before?
I joined Facebook in the summer of 2008 – before that I was at Mozilla.

What was the attraction?
It was a combination of amazing influence – I had done interesting things but there were a lot of unanswered questions about where the company [Facebook] could go.

You’re in charge of a team at a company that famously does things fast. How far out do you plan what projects you’re going to assign people to?
In general our projects are very iterative – often they last one or two months. Facebook Messages was bigger and longer [Mark Zuckerberg said that a team of more than 15 people worked on it more than a year] so we had to make a longer-term commitment. Others are about working on a piece of technology that has a huge effect on [site] performance. So for example there’s Hiphop [which compiles the interpreted scripting language PHP into runtime C++] which took two years to finish.

So we’re looking at making the web servers more efficient, which is something we have to invest in.

How many projects do you have going at once?
It’s hard to tell, because we have them running all the time, but they might be just a singe person or two. The answer I guess would be somewhere between several dozen to 100 at once.

How do you know if they’re running to plan? The big problem as organisations like Google or Microsoft get larger is keeping what they’re doing synchronised.
Well, intuition is what gives us the ideas for what to do, and data tells us if we’re getting it right. We iterate to find out if a project’s doing it right. Or you might make something live and then you look at whether people are using it frequently, or whether they use it once and don’t come back.

If they don’t come back then we probably didn’t get it right. It’s a constant process of iteration. The longer it gets before you get in data from the outcome, the worse it’s going to be if it’s not right.

Where’s the centre of gravity in the projects among the developers? Are people writing for mobile, or for HTML5, or what?

Mobile is certainly important, and increasingly important. From the products we have launched there’s increasing emphasis on mobile. So when we had Facebook Messages it launched on mobile and on the web at the same time. We have 200m people using Facebook on mobile; in some countries they only access it via mobile.

We’re also making huge enhancements to the systems through investments such as Hiphop and server software, because that makes the cost of operating lower, or leads to speed improvements.

The third thing that’s important is capability in machine learning, because it allows better creative experiences.

How do you decide on the getting the balance between what’s commercially important – what will bring in cash from ads – and what’s just “cool”?
We’re trying to run the company in a growth and investment phase. We can invest in R+D but there do some opportunities where you can show more ads, so how do we balance it. A lot of projects that we are working on focus more on getting more users to the site and have an engaging experience so that they come back and recommend it to their friends.

It’s easier to build a business model on a big site than a small one. So our efforts are aimed at getting more people to come to the site for longer. We’re trying to make sure that we generate enough revenue that we’re never blocked from doing something by commercial considerations.

But for example we’re the largest photo sharing site on the web. So that has some benefits.

How do you balance the cost of doing something against the time it will take and the benefits it might bring? How do you decide the balance?
It’s about 80-20 – the 80% on user-focussed growth, 20% on cost-saving, revenue-growing projects. So for example we’ve now added high-resolution photos to the site, which adds to the cost of the infrastructure, because we have to store those photos.

But we have people who are very good at projecting out into the future how many photos people will upload, where the cost of disk space is going, how much we can afford to allow, and so on. Sure, the cost of disk space is halving every year, but if we use up twice as much per photo… and so on.

You said earlier that you’re focussing on machine learning. What’s that for?
Deciding what stories to show you when you log on and look at your wall, for example. There’s only a limited number of stories we can display there, so we need to know based on what you like and what there is what to show you. Or to figure out what ads to show you. And for security, to detect whether somebody is sending out a suspicious number of friend requests, or spamming people, or sending spam links.

You bought a company called Dropio earlier this month. What did you buy them for?
It was a file-sharing site, but we bought it for the technical talent and the team. They’re going to work on different projects. They’re led by an exceptional entrepreneur.

In practice, most of our teams are about three to five people. So that if you build a service then you can leverage our huge user base. Early startups have the problem of getting the product out into the market. We don’t have that problem. So by joining us, they trade up to our huge leverage.

As companies get bigger, they face the problem of decisions having to flow up and down management, and inevitably things ossify – it’s been like that for Microsoft, and there are signs of it at Google. Is there a way to avoid that at Facebook?
(laughs) Yes, we don’t have the layers of management approval! We don’t pass things up and down the chain. The team working on the product development makes the decisions. If there’s a problem or if they think it merits it then they will talk to Mark [Zuckerberg] directly. We try to do a good job of setting out the context of the task and release people to get on and do it. People are pushing new features and code to the site every day. It’s really about trying to remove barriers and reduce friction in development.

Doesn’t that though leave a huge potential for a really catastrophic mess?
Yes, and that has happened. But with financial information or sensitive data, we have automated tests before we release code that will change what’s done there, and we make changes on those things only very carefully.

But most of the things we do aren’t in that category. The precise size of profile pictures, the colour of a link or element, or how photo uploads work: that’s not going to cause a disaster. The worst that will happen is that you lose a photo or you need to take something off. That’s bad, but it’s not a company-ending catastrophe.

And we aren’t changing privacy controls very often, and we take great care with that. Great care.

How do developers get oriented when they join?
We have an introduction to the team culture, where we’ve set up a boot camp, which lasts the first six weeks. No matter what role someone comes in as, they go on it, so they understand the code stack, and we encourage people to try their hand, to push changes up to the site; so that if you haven’t made a change to the live site in first week, well, we’d think actually that’s wrong.

What tools do you develop in?
There’s a wide proliferation – we use C++, Java and PHP. The back end is built in C++, the messaging in Java. We build in and leverage open source where we can, and when we build code then we try to release it as open source where we can.

What do the team develop on?
Everyone gets access to a development Linux server. For themselves, they get a choice of Windows or Mac laptops. Most of the developers are using Macs – I think it’s because they have a good selection of command-line tools.

A lot of people have suggested that you’re biting off more than you can chew with Facebook Messages – the developer of MIME suggested in a blog post that it’s a world of hurt. Do you think you can do it?
We’re trying to get enough of it done so that people understand it. Messaging is a challenging problem in general. I think that more than integrating email, what’s interesting is focussing on the people side, so it’s not one app for IM, one app for email, one app or phone for SMS. Rather than doing things by subject, or thread, it’s done by person.

But email’s going to be a bigger problem?
We’re already supporting POP email. IMAP is just a matter of time. The difficult thing is actually the user experience.