How Engineering Decisions Get Made

Now that we're a small team, patterns are emerging on how we make technical decisions. Team dynamics has always been pretty fascinating to me, and tons has been written about good engineering practices (e.g. Code Complete) combined with good project management (e.g. Getting Results).

Let's say you're working on a big project, and need a design decision. For example, your team is trying to decide whether to go with Rails or Django, or whether to use a database, flat files, or write your own storage system. Maybe you need to define a communication protocol. There are many ways to decide, but it comes down to three basic patterns:

1. "The Man in the Room" – One of the star developers thinks "I know how to do this," goes off into a room, and reemerges three days later with a prototype. The other developers are shocked and awed. When one discovers a flaw in the decision, the concern is brushed away with "but it's already working!" The design is adopted, partially to not hurt the star developer's feelings. Because the initial decision was never discussed, the developer ended up reinventing wheels, and forgot some of the requirements.
2. "Let's chat" – An hour-long discussion ensues, dominated by the most outspoken developers. Positions are taken early. The group is by the quieter developers, as initial arguments are exchanged. Eventually, the group moves to the whiteboard, and the different options are sketched out. One group eventually wins over, the other feels beaten. A decision is seemingly reached, but is quickly forgotten, since no one remembered to write it down.
3. "Committee rodeo" – An overeager project manager proposes to form a committee that makes a decision. Opinions are solicited. Specifications are written, then rewritten. At the scheduled time, all parties involved meet in a room, with a domain expert flown in from across the country to help out. After a day of deliberation, a decision is reached, and the specification becomes forever written in stone.

Well, those are the worst cases, actually, and you can probably make every single one work for you. I prefer (2.), combined with the discipline of writing things down.

I Like this Quote

"Lucky things happen to entrepreneurs who start fundamentally innovative, morally compelling, and philosophically positive companies, creating something interesting and valuable rather than strictly seeking money." - Bo Peabody, Lucky or Smart

I haven't read the book, but I find this one quote nicely summarizes what we'd like to build: A useful product, a positive philosophy, and a great place to work.

A New Setup

A little more than a week ago, we moved into an apartment in Foster City/San Mateo. The big plus here is that we don’t have to rush through an hour of traffic to get to meetings in the Valley. Suburbia provides fewer distractions here than you would have in San Francisco, but the hipness level is low, and a good bar is hard to find.



Fewer distractions have allowed us to make progress on the prototype. It turns out that this is more productive than working in a coffee shop. My commute now takes 10 seconds, and my workspace has a nice Dell flatscreen monitor and my favorite ergonomic keyboard. In addition, we have blocked YCombinator’s Hacker News and similar sites on our router, which resulted in an additional productivity boost.



Things are progressing well on other fronts, too - I’ll keep you updated. For now, it’s back to work.

Giants vs. Sprinters

This is a crosspost of an article I wrote for Swiss Entrepreneurship blog Synetgies.

Most revolutionary new technology products and Internet services come from a handful of large companies and small startups. What's the secret sauce?

Successful and profitable large companies such as Apple and Google invent and produce such products as the iPod, the iPhone, Google Maps, and Gmail. In contrast, startups have developed products and services such as Google Search (back when Google was a startup), Hotmail, PayPal, YouTube, Blogger, Facebook, and Twitter.

Users and the press rave about these products, and they have generated large valuations and profits. How does this kind of product innovation happen?

In this article, I'll contrast product development at large and small companies. I've experienced product development at Google (where I worked on Gmail and some unmentionable projects), Yahoo (where I interned at the end of the last bubble). I'm currently working on my startup, Xobni, where my role involves development as well as setting engineering and product priorities. We're a small team of 10 people and are building new ways to search and navigate your email. Thus, I've seen both ends of the spectrum.

Between the two extremes of small and large companies, there are a few common denominators:

• Both types of companies start with good people who are smart, well-educated, and passionate.


• They provide good tools: High-end workstations, great infrastructure, and good benefits. For example, Google will pay employees for health insurance, serve free food and drinks, coupons towards buying hybrids, gym memberships, and the like.


• They set a culture that is centered on engineers. Engineering and is the scarce resource, as it hard to find excellent engineers who can create great products. Openly or covertly, PR, Marketing, Sales, and HR are seen as second-rate citizens. This is most clear at Google, where engineering is kept on the main campus, but HR and PR are located in "Siberia": office buildings so far away that employees have to use bikes and scooters to commute to meetings.

Most people think of "innovation" as "ideas". But there's no lack of good ideas. At Xobni, we have an internal Wiki page with hundreds of product ideas. At Google, there's the ideas mailing list on which you can find thousands of employee-submitted proposals for new features and new products. I'm sure that Microsoft has an equivalent tool. But anyone who has added to that Wiki, or written to the ideas list knows that they are the place that ideas go to die.

What really counts is execution: At large companies, the ideas that survive have a strong proponent who will get support for the idea, find colleagues to work on it with, defend it in meetings, and launch it to a public. This is what happened at Gmail: Paul Buchheit started working on a webmail client, found others to work with, defended it against VPs who said that an ad-supported model would never work, and managers who said that it is prone to extinction because of Microsoft's control of JavaScript. At startups you'll find the same process (but less meetings): Xobni's most popular feature is search, but it was two of us who took it from a feature added as an afterthought one of the core pieces of our functionality.

Yet, there are many differences. Technology giants and startups both have their own set of advantages that play in their favor when executing against ideas:

Large technology companies

• Resources: As the name says, large companies have tons of people. Once management is convinced of the viability of a project, they can put people, infrastructure, and money to work to make the idea become reality. Giants move slowly, but once they do, the earth starts shaking.


• Experienced management: In Silicon Valley, senior managers at large companies typically have startup experience. They started or joined small companies that got bought or went public. They know how to manage innovation and push interesting projects forward.


• Instant credibility: When Apple, Google, Microsoft, or Yahoo launches a new product, the world listens. If a startup had released Google's phone SDK, there would not have been weeks of Google phone speculation in the press, weeks before launch. Consumers will feel safe buying new Apple products when they're launched, because they would know where to buy and what level of quality to expect. Startups have to build a really good product and build it from the start, build press relationships, and resort to guerilla marketing as needed.

Startups

• Focus, clear priorities: You'll never see a company as focused on progress as a startup. At Xobni, the number one priority is to get high-quality software out the door. There's only this one project: There are no distractions, no talks to attend, no other projects for engineers to switch to. We're all sprinting towards a clear goal.


• Aligned incentives: At startups, employees have significant amount of stock in the company, and their financial future is highly correlated with the success of the company. Thus, there is only one controlling variable: Their contribution to the product. If they can add or improve features, they will. On the other hand, large corporations attract resume stampers who are sometimes guided by self-interest: Their priority is to rise in the ranks, not contribute to overall success.


• No lockstep development: Startups have small numbers of people working on small products. Large companies work on large products with lots of people. These people require coordination and planning. For example, I've heard that the feature sets of Microsoft's Office suite are planned out two releases in advance, with two years between each release. This means that a product manager on Word knows what features the product will have in 2011. If you're an engineer at Microsoft and have an idea, it may not get executed upon until four years from now! In addition, there's the burden of reverse compatibility: Every new feature must be compatible with versions of the software that are decades old.

In summary, we explored differences in how startups and large companies run innovation and product development. There are some commonalities - great people, focus on engineering, and good tools - but startups have large advantages because they are more focused and have no existing customers, products, and profit lines to look after.

--

If you liked this article, you will also like Career Advice for High Achievers.

What We've Been Up To

I haven’t blogged about anything very substantial recently. Since we launched Xobni Insight in September, we’ve been very busy improving it. The responses have been phenomenal, but Xobni isn't perfect yet.



Here's what we've been working on:

1. Improving performance: Inside Xobni Insight, there is a powerful data engine we call XobniCommon. In some ways, it’s Xobni’s secret sauce. The data you see in the sidebar comes from this component. When we launched a month ago, we hadn't fine-tuned all parts of our racecar. For example, we use a number of small in-memory indexes that allow us to quickly aggregate and filter information. But we weren’t using them in all places. For example, when you clicked on a person, it would take us seconds to load that profile. We’ve since reduced that time by a factor 100.
2. Improving memory footprint: Our first version used about 2x as much memory as we are taking up now. Adam and Greg made tons of optimizations here, mostly around choosing the right memory structures that don’t waste space. While our memory footprint never was an issue for people with a normal amount of emails, things got uncomfortable if you had hundreds of thousands of them. They’s much better now.
3. Unconventional configurations: There are about the same number of Windows configurations as there are Windows machines. While we had done considerable configuration testing on different computers before launch, we weren’t expecting to see so many outliers. To give you a feel for what Xobni must deal with, here are some of the variables: We support Windows 2000, XP, Vista, in both 32 and 64 bit, on Outlook 2003 and 2007. We’ve seen conflicting plugins, messed-up Outlook registry keys, graphics card drivers that wouldn’t let us draw gradients, code access security settings that wouldn’t let us run, and we’ve seen Outlook disable us without reason. We’ve investigated these issues and have been solving them.

This list also demonstrates the difference between desktop and server apps: On the server side, you have a known configuration, and can simply add more and faster machines with more memory. On the desktop, however, you’re dealing with all the restrictions of the average consumer box. But you're not paying for that expensive colo.

Where do we go here? For us developers, it's much more exciting to work on new stuff than to massage and fix old code. Thus, I’m very happy that every fixed bug brings us closer to working on new, exciting features.

Oh, and there's another thing we've been working on: Expanding the Xobni team. If you're an exceptional developer, visit our jobs page. Xobni wants you!

More on this on the Xobni blog:
Xobni - Gearing up for Success by Matt
Deep in the Trenches: Support at Xobni by Skyler

A Healthy Disregard for the Impossible

Two weeks ago, I quit my job at Google. Later this year, I will join Xobni ("inbox backwards") in San Francisco, California!

Why leave Google? It's a fantastic company: They have the smartest people, enlightened management, great projects and pamper employees to no end. But most importantly, they're one of the few companies that stick to their values: When they say "don’t be evil", they actually mean it. I can only recommend working there.

But what I really wanted to do after school is to do a startup.

When I visited Adam and Matt in August last year, I was impressed: They were also interested in email, they seemed wicked smart, and had all the right connections. But above all, they had a healthy disregard for the impossible. These guys are willing to do what it takes to succeed.

Back then, it was hard to lure me away from my Google offer, and they didn't succeed. Since then, the company, software, and goal have evolved and they've recently received significant VC funding (as the media found out today).

Matt and Adam don't take no for an answer. After another offer earlier this year, it took me quite a while to make up my mind and actually quit. Back at Google Zurich, I was working on an awesome project with great people. The office was growing fast. Google had just won another award for being the best place to work, ever. It didn’t seem like a smart, mainstream move at the time.

What pulled the trigger was reading Jessica Livingston's Founders at Work on a plane ride back from the States. If all of these guys had done it, so could I. Around the same time, I got an email from Paul Buchheit who wrote: "The great thing about a good startup is that even if it doesn't work out, you still end up learning a lot more and meeting more interesting people than you otherwise would." That's true: The learning curve will be much steeper at Xobni than at Google; my impact will be much larger. That's the kind of environment I enjoy.

As one of Xobni's earliest employees, I'll be heading up their engineering effort. We're looking for a few superstars. If you're one of them, let me know.

The World of Work in 2012

While this post mentions a number of Google products, the opinions expressed here are mine, not those of my employer! I do not work on any of the products mentioned in this article.

I believe that how we work and collaborate will fundamentally change over the next 5 years. This is no far-fetched vision, but pretty obvious stuff, and I'm even not the first to connect the dots and point it out. In the space of five years, most technically inclined consumers will see:

• their lightweight communication and collaboration move almost entirely to the mobile phone,


• most document editing and heavyweight collaboration move to the browser,


• with just a few heavyweight applications remaining on the desktop.

Last weekend, I went to visit Lucerne, Switzerland, with friends. Among them was Fabian, who is a strong believer in using the cell phone solely for calling and sending text messages. He still owns a trusty old Nokia 6210 . We had no city map, and I had only a vague idea of the city's layout, so I whipped up Google Maps Mobile on my 6280 and searched for Lucerne. Fabian was hooked. (The only thing he's worried about now is data fees.)

Back in my student days, there used to be these group assignments: Each group had to come up with some form of document and hand it in. This is where the world split into nerds and normal people. The nerds used Latex files or HTML pages checked them into CVS. The normal people used Word documents that were mailed around and eventually unified into one by the last person. Today, the obvious choice is to put this into Google Documents.

These two examples illustrate how we'll see user behaviors change on cell phones, in browsers, and on the desktop.

The World of Cell Phones

There are many possible killer apps on cell phones:

• maps, directions, traffic information, timetables,
  
• shopping comparisons,
  
• fact lookups on Wikipedia,
  
• email

Some of these are already available, but are either unusable or usable only on a few devices, such as BlackBerries (CrackBerries?). This will change.

Have you seen the iPhone keynote? When I heard the announcement, I was a bit skeptical at first, since my current cell phone does almost everything theiPhone can do: It doesn't have WiFi access, but matches Apple's product on all other counts. But the iPhone's goal is not to introduce new functionality: Its goal is to make existing functionality usable. If you make something 10x more usable, 1000x more people will use it. But Nokia, Motorola, and Sony Ericsson aren't stupid - in a year or so, expect their phones to get significantly better, too.

In the next few years, the average cell phone user's device will have a great email client, a decent browser that displays the same version Internet as a desktop browser, and access to her online documents with lightweight editing. Some of this software won't come from the phone vendor.

In 2012, the most important part of your cell phone plan will be the price per transferred megabyte, not call minutes. You'll leave your house without a timetable printout, a clear idea of what gift you want to buy for your girlfriend, or even where the shop you want to visit is. You'll read all your email on your cell phone first, and only use a computer when the response needs to be more than a couple of lines.

But sometimes, you'll still need to sit down in front of a screen.

The World of Browsers

Let's say you're putting together your company's budget for 2013, or maybe you're writing a memo about how your car fleet should move entirely to hybrids. In 2012, you'll be working on this inside the browser.

Creating, editing, and revising documents are the prototypical activities of the office worker. Today, these documents are created in Word or Excel, after which they are emailed around to get feedback and iterate on the original draft.

The reason why Word and Excel are used is because of network effects: If you have the same software as everyone else, you can be sure the document will look the same as on the sender's machine. Most office users have also gotten used to the clumsiness of emailing around documents as just another aspect of drowning in email.

Even today, web office suites such as Google Documents or Microsoft's Office Live get rid of the need to sink thousands of dollars into desktop software. Documents look the same everywhere, and instead of emailing around revisions, all users can edit the same document at the same time, for free. In addition, you have access to these documents from every web browser anywhere in the world, instead of carrying a single copy of them on your laptop.

By 2012, these web applications will have evolved to a point where the default way of dealing with documents is by loading up the web app. But there are some applications that will remain on the desktop for a long time to come.

The World of the Desktop

Aaaah the power of the desktop. If you run applications right on the machine, you lose mobility, but you gain processing power, storage capacity, and the ability to build richUIs . That's why some applications will stay here: They require processing a lot data in real time, or need specialized user interfaces that cannot be replicated in the browser or the mobile phone. Examples are:

• Programming
  
• Desktop publishing
  
• Audio/Video editing
  
• Computer games

Let's take video editing, for example: If you're a video artist in 2012, you'll still be working with Gigabytes of data, and many work steps will still require huge amounts of processing power. It's unlikely that there will ever be enough bandwidth to handle all this data remotely. The same applies to anything from desktop publishing to computer games. Even five years down the road, you'll see these things happen on the desktop.

How to Get There

Plenty of things will have to happen before this becomes reality.

There are technical hurdles: For example, anyone who has worked with Java on Mobile Phones will happily attest that the UIs you can implement with the standard libraries aren't that compelling. On the browser side, there is also room for improvement. For example, making rich editing work right on both IE andFirefox is a nightmare. And then there's the offline problem, which I've written about before: No amount of technical innovation or investment will ensure 100% coverage of the planet. We need to build cell phone and web apps so they can deal with being offline.

Then there are the economic problems: For all this to work out, the price for data communication needs to drop very significantly. In countries where there's healthy competition between network operators, this will happen - all others will lag behind.

In addition, will users need to trust Google, Yahoo, Microsoft, et al enough to store their documents' data in their datacenters? Corporations in particular are very sensitive about security and privacy issues.

But the greatest hurdle is that non-technically inclined masses tend to stick with what they know, even if something better comes along. (People clinging to the old ways are hardly ever convinced later; they just die out.) If email in Outlook works, why switch? According to Bob Cringley, the masses only switch quickly if something is 10x as good - so these mobile and web apps will have to really kick ass!

So if my prediction doesn't become true, I have plenty of parties to blame. Still, I hope that this works out.

--

Thanks to Douwe Osinga, Fabian Siegel, Bálint Miklós, Julia Ferraioli, and Keno Albrecht for their ideas and comments on initial drafts of this.

A Month of Google Zurich

I re-joined Google a little more than a month ago. My friends have been asking me how it is. Obviously, I can't say about what I actually work on, can't talk about the Endoxon deal and also won't be able to disclose the Master Plan. But I can say a couple of things.

Above all, I love the team I work with. There's my friend Douwe, best known for inventing Google Trends. He has about two multi-billion-dollar ideas per week. We should hire an intern just to follow him around and write up all the stuff he comes up with. My manager, Oliver, won the best-PhD-thesis-award in Germany before he joined. Jonas, my super-friendly co-worker, has churned out a cool demo of what we're building. Then there's our charming Austrian product manager Chris, whom we should really get to work less. Our intern Alex should also get an award for his achievements: He produces tons of code, but is hauntingly quiet. When he does point out something, it's usually some tiny detail we forgot about but would have cost us days of debugging later on. Last week, he returned to his native Eastern Europe to get his degree, but our recruiters 'convinced' him to return afterwards.

When US companies put engineering offices in Europe, it's usually the dull work that they're concerned with: Localize this, translate that. Google's EMEA Engineering HQ in Zurich is different. I'm happy to report that we work on really crazy, brave, and fun things. You don't need to be in Silicon Valley to do that.

The most popular pastime at the office is foosball: I really need to get some mad skillz.