The State of the Internet Operating System

I've been talking for years about "the internet operating system", but I realized I've never written an extended post to define what I think it is, where it is going, and the choices we face. This is that missing post. Here you will see the underlying beliefs about the future that are guiding my publishing program as well as the rationale behind conferences I organize like the Web 2.0 Summit and Web 2.0 Expo, the Where 2.0 Conference, and even the Gov 2.0 Summit and Gov 2.0 Expo.

Ask yourself for a moment, what is the operating system of a Google or Bing search? What is the operating system of a mobile phone call? What is the operating system of maps and directions on your phone? What is the operating system of a tweet?

On a standalone computer, operating systems like Windows, Mac OS X, and Linux manage the machine's resources, making it possible for applications to focus on the job they do for the user. But many of the activities that are most important to us today take place in a mysterious space between individual machines. Most people take for granted that these things just work, and complain when the daily miracle of instantaneous communications and access to information breaks down for even a moment.

But peel back the covers and remember that there is an enormous, worldwide technical infrastructure that is enabling the always-on future that we rush thoughtlessly towards.

When you type a search query into Google, the resources on your local computer - the keyboard where you type your query, the screen that displays the results, the networking hardware and software that connects your computer to the network, the browser that formats and forwards your request to Google's servers - play only a small role. What's more, they don't really matter much to the operation of the search - you can type your search terms into a browser on a Windows, Mac, or Linux machine, or into a smartphone running Symbian, or PalmOS, the Mac OS, Android, Windows Mobile, or some other phone operating system.

The resources that are critical to this operation are mostly somewhere else: in Google's massive server farms, where proprietary Google software farms out your request (one of millions of simultaneous requests) to some subset of Google's servers, where proprietary Google software processes a massive index to return your results in milliseconds.

Then there's the IP routing software on each system between you and Google's data center (you didn't think you were directly connected to Google did you?), the majority of it running on Cisco equipment; the mostly open source Domain Name System, a network of lookup servers that not only allowed your computer to connect to google.com in the first place (rather than typing an IP address like 74.125.19.106), but also steps in to help your computer access whatever system out there across the net holds the web pages you are ultimately looking for; the protocols of the web itself, which allow browsers on client computers running any local operating system (perhaps we'd better call it a bag of device drivers) to connect to servers running any other operating system.

You might argue that Google search is just an application that happens to run on a massive computing cluster, and that at bottom, Linux is still the operating system of that cluster. And that the internet and web stacks are simply a software layer implemented by both your local computer and remote applications like Google.

But wait. It gets more interesting. Now consider doing that Google search on your phone, using Google's voice search capability. You speak into your phone, and Google's speech recognition service translates the sound of your voice into text, and passes that text on to the search engine - or, on an Android phone, to any other application that chooses to listen. Someone familiar with speech recognition on the PC might think that the translation is happening on the phone, but no, once again, it's happening on Google's servers. But wait. There's more. Google improves the accuracy of its speech recognition by comparing what the speech algorithms think you said with what its search system (think "Google suggest") expects you were most likely to say. Then, because your phone knows where you are, Google filters the results to find those most relevant to your location.

Your phone knows where you are. How does it do that? "It's got a GPS receiver," is the facile answer. But if it has a GPS receiver, that means your phone is getting its position information by reaching out to a network of satellites originally put up by the US military. It may also be getting additional information from your mobile carrier that speeds up the GPS location detection. It may instead be using "cell tower triangulation" to measure your distance from the nearest cellular network towers, or even doing a lookup from a database that maps wifi hotspots to GPS coordinates. (These databases have been created by driving every street and noting the location and strength of every Wi-Fi signal.) The iPhone relies on the Skyhook Wireless service to perform these lookups; Google has its own equivalent, doubtless created at the same time as it created the imagery for Google Streetview.

But whichever technique is being used, the application is relying on network-available facilities, not just features of your phone itself. And increasingly, it's hard to claim that all of these intertwined features are simply an application, even when they are provided by a single company, like Google.

Keep following the plot. What mobile app (other than casual games) exists solely on the phone? Virtually every application is a network application, relying on remote services to perform its function.

Where is the "operating system" in all this? Clearly, it is still evolving. Applications use a hodgepodge of services from multiple different providers to get the information they need.

But how different is this from PC application development in the early 1980s, when every application provider wrote their own device drivers to support the hodgepodge of disks, ports, keyboards, and screens that comprised the still emerging personal computer ecosystem? Along came Microsoft with an offer that was difficult to refuse: We'll manage the drivers; all application developers have to do is write software that uses the Win32 APIs, and all of the complexity will be abstracted away.

It was. Few developers write device drivers any more. That is left to device manufacturers, with all the messiness hidden by "operating system vendors" who manage the updates and often provide generic APIs for entire classes of device. Those vendors who took on the pain of managing complexity ended up with a powerful lock-in. They created the context in which applications have worked ever since.

This is the crux of my argument about the internet operating system. We are once again approaching the point at which the Faustian bargain will be made: simply use our facilities, and the complexity will go away. And much as happened during the 1980s, there is more than one company making that promise. We're entering a modern version of "the Great Game", the rivalry to control the narrow passes to the promised future of computing. (John Battelle calls them "points of control".) This rivalry is seen most acutely in mobile applications that rely on internet services as back-ends. As Nick Bilton of the New York Times described it in a recent article comparing the Google Nexus One and the iPhone:

Chad Dickerson, chief technology officer of Etsy, received a pre-launch Nexus One from Google three weeks ago. He says Google's phone feels connected to certain services on the Web in a way the iPhone doesn't. "Compared to the iPhone, the Google phone feels like it's part of the Internet to me," he said. "If you live in a Google world, you have that world in your pocket in a way that's cleaner and more connected than the iPhone."

The same thing applies to the iPhone. If you're a MobileMe, iPhoto, iTunes or Safari user, the iPhone connects effortlessly to your pictures, contacts, bookmarks and music. But if you use other services, you sometimes need to find software workarounds to get access to your content.

In comparison, with the Nexus One, if you use GMail, Google Calendar or Picasa, Google's online photo storage software, the phone connects effortlessly to these services and automatically syncs with a single log-in on the phone.

The phones work perfectly with their respective software, but both of them don't make an effort to play nice with other services.

Never mind the technical details of whether the Internet really has an operating system or not. It's clear that in mobile, we're being presented with a choice of platforms that goes far beyond the operating system on the handheld device itself.

With that preamble, let's take a look at the state of the Internet Operating System - or rather, competing Internet Operating Systems - as they exist today.

(continue reading)

Where 2.0: Ignite & NAVTEQ's LBS Challenge Tuesday Night

The Where 2.0 Conference starts tomorrow with workshops all day. In the evening we will have our kick-off event in the Ballroom at the San Jose Marriott. As always we start off with a series of Ignite talks. This year in the middle of Ignite we are going to host the NAVTEQ LBS Challenge Awards.

The Ignite talks look great this year and touch on major issues in the mobile, mapping and local industries. Patrick Meier will be talking about Ushahidi's work in Haiti. Paul Ramsey will inform us why our data sucks. Jonathan Stark will address the problems with App Stores. It should be a fun night. The complete list of talks is after the jump

Ignite Where kicks off at 7:00 p.m. and NAVTEQ’s seven year old Global LBS Challenge developer competition takes the main stage live at 8:00 p.m. All twelve finalists will make a one minute pitch of their location-based applications, giving the audience a taste of what it takes to make it to the Challenge finals. They are competing for prizes valued at $10 million. The audience will provide input via SMS.

Where 2.0 is in San Jose from 3/30-4/1. There are still seats available, but they are filling up.

(continue reading)

The iPad needs its HyperCard

Easy to use content creation tools are key to the iPad's long-term success.

(Note: With the iPad scheduled to arrive this week, we reached out to a number of folks across a variety of industries to get their take on the device and the changes it may usher in. We'll be featuring these pieces over the next few weeks. -- Mac)

Dale Dougherty, editor and publisher of MAKE:

When I think about opportunities around the iPad, I recall the CD-ROM market of the late 1980s. CD-ROM followed packaged software but created a number of innovative "content" packages, creating new categories such as "edutainment" with products like Reader Rabbit.

My favorite CD-ROM product, which I thought held such promise as a landmark approach to multimedia, was Beethoven's 9th Symphony by Robert Winter, a UCLA music professor. While listening to the symphony, you saw notes appear that explained the music and some of its features. This was liner notes on steroids. This CD-ROM offered something you couldn't get on a TV or a stereo.

The Beethoven 9th application was written in Hypercard and it was produced by Bob Stein at The Voyager Company. Stein also combined documentary or feature films with the critical commentary in products like "A Hard Day's Night" and children's books such as "The Amanda Stories."

The CD-ROM market also consisted of "productivity" titles from Broderbund such as Family Tree Maker and Print Shop. It also gave birth to a game market that culminated in Myst (also based on HyperCard.)

The market for CD-ROMs collapsed because the distribution channel for boxed software went away, and the web became the primary means for users to find entertainment, games and productivity apps. It was also true that the web lowered the bar for creating applications, even though it was much less capable of delivering rich content. (Nothing like the Beethoven's Symphony app has been created on a website that I know of.)

(continue reading)

Four short links: 29 March 2010

Distributed Comments, Graph Exploration, Body as UI, and Genomic Advertising

1. Salmon Protocol -- protocol to unite comments and annotations with original web pages. A distributed solution to the problem that Disqus tackles in a centralised fashion. Important because we'll all be historians of our earlier lives and dissipated prolific micro-content is a historian's nightmare.
2. Gephi -- open source (GPLv3) interactive visualization and exploration platform for all kinds of networks and complex systems, dynamic and hierarchical graphs. I believe tools for data exploration, versus static infographics, are the only way to develop a new sense for data. (via mattb on Delicious)
3. Skinput -- a bio-acoustic sensor lets you use your skin to write, tap, drag, etc. See also BBC article. (via Mike Loukides)
4. First Synthetic Genome Secret Messages Decoded (Wired) -- the first synthetic genome contained advertisements ("VENTNERINSTITVTE", "CRAIGVENTNER"). I can't figure out whether it's a cheeky easter egg in the finest geek tradition, or whether it's as if the Apollo 11 had "BUY COKE" on the side or Magellan's yachts had sails emblazoned with "VENETIAN GLASS: BEST IN THE WORLD!". (via christianbok on Twitter)

News from Appland

A look at the early momentum of iPad applications

Carsonified -- a site that proclaims on its home page, "We're hugely passionate about the web" -- declared the death of the web this week in, "Bye Bye Web, Hello Apps." The post makes the case for mobile apps, especially iPhone apps, and their advantages over web apps. I certainly won't be calling web apps dead for a while, but I'm impressed by how much activity has been unleashed by the app world. Here are some stories from Appland this week.

• My friends at 37signals released their first official iPhone app this week, Highrise. It's a great little app, highlighting the best parts of mobile applications, like on-the-go voice recording, and the worst, like a lengthy install period (it took 10 minutes for me) -- which they mitigated with a built-in "shall we play a game?" to keep you entertained during the download. This is one of several cases I know of where a company with a set of third party iPhone apps has decided to go official, which to me is a sign of the increasing importance of these apps. In the Highrise case, the existing apps weren't very good at all (I'd tried them all), and the lack of a great iPhone app made the web app less useful.
• How seriously are app developers taking the iPad? Check out this preview of the iPad App Store on MacStories. I'm psyched to see OmniGraffle, for instance, on there, and a little shocked to see it priced at $49.99. It's pretty obvious that app developers are treating this as a full-fledged application platform, not just a big phone. When I first saw a preview of the iPad App Store, my thought was, "I wonder how long it will be before Microsoft Office is available here?"
• Marco Arment's preview of Instapaper on iPad was great fun to read:

  [...] then I saw the pixel-doubled version of my app in the simulator. It sucked, and it was completely unusable by my standards. I don’t think I’ll want to run any pixel-doubled apps on my iPad in practice. [...] While I could have taken the conservative option and waited until a month or two after the iPad’s release before launching Instapaper for it, an iPad without native Instapaper Pro is not a device I want to own.

  I agree with Marco about pixel-doubling; it looks so bad I don't think we should count the iPad as having 150,000 apps on day one. Widgets? Demos? Sure. But I bet a good count of how many apps are still "alive" in the App Store will be how many of their developers bother to make a universal iPhone/iPad app. Anyone who doesn't, doesn't care about the app at all.
• Speaking of bad, links to the iTunes web site -- which launches the iTunes application when you have the misfortune of landing on it -- are my nominee for the new Web Links of Great Regret (unlabeled PDF links being the past champion). Oh no, I'm on iTunes! Halp! Such a disaster. Here's some help: How-To: Stop iTunes Web Links From Opening iTunes.

Some Highlights From This Year's Adas

Ada Lovelace Day turned up hundreds of interesting people. Here's a couple of interesting pointers:

1. Mileva Maric is an uncomfortable read, but worth it for the perspective it gives to heroes.
2. Over on Best of Three you get a reading list for Caroline Herschel, Marie Curie, and Lise Meitner.
3. Margaret Moth is lauded on Elpie's blog.
4. Adafruit Industries posted about one great woman every hour for the whole day. The post I link to has pointers to all 24 articles.

Base Map 2.0: What Does the Head of the US Census Say to Open Street Map?

Ian White, the CEO of Urban Mapping, makes his living collecting and selling geo data. For next week's Where 2.0 has put together a panel of government mapping agencies (the UK's Ordnance Survey and the US's Census Department) and community-built mapping projects (Open Street Map and Waze). Crowdsourced projects like Waze and Open Street Map have forced civic agencies to reconsider their licensing. They have similarly encouraged larger companies like Google, NAVTEQ and Tele Atlas to implement their own crowdsourcing platforms (like Google Mapmaker and Tele Atlas' MapShare). Ian and his panelists will discuss all of this in Base Map 2.0 on Thursday at Where 2.0 - you can consider their conversation as Part 1 of the panel.

Ian gathered the panelists (listed below) for a pre-conference phone call. Here's Ian's descriptions of the panelists:

• Peter ter Haar - the Head of Products for the Ordnance Survey which is the UK's national mapping agency.
• Tim Trainor - the Head of Geography at the Bureau of Census within the Department of Commerce in the US Government. His group is responsible for TIGER which is a lot of the line geometry as the US as well as the master addressing file, a lot of improvements there in the 2010 Census.
• Steve Coast - the Founder of OpenStreetMap and also Cofounder of the commercial entity CloudMade whose role is to commercialize the OpenStreetMap data, although I believe he's speaking primarily with the OpenStreetMap hat on today.
• Noam Bardin - the CEO of Waze, a company which effectively offers a crowd source navigable base map with an emphasis on real-time data.

The edited transcript is after the jump:

(continue reading)

Why health care is coming to the Open Source convention

This year for the first time, O'Reilly's Open Source convention contains a track on health care IT. The call for participation just went up, soliciting proposals on nine broad areas of technology including health data exchange, mobile devices, and patient-centered care.

One correspondent asked a bit timidly whether it would be all right to submit a proposal if her company didn't use open source software. Definitely! The Open Source convention has always been about a wide range of computing practices that promote openness in various ways. Open source software is a key part of the picture but not the whole picture. Open data, standards, and collaborative knowledge sharing are also key parts of the revolution in today's health care.

This new track is as much a response to urgings from friends and colleagues as it is an O'Reilly initiative. We could use help spreading the word, because the deadline for proposals is tight. In this blog I'll explain why we created the track and why OSCon is a promising venue for trends that will move and shake health care in positive ways.

(continue reading)

How do we measure innovation?

In response to the IEEE's report on Patent Power, which lists the top companies ranked by number of patents, Ari Shahdadi and Brad Burnham made trenchant comments in email that I thought were worth sharing (with their permission):

Ari wrote:

The main article is sad to read, with choice quotes like this: "Clearly, the global recession seriously hampered innovation in the United States." If I'd like to do anything, it's end the use of patenting statistics as a metric for innovative activity, especially by groups like the IEEE.

Amen - R&D; spending is also a bad indicator because so much is wasted in big companies. The methodology should have something to do with end user utility. Facebook has had a bigger impact on more lives than IBM and they don’t spend a fraction of what IBM spends on R&D; or on patents.

I've long been mindful of the power of synthetic indexes. How many people who religiously check the Dow or the Nasdaq know which companies it actually represents?

It seems to me that there ought to be a way to measure the introduction of new products, and rank them by novelty and by widespread acceptance, in some way that reflects a more substantial measure of innovation and its impact on the economy.

I'd love your thoughts about what could go into such a measure.

Four short links: 26 March 2010

Chrome Extensions in Firefox, AUI Opened, Closing Open Hardware, Fixing Science Metrics

1. Chrome Extensions Manager for Firefox -- lets you run Chrome extensions in Firefox. I don't think, though, that people choose Chrome over Firefox for the extensions (quite the opposite, in fact).
2. Atlassian User Interface -- Javascript HTML UI toolkit, opensourced by Atlassian. (via lachlanhardy on Twitter)
3. Open Source Ethics and Dead End Derivatives -- open source hardware is dealing with the problem of people changing open source designs but not publishing their modified source. Open source software hasn't found an efficient and reproducible mechanism for dealing with this, though I'd love to be shown one. (via bre on Twitter)
4. Let's Make Science Metrics More Scientific (Nature) -- excellent paper about the problem of the metrics for measuring scientific performance are based around papers and citations, but fail to take into account teaching, mentoring, communicating, etc. (via dullhunk on Twitter)

Web-TV convergence is already here, just not the way we expected

New data from Nielsen shows consumers are multitasking on their own terms

Remember WebTV? It was supposed to combine the web and television experiences into a media consumption firehose. Or something like that.

WebTV didn't work out, but that hasn't kept other companies from pursuing similar dreams of web-television utopia. Yahoo is building TV-based widgets. TiVo is banking on search across media types. Boxee wants to entwine browsing and viewing. Even Google is getting in on the act.

But new data from Nielsen suggests they're all headed in the wrong direction. The convergence between television and web has already happened, but it's not occurring in a standalone box:

In the last quarter of 2009, simultaneous use of the Internet while watching TV reached three and a half hours a month, up 35% from the previous quarter. Nearly 60% of TV viewers now use the Internet once a month while also watching TV.

There's a couple things I find notable about this:

(continue reading)

Four short links: 25 March 2010

Against Open Data, Singalong Selection, Library Release, and Twitter Analysis

1. Aren't You Being a Little Hasty in Making This Data Free? -- very nice deconstruction of a letter sent by ESRI and competitors to the British Government, alarmed at the announcement that various small- and mid-sized datasets would no longer be charged for. In short, companies that make money reselling datasets hate the idea of free datasets. The arguments against charging are that the cost of gating access exceeds revenue and that open access maximises economic gain. (via glynmoody on Twitter)
2. User Assisted Audio Selection -- amazing movie that lets you sing or hum along with a piece of music to pull them out of the background music. The researcher, Paris Smaragdis has a done lot of other nifty audio work. (via waxpancake on Twitter)
3. Cologne-based Libraries Release 5.4M Bibliographic Records to CC0 -- I see resonance here with the Cologne Archives disaster last year, where the building collapsed and 18km of shelves covering over 2000 years of municipal history were lost. When you have digital heritage, embrace the ease of copying and spread those bits as far and wide as you can. Hoarding bits comes with a risk of a digital Cologne disaster, where one calamity deletes your collection. (via glynmoody on Twitter)
4. ThinkTank -- web app that lets you analyse your tweets, break down responses to queries, and archive your Twitter experience. Built by Expert Labs.

Recent Posts

• At the Forefront of the Next Industrial Revolution | by Tim O'Reilly on March 24, 2010
• First impression: Health reform and Health IT | by Brian Ahier on March 24, 2010
• Four short links: 24 March 2010 | by Nat Torkington on March 24, 2010
• Lady Ada Day 2010 | by Nat Torkington on March 24, 2010
• Joe Stump on data, APIs, and why location is up for grabs | by James Turner on March 23, 2010
• Web 2.0 Expo NYC CFP is Open Plus Webcast For Submission Tips This Week | by Brady Forrest on March 23, 2010
• Four short links: 23 March 2010 | by Nat Torkington on March 23, 2010
• Four short links: 22 March 2010 | by Nat Torkington on March 22, 2010
• Architecture is Destiny: A Tale of Two Cities and Lessons for the Social Business | by Joshua-Michéle Ross on March 19, 2010
• Current activities at the Electronic Privacy Information Center | by Andy Oram on March 19, 2010
• Trapping content on the iPad won't work, even if it's pretty | by Mac Slocum on March 19, 2010
• Four short links: 19 March 2010 | by Nat Torkington on March 19, 2010