John Hagel on The Social Web

Subscribe to this video podcast via iTunes. Or, you may download the file.

I am releasing my conversation with John Hagel in three segments. In the first segment we discussed the real-time web. Here we discuss the move from the information web to the Social Web.

John makes the point that the rise of the Social Web feels “a bit like Back to the Future” for people who have a long history with the Internet. In the early days the Internet functioned to link people - scientists, researchers etc. The advent of the World Wide Web saw the Internet functioning more as a publishing platform. Now, with the Social Web, we are back full circle to a network that connects people together. When you connect people to people (as opposed to just brokering information) you are able to surface valuable tacit knowledge that is difficult to express in documents.

Building Searcher Personas For Greater Customer Engagement and Acquisition

When we want to find more information about something, hear about something interesting from our friends, see a compelling television commercial, or need a local mechanic, chances are the first place we turn is the Google search box. Fifty percent of us in the United States use search engines every day and over 90% of us search every month. These days, your search strategy is your business strategy, whether you realize it or not, because that’s how potential customers are trying to find you. Search is the new yellow pages, 800 number, Sunday circular, card catalog, and cash register. No matter what kind of web site you have--whether it’s a media property like a blog, an ecommerce site, or the online arm of multinational corporation--you want to connect with as many of your potential audience as possible, and organic search can help make that happen.

Effective search acquisition strategy
To effectively use organic search as an acquisition channel, you should ensure that your web site:

• Can be discovered by search engines so the pages can be added to the list the crawler uses to traverse the web.
• Uses site architecture that doesn’t introduce obstacles that make the pages inaccessible to the crawlers.
• Presents all content in a way that’s extractable for search engines to index.
• Is relevant and useful to searchers

Sounds easy, right? The good news is that the path to success is fairly straightforward. But as you might imagine, once you dive into the details, the road starts taking a few hairpin turns. The first three points focus on the web site infrastructure and the last point is all about what’s actually on the site.

(continue reading)

Abandon Stocks, Embrace Flows - A Conversation with John Hagel

Subscribe to this video podcast via iTunes. Or, you may download the file.

John Hagel spoke yesterday at the Web 2.0 Summit on the panel, Web Squared and the Economy of Work I met with John beforehand and wanted to discuss three “Big Shifts” that have dominated 2009 (1) The move to the real-time web, (2) the move from the information web to the Social Web and (3) the rise of mobile. Since John co-chairs Deloitte’s Center for the Edge I wanted to get his take on each in terms of its impact on larger organizations.

This first video covers the Real-Time Web.

Four short links: 23 October 2009

Beautiful Information, Teen Game Designer, Creative Science Writing, Open Source Schools

1. Information is Beautiful -- gorgeous descriptions of the design of infographics. For once, a design discussion that might be useful to mere mortals like me.
2. Australian Teen Crafts "Sneaky" Games -- video interview with a 16 year-old winner of the IFTF, Sun, and BoingBoing Digital Open. Great to see game design, a topic we've followed on Radar, getting uptake by the people about to enter the workforce. "I love index cards," says Harry, "And I was thinking -- hmm, how can I incorporate them into a project?" So he designed and printed these game cards, and "spread the seeds of sneakiness and espionage" into the unsuspecting pockets, math books, binders and bags and jackets of his schoolmates. (via BoingBoing)
3. Science Writing Shortlist -- the Manhire Prize is New Zealand's most prestigious award for creative science writing. The shortlisted entries are available via this link, and make for enlightening reading. Interestingly, there are two prizes awarded: one for fiction and another for non-fiction; New Zealand has a tradition of encouraging interaction between the arts and sciences.
4. Fedena -- an open source school management system, built in India, using Ruby on Rails. (via Brenda Wallace)

Four short links: 22 October 2009

Cognitive Surplus, Scaling, Chinese Blogs, CS Education for Growth

1. Eight Billion Minutes Spent on Facebook Daily -- you weren't using that cognitive surplus, were you?
2. How We Made Github Fast -- high-level summary is that the new "fast, good, cheap--pick any two" is "fast, new, easy--pick any two". (via Simon Willison)
3. Isaac Mao, China, 40M Blogs and Counting -- Today, there are 40 million bloggers in China and around 200 million blogs, according to Mao. Some blogs survive only a few days before being shut down by authorities. More than 80% of people in China don’t know that the internet is censored in their country. When riots broke out in Xinjiang province this year, the authorities shut down internet access for the whole region. No one could get online.
4. Congress Endorses CS Education as Driver of Economic Growth -- compare to Economist's Optimism that tech firms will help kick-start economic recovery is overdone.

Why Google and Bing's Twitter Announcement is Big News

Tweets will finally become first class web citizens

Lurking innocently on Google's blog this afternoon, like many of their big announcements, was the bombshell that they have reached an agreement with Twitter to make all tweets searchable. This followed an earlier announcement at the Web 2.0 conference by Microsoft that Bing has also arranged to make tweets searchable.

This is not only a huge thing for Twitter, it is also well past due. Until now, Twitter really hasn't been a first class web citizen, because you're not really part of Web 2.0 until you're searchable by Google (and, I suppose, Bing). Sure, you can read someone's tweets from Twitter, or get a thread via a #tag, but the full text searching capabilities that make things really usable on the web, largely powered by Google, have been missing.

Making tweets searchable is a major usability improvement as well. Twitter handles are cute, but sometimes obscure as well. Perhaps people will start using more full names in their tweets in addition to @ references, which would let you find tweets about people without having to know what their handle happened to be.

It appears that Twitter is going out of their way not to play favorites in the search space, by cutting deals with both Microsoft and Google. Microsoft seems to be ahead of the game right now, since they have a live site up, whereas the announcement from Marissa Mayer of Google only hints at things to come over the next few months.

The Bing interface is interesting, it seems to be a hybrid of a web search engine and a twitter search. Typing in a term gets you back both the latest tweets that match the keywords, as well as web pages that more than one tweet share in common that also match the keywords. This is a tacit acknowledgement that a lot of the useful content of Twitter is found in the web pages that are linked from the Tweets.

If I had to guess, I'd say that Tweets will show up more traditionally on Google, as just another kind of search result, that can be narrowed in the same way that you can narrow results to just images or movies. I guess we'll have to wait and see on that.

Four short links: 21 October 2009

Battlefield Android, DIY Leukemia Hacking, Localisation, Bus Pirates

1. Raytheon Sends Android to Battlefield -- Google's OS sees deployment. Using Android software tools, Raytheon ( RTN - news - people ) engineers built a basic application for military personnel that combines maps with a buddy list. [...] Every part of RATS is tailored for use on a battlefield. A soldier could make an unmanned plane a "buddy," for instance, and track its progress on a map using his phone. He could then access streaming video from the plane, giving him a bird's eye view of the area. Soldiers could also use the buddy list to trace the locations of other members of their squad. (via Jim Stogdill)
2. The Kanzius Machine (CBS News, video) -- inventor lost the race against leukemia, but his DIY RF therapy device is being developed "for real". (via Jim Stogdill)
3. Lost in Translation -- Will Shipley shows how to handle internationalisation and localisation. In this post I'm going to explain to you what internationalization and localization are, how Apple's tools handle them by default, and the huge flaws in Apple's approach. Then I'm going to provide you with the code and tools to do localization in a much, much easier way. Then you're going to think, 'That will never work, because of blah!' and I'm going to respond, as if I can read your mind or I've already had this argument with a dozen developers, 'It already did - I used these tools in Delicious Library and Delicious Library 2 and they've won three Apple Design Awards between them. (via migurski on Delicious)
4. The Bus Pirate -- interfaces to a heap of embedded hardware. The ‘Bus Pirate’ is a universal bus interface that talks to most chips from a PC serial terminal, eliminating a ton of early prototyping effort when working with new or unknown chips (via joshua on Delicious)

Pipelining and Real-time Analytics with MapReduce Online

Most of the news related to the real-time web these days centers around the adoption of decentralized, push-oriented^† protocols (pubsubhubbub, rsscloud) designed to reduce latency in web publishing. Less discussed are the analytic tools that can are capable of crunching through data in real-time. As more of the web moves towards these types of publishing tools, data-driven organizations will demand low latency analytic tools.

Some organizations create their own real-time analysis tools, while others turn to specialized solutions^††. The Huffington Post developed in-house tools that let editors optimize headlines in near real-time. In some domains, the need for real-time analytics isn't new and companies have moved in with targeted products: SF-based Splunk is a popular real-time analytic tool for IT organizations.

In a previous post, I highlighted SQL-based real-time analytic tools that can handle large amounts of data. Tools like Truviso (based on the Postgres database) and streambase are attractive in that they require little adjustment for developers already familiar with SQL. In the same post, I noted that other big data management systems such as MPP databases and MapReduce/Hadoop were too batch-oriented (load all the data, then analyze) to deliver analysis in near real-time.

At least for MapReduce/Hadoop systems things may have changed slightly since my last post. A group of researchers from UC Berkeley and Yahoo recently modified MapReduce to allow for pipelining between operators. Rather than waiting for a Map or Reduce operator to complete (or "materialize to stable storage") before kicking off a subsequent operation, their solution is to modify MapReduce to allow intermediate data to be pipelined between operators. As they noted in their paper, pipelining holds several advantages:

A downstream dataflow element can begin consuming data before a producer element has finished execution, which can increase opportunities for parallelism, improve utilization, and reduce response time.

Since reducers begin processing data as soon as it is produced by mappers, they can generate and refine an approximation of their final answer during the course of execution. This technique, known as online aggregation, can reduce the turnaround time for data analysis by several orders of magnitude.

Pipelining widens the domain of problems to which MapReduce can be applied. This allows MapReduce to be applied to domains such as system monitoring and stream processing.

In a recent conversation with lead authors Tyson Condie and Neil Conway, they highlighted a few other features of HOP that would make it attractive to current Hadoop users. First, HOP not only preserves Hadoop's public interfaces, it also allows for jobs to be co-scheduled and pipelined, thus reducing the need to write results to HDFS. Second, pipelining leads to preliminary results and early feedback, resulting in faster debugging cycles. Upon seeing early results, a developer can either kill a task, or toggle between pipeline and block mode. Third, HOP does a better job of handling stragglers (slow running tasks) by using previous results to kick-off smart re-starts. Finally, they are currently incorporating a continuous and adaptive optimizer that for a given task, will let HOP converge to the optimal degree of parallelism. The optimizer will allow HOP to scale up/down, dynamically adding/dropping mappers & reducers, based on data being pipelined. In preliminary experiments, they found that superior cluster utilization via pipelining can mean substantial reductions in job completion times.

For those interested in performing real-time analytics within Hadoop, Tyson and Neil informed us that they will make the HOP code publicly available within a month. When asked if HOP can handle large data sets, they confirmed that researchers inside Yahoo have ongoing (successful) experiments using HOP on "Hadoop scale" data. Over the long-term, they predict some form of pipelining will become standard within Hadoop.

So how does HOP compare with the real-time SQL databases I described in an earlier post? For domains where the latency required is in the order of (sub) milliseconds (e.g. algorithmic trading), HOP probably won't help. OTOH, solutions like Truviso and streambase have shown they can handle those types of problems. But for a broader class of problems where a delay of a few seconds is acceptable, HOP will be a suitable analytic engine. In terms of usability, tools like Truviso and streambase look and work like standard SQL, making them fairly accessible to a broad class of users. To make HOP more accessible, Tyson and Neil noted that one interesting side project is to modify equivalent MapReduce tools (Hive and Pig) to incorporate "continuous and real-time queries".

(†) Traditional pull-oriented sytems require subscribers to nag publishers regularly ("Do you have something new?"). Push models deliver content to clients automatically as soon as new content is published ("Don't call us, we'll call you.").
(††) For real-time structured data analysis, enterprises favor the term complex event-processing (CEP). An example is TIBCO's CEP software.

Web 2.0 Summit Starts Today

Last year at Web 2.0 Summit, one prominent tech executive responded to our focus on "Web meets World" -- the way web technology is being used to attack the world's problems -- by saying "I don't come to this conference to learn how to do good. I come to learn about trends that are going to affect my business."

As it turns out, the "Web meets World" theme was in fact exactly on point with the trends that were going to affect his business. What Fred Wilson calls "the golden triangle" of Web meets World trends -- mobile, social, and real-time -- are at the heart of many of the cutting edge non-profit activities we showed last year, and they are very much at the heart of the for-profit companies following hard on their heels.

I've written a much longer paper on this subject - Web Squared: Web 2.0 Five Years On, and I won't repeat that there. But that's the theory. The practice is how entrepreneurs are taking advantage of these disruptive trends, how big companies are responding, and what kind of infrastructure changes we'll need to support the future that is coming at us.

This year at the Web 2.0 Summit, we'll be hearing how real-time, social, and mobile play out in the strategy of Google, Microsoft, Intel, Facebook, Twitter, Yahoo!, News Corp, AOL, Comcast, Nokia, and even GE, but we'll also be hearing from entrepreneurs, and yes, even some more innovative hackers who are helping birth the future away from the commercial limelight.

The official sessions are great, but it's the hallway conversations that can really set your mind off in a new direction. For example, at a pre-Summit event last night, I had a fascinating conversation with Marc Pincus of Zynga last night about his belief that the third great internet business model has arrived. Fortunately, you don't need to bump into Marc to hear what he thinks: he's speaking this afternoon at 4:15. He's put his ideas about social selling into practice, with 129 million users playing Zynga games each month, spending millions of dollars on virtual goods. But what's most fascinating is how Marc sees the potential to apply social gaming principles to all of e-commerce. His riff on how what's he's learned applies to Amazon (and anyone else selling on the web) is worth the price of admission to the Summit.

I hope to see you at the Summit. John Battelle and I kick off the show with opening remarks at 2 pm at the Westin Market Street in San Francisco.

The Fun Theory

In one of my favorite reads this last year, Play: How it Shapes the Brain, Opens the Imagination, and Invigorates the Soul, author Stuart Brown talks about play as “an un-realized power that can transform our social and economic lives.”

As I read, I realized:
- We have replaced play with homework--big mistake
- Play is how our passions find us
- Play is where happiness finds us, and
- Play is where failure isn’t failure and isn’t emotionally charged. Play is all about iteration and we iterate on the emerging questions that arise from within us and that we are driven to understand.

Some of the most accomplished people I’ve met--Dean Kamen, Roderick MacKinnon, Charles Zuker, and Nathan Myhrvold--talk about their work as play. When I ask them how they played as children, they often describe activities that explore the same questions and ideas they are exploring today in their work. At a gathering of molecular scientists, more than a few whispered that, as children, they'd electrocuted bugs--they had to know what would happen. Interaction, reaction. One of them told me that his MBA sister had a chemistry set that looked as if it had never been used, while his was trashed shortly after he opened it. He tried everything.

Recently, VW launched a campaign, The Fun Theory. The videos on the site show people:
- Choosing stairs over an escalator in a subway station when the stairs are turned into piano keys
- Recycling glass when the glass recycle bin is like a slot machine, and
- Clearing litter when a trash can offers sound effects as trash is pitched in.

The Fun Theory Award competition is accepting entries until November 15. Short window, competition opened October 1. Radar readers are brainy and creative. A winning combination!

Four short links: 20 October 2009

Politics in The Age of Social Software, Ethernet Patents, Free Book Fear, Programming Exercises

1. Poles, Politeness, and Politics in the Age of Twitter (Stephen Fry) -- begins with a discussion of a UK storm but rapidly turns into a discussion of fame in the age of Twitter, modern political discourse, the "deadwood press", and The Commons in Twitter Assembled. There is an energy abroad in the kingdom, one that yearns for a new openness in our rule making, our justice system and our administration. Do not imagine for a minute that I am saying Twitter is it. Its very name is the clue to its foundation and meaning. It is not, as I have pointed out before, called Ponder or Debate. It is called Twitter. But there again some of the most influential publications of the eighteenth century had titles like Tatler, Rambler, Idler and Spectator. Hardly suggestive of earnest political intent either. History has a habit of choosing the least prepossessing vessels to be agents of change.
2. Apple and Others Hit With Lawsuit Over 90s Ethernet Patents -- unclear whether the plaintiff is 3Com (who filed the patents) or a troll who bought them. "We strongly believe that 3Com’s Ethernet technologies are being regularly infringed by foreign and some US companies," said David A. Kennedy, Chief Executive Officer of U.S. Ethernet Innovations. "We believe that the continued aggressive enforcement of the fundamental Ethernet technologies developed by 3Com against the waves of cheap, knock-off, foreign manufactured equipment is a necessary step in protecting the competitiveness of this American technology and American companies in general." (via Slashdot)
3. The Point -- someone's publishing Mark Pilgrim's "Dive into Python", which was published by APress under an open content license. Naturally this freaked out APress (it's easy to imagine many eyelids would tic nervously should such a thing happen with one of O'Reilly's open-licensed books). Mark's response is fantastic. Part of choosing a Free license for your own work is accepting that people may use it in ways you disapprove of. There are no “field of use” restrictions, and there are no “commercial use” restrictions either. In fact, those are two of the fundamental tenets of the “Free” in Free Software. If “others profiting from my work” is something you seek to avoid, then Free Software is not for you. Opt for a Creative Commons “Non-Commercial” license, or a “personal use only” freeware license, or a traditional End User License Agreement. Free Software doesn’t have “end users.” That’s kind of the point.
4. Programming Praxis -- programming exercises to keep your skills razor-sharp, with solutions.

Life With TED - Micromanaging Your Carbon Footprint

I've spent three days watching my power consumption like a hawk, here's how it's going

I've been interested in having a better handle on my electrical consumption for a long time. Our family regularly goes through 1100-1200 kWh a month, and it's been frustrating that I couldn't really get a grip on where or when the power was really being used. I want to get my power usage under control for three reasons:

1. I want to reduce my $180-a-month-and-climbing power bill. Public Service Company of NH (PSNH) has one of the higher electricity rates in the country (we have a nuke we're still paying off, among other things.)


2. I'm seriously investigating adding solar to the mix, now that a 30% federal tax credit, a $6,000 state rebate, and lower prices for the panels have converged. It would be great to get my usage down into the 600-800 kWh average output I've been told I can expect a month from a system, and zero out my PSNH bill on a yearly basis.


3. I'm a firm believer in reducing carbon emissions, I'd like my 14 year old son to have a world to grow up in. I've already cut my fuel oil use in half (to a still awful 250 gallons a month in the winter, but it's a huge house...) Cutting my electricity is the next low-hanging piece of fruit on the tree.

I had been tracking Google PowerMeter, a Google initiative that lets people monitor their energy usage online, but it was only available to customers of electric providers who were using so-called "Smart Meters". Smart Meters send usage data back to the provider, and PSNH isn't one of them.

Then, this week, Google announced on their blog that normal mortals could now order a device called The Energy Detective (or TED, as he's known by his friends...) TED is made by Energy, Inc. out of South Carolina, and consists of a minimum of two components. The first piece is an inductive current measuring device that lives out in your circuit breaker box. The second is a gateway device that plugs into a wall socket and has an Ethernet jack. Optionally, you can also get a stand-alone display, so that you don't need a computer to view your usage.

Wiring the sensor device into your box is fairly straightforward. You clamp the two sensors around the mains as they come into the box. You also have to wire the device to the two "hot" phases of your 220V service (which requires two free breakers in your box on different phases), and a third wire running to neutral. If you have some basic electrical savvy, you can do it yourself, but I decided to wimp out, since my box is so crowded (after-effects of having a transfer switch put in for a generator...), so I shelled out the $85 to have an electrician put it in.

The gateway unit communicates with the sensor unit via signals sent over the house AC. As with anything using the power lines to communicate, I found the unit was very particularly to which outlet I plugged it into. It really doesn't like to share a circuit with a computer, for example. Neither of the two plugs which was actually next to a network hub would pick up a signal, but one in an adjacent room that happened to have a network jack did.

Once you have the gateway talking to the sensors and plugged into the network (it uses DHCP to get an address), you can surf to it using any browser. I can even get to it using Safari on my iPhone. The "home" screen is a dashboard, showing various statistics about current demand and your daily, weekly and monthly averages. You can view the data in terms of kWh, dollars (once you tell TED how much you pay for power, it can even handle peak period and tiered pricing models), or pounds of CO2.

All of the ranges on the dials and bar-graphs are configurable, so if you want 3kWh to be "red", you can set it up that way. You can also configure refresh rates. Clicking on the "Graphing" tab lets you view your usages second to second, minute by minute, or by daily or weekly aggregates.

It's these graphs that I have found to be most useful. You can start to see all sorts of interesting patterns, like the "heartbeat" of my furnace turning on and off at night, when the rest of the house is otherwise quiet.

I can also see the huge hump when my son wakes up in the morning, and proceeds to turn on every first floor light in the house. I was even able to tell that my wife had turned on the dishwasher before she left for school one morning.

(continue reading)

Recent Posts

• George Dyson's "Among the Machines" in Mountain View | by Dale Dougherty on October 19, 2009
• Why Posterous Is a Smart Tool For Informal Government Blogging | by Mark Drapeau on October 19, 2009
• Four short links: 19 October 2009 | by Nat Torkington on October 19, 2009
• Only Connect - Should Broadband Access Be a Right? | by Joshua-Michéle Ross on October 17, 2009
• A Conversation with Dr. Walter Scott of DigitalGlobe | by Joshua-Michéle Ross on October 17, 2009
• Ignite Show: Kathy Sierra on Feeling Better is Better | by Brady Forrest on October 17, 2009
• Social Networking is the Means to Achieve Workplace Collaboration | by Mark Drapeau on October 16, 2009
• Four short links: 16 October 2009 | by Nat Torkington on October 16, 2009
• Shopping with AR | by Christine Perey on October 15, 2009
• Random Hacks of Kindness: Disaster Relief Codejam | by Brady Forrest on October 15, 2009
• Snow Leopard Is Location-Aware | by Brady Forrest on October 15, 2009
• Blog Action Day 2009: Climate Change | by Tim O'Reilly on October 15, 2009