How Big Data Impacts Analytics

Research for our just published report on Big Data management technologies, included conversations with teams who are at the forefront of analyzing massive data sets. We were particularly impressed with the work being produced by Linkedin's analytics team. [We have more details on Linkedin's analytics team, in an article in the upcoming issue of Release 2.0.]

At the second Social Web Foo camp, I had a chance to visit with Linkedin's Chief Scientist DJ Patil. As a mathematician specializing in dynamical systems and chaos theory, DJ began his career as a weather forecaster working for the Federal government. Years later, he ended up in an analytics role at Ebay where his prior experience with massive data sets came in handy. In the short video below, DJ shares his observations on how analytics has changed in recent years, especially as Big Data increasingly becomes common. Companies are casting a wider net, and are hiring scientists from fields not traditionally known as fertile recruiting grounds for data intelligence teams.

DJ also talks about his personal journey from mathematics to e-commerce and social networks. Among his previous stints, DJ worked with the DOD and used "... social network analysis to identify terrorists."

Other short videos from Social Web Foo camp:

Big Data: Technologies and Techniques for Large-Scale Data

Our belief that proficiency in managing and analyzing large amounts of data distinguishes market leading companies, led to a recent report designed to help users understand the different large-scale data management techniques. Our report on Big Data Technologies was the result of interviews with over thirty experts, including research scientists, (open-source) hackers, vendors, data analysts, and entrepreneurs. Rather than endorse specific vendors and technologies, we provide a framework to help readers navigate the wide variety of options available. (NOTE: If you're interested in purchasing the report as a single-issue of Release 2.0, we can provide you with a DISCOUNT CODE. Contact information is at the end of the video clip below.)

I recently sat down with my co-author, Roger Magoulas (Director of Research at O'Reilly), who agreed talk about our report and Big Data in general. Roger begins by speaking passionately of the importance of data management and analysis. He proceeds to highlight what we believe to be the key technology dimensions for evaluating data management solutions. The video ends with a glimpse into future technologies and general advice to organizations interested in improving their proficiency in handling data.

The full program is available in four extended clips:

[ Head over to O'Reilly Media's Youtube channel for other interesting videos. ]

Ignite Show: Kati London on Botanicalls: Homegrown Terra-rists

Today we are releasing "Botanicalls: Homegrown Terra-rists" the second episode of the Ignite Show. This week's speaker is Kati London, the co-creator of Botanicalls, a device that will let you know when your plant needs watering via Twitter. Kati doesn't spend much time explaining Botanicalls instead she talks about some fun(ny) uses for it.

If you want to learn more about Botanicalls check out their site. Or you can see what the Twitter feed of one Robert Plant (an orchid) is like. Or you can pick up a kit at Maker Shed or ThinkGeek. Botanicalls was created by Kati London (the speaker), Rob Faludi, Rebecca Bray and Kate Hartman.

Enjoy the show!

Hope Art

The Capitol Hill Seattle blog has produced a short video about the surge of Obama art around Seattle. On the street you can find Shepard Fairey's Hope image has been put on garage doors and merged with donuts. It's also being used to advertise for local businesses (as seen in these coffee posters).

The other day I heard an interview with Shepard on NPR about how his art went from the street to the inauguration. It's definitely worth a listen.

Software for Civic Life: An Interview with Mike Mathieu of Frontseat.org

In this interview Mike Mathieu, founder of Frontseat.org, discusses how he is helping to build “software for civic life”. Using publicly available data and web services (many of their applications use S3 and EC2) Frontseat creates simple, highly functional tools like Walkscore (rating neighborhood walkability) and Countmore (helping students in the recent elections decide which state to cast their vote in). Mike is also behind obamaCTO where you can add your opinion and cast your vote for what the new CTO of the USA’s priorities should be.

With the recent election there has been a lot of talk and enthusiasm for the possibility of a more open, modern government that operates with transparency and makes data available for remixing by it’s citizens. People have their eye on government to change…This is a worthy goal to push for but don’t hold your breath. The government of the United States is a behemoth that, all told, employs 12 million people and is preternaturally territorial and risk averse…

Pressing government to change is necessary but is not the only bet we should place. Mike makes the point in this video that we don’t need to wait for data that can improve civic life or increase transparency in government.

If you know of other examples of citizens improving civic life that deserve mention, please share them in the comments.

Part one of this interview is available here.

GeoData Explorations: Google's Ever-Expanding Geo Investment

Google has been investing lots of money in geodata acquisition. Some of the money is being spent externally: they've inked an exclusive satellite imagery deal with GeoEye (Radar post) and a data sharing deal Tele Atlas (Radar post). And some is being spent internally with Mapmaker, Street View and the web. Over the past week Google has been sharing visualizations of their internally gathered geodata. Here's a round-up of them.

The image above was released on December 9th. It shows how much of the US is available via Street View. According to the post Street View imagery increased 22 fold around the world in 2008.

The dark image above was released on December 11th. It highlights the parts of the world that are being mapped on Google's Mapmaker by users (Radar post). Mapmaker is now live in 164 countries. According to the map it has gained the most traction in Africa and the Indian sub-continent. The Google Mapmaker team has released timelapse videos of Mapmaker building cities on the Mapmaker YouTube Channel. I've embedded one after the jump.

This final image shows all the points described by GeoRSS and KML all over the world. It was shown at Where 2.0 2007 by Michael Jones (video). Unsurprisingly, this image and the Mapmaker image show opposite data density concentrations.

In some more GeoData Explorations posts this week I will look at OSM vs Google and some surprising trends in KML.

(continue reading)

“Technology is the 7th Kingdom of Life” - A conversation with Kevin Kelly

Or, you may download the file.



Kevin Kelly doesn’t need much in the way of introduction to Radar readers. He is a big thinker looking at the intersection of biology, technology and culture.

Kevin gave a great High Order Bit at the Web 2.0 Summit and I caught up with him afterward. This interview covers:

• The impact of the web on our recent elections
• The rich new possibilities for interaction and collaboration afforded by the web
• The Wisdom of the Crowds vs. the Stupidity of the Mob
• Technology is the 7th Kingdom of Life… looking into “what technology wants”

Huffington, Newsom, and Trippi talk politics in a Web 2.0 world

"Were it not for the Internet, Barack Obama would not be President," declared Arianna Huffington from the stage at Web 2.0 Summit, the day after the election. In "The Web and Politics" session, moderator John Heileman explores the new world of running for office--and governing once you win--with Huffington, San Francisco Mayor Gavin Newsom, and veteran politico Joe Trippi.

Politicians and pundits love to talk, and in this case, they're good at it. A few bon mots and surprising stats to whet your appetite:

* Trippi: People watched 14.5 million hours of the official video created by Obama campaign (not including supporters' videos). Obama would have had to spend $47 million dollars to buy those eyeballs on TV. Plus, YouTube viewers went there on purpose, to watch Obama's videos--they weren't interrupted in the middle of their football game.

* Huffington: "The Internet has killed Karl Rove politics." (You'll have to watch the video to catch her funniest line. Hint: it has an Alaska connection).

* Newsom: "I have to watch myself sing "I Left My Heart in San Francisco" on YouTube, and it can't go away. I am desperate to get it to go away....we're in a reality TV series now, in politics, 24/7."

[NOTE: Web 2.0 summit videos are available on YouTube.]

Online Communities: The Tribalization of Business

Or, you may download the file.



Recently I spoke with Francois Gossieaux of Beeline Labs about the role of online communities in the enterprise. Francois has been evangelizing the learning gained from his recent study “The Tribalization of Business” (see here for the Slideshare presentation).



The interview is broken into three parts. Francois is a great storyteller, bringing case studies in to support nearly every point. Here are a few insights I took away from our conversation:
Community for community’s sake: most businesses begin planning a community with traditional objectives (lower support costs, drive innovation, increase customer loyalty etc.). On the Social Web this is the equivalent of entering a personal relationship with an ulterior motive (which never works out quite right). Businesses should begin with the question, “how can I satisfy the needs of this community?”- and then follow the community’s lead. Be open to the unexpected.

In my experience this is one of the hardest things for companies to get behind and relegates this kind of "enlightened" community effort to either top-level leadership or skunk works development. Middle management is typically the most reluctant to deviate from standard practice and place a bet on community for the community’s sake.

Communities require a social framework to thrive - most companies have a mindset that reflects the legal, contractual and hierarchical underpinnings of their business and carry these behaviors with them into the community. This informs their planning, measurement and how they encourage contribution. These incentives have little sway on the Social Web where the mindset is social and trust, reputation and relationship are big drivers of contribution. As Francois says, “The most successful communities occur when you tap into that social framework”

Consider stories as a success metric: While there is a fair amount in this interview about measurement - this was my favorite: A great anectdote about how one company views the stories that emerge from their community as a key metric of success. Great stories are inherently viral and can have a profound impact on decision making in an organization.

Think Bigger: Most large companies are satisfied to have small communities; basically bringing a focus group online. Doing so misses the potential of the online community to transform your business. Consider how Intuit is now embedding live community directly into their application - allowing users to seek help and get questions answered directly.

Transformative communities blur the lines between company and customer and portend a future where retail ecommerce sites go well beyond ratings and reviews and provide problem solving, shopping mentors, product development and other services directly from the community. Where internet sites are co-evolved (from interface to feature-sets to codebase) in cooperation with community, where complex applications (desktop and cloud-based) meld standard functions with community functions. Communities are certainly helpful in providing feedback on customer behavior but that is just one small part of the story.

Shai Agassi on Electric Cars

One of my favorite sessions at the recent Web 2.0 summit was Tim's half-hour conversation with Shai Agassi, the CEO of Better Place. Better Place aims to make electric cars widespread ("the electric car as the de facto standard") by addressing major issues that have held back electric vehicles: affordability and convenience.

In a relaxed conversation with Tim, Shai described an electric car industry that resembles the mobile phone business. Just as telecom companies sell mobile handsets at a discount if one is willing to commit to a contract, their subscription-based model will allow consumers to purchase an electric car at the fraction of the normal price. Car owners will pay additional fees based on the amount of miles they drive and the type of car they choose to own. To support their subscribers, Better Place will also build extensive networks of charging spots and battery exchange stations. They will build the first "Electric Recharge Grids" in Israel and Denmark.

Prior to starting Better Place, Shai was a president at software vendor SAP. The interview briefly touches on IT and enterprise computing.

[NOTE: Web 2.0 summit videos are available on YouTube.]

Web 2.0 Expo Europe Videos Up

Many of the keynote videos from last week's second Web 2.0 Expo Europe are available. The highlight for me was definitely Tim's conversation with Martin Varsavsky, the CEO of Fon. He discussed his path from Argentina to Spain, his handling of the credit crisis a year before Sequoia's warning and his philosphy as an entrepreneur.

Other mainstage highlights included:

A Conversation Between Yossi Vardi (International Technologies) and Tim O'Reilly (O'Reilly Media)
Alexandra Deschamps-Sonsino (Tinker.it!), In Case of Turbulence: Open Source Hardware
Saul Klein (Index Ventures), The European VC Market
Leisa Reichelt (www.disambiguity.com): Redesigning Drupal.org: An Exercise in Open Source Design

Unfortunately, only the mainstage sessions were recorded by us. You can find many speaker's slides online. More videos will be coming this week, including my conversation with John Lilly (CEO of Mozilla) on their product's future and the Tele Atlas's CTO's talk on their crowdsourced mapping process.

Updated: Dopplr provides a visualization of where our attendees came from. It's great to see most of Europe so well represented.

Technology, Politics and Democracy

Or, you may download the file.


Recently I spoke with Jascha Franklin-Hodge, CTO and co-founder of Blue State Digital about how technology is affecting politics and democracy in the U.S.

Blue State Digital was born out of Jascha's experience helping Howard Dean’s seminal run for the White House in ’04. and is the technology and strategic services company powering Barack Obama (and many other Democratic leaders and social justice causes like Save Darfur and We Can Solve It).

These videos (there are three total) are timely in light of the staggering September figures from the Obama campaign:

• 630,000 new donors (bringing total donors to 3.1 million)
• 150 million dollars raised
• Average contribution: $86

(continue reading)

Recent Posts

• Live Stream of MSR's Social Computing Symposium | by Brady Forrest on October 13, 2008
• Ignite Boston 4 -- Videos Uploaded | by Mike Hendrickson on September 23, 2008
• Breaking Down What's Happening on the Social Web | by David Recordon on July 18, 2008
• Video of Rich Wolski's EUCALYPTUS talk at Velocity | by Jesse Robbins on June 24, 2008
• code_swarm - visualizing the life of open source | by Jesse Robbins on June 18, 2008
• DisasterTech from Where2.0 | by Jesse Robbins on May 30, 2008
• Learn More About Ugandan Hip Hop with Diamonds in the Rough | by Brady Forrest on May 20, 2008
• Where 2.0 Video: Google/ESRI Keynote | by Brady Forrest on May 13, 2008
• You Become what You Disrupt - (part two) | by Jesse Robbins on April 13, 2008
• Jill Bolte Taylor's amazing TED talk | by Jimmy Guterman on March 16, 2008
• DIY Multitouch with the Wiimote | by Jesse Robbins on February 25, 2008
• Mainstream acceptance of Twitter for disaster communication... | by Jesse Robbins on January 5, 2008