Wordle visualization of my Tweetstream

Mike Hendrickson (@mikehatora) sent me a nice Wordle visualization of my tweet stream, dating back as far as Twitter keeps it. As you can see, I retweet a lot. It's also interesting that many of the repeated words are not concepts or topics, but people's names (in the form of twitter handles.) This is one of the interesting things about twitter: it's a reflection of a community of shared minds, rather than of shared ideas.

(Tweets were retrieved using this python script.)

Reinventing the Book in the Age of the Web

There's a lot of excitement about ebooks these days, and rightly so. While Amazon doesn't release sales figures for the Kindle, there's no question that it represents a turning point in the public perception of ebook devices. And of course, there's Stanza, an open ebook platform for the iPhone, which has been downloaded more than a million times (and now has been bought by Amazon.)

But simply putting books onto electronic devices is only the beginning. As I've said for years, that's a lot like pointing a camera at a stage play, and calling it a movie. Yes, that's pretty much what they did in many early movies, but eventually, the tools of production and consumption actually changed the format of what was produced and consumed. Camera angles, pacing, editing techniques, lighting, location shooting, special effects: all these innovations make the movies (and television) of today very different from the earliest movies. YouTube is pushing the envelope even further. Why should books be any different? (Aside: Bruce Sterling just published an amazing rant on this topic - how the context of pulp magazines shaped the content of early science-fiction.)

In our work at O'Reilly as authors and publishers, we've long been interested in exploring how the online medium changes the presentation, narrative and structure of the book, not just its price or format.

A sample from my latest experiment, The Twitter Book, can be seen below.

Now, you might ask, how is a book authored in powerpoint a web publishing experiment? It boggles the mind!

The web has changed the nature of how we read and learn. Most books still use the old model of a sustained narrative as their organizational principle. Here, we've used a web-like model of standalone pages, each of which can be read alone (or at most in a group of two or three), to impart key points, highlight interesting techniques or the best applications for a given task. Because the basics are so easy, there's no need to repeat them, as so many technical books do. Instead, we can rely on the reader to provide (much of) the implicit narrative framework, and jump right to points that they might not have thought about.

Perhaps the biggest driver, though, was the need for speed. We couldn't imagine writing a book about twitter that wouldn't be immediately out of date, because there are so many new applications appearing daily, and the zeitgeist of twitter best practices is evolving equally quickly. So we needed a format that would be really easy to update. (Again, modular structure helps, since new pages can be inserted without any need to reflow the entire document.) We plan to update The Twitter Book with each new printing.

The idea to write the book in powerpoint came to me while I was thinking about how quickly I write a new talk: I generally use pictures as visual bullets, to remind me about the order of my main points; I know what I want to talk about when I see each picture. And pictures are a memorable, entertaining way to tell a story. All I needed to do, I realized, was to write down some notes equivalent to what I'd be saying if I were giving this as a talk. (And in fact, I will be using portions of the book as the basis for my talk later today at the Inbound Marketing Summit, and a few weeks later at the Twitter Boot Camp.)

Of course, having the amazing Sarah Milstein as a co-author really helped. She immediately grasped the concept, and because she knows just about everything there is to know about the twitter app ecosystem, tools, and techniques, she actually provided much of the meat of the book. This allowed me to spend time on giving my perspectives on points that particularly matter to me, or that demonstrate my approach to twitter.

But even there, we saw real benefit in the format of the book. As wikipedia has demonstrated, collaboration is easiest when documents are constructed using a modular architecture. It's hard to coordinate a complex narrative (even single authors sometimes lose track of their plot details); much easier to work on things in standalone units that share a common, "interoperable" format.

I first explored this modular approach to the book in Unix Power Tools, a book I wrote in 1993 with the explicit goal of emulating the hypertext style of the web in a print book. The book consists of a thousand inter-linked articles. In the print book, the "hyperlinks" were in the form of cross references to individually numbered articles. In online versions such as the one at Safari books online, the cross references are expressed as real hyperlinks.

Similarly, our "Cookbook" series of technical books (whose format was originated by Nat Torkington in 1998 with the first edition of the Perl Cookbook), effectively creates a database of answers to common problems.

In 2003, Dale Dougherty and Rael Dornfest developed the Hacks series, another approach to books as collections of loosely-related pages. The Hacks books provide a collection of tips, tricks, and documentation on the problem-solving approaches of cutting edge users.

Of course, modularity isn't the only thing that publishers can learn from new media. The web itself, full of links to sources, opposing or supporting points of view, multimedia, and reader commentary, provides countless lessons about how books need to change when they move online. Crowdsourcing likewise.

But I like to remind publishers that they are experts in both linking and in crowdsourcing. After all, any substantial non-fiction work is a masterwork of curated links. It's just that when we turn to ebooks, we haven't realized that we need to turn footnotes and bibliographies into live links. And how many publishers write their own books? Instead, publishers for years have built effective business processes to discover and promote the talents of those they discover in the wider world! (Reminder: Bloomsbury didn't write Harry Potter; it was the work of a welfare mom.) But again, we've failed to update these processes for the 21st century. How do we use the net to find new talent, and once we find it, help to amplify it?

I don't exempt O'Reilly from that criticism. While we've done many pioneering projects, we haven't fully lived up to our own vision of the ebook of the future. For example, Safari Books Online, our online library, recognizes that the reference work of the future is far larger than a single book. But we've done a poor job of updating the works in that library to be more "web like" in the way I've just outlined. It is still primarily a collection of books online. (We're adding video, more web content, and working to update books to be more link-rich, but we're not as far along as I'd like.)

Take a look at any ebook, and ask yourself how it could be richer, more accessible, more powerful, if it approached the job it was trying to do with fresh eyes, and a fresh approach.

Many of the products that result won't look like books at all. After all, Google Earth is the new Rand McNally, Wikipedia is the new Brittanica, Google itself is the new competitor to many reference works, YouTube is becoming a vehicle for just-in-time learning, and World of Warcraft is the new immersive fantasy novel. What job do publishers do? And how can new media help us do it better?

Why Aneesh Chopra is a Great Choice for Federal CTO

The news has now been leaked that President Obama intends to nominate Aneesh Chopra as the nation's first Chief Technology Officer. The Federal CTO will be an assistant to the President, as well as the Associate Director of the White House Office of Science and Technology Policy. He will work closely with Vivek Kundra, the recently-named Federal CIO, to develop and implement the President's ambitious technology agenda.

According to one background document I was given access to, the White House describes the relationship between the CIO and CTO roles as follows:

The responsibilities of the CIO are to use information technology to transform the ways in which the government does business. The CTO will develop national strategies for using advanced technologies to transform our economy and our society, such as fostering private sector innovation, reducing administrative costs and medical errors using health IT, and using technology to change the way teachers teach and students learn.

Some in Silicon Valley have hoped for one of their own, a CTO with a deep technology pedigree and ties to the technology industry. For example, the Techcrunch coverage leads with the title Obama Spurns Silicon Valley. This is a narrow view.

I've been working for much of the past year to understand what many have been calling Government 2.0, and in that process, Chopra has been one of those who have taught me the most about how we can build a better government with the help of technology. He is an excellent choice as Federal CTO, for many reasons:

1. Chopra has been focused for the past three years on the specific technology challenges of government. Industry experience does little to prepare you for the additional complexities of working within the bounds of government policy, competing constituencies, budgets that often contain legislative mandates, regulations that may no longer be relevant but are still in force, and many other unique constraints. In his three year tenure as Secretary for Technology for the Commonwealth of Virginia, Chopra has demonstrated that he has these skills. In fact, last year, the National Association of State Chief Information Officers ranked Virginia #1 in technology management.

2. The role of the CTO is to provide visionary leadership, to help a company (or in this case, a government) explore the transformative potential of new technology. Try a few of these Virginia technology initiatives on for size:
   • the first officially-approved open source textbook in the country, the Physics Flexbook.
   • integrating iTunes U with Virginia's state education assessment framework;
   • the Learning Apps Development Challenge, a competition for the best iPhone and iPod Touch applications for middle-school math teaching;
   • a Ning-based social network to connect clinicians working in small health care offices in remote locations;
   • a state-funded "venture capital fund" to allow government agencies to try out risky but promising new approaches to delivering their services or improving their productivity;
   • a lightweight approval and testing process that allows the government to try out new technologies before making a full, expensive commitment.

3. Chopra demonstrates a deep understanding of the idea that the government is an enabler, not the ultimate solution provider. From the list of initiatives above, you can see that Chopra grasps the power of open source software, Web 2.0, user-participation, and why it's better to harness the ingenuity of a developer community than to specify complete top-down solutions. In a conversation with me a couple of months ago, he expressed his enthusiasm for the idea of a "digital commonwealth," a recognition that technology can help us to come together as a society to solve problems and create value through common effort. (See my post yesterday, The Change We Need: DIY on a Civic Scale.)

   This digital commonwealth approach can be seen in Virginia's approach to rural broadband. The Virginia Information Technology Agency has developed a "broadband toolkit" that fosters cooperation between public agencies and private companies, identifying the location of public sector radio towers that can be used for free by broadband providers in order to reduce their costs, and areas with zoning rules that allow for public sector use of private radio towers.

   The digital commonwealth reflects an understanding of the dynamics that have led to technology successes like Google Maps, Facebook, Twitter, and the iPhone app store: the platform provider creates enabling technology, "rules of the road," and visibility for participants, and then gets out of the way, leaving room for third parties to create additional value. This is a great model for all future government technology efforts.

4. Chopra understands that government technologists need to act more like their counterparts in Silicon Valley. As Micah Sifry notes in the TechPresident blog, quoting from a Governing magazine article about the Virginia venture capital experiment:

   "More important, and more unusual for the bureaucrats," says Governing, "he gives them permission to fail. You can't innovate, Chopra tells them, without taking a gamble every now and then." He adds, "We need to fundamentally change the culture of government in which change is measured in budget cycles to one in which change is measured in weeks or months."

5. Chopra is a practical innovator. He's not chasing technology for its own sake. I like this quote from a recent Federal Computer Week story:

   Understanding the process or service is always the most important factor, with technology running second. “Service sector innovation is the most important question,” Chopra said. “I’m not as excited about whether or not it’s emerging technology.”

   In my own conversations with Chopra over the past few months, this focus on "service sector innovation" has seemed particularly fertile. Our economy increasingly consists of service jobs. Improving the effectiveness of those jobs is one of the great challenges of the 21st century. Chopra wants to put technology to work to make us better at health care, at education, at creating a vibrant economy. These are also, incidentally, the goals of the Federal CTO job, as described in one briefing document I reviewed:

   One of the primary responsibilities of the CTO will be to leverage American ingenuity generally and new technologies in particular as engines for job creation and economic growth. The CTO’s priorities will include expanded use of technology to boost broadband access, reduce health care costs, enable novel job-producing industries, remove barriers to technological progress, and create a more transparent and interactive government.

6. Chopra has a real focus on measurement, and on figuring out what really works. For example, the social network for remote health care workers mentioned above was the result of data showing an unusually high turnover rate for these workers. As I once wrote, in If Google Were a Restaurant,

   Web 2.0 (or "Live Software", as Microsoft has so insightfully called it) depends on creating information feedback loops. This is the practical plumbing that makes possible Web 2.0 systems that get better the more people use them.

   If we are truly to remake America's economy with the aid of technology, as the Obama administration has promised, we need to embrace the culture of transparency and feedback. The Federal transparency initiative is a central part of the plan. While there's a long way to go, the recovery.gov initiative, to report on the progress of Federal stimulus spending, is a critical step in building the electronic nervous system that will help us to understand what we're spending, where it's going, and what we're getting for all that money. Under Chopra's leadership, Virginia has been in the forefront of driving stimulus transparency down to the state level. stimulus.virginia.gov was one of the first state-level stimulus sites, and has served as a model for other states.

7. Chopra has specific expertise in Health Care IT. This is one of the most critical areas where we need to see technology innovation in the coming years. Unless we can get Moore's Law working in health care, it will eventually bankrupt our already-tottering economy. $19 billion has been allocated to Health Care IT in the stimulus package. We need someone who can help us spend it wisely!

8. Chopra is incredibly charismatic. This is essential in a role that depends on persuasion rather than outright authority. As Sean Garrett said in an excellent post on the 463 blog:

   I highly recommend watching a good portion of the video below. It's from this year's Congressional Internet Caucus conference in September.

   Chopra may not be a Valley guy, but Silicon Valley is going to like him a lot. He's energetic, insightful and can speak the language (again, watch the video). He's no bureaucrat.

   And, just because you didn't previously work for a chip company or an Internet start-up doesn't mean that you "are not a tech guy" as I just read another blog. Chopra spent a bulk of his career seeing technology in action (for better or worse) in his work in the health care industry and knew that it could and should do better to bring change to the massive sector.

   I couldn't agree more. Aneesh Chopra is a rock star. He's a brilliant, thoughtful change-maker. He knows technology, he knows government, and he knows how to put the two together to solve real problems. We couldn't do better.

The Change We Need: DIY on a Civic Scale

I've been working a lot lately to imagine what Government 2.0 might look like. One of the most inspiring and thought-provoking stories I've read recently might not look like a Gov 2.0 story, but it is: Island DIY: Kauai residents don't wait for state to repair road.

Their livelihood was being threatened, and they were tired of waiting for government help, so business owners and residents on Hawaii's Kauai island pulled together and completed a $4 million repair job to a state park -- for free.

In fact, governments, like corporations, are vehicles for collective action. We pay a government, or a business, because it's an efficient way to tackle projects that are larger than a single person or group of friends can take on. But let's not forget that we ourselves are the raw material of collective action.

Traditional communities still remember how to do a barn raising. Those of us who spend our time on the internet celebrate wikipedia, but most of us have forgotten how to do crowdsourcing in the physical world.

The internet provides new vehicles for collective action. A lot of people pay attention when social media is used to organize a protest (as with the recent twitter-fueled protests in Moldova.) But we need to remember that we can organize to do work, as well as to protest!

Especially striking in the story are the cost and time-savings:

"It would not have been open this summer, and it probably wouldn't be open next summer," said Bruce Pleas, a local surfer who helped organize the volunteers. "They said it would probably take two years. And with the way they are cutting funds, we felt like they'd never get the money to fix it."

And if the repairs weren't made, some business owners faced the possibility of having to shut down....

So Slack [owner of a kayak tour business in the park], other business owners and residents made the decision not to sit on their hands and wait for state money that many expected would never come. Instead, they pulled together machinery and manpower and hit the ground running March 23.

And after only eight days, all of the repairs were done, Pleas said. It was a shockingly quick fix to a problem that may have taken much longer if they waited for state money to funnel in....

"We can wait around for the state or federal government to make this move, or we can go out and do our part," Slack said. "Just like everyone's sitting around waiting for a stimulus check, we were waiting for this but decided we couldn't wait anymore."

Now is the time for a renewal of our commitment to make our own institutions, our own communities, and our own difference. There's a kind of passivity even to our activism: we think that all we can do is to protest. Collective action has come to mean collective complaint. Or at most, a collective effort to raise money.

What the rebuilding of the washed out road in Polihale State Park teaches us is that we can do more than that. We can apply the DIY spirit on a civic scale.

Aneesh Chopra, the Secretary for Technology of the Commonwealth of Virginia, recently told me why he liked the term "commonwealth" better than "state": commonwealth emphasizes the value that we create by coming together. Technology provides us with new ways to coordinate, new ways to govern and to regulate, but we should never forget that these are merely means. The end is a better society. And that starts with us.

We need to rediscover government as an enabler, not a solution provider; as a platform for our own innovation, a lever for our own work, not as the deus ex machina that we've paid to do for us what we could be doing for ourselves.

If you know of other great civic DIY stories, let me know. I want to feature technology in my Government 2.0 activism, but I also want to feature the simple DIY spirit.

What Publishers Need to Learn from Software Developers

There was a great exchange on the O'Reilly editors' backchannel the other day, so illuminating that I thought I should share it with the rest of you. We've been discussing the fast-track development we're using to produce The Twitter Book. (We're basically authoring the book as a presentation, after I realized how much more quickly I am able to put together a slide deck to make my points than I am a normal book. Twitter is also such a fast-moving topic that we need to be able to update the book every time we reprint it.)

Sarah Milstein wrote:

Apropos of everything, the NYT on publishers' speeding up the production process, especially with eBooks:

“If this book had gone through the normal publishing procedures,” Mr. Kiyosaki said, “it wouldn’t be worth writing.”

The more I think about it the more obvious it's becoming to me that the next generation of authoring/production tools will have much more in common with today's software development tools than with today's word processors.

Software developers spend enormous amounts of time creatively writing with text, editing, revising, refining multiple interconnected textual works -- and often doing so in a highly distributed way with many collaborators. Few writers or editors spend as much time as developers with text, and it only makes sense to apply the lessons developers have learned about managing collaborative writing and editing projects at scale.

'Nuff said. I await said next generation of authoring/production tools.

It's Always Ada Lovelace Day at O'Reilly

I had a hard time choosing just one of the many marvelous women in tech that I might write about for Ada Lovelace Day, because, frankly, I'm surrounded by those women! Where so many think of the tech world as male-dominated, women have always played a major role at O'Reilly. A large part of our management team has always consisted of women, and women are the creators of some of our best known products and brands. I want to acknowledge their contributions, highlighting the fact that they have been so central to the success of my company. I also want to emphasize that there are many ways to contribute to a tech community, and that being a coder is not the only way to have an impact on the world of computing.

My first hat tip has to go to my wife, Christina O'Reilly. She's a playwright and choreographer, not a techie. But if you've been influenced by me, you've also been influenced by her. The company, its values, and much of its unusual nature have been profoundly shaped by our relationship. (You can see her influence in some of the early company documents you'll find here,.) In more ways than I can count, I've built the company to be one that she would be proud of. We met when I was nineteen, and she's been part of everything I've ever done, in the same way that Elizabeth Barrett Browning said of her husband, Robert Browning:

What I do and what I dream include thee,
As the wine must taste of its own grapes

From the earliest years of the company, most of my key managers have been women. From 1985 till about 2000, there was a troika--Linda Walsh, Linda Lamb, and Cathy Brennan--who helped me shape the corporate culture, and for many years were touchstones for the values I still espouse.

Linda Walsh was my first employee. She helped me build my original documentation consulting business, and helped me come into my own as a leader when I broke up with my original business partner. She was also the key business leader for our digital books initiative in the late 1990s and one of the founders of Safari Books Online.

Linda Lamb was a key member of the team (along with me and Dale Dougherty) that developed our original publishing program back in 1985. Linda also served as our director of marketing for many years, with a delicious, quirky sense of humor. (I still remember one of her earliest trade show pieces, a wonderful riff on the National Enquirer, in which she reported on abductions by strange aliens with big eyes, programmers forced to participate in camel races, and exorcisms performed after errors in programming with curses.) She was also the original author of one of our first books, Learning the Vi Editor, and later creator (with Nancy Keene) of our series of Patient-Centered Guides.

I hired Cathy Brennan (now Strider) as a receptionist in 1985 or 1986, but her common sense soon made her one of my most trusted advisers and her ability to get things done made her one of my senior executives. When I decided to move to California from Boston in 1987, Cathy was the one who persuaded me to move our then-fledgling publishing business along with me, and agreed to move out herself to set up customer service and operations for this new line of business. She built and ran our operations as we grew from a tiny startup to a publishing powerhouse. She also managed the design and construction of our office complex in Sebastopol.

Laura Baldwin joined the company as CFO in 2001, just as we were crashing into the wall of the dot com bust. I can with confidence say that we wouldn't have survived without her. I had always held up Harold, the last of the Saxon kings of England, as one of my management heroes. He went down to defeat by William the Conqueror rather than abandon his people to fight another day. Laura convinced me that I needed to do layoffs--and when I did, I found that the people we laid off moved on to what were often better jobs for them, and the company itself became leaner, more creative, and more effective. Laura brought financial discipline to the company. She helped bring us back from the brink, and built a bigger, more profitable, and more successful company. Living to fight another day helped us to birth the Web 2.0 movement, Foo Camp, Make: magazine, the Missing Manuals, and many of the other post-2001 products we are known for today.

Laura is now our COO as well as CFO, and is the day-to-day manager of the business. Those of you who wonder how I find so much time to spend on twitter while running a business, look no further!

I have learned more about the nuts and bolts of business from Laura than from any other person. Harold Geneen once said "The skill of management is to achieve your objectives through the efforts of others." Laura knows how to do that in spades - she's a great manager. But she's also the most amazingly productive person I've ever met. Usually, you have to choose between an effective manager and an effective individual contributor. Laura somehow manages to be both.

The list goes on. CJ Rayhill was our CIO for seven years, working with Laura to build the information systems that turned O'Reilly into a "real company." (She was also one of the first women to graduate from the Naval Academy - here's an interview about her history in the tech industry.) She's now the COO for Safari Books Online, where she's managing the extremely cool features that will be appearing in Safari 6.0 later this year.

Gina Blaber was the original managing editor for GNN, the world's first commercial website. She then ran our software group (remember Website Pro, the first PC-based web server?) and now runs the O'Reilly Conferences group. If you've ever been to an O'Reilly event, it's easy to think that the speakers and program chairs do all the work, forgetting that it's Gina and her team (mostly women) who make it all look so easy.

Laurie Petrycki ran several of our publishing divisions (notably Head First and Missing Manuals) before taking on the challenge last year of launching our new education division.

And that's leaving out the many women who've worked as editors, copyeditors, designers, and production staff at O'Reilly over the years. If you've ever read and enjoyed an O'Reilly book, take a moment to appreciate how many hands, how many eyes, read it before you did, to shape it into its final form.

Sara Winge is the creator of Foo Camp, one of O'Reilly's quirkiest and most influential initiatives. While Wikipedia claims me as a co-creator of the event, I have to say that it has always really been Sara's brainchild. I had wanted to do some kind of events at O'Reilly after the dot com bust left us with a lot of unused space, and I might have even proposed residential events, but Sara is the one who picked up this idea from the heap where we tend to leave good ideas that don't have anyone to make them real.

Sara conceived and developed the format (inspired in part by Open Space, she says, but to my mind, all the best parts were original.) I've just been the front-man and impresario. So if you've been to Foo Camp, or to Bar Camp, or any of the many other "Camp" spinoffs, you owe a big round of thanks to Sara. Foo Camp also demonstrates a uniquely feminine sensibility. Sara didn't charge to the front; she created a context where other people can shine, quietly facilitating. As Lao Tzu said, "When the best leader leads, the people say, 'We did it ourselves.'"

Edie Freedman is the creator of the distinctive O'Reilly animal brand. Many people know a bit about the story of how strange animals came to be the symbols of so many technologies, but what they probably don't know that behind this brand, so central to the company's heritage, was an act of generosity by a complete stranger.

Our first books, published late in 1985, all had the same simple cover, featuring the image of a nutshell. The idea was that these small books had everything you needed, in a nutshell. In 1987, with seven books in print, we realized that people at trade shows weren't recognizing that we had more than one book, so we hired a designer to produce some new covers. She developed a treatment that was colorful, geometric and high-tech. We had an all-hands meeting on a Friday afternoon to review the new cover treatment. I just couldn't go for it. It was too expected. I said we'd have to try again.

Linda Lamb shared our plight with her housemate, Edie, who at the time was a designer at Digital Equipment Corporation. Edie thought that Unix program names sounded like weird animals. She also realized that 19th century woodcuts provided a unique, low-cost design option. Fair enough. But she went further than that. She produced and laid out seven mechanicals of possible covers over the weekend. Linda brought them in on Monday, free of charge. Here's one of the original designs, for Sed & Awk, a book that didn't even exist yet.)

How cool is that? One of the great brands in tech was a gift from a stranger! (Edie did later come to work for us, and served as our Creative Director for many years. She is still with the company.) That is one of the many experiences that made me receptive to the idea of open source as a gift culture when Eric Raymond wrote about that in 1997.

(While I'm on the subject of the kindness of strangers, I have to call out the contributions of Danese Cooper and Linda Stone, two people who've never worked for O'Reilly but who might as well have done, for all the tireless work they do on our behalf. I met Danese through our mutual work on open source when she was the "open source diva" at Sun, and later at Intel. She's the person I always call when anyone asks me for advice on open source licensing, community, or corporate adoption. Linda Stone, former maven at Apple and Microsoft, is a mentor and inspiration on the value of making connections between people who ought to know each other. Linda is also the one who had the brilliant idea of Science Foo Camp when Timo Hannay of the Nature Publishing Group and I were scratching our heads trying to find a project to do together. She's also the one who suggested we ask Google to host the party. Never mind random acts of kindness; there's real power in random acts of connection. When I said a few years back that our purpose at Foo Camp is to create new synapses in the global brain, I was channeling what I learned from Linda.)

I could go on and on. There's Allison Randal, who Nat Torkington already wrote about in his own Ada Lovelace Day post earlier today. There's Kathy Sierra, the creator of the amazing Head First series of books, and also the subject of many another Ada Lovelace Day post. There's Carla Bayha, who for many years was the computer book buyer at Borders, and whose penetrating judgment helped books on many an obscure topic find space on shelves across America. Carla is truly an unsung hero of the industry.

And of course, I celebrate the many other women authors, conference speakers, and coders who've been a part of O'Reilly's story over the years. Truly, we could never have done it without you. As far as I'm concerned, it's always Ada Lovelace Day at O'Reilly.

The Paradox of Transparency

In his memo on transparency and open government, President Barack Obama said:

"My Administration is committed to creating an unprecedented level of openness in Government. We will work together to ensure the public trust and establish a system of transparency, public participation, and collaboration. Openness will strengthen our democracy and promote efficiency and effectiveness in Government."

Then reality intervened. Last week, the FBI raided the offices of Kundra's former employer, the Office of the CTO in Washington D.C., arresting a mid-level manager and the head of a long-time IT contracting firm who were involved in a bribery and kickback scheme. The White House promptly suspended Kundra, three days into his new job as Federal CIO, "out of an abundance of caution," despite the fact that the FBI made clear that Kundra was not a target of the investigation, and that the corrupt official in question, Yusuf Acar, had been working for the city since 2002, well before Kundra took on the job in 2007.

While Acar's frauds were not revealed by technological means, but by an old-fashioned whistleblower, they are exactly the kind of procurement shenanigans that Kundra set out to uncover in D.C. Without talking to CW (the "Cooperating Witness", as he was named in the FBI request for an arrest warrant), it's impossible to know what role Kundra's emphasis on cleaning up the DC procurement process played in encouraging the whistleblowing, but this conviction is certainly in line with Kundra's goals, as stated in his December 2007 testimony at the Public Roundtable on Theft and Fraud Prevention in District Government Agencies (pdf).

The paradox of transparency is that it may indeed reveal waste, fraud, and malfeasance, making things appear worse before they begin to get better. This is not something to be afraid of. It's a sign of success.

Nonetheless, the political atmosphere in Washington has grown so sensitive that the Obama administration initially felt the need to distance itself from Kundra, lest they be touched by even the faintest whiff of the D.C. scandal.

We need to make sure that the transparency mission is not going to be hijacked by political considerations. What cabinet secretary, what governor, what mayor, what IT manager in local government, what supplier will support the Federal transparency initiative if whatever is uncovered will have to be weighed against the risk that the other party will take advantage?

We need a bipartisan commitment to transparency. It's ridiculous to think that we won't turn up things that we don't like, but we need the message to be: we're all in this together. We need to make sure that transparency doesn't become a political weapon, or "out of an abundance of caution," we'll abandon the mission before it has a chance to succeed.

Fortunately, we saw news today that as of today, Kundra is back at his White House desk.

Clay Shirky's "Newspapers and Thinking the Unthinkable"

Sometimes Clay Shirky astounds us by articulating something we've never thought of, and sometimes he astounds us by telling us something many have thought, but never so clearly and so compellingly. But always, he astounds.

Into the first category falls the claim that he made in his keynote at the last Web 2.0 Expo that "the critical technology of the 20th century...was the sitcom." Who would have thought that so penetrating an analysis could hinge on such a preposterous assertion! (If you haven't already done so, read the transcript or watch the video.)

Yesterday's piece, Newspapers and thinking the unthinkable, falls into the second category. When I said the other day that "Twitter is the most minimal newspaper," or when I talked to the New York Times about rediscovering what is essential in what they do, I was speaking to this same topic, but like so many others, I was still framing the dialogue around "saving the newspaper." By contrast, Clay cuts the Gordian knot:

That is what real revolutions are like. The old stuff gets broken faster than the new stuff is put in its place....

And so it is today. When someone demands to be told how we can replace newspapers, they are really demanding to be told that we are not living through a revolution. They are demanding to be told that old systems won’t break before new systems are in place. They are demanding to be told that ancient social bargains aren’t in peril, that core institutions will be spared, that new methods of spreading information will improve previous practice rather than upending it. They are demanding to be lied to.

There are fewer and fewer people who can convincingly tell such a lie.

That being said, when I speak to this topic myself, I offer this hope: that while institutions may be overwhelmed by the tide of change, new institutions do arise. The deep needs that newspapers serve aren't going away. We will find new ways to serve those needs. As Clay says:

When we shift our attention from ’save newspapers’ to ’save society’, the imperative changes from ‘preserve the current institutions’ to ‘do whatever works.’ And what works today isn’t the same as what used to work.

And to be quite frank, we can already see the shape of that reinvention in specialized fields. In the mid-1990s, Michael Leeds, the CEO of CMP, in its day one of the titans of specialty computer newspapers, told me that if he couldn't get one of his papers to $50 million in revenue in 3 years, he would shut it down as not worth doing. Today, many of the papers he owned are gone, yet small firms like Techcrunch, Mashable, and ReadWriteWeb are successful (and doing at least as good a job of covering computer industry news) at an order of magnitude less revenue than CMP would once have thrown away.

Jobs that matter get done. But no one is guaranteed that their business as they conceive of it today will be preserved, especially at any given scale or profitability. So, have faith. The world as we know it is being broken. Now, let's get on with reinventing it!

The Social Nervous System Has More Than One Sense

Radar's Joshua Michele-Ross published a fabulous piece on Forbes entitled The Rise of the Social Nervous System. His premise:

...communication is the foundation of society, business and government. When you scale up communications, you change the world....As ever more people get connected, we see an acceleration in the way the Internet is used to coordinate action and render services from human input. We are witnessing the rise of a social nervous system.

Watch the news, and you will see daily evidence of how a system that connects billions of people is influencing the physical world--from recent protests in California against Proposition 8 organized by Facebook to the riots in my hometown of Oakland after several witnesses uploaded video taken from their mobile phones of a police shooting.

The one area where I disagree with Josh's analysis is in his dismissal of purely machine-mediated sensing.

It is easy to confuse this concept with the emerging field of machine learning such as the smart energy grid, traffic control using the sensor Web or the Planetary Skin Initiative recently announced by Nasa and Cisco. Machine optimization is useful but hardly social: Human beings do not contribute the data, share it or act upon it. And the implications of a social nervous system are far more profound than simply a "smart" grid.

But where do you draw the line? When we make a phone call from one location rather than another, we don't think we are contributing our location, but our phone is quietly doing so nonetheless. When we make a credit-card purchase, we don't think we are contributing, but software at the bank, the merchant, and our personal finance application is listening to that credit card reader. When we turn on a light switch in a Smart-grid connected house, we won't think we are contributing, but we will be. And the refrigerator waking up and deciding to turn on its compressor will be making exactly the same kind of contribution. The Smart Grid is in fact intended to be just such a sensing-and-responding system, connecting people and machines into a new kind of super-organism.

It's important to remember that even the human brain has more than one sense. Computers will have a rich new sensorium of their own, driving increasingly autonomic applications. Those applications will share and sense not just words passed from human to human across services like twitter, or our search behavior as captured in the database of intentions, but sounds, pictures, and increasingly, data from senses that unaided humans don't possess at all, or less precisely: a sense of precise location, or the rate of speed at which we move, the power we consume, the carbon we emit, the approaching weather, the state of the financial markets, the unique sequence of our genome, or even the way we smell. I'd bet some of the next great fortunes will be made by someone discovering how to build a system that reacts to one of the internet's new senses.

Still, Josh's analogy is a powerful reminder that "collective intelligence" is not cerebral, but ultimately becomes visceral, that it affects not just what we think but what we do. I expect that many of the "new" senses that currently appear to be merely mechanical will soon develop social dynamics of their own.

Bulk Data Downloads: A Breakthrough in Government Transparency

As those of you who follow my tweets know, I spent last week out in Washington, D.C. meeting with various folks, attending Transparency Camp, and giving a couple of talks. One of the more interesting meetings was with staffers from Congressman Mike Honda's office. He represents the area around San Jose, including much of Silicon Valley.

Honda sits on a very interesting subcommittee of Appropriations, the one responsible for the legislative branch, an enormously powerful place to be since they hold the purse strings for everything from how much money members get for office supplies to how much money employees get paid. Honda's staff told me about an interesting rider the subcommittee was working on which would require the agencies that the U.S. Congress to distribute their data in bulk.

John Wonderlich from the Sunlight Foundation wrote to me this morning to tell me the provision made it into the Omnibus Appropriations Bill. This is big news. Honda's staff told me that the Congressman had been working on this for a year. Here's a link to the appropriations bill. (I sure wish they gave us the ability to pull these things up in HTML, go directly to a bookmarked section, and use change control to see what has changed, but that's another post).

The money quote is this paragraph right here:

*Public Access to Legislative Data* - There is support for enhancing public access to legislative documents, bill status, summary information, and other legislative data through more direct methods such as bulk data downloads and other means of no-charge digital access to legislative databases. The Library of Congress, Congressional Research Service, and Government Printing Office and the appropriate entities of the House of Representatives are directed to prepare a report on the feasibility of providing advanced search capabilities. This report is to be provided to the Committees on Appropriations of the House and Senate within 120 days of the release of Legislative Information System 2.0.

Advanced search is great, and the Legislative Information System 2.0 thing sounds very good as well, but I was struck by the phrase "bulk data downloads and other means of no-charge digital access to legislative databases" and the specific reference to agencies. What would it mean if all the bulk data from the Library of Congress, Congressional Research Service, Government Printing Office, and "the appropriate entities of the House of Representatives" were made available? I asked Carl Malamud, who has worked with many of these databases, if this looked like something real or just another report.

Carl replied:

Wow! This is huge. The language only requires a report, but a report to an Appropriations subcommittee means a whole bunch, because if they don't like your report, you don't get money. (Appropriations was where the action occurred when we took on the Smithsonian over the Showtime deal. Once they cut the budget $28 million, they had their attention.) Here's what this means in practice:

• The Library of Congress sells a series of expensive bulk data products including the Copyright Database, card catalog information in XML, and what are known as "authority files" which are lists of names, subjects, and other classification headings so that all libraries can call things by consistent names. Even though the data is public, it is very expensive today. The Copyright Database, for example, costs $86,625 for the retrospective and a one-year feed (we harvested this in 2007 as you reported, but this would be much easier if they simply provided an FTP server and rsync!)
• The Government Printing Office sells the Official Journals of Government, which we've been working very hard on harvesting and purchasing. If we had $100,000, we would have bought one of each long ago. This stuff is the official record of the United States. Here's the full list of databases, including the Congressional Record, the Compilation of Presidential Documents, the United States Code, and much more. [Editorial note: The NY Times blog just did a piece earlier today about Carl's quest to reinvent the mission of the Government Printing Office for the 21st Century.]
• The Congressional Research Service is such a no-brainer. With the exception of classified information, who can afford the luxury of paying for some of the best research in the world and then just bury it! Taxpayer dollars paid for CRS reports and they need to be available. (More on this at the Sunlight Foundation.)
• Other Entities of the House is the most impressive clause in that whole paragraph. My reading is that this clause includes bulk access to broadcast quality video from every congressional hearing. And if it doesn't include that, I wish they'd make it clear in report language. In this day and age, you can't say a committee hearing is public if you can't access it on the Internet. Itty-bitty streaming video using some proprietary client/format just doesn't cut it any more. We ran a pilot with 4 house committees to show that this is very doable and makes a huge difference (check out the before and after shots on this video of Chad Hurley testifying before Congress,)

On video, I want to add one more note. Policy on what gets archived and distributed from a committee hearing is up to committee chairmen. It's a very decentralized system. So, if we're serious about putting broadcast quality video from congressional hearings on-line, the Legislative Branch Subcommittee of Appropriations would be a wonderful place to start. Happy to help if they need a hand!

And, if we're going to do video, there is one more administrative entity in the House that we should call out. The House Broadcast Studio has a huge archive of prior hearings. We asked Speaker Pelosi is we could run FedFlix on that archive and her staff sounded very supportive. (FedFlix is our program to help government agencies: they send us video, we digitize it, send it back to them. No cost to the government, more data for the public domain!) It would be great if the report from the House Broadcast Studio specifically dealt with how they're going to make their archive of several thousand hearings available as high-resolution, downloadable video. Again, happy to help if they need a hand.

Bottom line? This is really great if they can pull this off. Congratulations to Congressman Honda, as well as to the Sunlight Foundation which I know did some heavy lifting on this issue. (Sunlight has turned into a remarkably effective lobbyist in favor of transparency. They're outgunned by K-Street, but they're definitely holding their own!)

I'd love your thoughts on what government data should be made available, what formats it should be available in, and what you'd do with it if you had access to it. When I spoke with Congressman Honda's staff, they made clear that they'd love Silicon Valley's best ideas for other technological reforms that they can include in future legislation. When you've got a Congressman who's paying attention, that's a great opportunity! I'm fairly sure that the Congressman will be checking the comments on this post, so it's your chance to let him know what you think.

P.S. Rob Pierson, Congressman Honda's Online Communications Director, actually gave a Q&A; session at Transparency Camp in which he asked for ideas about how to redesign the Congressman's web site. He got lots of suggestions, including ways to incorporate twitter and facebook feedback, but I'm sure that there are many more ideas. So in addition to responding with ideas about the bulk data provision in the legislation, this is a great chance to give the Congressman feedback on how he can do a better job listening to you.

Update: While it isn't clear whether CRS reports would be covered under this provision, Senator Lieberman yesterday wrote a letter to the Senate Rules Committee Chairman asking for greater public access to CRS reports. With the new democratic majority in Congress and the White House, transparency measures are sprouting up everywhere.

Recent Posts

• Change Happens on February 14, 2009
• Stuff That Matters: Non-profit to For-profit on February 1, 2009
• Privatizing Success, Socializing Failure: FAIL on January 30, 2009
• Twitter Quote of the Day: 29 Jan 2009 on January 29, 2009
• Twitter Quote of the Day: 28 January 2009 on January 28, 2009
• Twitter Quote of the Day: 26 January 2009 on January 26, 2009
• Free The Facts: Critical Issue, Killer Presentation on January 26, 2009
• Competition in the eBook Market on January 25, 2009
• Quote of the Day from Twitter: 25 January 2009 on January 25, 2009
• Inauguration Moments and Links on January 20, 2009