How Hackers Show it's Not All Bad News at the New York Times

Andrew Savikas
July 30, 2008 | Permalink | Comments (3) | Listen

News of a looming downgrade of NYT stock to "junk" status by Standard & Poor's sadly isn't all that shocking. I'm certainly glad I'm not an investor holding any NYT.

But there's something going on at the Times that probably won't make it to Silicon Alley Insider, much less the mainstream business press, and it's something that's starting to make me think the Times just might succeed in adapting to the changing rules of the media and publishing game (though there will almost certainly be many more casualties before it's over).

So what's the Times doing that's so important? They're hacking.

Not hacking in the nefarious sense, but in the original sense of experimentation, and curiosity, and solving interesting problems (as Paul Graham put it, "Great hackers think of it as something they do for fun, and which they're delighted to find people will pay them for.") How many other publishers are running blogs about their work with open source software? Even fewer are developing and releasing their own high-quality open source software:

Quite frankly, we wanted to scale the front-end webservers and backend database servers separately without having to coordinate them. We also needed a way to flexibly reconfigure where our backend databases were located and which applications used them without resorting to tricks of DNS or other such "load-balancing" hacks. Plus, it just seemed really cool to have a JSON-speaking DB layer that all our scriptable content could talk to. Thus, the DBSlayer was born.

That is not typical newsroom conversation.

But this isn't just about open source software, or even about some developers building cool software to run backend system. The Times has put developers right in the middle of the newsroom. At a MediaBistro event in May, Aron Pilhofer from the "Interactive News Technology" group at the Times (sharing the stage with their Editor of Digital News, Jim Roberts), talked about how the Minnesota bridge collapse was when they realized they needed to develop their own tools to cover the news with the web, and not just on the web. Less than a year later, when Hillary Clinton's infamous public schedule was released, they had the people and the skills in place to crunch 12,000 PDF documents (containing images of scanned documents) through a text-recognition program, on to Amazon's "Elastic Computing Cloud" and finally into a Ruby on Rails Web application providing full-text search across all eight years of calendars.

Just this week, the Times' Derek Gottfrid gave a talk at O'Reilly's Open Source Convention (OSCON) titled "Processing Large Data with Hadoop and EC2" based on work he'd done on the Times' archives. Again, this is the kind of talk you're not likely to hear at most newspapers (or magazines, or book publishers) these days:

I was able to create a Hadoop cluster on my local machine and wrap my code with the proper Hadoop semantics. After a bit more tweaking and bug fixing, I was ready to deploy Hadoop and my code on a cluster of EC2 machines. For deployment, I created a custom AMI (Amazon Machine Image) for EC2 that was based on a Xen image from my desktop machine. Using some simple Python scripts and the boto library, I booted four EC2 instances of my custom AMI. I logged in, started Hadoop and submitted a test job to generate a couple thousands articles — and to my surprise it just worked.

Earlier this month at FOO Camp I had the pleasure of meeting another hacker from the Times, Nick Bilton, part of the Times R&D lab -- the folks who built the impressive NYT iPhone App.

UPDATE: Nick Bilton points out via email that:

There were people from nytimes.com that were instrumental in building the NYT iPhone app also ... Is there anyway you can add a couple of words that the R&D Group 'worked with nytimes.com' to help build the iPhone app?

If you're worried about EBITDA and EPS, then you're rightly worried about the Times right now. But if you're worried about the future of journalism, and about the ability of established media companies to adapt to a digital world, there's also reason to be excited about the Times right now too.

Related Stories:

• Silicon Alley Insider: "Our Offer For The New York Times Company"
• Guardian Blazes New Media Trail with paidContent.org Acquisition
• The Media Industry's Perspective Problem
• Lessons for Publishers in IDG's Digital Success

Technology's "Killer" Distraction

Mac Slocum
July 29, 2008 | Permalink | Comments (3) | Listen

A new search engine, Cuil, is attracting the requisite "Google killer" coverage. Thankfully, Seth Godin provides some much-needed perspective:

I have no doubt that someone will develop a useful tool one day that takes time and attention away from Google, but it won't be a search engine. Google, after all, isn't broken, not in terms of solving the iconic "how do I find something online using my web browser" question.

I have no beef with Cuil itself (the handful of queries I ran worked fine), but this "killer" business is another matter. In the history of tech prognostications, has an upstart killer ever successfully terminated its target? More importantly, what possible benefit do any of us get from this type of analysis?

I can only imagine the useful commentary we would see if the killer oeuvre could be stricken from the record. The bombastic flavor-of-the-day cycle might be replaced with actual thoughts about the future of particular applications and their accompanying industries. Perhaps we'd even stop shoehorning lightning-in-a-bottle success stories into unrelated products (e.g. the Kindle/iPod comparisons). And maybe we'd finally see that the exciting developments -- the products and experiments that really stir things up -- come from people who focus on creation rather than dominance.

As Seth eloquently notes:

... success keeps going to people who build new icons, not to those that seek to replace the most successful existing ones.

Related Stories:

• Lamenting the Digital Decline is a Dangerous Path
• The Pitfalls of Publishing's E-Reader Guessing Game
• Calling Google a Publisher Underestimates its Platform

The Media Industry's Perspective Problem

Mac Slocum
July 23, 2008 | Permalink | Comments (0) | Listen

A newsroom survey conducted by the Pew Research Center's Project for Excellence in Journalism touches on one of the major issues -- and failings -- affecting mainstream media: the power of flawed perspective. Here's an excerpt from "The Changing Newsroom" report:

Staffing for coverage of sports, local government and politics, police and investigative reporting, all grew in 30% of the newsrooms surveyed. Although not specifically measured in the survey, anecdotal evidence suggests that at least some of these gains have been driven by pressure to provide web content during the course of the day. Some of this content is often then "reversed published" back into the newspaper. [Emphasis added.]

There's a huge difference between "published" and "reversed published." A published piece of content -- be it an article, a podcast, a broadcast, or even a book -- is pushed into the world with a clear intent (inform, entertain, influence, etc.). But reversed published content has been stripped of intent. Its sole purpose is to fill space; whether it entertains, informs, or influences is secondary.

The whole concept of "reversed published," and the adjacent issues of print vs Web vs mobile vs broadcast, illustrates a fundamental flaw in the media perspective. Content should be defined by its audience, not by its container. If an article is initially published on the Web, that article must be geared toward the Web audience. If the same material later appears in the paper, that material needs to be geared toward the newspaper audience. Same goes for mobile consumers and broadcast consumers.

Repurposing material without regard for its audience is a luxury the media industry used to enjoy when it was a primary information conduit. The only difference is that years ago the Web was where rehashed shovelware was dumped ("Story continues on A12", anyone?). Early Web users quickly tired of media's detritus, so they looked elsewhere for useful information. Apparently, media organizations didn't learn from this past mistake because now they're pulling the "repurposed content" maneuver with traditional audiences. No one wants rehashed bits.

This is where perspective comes in. If a media organization continues to think in terms of content containers rather than content consumers, then it will inevitably default to "reverse publishing" and other bad habits. These days, as audiences scatter and company valuations plummet, every piece of content needs the justifications and intentions of fully published material.

Related Stories:

• Full Pew Report: "The Changing Newsroom"
• Guardian Blazes New Media Trail with paidContent.org Acquisition
• Lessons for Publishers in IDG's Digital Success
• Acknowledge and Move On: A Useful Debate Format
• Lamenting the Digital Decline is a Dangerous Path

Cloud Computing's Potential Impact on Publishing

Mac Slocum
July 22, 2008 | Permalink | Comments (0) | Listen

If you use Google Docs or access email via a Web browser, you're already versed in cloud computing. Access to Web-based material is taking the place of downloads.

Cloud computing focused in the early going on software as a service (SaaS) applications, but Amazon, Netflix, Google, Apple, Microsoft and others are now tapping the cloud for content delivery (some of these companies focus on streaming entertainment, while others focus on content creation/management).

An interesting conversation about the cloud's impact on content publishers popped up recently on Peter Brantley's Reading 2.0 list. Peter, by way of an an article link, noted that Amazon is moving some of its video distribution business into the cloud. From Last100:

Not only is Amazon utilizing streaming in order to deliver "instant" playback but it also means that content doesn't have to be permanently stored on a user's hard drive. As a result, Amazon is able to offer another potential benefit to customers: a virtual video library of previously purchased content, stored in the 'cloud' (on the company's own servers) ready to be streamed as many times and to as many compatible devices as the user has access to. While this will initially consist of PCs running Mac OSX or Windows, along with select TVs from Sony, in the future this could extend to many different devices, either through specific partnerships like the one currently forged with Sony, or by utilizing browser-based standards or any other technology or protocol Amazon chooses to support.

Expanding on Peter's post, Mike Shatzkin said the centralization of cloud-based content raises issues around digital rights management (DRM) and other access limits:

The cloud changes everything in terms of piracy and copyright. We are living in a transitional period where computer storage is decentralized. When that period is over, and the time is now not far off, everything is accessed from the cloud and it will be a relatively easy matter for rules about content access to be enforced by the content originator or distributor.

As others on the Reading 2.0 list pointed out, cloud computing brings up additional questions around copyright and ownership. Toss in concerns about system reliability, open vs. closed clouds, and the potential for lock-in (or lock out) and you can see this rabbit hole growing deeper.

Cloud adoption may also represent an important moment in book publishing's digital transition. Publishers have enjoyed the past luxury of learning digital lessons from the media, music and film industries, but the wait and see approach may not work this time. If consumers come to expect access to their content -- all their content -- anywhere/anytime, publishers will need to meet that expectation ... or risk watching an unaffiliated company or industry step in.

Related Stories:

• HowStuffWorks: "How the Google-Apple Cloud Computer Will Work"
• Wired: "Cloud Computing. Available at Amazon.com Today"
• Rough Type: "The cloud's not-so-silver lining"
• The battle for the cloud
• Open Question: How Can Ebooks Improve the Reading Experience?

Open Question: Should Publishers Develop Software Apps?

Mac Slocum
July 21, 2008 | Permalink | Comments (6) | Listen

Book publishing's response (or lack thereof) to the iPhone 3G and the App Store has stirred up an interesting question around publishing and software development: namely, should publishers create their own software applications?

Sara Lloyd from thedigitalist says a focus on content, not software, is key:

Interestingly the price of apps [in Apple's store] is already plummeting as free apps get more highly and more frequently rated and the paid-for apps drop down the ratings. Perhaps this suggests even more strongly that the App is not The Thing; it is merely a container or a channel for the content, which will still be The Thing.

On the other side, James Bridle from booktwo.org says publishers are the natural source for e-reader apps:

Most ereader technologies are built by techies who put the technology before the reading experience: the combined skills of typesetters, print designers, editors and technologists that only publishers possess could, with the right direction, produce a far superior ereader app than any we've seen so far.

What's your take? Should book publishers move into the software domain? Please post your thoughts in the comments area.

Survey of Book Industry Reaction to New iPhone and App Store

Mac Slocum
July 17, 2008 | Permalink | Comments (0) | Listen

Kassia Krozser struck a nerve earlier this week with criticism of the publishing industry's slow approach to the new iPhone and the just-opened App Store. From Booksquare:

Call me crazy, but I'd expect an industry that salivates over moving 150,000 units to be all over the potential for reaching seven million "mobile is the future" customers. Are you not out there, listening to readers, gauging their interest? They want, you have, and you're still hiding the goods. I get this isn't the largest market you have, but is that an excuse to sit on the sidelines?

Sara Lloyd doesn't see long-term value in this current burst of iPhone excitement. From thedigitalist:

... apart from a few digital PR points scored against competing publishers, there doesn't seem to me to be any huge value in first mover advantage here for publishers, unless we want to make the decision to become software developers. The perception is that the App Store has 'opened up' the iPhone to publishers and to e-reading. The reality is that the iPhone has always been enabled for e-reading ... So, whilst we have been awaiting the launch of the App Store with interest, we didn't see enormous advantage in, for example, creating a reading app ourselves or Being There on Day One, just for the sake of it.

Expanding on the software theme, James Bridle says book publishers are uniquely positioned to develop ebook applications that meet consumer needs. From booktwo.org:

... who better than publishers to craft such software? Most ereader technologies are built by techies who put the technology before the reading experience: the combined skills of typesetters, print designers, editors and technologists that only publishers possess could, with the right direction, produce a far superior ereader app than any we've seen so far.

Broadening the analysis, Michael Cairns says the "silo" mentality displayed in this iPhone debate is a competitive obstacle that needs to be put aside. From PersonaNonData:

To bring us back to the iPhone circumstance, as long as publishers continue to think in terms of traditional functional silos and roles and responsibilities they limit themselves in their ability to leverage their assets. In contrast witness Amazon which has never considered any aspect of the publishing value chain to be off limits and more publishers need to think in this manner if they want to redress some of the advantages Amazon and others retain (or new competitors develop) in the marketplace.

(Many of the links and call-outs in this post were provided by Peter Brantley via his Read 20 list.)

Additional Coverage & Related Stories:

• Exact Editions: "Dozing at the Wheel"
• Teleread: "The iPhone excitement and e-books..."
• Ebooks Abound in New iPhone Apps Store
• iPhone as Ultimate Reader? Maybe. iPhone as Ultimate Mobile Browser? Definitely.
• Publishing Frontier: "ebooks and the iPhone"

Ebooks Abound in New iPhone Apps Store

Andrew Savikas
July 10, 2008 | Permalink | Comments (2) | Listen