How Hackers Show it's Not All Bad News at the New York Times

Andrew Savikas
July 30, 2008 | Permalink | Comments (3) | Listen

News of a looming downgrade of NYT stock to "junk" status by Standard & Poor's sadly isn't all that shocking. I'm certainly glad I'm not an investor holding any NYT.

But there's something going on at the Times that probably won't make it to Silicon Alley Insider, much less the mainstream business press, and it's something that's starting to make me think the Times just might succeed in adapting to the changing rules of the media and publishing game (though there will almost certainly be many more casualties before it's over).

So what's the Times doing that's so important? They're hacking.

Not hacking in the nefarious sense, but in the original sense of experimentation, and curiosity, and solving interesting problems (as Paul Graham put it, "Great hackers think of it as something they do for fun, and which they're delighted to find people will pay them for.") How many other publishers are running blogs about their work with open source software? Even fewer are developing and releasing their own high-quality open source software:

Quite frankly, we wanted to scale the front-end webservers and backend database servers separately without having to coordinate them. We also needed a way to flexibly reconfigure where our backend databases were located and which applications used them without resorting to tricks of DNS or other such "load-balancing" hacks. Plus, it just seemed really cool to have a JSON-speaking DB layer that all our scriptable content could talk to. Thus, the DBSlayer was born.

That is not typical newsroom conversation.

But this isn't just about open source software, or even about some developers building cool software to run backend system. The Times has put developers right in the middle of the newsroom. At a MediaBistro event in May, Aron Pilhofer from the "Interactive News Technology" group at the Times (sharing the stage with their Editor of Digital News, Jim Roberts), talked about how the Minnesota bridge collapse was when they realized they needed to develop their own tools to cover the news with the web, and not just on the web. Less than a year later, when Hillary Clinton's infamous public schedule was released, they had the people and the skills in place to crunch 12,000 PDF documents (containing images of scanned documents) through a text-recognition program, on to Amazon's "Elastic Computing Cloud" and finally into a Ruby on Rails Web application providing full-text search across all eight years of calendars.

Just this week, the Times' Derek Gottfrid gave a talk at O'Reilly's Open Source Convention (OSCON) titled "Processing Large Data with Hadoop and EC2" based on work he'd done on the Times' archives. Again, this is the kind of talk you're not likely to hear at most newspapers (or magazines, or book publishers) these days:

I was able to create a Hadoop cluster on my local machine and wrap my code with the proper Hadoop semantics. After a bit more tweaking and bug fixing, I was ready to deploy Hadoop and my code on a cluster of EC2 machines. For deployment, I created a custom AMI (Amazon Machine Image) for EC2 that was based on a Xen image from my desktop machine. Using some simple Python scripts and the boto library, I booted four EC2 instances of my custom AMI. I logged in, started Hadoop and submitted a test job to generate a couple thousands articles — and to my surprise it just worked.

Earlier this month at FOO Camp I had the pleasure of meeting another hacker from the Times, Nick Bilton, part of the Times R&D lab -- the folks who built the impressive NYT iPhone App.

UPDATE: Nick Bilton points out via email that:

There were people from nytimes.com that were instrumental in building the NYT iPhone app also ... Is there anyway you can add a couple of words that the R&D Group 'worked with nytimes.com' to help build the iPhone app?

If you're worried about EBITDA and EPS, then you're rightly worried about the Times right now. But if you're worried about the future of journalism, and about the ability of established media companies to adapt to a digital world, there's also reason to be excited about the Times right now too.

Related Stories:

• Silicon Alley Insider: "Our Offer For The New York Times Company"
• Guardian Blazes New Media Trail with paidContent.org Acquisition
• The Media Industry's Perspective Problem
• Lessons for Publishers in IDG's Digital Success

Tech Publisher Asks "Are Ebooks Ready for Technical Content?"

Andrew Savikas
July 29, 2008 | Permalink | Comments (4) | Listen

Dave Thomas from the Pragmatic Programmers is mulling whether to make their books available on the Kindle, and encountering many of the same issues we faced here at O'Reilly regarding technical content and the limitations of current ebook devices:

In fact, we've had a prototype form of that capability for a while now, but we've always held back. Frankly, we didn't think the devices worked well with our kind of content. Basically, the .mobi format used by the Kindle is optimized for books that contain just galleys of text with the occasional heading. Throw in tables, monospaced code listings, sidebars and the like, and things start to get messy.

Dave's post has sparked a great conversation within the comments, including one from Shelly Powers, whose book Painting the Web was among those included in our pilot program:

I think that providing the package deal that O'Reilly does (with PDF, epub, and mobi), in addition to downloadable code is the way to go. If you sell Kindle books, you definitely need to make both your figures and your source available, separately. For instance, I have my Painting the Web figures in an online gallery and the examples are available at O'Reilly--takes care of a lot of issues related to Kindle. Another approach could be to make available (for no additional cost) a PDF of just the figures, or the figures and code.

Preparing a book for the ebook market may seem like a lot of work, but you have the potential to reach a new audience of book buyers. Buyers used to the internet and having access to immediate information; who may not want to order a book and wait a week for it to arrive, but who will buy a book if it means they can have access to it now. I wouldn't have considered myself an "impulse buyer" when it comes to books, but I have probably at least a dozen books I bought because the ebook format was cheaper (that's a key element), and I could get the book _right now_.

On one hand, merely working to replicate a print experience isn't the right way to exploit the benefits of the new platform; on the other hand, publishers (and as usual, I use that term quite loosely) should be able to expect at least minimal rendering of common elements like tables, along with support for at least the same core 14 fonts available in Acrobat (speaking of fonts, if you're looking for a laugh check out this mock "font conference").

Related Stories:

• Sony Reader Now Supports EPUB and Digital Editions
• 30 O'Reilly Titles Now Available as Ebook Bundles
• Treating Ebooks Like Software
• AAP Passive-Aggressively Endorses EPUB

POD Opens Door to Magazine Experiments and Customization

Mac Slocum
July 28, 2008 | Permalink | Comments (0) | Listen

MagCloud is a new print-on-demand (POD) service targeting the magazine industry. In the following Q&A;, MagCloud consultant Derek Powazek -- co-founder of JPG Magazine and founder of Fray -- discusses the utility of POD and the evolving relationship between print and Web content.

How did you get involved with MagCloud?

I came into the project over a year ago -- it had been percolating in HP Labs for a long time before that, led by Andy Fitzhugh, Udi Chatow, and Andrew Bolwell. Andy is the one who brought me in. We had this meet and greet lunch to talk about the future of publishing and it turned out we had the same vision. He kept saying, "Right, now push that further."

When did you first encounter POD?

Years ago, when Heather [Champ] and I were exploring ways to make a photography magazine, Lulu was really the only game in town. We learned so much creating JPG there, and starting with a POD service allowed us to experiment, develop the voice and vision of the magazine, and build an audience. I think it's a very natural way to start a magazine.

How did you gravitate toward a POD model for magazines?

It's all about the Giant Pile. I've worked on a lot of newspaper and magazine projects, and they all had one thing in common: A huge print run, followed by the slow, terrible realization that you've gotta get rid of all that paper.

POD banishes the Giant Pile to the dustbin of history where it belongs. Because, with a POD system, you don't print it until somebody wants it. It avoids the pile. It avoids creating trash (70 percent of all magazines are never bought). It brings some of the elegance of the Internet to this very old industry.

But mostly it was just a financial decision. Heather and I weren't out to become publishing magnates. We just had an idea that we thought people would like. We wouldn't have been able to do it at all if not for POD.

What types of magazine publishers (large, small, individuals, etc.) are best suited for MagCloud?

I think that magazines are about nurturing a community. If you look at the most successful magazines (Rolling Stone in the '60s, Wired in the '90s, Make now), they've always been the ones that surfed the zeitgeist. They found a growing community of people and reflected it, and in that reflection, began to lead it for a time.

But if you tell people in the publishing industry that they're really in the community business, they'll say "shut up, hippy" and go back to monetizing their audience metrics.

So the trick is to find those niche audiences that need a voice. And there are a lot of them. And the truth is, they know who they are better than we do. So, with MagCloud, the idea is to open up the tools so that those communities can create their own magazines. We think they're going to make amazing things.

Do you see larger magazine publishers eventually moving to POD, or will this be a niche option?

Not only do I think that large magazine publishers will move to digital printing, but I think that the idea that we used to print millions of things that were exactly the same will someday be seen as a cute historical artifact. "You mean every copy of this magazine was the same for everyone, Grandpa? Weird!"

For the biggies, it's just a matter of economics. As soon as the price per page for printing on digital is cheaper than traditional offset printing, the biggies will move. The quality of POD is already the same or better than offset.

It'll start with smaller publications because they're the most agile, and they don't see the real price savings of scale anyway. Right now, if you're printing a few thousand copies, digital printing is the same cost as traditional offset. (I've been wrestling with this for Fray.com -- we're right at the cusp. Our first issue was printed via traditional offset, but issue two will be printed with MagCloud.)

And once magazines move to POD, they'll realize it opens up opportunities they never had before. When you can really tailor each issue for each subscriber, what will you do? Exciting, huh?

Book publishers often focus on the short-term elements of POD, most notably POD's higher cost per page. Some industry folks try to cite the long-range benefits, such as efficiency, higher retail prices via customization, etc., but the per-page discrepancy continues to be a sticking point. Have you encountered similar obstacles on the magazine side?

Magazines are a better fit for POD because, unlike books, they're usually all color and timeliness is much more of a factor. Plus, the price per page for digital print is falling fast, while the price per page of traditional offset has remained very steady. Still, the exciting part is all the opportunities digital printing enables. Ultimately, POD services like MagCloud will enable a degree of customization that is not only cheaper, but just plain impossible to do via traditional means.

Beyond strict numbers, what do you see as the upside to print editions? Does a print product carry a higher level of esteem for a writer or consumer?

I love the Web. I think it's still a publisher's dream come true. But, inconveniently, we humans are still real world creatures. And no matter how much connectivity blankets the planet, and how good our devices get, there will still be a role for print.

I don't say this because I'm some ancient technology fetishist. I don't own a tube amp. I sold all my CDs. It's just that print is a really good delivery mechanism for some kinds of experiences. Reading a physical magazine is a different experience than surfing hypertext online.

And, yes, I think the scarcity of print does give it a higher level of importance for its creators and consumers. On the Web, where every page is just a click away from any other, there's no relative importance communicated. But in a magazine, you know that a team of writers and editors picked this story to go here. That has a profound effect on how that media is consumed.

Related Stories:

• HP Indigo On-Demand Digital Presses
• Open Question: What is the Best Use for Print on Demand?
• Magazine POD Service Looks to Help Publishers Experiment
• Time Inc. Prepping Mix and Match Magazine Service
• Budding Authors Use Espresso Book Machine to Publish

The Media Industry's Perspective Problem

Mac Slocum
July 23, 2008 | Permalink | Comments (0) | Listen

A newsroom survey conducted by the Pew Research Center's Project for Excellence in Journalism touches on one of the major issues -- and failings -- affecting mainstream media: the power of flawed perspective. Here's an excerpt from "The Changing Newsroom" report:

Staffing for coverage of sports, local government and politics, police and investigative reporting, all grew in 30% of the newsrooms surveyed. Although not specifically measured in the survey, anecdotal evidence suggests that at least some of these gains have been driven by pressure to provide web content during the course of the day. Some of this content is often then "reversed published" back into the newspaper. [Emphasis added.]

There's a huge difference between "published" and "reversed published." A published piece of content -- be it an article, a podcast, a broadcast, or even a book -- is pushed into the world with a clear intent (inform, entertain, influence, etc.). But reversed published content has been stripped of intent. Its sole purpose is to fill space; whether it entertains, informs, or influences is secondary.

The whole concept of "reversed published," and the adjacent issues of print vs Web vs mobile vs broadcast, illustrates a fundamental flaw in the media perspective. Content should be defined by its audience, not by its container. If an article is initially published on the Web, that article must be geared toward the Web audience. If the same material later appears in the paper, that material needs to be geared toward the newspaper audience. Same goes for mobile consumers and broadcast consumers.

Repurposing material without regard for its audience is a luxury the media industry used to enjoy when it was a primary information conduit. The only difference is that years ago the Web was where rehashed shovelware was dumped ("Story continues on A12", anyone?). Early Web users quickly tired of media's detritus, so they looked elsewhere for useful information. Apparently, media organizations didn't learn from this past mistake because now they're pulling the "repurposed content" maneuver with traditional audiences. No one wants rehashed bits.

This is where perspective comes in. If a media organization continues to think in terms of content containers rather than content consumers, then it will inevitably default to "reverse publishing" and other bad habits. These days, as audiences scatter and company valuations plummet, every piece of content needs the justifications and intentions of fully published material.

Related Stories:

• Full Pew Report: "The Changing Newsroom"
• Guardian Blazes New Media Trail with paidContent.org Acquisition
• Lessons for Publishers in IDG's Digital Success
• Acknowledge and Move On: A Useful Debate Format
• Lamenting the Digital Decline is a Dangerous Path

Cloud Computing's Potential Impact on Publishing

Mac Slocum
July 22, 2008 | Permalink | Comments (0) | Listen

If you use Google Docs or access email via a Web browser, you're already versed in cloud computing. Access to Web-based material is taking the place of downloads.

Cloud computing focused in the early going on software as a service (SaaS) applications, but Amazon, Netflix, Google, Apple, Microsoft and others are now tapping the cloud for content delivery (some of these companies focus on streaming entertainment, while others focus on content creation/management).

An interesting conversation about the cloud's impact on content publishers popped up recently on Peter Brantley's Reading 2.0 list. Peter, by way of an an article link, noted that Amazon is moving some of its video distribution business into the cloud. From Last100:

Not only is Amazon utilizing streaming in order to deliver "instant" playback but it also means that content doesn't have to be permanently stored on a user's hard drive. As a result, Amazon is able to offer another potential benefit to customers: a virtual video library of previously purchased content, stored in the 'cloud' (on the company's own servers) ready to be streamed as many times and to as many compatible devices as the user has access to. While this will initially consist of PCs running Mac OSX or Windows, along with select TVs from Sony, in the future this could extend to many different devices, either through specific partnerships like the one currently forged with Sony, or by utilizing browser-based standards or any other technology or protocol Amazon chooses to support.

Expanding on Peter's post, Mike Shatzkin said the centralization of cloud-based content raises issues around digital rights management (DRM) and other access limits:

The cloud changes everything in terms of piracy and copyright. We are living in a transitional period where computer storage is decentralized. When that period is over, and the time is now not far off, everything is accessed from the cloud and it will be a relatively easy matter for rules about content access to be enforced by the content originator or distributor.

As others on the Reading 2.0 list pointed out, cloud computing brings up additional questions around copyright and ownership. Toss in concerns about system reliability, open vs. closed clouds, and the potential for lock-in (or lock out) and you can see this rabbit hole growing deeper.

Cloud adoption may also represent an important moment in book publishing's digital transition. Publishers have enjoyed the past luxury of learning digital lessons from the media, music and film industries, but the wait and see approach may not work this time. If consumers come to expect access to their content -- all their content -- anywhere/anytime, publishers will need to meet that expectation ... or risk watching an unaffiliated company or industry step in.

Related Stories:

• HowStuffWorks: "How the Google-Apple Cloud Computer Will Work"
• Wired: "Cloud Computing. Available at Amazon.com Today"
• Rough Type: "The cloud's not-so-silver lining"
• The battle for the cloud
• Open Question: How Can Ebooks Improve the Reading Experience?

Open Question: Should Publishers Develop Software Apps?

Mac Slocum
July 21, 2008 | Permalink | Comments (6) | Listen

Book publishing's response (or lack thereof) to the iPhone 3G and the App Store has stirred up an interesting question around publishing and software development: namely, should publishers create their own software applications?

Sara Lloyd from thedigitalist says a focus on content, not software, is key:

Interestingly the price of apps [in Apple's store] is already plummeting as free apps get more highly and more frequently rated and the paid-for apps drop down the ratings. Perhaps this suggests even more strongly that the App is not The Thing; it is merely a container or a channel for the content, which will still be The Thing.

On the other side, James Bridle from booktwo.org says publishers are the natural source for e-reader apps:

Most ereader technologies are built by techies who put the technology before the reading experience: the combined skills of typesetters, print designers, editors and technologists that only publishers possess could, with the right direction, produce a far superior ereader app than any we've seen so far.

What's your take? Should book publishers move into the software domain? Please post your thoughts in the comments area.

Survey of Book Industry Reaction to New iPhone and App Store

Mac Slocum
July 17, 2008 | Permalink | Comments (0) | Listen

Kassia Krozser struck a nerve earlier this week with criticism of the publishing industry's slow approach to the new iPhone and the just-opened App Store. From Booksquare:

Call me crazy, but I'd expect an industry that salivates over moving 150,000 units to be all over the potential for reaching seven million "mobile is the future" customers. Are you not out there, listening to readers, gauging their interest? They want, you have, and you're still hiding the goods. I get this isn't the largest market you have, but is that an excuse to sit on the sidelines?

Sara Lloyd doesn't see long-term value in this current burst of iPhone excitement. From thedigitalist:

... apart from a few digital PR points scored against competing publishers, there doesn't seem to me to be any huge value in first mover advantage here for publishers, unless we want to make the decision to become software developers. The perception is that the App Store has 'opened up' the iPhone to publishers and to e-reading. The reality is that the iPhone has always been enabled for e-reading ... So, whilst we have been awaiting the launch of the App Store with interest, we didn't see enormous advantage in, for example, creating a reading app ourselves or Being There on Day One, just for the sake of it.

Expanding on the software theme, James Bridle says book publishers are uniquely positioned to develop ebook applications that meet consumer needs. From booktwo.org:

... who better than publishers to craft such software? Most ereader technologies are built by techies who put the technology before the reading experience: the combined skills of typesetters, print designers, editors and technologists that only publishers possess could, with the right direction, produce a far superior ereader app than any we've seen so far.

Broadening the analysis, Michael Cairns says the "silo" mentality displayed in this iPhone debate is a competitive obstacle that needs to be put aside. From PersonaNonData:

To bring us back to the iPhone circumstance, as long as publishers continue to think in terms of traditional functional silos and roles and responsibilities they limit themselves in their ability to leverage their assets. In contrast witness Amazon which has never considered any aspect of the publishing value chain to be off limits and more publishers need to think in this manner if they want to redress some of the advantages Amazon and others retain (or new competitors develop) in the marketplace.

(Many of the links and call-outs in this post were provided by Peter Brantley via his Read 20 list.)

Additional Coverage & Related Stories:

• Exact Editions: "Dozing at the Wheel"
• Teleread: "The iPhone excitement and e-books..."
• Ebooks Abound in New iPhone Apps Store
• iPhone as Ultimate Reader? Maybe. iPhone as Ultimate Mobile Browser? Definitely.
• Publishing Frontier: "ebooks and the iPhone"

"Lost" Builds Community through Book Club and Web Games

Mac Slocum
July 14, 2008 | Permalink | Comments (0) | Listen

Producers of ABC's "Lost" often sneak books into the fabric of episodes so die-hard fans can hunt for clues (or red herrings) in external literary sources. Seeing an opportunity, ABC is launching the official "Lost Book Club" through ABC.com and iTunes. From UPI:

Also available on ABC.com will be a message board to discuss the titles, a synopsis of each book, along with when and how it was referenced in the show, and an introduction by co-creator/executive producer Damon Lindelof and executive producer Carlton Cuse, ABC said.

Two years ago, Hyperion published Bad Twin, a book "written" by one of the passengers on "Lost's" ill-fated flight Oceanic 815 (if you're a fan of the show, you'll recognize the author as the guy who got sucked into the engine moments after 815 crashed).

Response to Bad Twin was tepid, but the universe beyond "Lost" episodes has been successfully mined through a number intricate alternate reality games that reveal clues about the show's secondary mysteries. Speaking as a full-fledged "Lost" junkie myself, I know of a number of folks who spent dozens of hours playing these games.

Book publishers with mythology-laden source material may want to take a note from "Lost," "Harry Potter," "Star Wars" and other series. These franchises create organic affinity communities that thrive on interactivity and story expansion, and they can be fostered through forums, social networks, and real-world meetups at related events. Outside observers and casual viewers may not understand the impulse to dress like Boba Fett or write "Lost" fan fiction, but the ardent enthusiasm of a dedicated community presents opportunities that should not be tossed off.

(Via Publishers Weekly)

Related Stories:

• Storytelling 2.0: Alternate Reality Games
• Web Community Lessons from Stephen Colbert
• Opportunities in Book Publishing and Web Communities
• Q&A; with WEbook President Sue Heilbronner
• Nabokov's Final Novel: The Perfect Mash-Up Source

Open Question: Do You Use Twitter?

Mac Slocum
July 1, 2008 | Permalink | Comments (4) | Listen

Mediabistro recently conducted an informal round-up of publishers and authors who use Twitter to publicize titles and interact with readers. Within TOC, we use Twitter (plug: follow us here) to exchange quick bursts of information and story ideas, and we've also found it to be a surprisingly effective beat coverage tool -- breaking stories and new memes often appear on Twitter before they hit the blogosphere and mainstream media outlets.

This anecdotal evidence suggests Twitter is gaining steam in the publishing world, but is that really the case? Are you using Twitter? Have you even heard of Twitter? Please share your thoughts in the comment area.