Google Book Settlement Round 2

Don't Hold Your Breath

The US government filed its Statement of Interest regarding the revised Google settlement yesterday with the District Court in New York. While the statement was signed by an attorney from the Antitrust Division of the Justice Department, several agencies including the Copyright Office reportedly contributed to it.

As you may recall, the judge has only 2 choices: he can approve the settlement, or send it back to the parties for revision. He cannot modify it himself.

The US government statement advises the judge that the public interest would be best served by sending the settlement back, and points out that the revised version still suffers from the "same core problem" that afflicted the first version: "an attempt to use the class action mechanism to implement forward-looking business arrangements that go far beyond the dispute before the Court in this litigation."

The press reports that I've seen take the government's statement as an emphatic thumbs down.

The judge has scheduled a hearing for February 18 in his Manhattan courtroom.

It is very unlikely that the judge will approve this version of the settlement. Also, he may once again decide to postpone a full-fledged fairness hearing-although the many objectors, large and small, are eager to have their day in court. Because the parties withdrew the proposed settlement before the originally scheduled fairness hearing occurred in October 2009, the judge has not yet formally considered the many objections filed to date on the revised settlement or those filed in anticipation of the fairness hearing cancelled last October.

Bottom line for the long term: even if the judge sends the settlement back, and even if the parties agree to deadlines as short as the deadlines for this presumably ill-fated revision, there is no resolution in sight for the litigation.

Whether the case is tried or the settlement discussions continue, the legal end point will not be the trial judgment or settlement approval issued by the district court judge. The end point will be the disposition of the final appeal from that district court judgment or approved settlement, and that disposition is years away.

It's hard to imagine what relevance the final legal disposition would have then, as public and private innovators are not sitting on their hands, waiting for the judge to sort this out.

Feedback and analysis: the missing ingredients in local's recipe

Access to local information is great, but context is even better

There's plenty of enthusiasm for local / hyperlocal projects, but the sweepstakes has yet to be won. PaperG CEO Victor Wong digs in to some of the missed opportunities in a paidContent.org guest column.

I found this excerpt intriguing:

How useful would it be to know when local used-car dealerships have a large increase in inventory (and thus are probably more willing to sell at a lower price)? Other data like new-car listings could show what the local population is buying by examining what is posted and taken down by the dealers. Publishers can even create new content by encouraging users to input data about what sorts of deals and treatment they got, which would be useful for other local buyers and could be turned into a local car-buying guide.

Wong has a stake in the local game -- PaperG focuses on local advertising -- but that doesn't diminish the point he alludes to in the excerpt: feedback and analysis are the missing parameters in the local equation.

So many of these local efforts rely on traditional information delivery through news articles or databases. That material has use, no doubt. Yet few projects take the extra step and put that data into context. They don't explain why the information is important. They don't connect the dots.

A lot of this reminds me of web analytics. It's easy to grant access to traffic data, and the access itself has a low level of value. But the insight that guides decisions comes from deeper analysis. You need to know why a particular keyword or topic is resonating.

Four short links: 8 February 2010

Kindle SDK, Javascript eBook Reader, Peer Review Review, eBook Moments

1. Kindle Development Kit APIs -- Amazon will release a Kindle SDK. These are the API docs. (via obra on Twitter)
2. rePublish -- all-Javascript ebook reader. (via kellan on Twitter)
3. Peer Review: What's it Good For? (Cameron Neylon) -- harsh and honest review of peer review with some important questions for the future of science. But there is perhaps an even more important procedural issue around peer review. Whatever value it might have we largely throw away. Few journals make referee’s reports available, virtually none track the changes made in response to referee’s comments enabling a reader to make their own judgement as to whether a paper was improved or made worse. Referees get no public credit for good work, and no public opprobrium for poor or even malicious work. And in most cases a paper rejected from one journal starts completely afresh when submitted to a new journal, the work of the previous referees simply thrown out of the window. Some lessons in here for social software, too.
4. Analog IMDB -- The transition is moving slowly, but it’s moving. It’s a fascinating thing to watch. The technology is the dull part: what’s interesting is the shift in perception. You know how sometimes you turn off a certain section of your brain and force yourself to see a word not as a piece of language with meaning, but as a sequence of black shapes and white spaces? It’s like you’re seeing that image for the very first time and suddenly “bird” seems like a very odd thing. I’ve been buying all of my in-print books electronically for a couple of years. Physical books aren’t weird to me yet. But damn, that old copy of the Maltin guide was a freaky and bizarre object. It’s the first time I looked at a book and didn’t see a container for information. I saw dead wood.

Four short links: 5 February 2010

Public Domain, Science Code, Bad Crypto, Javascript Grids

1. The Public Domain Manifesto -- eloquent argument in favour of the public domain. (via BoingBoing)
2. Clear Climate Code -- project to write and maintain software for climate science, with an emphasis on clarity and correctness. What a wonderful way for coders who aren't scientists to contribute to open and better science. (via the interesting OKFN blog)
3. Don't Hash Secrets -- One area of secure protocol development that seems to consistently yield poor design choices is the use of hash functions. What I’m going to say is not 100% correct, but it is on the conservative side of correct, so if you follow the rule, you (probably) can’t go wrong. You might be considered overly paranoid, but as they say, just because you’re paranoid doesn’t mean they’re not after you. So here it is: Don’t hash secrets. Never. No, sorry, I know you think your case is special but it’s not. No. Stop it. Just don’t do it. You’re making the cryptographers cry.
4. Javascript Grid Editors -- nice wrapup of available Javascript editable grid components, divided into "data driven", "light edit", and "spreadsheet". (via joshua on Delicious)

One hundred eighty degrees of freedom: signs of how open platforms are spreading

I was talking recently with Bob Frankston, who has a distinguished history in computing that goes back to work on Multics, VisiCalc, and Lotus Notes. We were discussing some of the dreams of the Internet visionaries, such as total decentralization (no mobile-system walls, no DNS) and bandwidth too cheap to meter. While these seem impossibly far off, I realized that computing and networking have come a long way already, making things normal that not too far in the past would have seemed utopian.

(continue reading)

Apple vs. Adobe vs. Content Creators

Lack of Flash support on the iPad could undermine publisher's tablet ideas

Remember when Wired's fancy tablet demo made the rounds a few months ago? That Adobe Air-driven prototype certainly stoked the fires of iPad enthusiasm.

There's just one problem: It won't work on the iPad. It won't work natively on the iPad.

Leander Kahney at Cult of Mac explains why:

Apple has rejected Adobe technologies like Flash and Air — with extreme prejudice. No one at Condé Nast appears to have seen that coming, even though the iPhone OS hasn’t supported Flash since its launch in 2007.

Maybe Condé Nast developers thought the iPad would run Mac OS. Or maybe they just got ahead of themselves.

Update 2/5: Wired editor-in-chief Chris Anderson notes in the comments, and in a follow-up piece at Cult of Mac, that the iPad's Flash limitations were known from the start. Wired will be available on the iPad, as well as Android and Windows.

Time Inc. ran into a similar problem just before the iPad's launch. Its Sports Illustrated tablet prototype was constructed around a wish list, not tech specs.

This is the first sign I've seen that the Apple vs. Adobe spat is spilling beyond the tech space. Content creators accustomed to the Adobe toolset -- particularly Air and Flash -- will have to recalibrate if they want to be on the iPad (and really, who doesn't want to be on that thing?). That means more development and a longer wait for consumers.

Four short links: 4 February 2010

Personal Ad Preferences, Android Kernel, EC2 Deconstructed, Symbian Opened

1. Google Ad Preferences -- my defaults look reasonable and tailored to my interest. Creepy but kinda cool: I guess that if I have to have ads, they should be ones I'm not going to hate. (via rabble on Twitter)
2. Android and the Linux Kernel -- the Android kernel is forked from the standard Linux kernel, and a Linux kernel maintainer says that Google has made no efforts to integrate. (via Slashdot)
3. On Amazon EC2's Underlying Architecture -- fascinating deconstruction of the EC2 physical and virtual servers, without resorting to breaking NDAs. (via Hacker News)
4. First Full Open Source Symbian Release (BBC) -- source code will be available for download from the Symbian Foundation web site as of 1400GMT. Nokia bought Symbian for US$410M in 2008 (for comparison, AOL bought Netscape for $4.2B in 1999 but the source code tarball had been escape-podded from the company a year before the deal closed). This makes Symbian more open than Android, says the head of the foundation: "About a third of the Android code base is open and nothing more,” says Williams. “And what is open is a collection of middleware. Everything else is closed or proprietary.” (quote from Wired's story).

What Facebook's HipHop means for developers and businesses

O'Reilly PHP author Kevin Tatroe puts Facebook's PHP project in context

Facebook's PHP overhaul, HipHop, reportedly cut CPU usage on the company's servers by around 50 percent. You don't have to be a programmer to understand that kind of result.

Facebook says it wants to share that efficiency -- and presumably influence a few things along the way -- so it's setting HipHop loose as an open source project. I asked Kevin Tatroe, co-author of O'Reilly's Programming PHP, to weigh in on HipHop's functionality and its broader applications.

Mac Slocum: How will HipHop help programmers?

Kevin Tatroe: One of PHP's greatest strengths is its expansive leniency. But for very large code bases, it can also be somewhat problematic. For example, while you can change the type of data stored in a variable mid-script, I'd wager that the vast majority of the time, it's a mistake.

Those are the kinds of things HipHop's analysis and type inference steps will find. For that reason, I can even see running sites through HipHop that don't need footprint savings, just as a sanity check.

MS: How about businesses that rely on PHP. Does HipHop offer them any utility?

KT: Certainly. At its best, PHP scales very well by running on teeming hordes of cheap servers. As great as that strategy has turned out to be, running on half as many teeming hordes of cheap servers has clear benefits in deployment costs and costs to maintain.

There's also benefits at the other end of the scale. I've seen more companies than I can count stick with one deployment server when they really ought to be scaling up to two or three. There's a "leap of faith" barrier there. Companies say: "Obviously, we can't deploy on zero servers, so one seems fine. But two? We're just a small operation. Can't you make your code work better?"

MS: Facebook calls HipHop a "source code transformer." In plain terms, what is that?

KT: It takes the PHP code written by PHP programmers and converts it to C++ code, which is then compiled by g++ into machine code.

In doing so, it has to disallow certain PHP language features, like eval(). And it runs a pass to determine what type each variable in your PHP code should be in C++.

PHP does not require you to state up front what kind of data you're going to store in a variable. It lets you change the type of data stored in a given variable willy-nilly, which is not necessarily the best idea.

MS: Facebook says HipHop reduced CPU load on its servers by about 50 percent. Any idea how it does that?

KT: Running native, compiled C++ takes less processing effort to run than PHP's scripts via an opcode virtual machine (such as using Zend Engine). That's because it's skipping the virtual machine entirely.

It's telling that Facebook didn't mention any HipHop speed increases. It'd be surprising if there weren't any measurable speed improvements, but their primary focus seems to be: use less cheap hardware to run the same site.

MS: How easy -- or hard -- do you think it will be for other companies to take advantage of HipHop?

KT: This all depends on the tools. Certainly, any organization with deployments large enough to really notice much out of this will have devs comfortable with the more traditional "write, compile, test, deploy" cycle.

PHP also attracts a large percentage of folks who've never had to compile anything in their lives. But these same people aren't necessarily afraid of the command line. If the tools are simple enough that people aren't scared off, it should be pretty simple to get a site up and running using HipHop.

Note: Kevin's comments were condensed and edited from a longer interview.

Four short links: 3 February 2010

Bad Census Data, Telephone Fraud, Math Art, and EBook Bugs

1. Bad Census Data for The Last Decade (Freakonomics blog) -- the "representative sample" of statistics data that the Census Bureau releases has apparently been flawed. It's been used in thousands of studies, and the Census Bureau has refused to correct it.
2. Modern Telephone Fraud -- it's actually an old fraud updated: an insecure digital PBX used to route expensive calls. Innocent company is whacked with bill at end of month. Interesting questions raised about what we expect company to do (pay?) and telco to do (forgive?). It's a good reminder that every electronic product is now an avenue for fraud or intrusion, but we don't plan or contract for these situations.
3. Found Functions -- Nikki Graziano adds mathematics to photographs. Her photos let me see the world through a mathematician's eyes. (via sciblogs)
4. Getting Past Good-Enough E-Books -- fantastic list of TODOs for ebook publishers.

Forget Google, social search is all about mobile

New research from Aardvark shows higher social search use on the mobile side

There's considerable chatter about a seismic shift in search. A lot of it's overblown, but the central idea is intriguing: Google's biggest problem -- the one that keeps company execs up at night -- isn't Bing or Ask or traditional search. The real threat is social search.

Or so the thinking goes ...

I've always dismissed the notion that Twitter or Facebook could knock Google from its throne. Those services are built for speed, not depth. And even though Google is a huge organization, it still has the agility and forward-thinking to fend off attackers.

Earlier today, I ran across a data point in Aardvark's new social search report that I find way more interesting than Google's theoretical downfall. It's not whether social search will displace Google. It's how -- and where -- social search can actually be useful

TechCrunch's breakdown of the Aardvark report includes this bit of analysis:

[Aardvark's] average query volume was 3,167.2 questions per day, with the median active user asking 3.1 questions per month. Interestingly, mobile users are more active than desktop users. The Aardvark team attributes this to users wanting quick, short answers on their phones without having to dig for anything. They also think people are more used to using more natural language patterns on their phones. [Emphasis added.]

The real seismic shift in social search will come from its commingling with mobile applications.

Why? Because mobile is a different animal than the desktop. No one wants to fumble around for queries. People on the go don't have time to scan listings. The screens are too small, and the input mechanisms -- improved as they are -- are way too clunky.

Mobile search has to be concise and targeted. Results that emanate from a trusted network of friends and associates certainly fit that bill. Toss in more geolocation features and improved speech recognition, and the utility of mobile-based social search could get really interesting.

Rethinking Open Data

Lessons learned from the Open Data front lines

In the last year I've been involved in two open data projects, Open New Zealand and data.govt.nz. I believe in learning from experience and I've seen some signs recently that other projects might benefit from my experience, so this post is a recap of what I've learned. It's the byproduct of a summer reflection on my last nine months working in open data.

Technologists like to focus on technology, and I'm as guilty of that as the next person. When Open New Zealand started, we rushed straight to the "catalogue". I was part of a smart group of top-notch web hackers--we know what a catalogue is, it's a web-based database and let's figure out the UI flow and which fields do we want and hey I can hack one up in Wordpress and I'll work on the hosting and so on. We spent more time worrying about CSS than we did worrying about the users.

This is the exact analogue of an open source software failure mode: often companies think they can get all the benefits of open source simply by releasing their source code. The best dinner parties are about the other people. Similarly, the best open source projects have great people, attract great people, and the source is simply what they're working on: necessary but not sufficient. You can build it but they won't come. All successful open source projects build communities of supportive engaged developers who identify with the project and keep it productive and useful.

Data catalogues around the world have launched and then realised that they now have to build a community of data users. There's value locked up in government data, but you only realise that value when the datasets are used. Once you finish the catalogue, you have to market it so that people know it exists. Not just random Internet developers, but everyone who can unlock that value. This category, "people who can use open data in their jobs" includes researchers, startups, established businesses, other government departments, and (yes) random Internet hackers, but the category doesn't have a name and it doesn't have a Facebook group, newsletter, AGM, or any other way for you to reach them easily.

This matters because it costs money to make existing data open. That sounds like an excuse, and it's often used as one, but underneath is a very real problem: existing procedures and datasets aren't created, managed, or distributed in an open fashion. This means that the data's probably incomplete, the document's not great, the systems it lives on are built for internal use only, and there's no formal process around managing and distributing updates. It costs money and time to figure out the new processes, build or buy the new systems, and train the staff.

In particular, government and science are often funded as projects. When the project ends, the funding stops. Ongoing maintenance and distribution of the data hasn't been budgeted for almost all the data sets we have today. This attitude has to change, and new projects give us the chance to get it right, but most existing datasets are unfunded for maintenance and release.

So while opening all data might be The Right Thing To Do from a philosophical perspective, it's going to cost money. Governments would rather identify the high-value datasets, where great public policy comment, intra-government optimisation, citizen information, or commercial value can be unlocked. Even if you don't buy into the cost argument, there's definitely an order problem: which datasets should we open first? It should be the ones that will give society the greatest benefit soonest. But without a community of users to poll, a well-known place for would-be data consumers to come to and demand access to the data they need, the policy-making parts of governments are largely blind to what data they have and what people want.

That's not to say that data catalogues aren't useful. We were scratching an itch--we wanted easier access to government data, so we built the tool that would provide it. The community of data users can be built around the tool. As Krishna was told by Arjuna, "a man must go forth from where he stands. He cannot jump to the Absolute, he must evolve toward it". I'm just noting that, as with all creative endeavours, we learned about the problem by starting to fix it.

Which brings me to the second big lesson: which problem are we trying to solve? There's an Open Data movement emerging around governments releasing data. However, there are at least five different types of Open Data groupie: low-polling governments who want to see a PR win from opening their data, transparency advocates who want a more efficient and honest government, citizen advocates who want services and information to make their lives better, open advocates who believe that governments act for the people therefore government data should be available for free to the people, and wonks who are hoping that releasing datasets of public toilets will deliver the same economic benefits to the country as did opening the TIGER geo/census dataset.

The one thing these groups don't share is an outcome. I can imagine an honest government where the costs of transparency overweigh the costs of corruption (think of the cost of removing every dirt particle from your house). I can imagine PR wins that don't come from delivering real benefits to citizens, in fact I see this in a recent tweet by Sunlight Labs's Ellen Miller:

Most of the raw data released by the OGD most likely isn't for you to use.

There are some promising signs. The UK government data catalogue had a long beta period where developers were working with the data. The UK team built a community as well as a catalogue. That's not to say that the UK effort is all gold--I saw plenty of frustration with RDF while I was observing the developers--but it stands out simply for the acknowledgement of users. Similarly, the UK's MySociety defined what success is to them: they're all about building useful apps for citizens, and open data is a means not an end to them.

So, after nearly a year in the Open Data trenches, I have some advice for those starting or involved in open data projects. First, figure out what you want the world to look like and why. It might be a lack of corruption, it might be a better society for citizens, it might be economic gain. Whatever your goal, you'll be better able to decide what to work on and learn from your experiences if you know what you're trying to accomplish. Second, build your project around users. In my time working with the politicians and civil servants, I've realised that success breeds success: the best way to convince them to open data is to show an open data project that's useful to real people. Not a catalogue or similar tool aimed at insiders, but something that's making citizens, voters, constituents happy. Then they'll get it.

My next project with Open New Zealand is to build a community of data users. I want to see users supporting each other, I want to build a tight feedback loop between those who want data and those who can provide it, to create an environment where the data users can support each other, and to make it easier to assess the value created by government-released open data. Henry Kissinger said, "each success only buys admission to a more difficult problem". I look forward to learning what the next problem is.

Four short links: 2 February 2010

Physical UIs, Code Visualization, Money Money Money, and Educational Screencasts

1. Phones That Touch Us (TEDxBerlin) -- excellent short (<5m) talk about ways that mobile phones can be designed to convey information in new ways. (via RussB on Twitter)
2. Code City -- an integrated environment for software analysis, in which software systems are visualized as interactive, navigable 3D cities. The classes are represented as buildings in the city, while the packages are depicted as the districts in which the buildings reside. The visible properties of the city artifacts depict a set of chosen software metrics. (via mikeloukides on Twitter)
3. Subscriptions Are the New Black (Dave McClure) -- high-octane rant that boils down to "deliver a good product, charge a fair price". Nobody tell 37Signals, they'll be pissed to discover they've been on the wrong track for all this time. Oh wait ...
4. Khan Academy -- not a Star Trek spinoff but a collection of easy-to-understand science, maths, and economics instructional YouTube screencasts. (via Jon Udell)

Recent Posts

• The iPad is real-life social | by Edd Dumbill on February 1, 2010
• Can open source guide a moon mission? | by Mac Slocum on February 1, 2010
• Four short links: 1 February 2010 | by Nat Torkington on February 1, 2010
• Web developers can rule the iPad | by Simon St. Laurent on January 29, 2010
• Four short links: 29 January 2010 | by Nat Torkington on January 29, 2010
• The iPad is the iPrius: Your Computer Consumerized | by Jim Stogdill on January 29, 2010
• Check Mate: Apple's iPad and Google's Next Move | by Mark Sigal on January 28, 2010
• Four short links: 28 January 2010 | by Nat Torkington on January 28, 2010
• The iPad and publishers: A survey of early reaction | by Mac Slocum on January 27, 2010
• When it Comes to News, Why Won't People Eat Their Vegetables? | by James Turner on January 27, 2010
• Four short links: 27 January 2010 | by Nat Torkington on January 27, 2010
• Four short links: 26 January 2010 | by Nat Torkington on January 26, 2010