Stop Fishing and Start Feasting: How Citable Public Documents Will Change Your Life

Putting government documents and data online is a great step towards making our government process more transparent to the people it serves, but in many ways simply making the material available is like serving someone dinner by giving them a pond full of fish. The pond is huge and the poor dinner guest doesn't have any tools. Worse, they're only looking for one particular bass, and every time someone sends them to where they last saw the fish it's long gone.

The recent healthcare bill was more than 1,000 pages long. The budget can often be half again that big. Commenting on these types of documents as they are currently implemented is extremely challenging. Pointing a finger at that big pond and telling someone that you swear you saw a fish isn't very effective. It's even worse when someone swears they saw a fish that isn't really there and it is effective because no one is willing to refute them. No one has time to wade around themselves and so they take it on faith. The recent "killing grandma" scare is an excellent example.

Citations, first, are a way of pointing at the fish. A simple paragraph level of granularity for references should be enough. This promotes ease of implementation and use and provides a tight enough zoom to bring someone right to the material being discussed.

The next problem is that fish move. If you're trying to point out a moving fish, and show it to someone later, you need to have a photograph with a timestamp. That line in the budget about forcing our children to manufacture chemical weapons might have moved to page three the next day, or a wily senator may have changed the wording and put it under a different heading. Proper citability requires an archived snapshot of the online material that maintains the integrity of any reference links.

Lastly, for someone to believe you about this fish, you need to have a way of pointing out where you saw it at the specified time. They'll want to know it was the same pond.

Making it possible to create timestamped permalinks at a paragraph level of granularity would be a huge leap forward in increasing government transparency through its online documents. The same principles apply when producing citable government data. When recovery.org decided to display visual representations of the data coming in about recovery money around the nation, it quickly became clear that some amount of data was erroneous. When the errors were reported and the data was later modified, there wasn't any way to go back and compare the two versions to see what changes had taken place. A blogger, reporter, statistician or scientist should be able to run a query against any specific collection of government data, as it was published, for a given version or moment in time.

WHAT WE'RE DOING

The nonprofit, nonpartisan League of Technical Voters has proposed a simple, easy to build and implement citability solution. Open source software development is underway and a wide range of government institutions are already on board. If you would like to help with this effort, consider being part of our upcoming codeathon or create your own codeathon.

Four short links: 7 April 2010

HTML5 Widgets, RDF and Unix, Movie Piracy, and Online Complaints

1. SproutCore -- open-source HTML5 application framework (i.e., lots of Javascript goodness) that'll work with any backend. To code for this, you put most of the logic in the front-end and leave the back-end much simpler.
2. RDF for Intrepid Unix Hackers -- an interesting series, showing how to use common Unix tools to manipulate RDF data from the commandline. (via Edd Dumbill)
3. How to Thrive Among Pirates (Kevin Kelly) -- a look at how indigenous movie-makers make money in countries like China, India, and Nigeria where piracy is rampant. In short, they make cheap movies, sell near the price of inferior-quality knockoffs, and take advantage of unique experiences that movie theaters offer (e.g., air-conditioning).
4. On Complaints (PublicStrategist) -- a very good analysis of complaints departments and expectations of people who complain. But there is also a vital question of what the organisation thinks the purpose of a complaints process is. If it is a safety valve, a means of finding and correcting the most egregious failures or a means of channelling immediate anger and dissatisfaction into a swamp of unresponsiveness, then it can’t provide any broader value. That’s where the Patient Opinion model starts to look really attractive. It is deliberately and carefully constructed to elicit feedback, not just complaints. More than half the stories it gets told are positive, even some of the most harrowing, and it therefore creates a picture which is as clear about what is valued as it is about what is seen as in need of improvement.

Four short links: 6 April 2010

Copytheft, Digg UI, HIV Detection, and Facebook Sueage

1. Thinking Further About Copyright (Confused of Calcutta) -- several nice illustrations of the "copying is not theft" distinction. Copying per se is not stealing. After Michael Jackson did his moonwalk, children the world over copied him. They were not stealing. Digital forms of music, film, book and newspapers are cheap to copy and to distribute, because of the internet. The internet is a commons, specifically designed for doing this. For copying and distributing. Throwing that away just to protect the “rightsholders” is questionable in the extreme. Digital assets are nonrival goods, shareable without affecting the rights of anyone else to enjoy the same thing.
2. DUI: Digg User Library -- Javascript UI library from the folks at Digg.
3. Building a Handheld HIV Detector -- gadget the size of an iPod, that detects the T-cells that HIV kills. Prototype cost $250 to make, orders of magnitude less than the typical medical instrument. This is just one of many approaches to the problem, including disposable test kits funded by the Bill and Melinda Gates Foundation. (via @parc)
4. How I Got Sued by Facebook (Pete Warden) -- he'd previously reported security holes to Facebook's security team, and that apparently saved him from a full-on lawsuit. Their contention was robots.txt had no legal force and they could sue anyone for accessing their site even if they scrupulously obeyed the instructions it contained. The only legal way to access any web site with a crawler was to obtain prior written permission. Obviously this isn't the way the web has worked for the last 16 years since robots.txt was introduced, but my lawyer advised me that it had never been tested in court, and the legal costs alone of being a test case would bankrupt me.

Four short links: 5 April 2010

Intelligence, Community, Compression, and UI Hackery

1. Wrong about the iPad (Tim Bray) -- I am actively ignoring the iPad drivel, but this line caught my eye: Intelligence is a text-based application.
2. Fertile Medium -- online community consultancy, from the first and former Flickr community coordinator. One to watch: Heather and Derek really know their community. Again I say it: understanding of how open source and other collaborative communities can function is rare and valuable. (via waxy)
3. pigz -- parallel gzip implementation. Voom voom, so fast! (via kellan on Delicious
4. Prefab: What If We Could Modify Any Interface? -- screen-scraping for GUIs to bolt on new functionality to user interfaces. This is incredible. Watch the demo, it's impressive!

What brand of freedom would you like?

Apple's restrictions and Google's openness have more in common than you might think.

There's been a ton of criticism, including from me, over Apple's restrictions on the App Store. How it restricts the freedom of developers by blocking applications users definitely want to use, or yanking apps that fail to meet a changeable standard of appropriateness or legitimacy. I originally thought I would never buy an iPhone because of that policy alone, and I felt like quite a hypocrite when I decided to buy one anyway. As many people have pointed out, the iPad, if successful, will further extend Apple's control over the code its users are able to run.

One of the main pitches for Google's Android platform, in contrast, is that it is more open, freer, more consistent with the principles that the open source world (and this blog, and I) espouse. For the code, and applications running on it, that certainly seems to be true. I can go to source.android.com and download the Android source. The Apache license applied to most of the project is very liberal in the use of that code. As a developer, I can publish applications for Android at www.android.com/market, where, Google says, "developers have complete control over when and how they make their applications available to users." How much more free could you get and still call it a platform? The Android stance is nearly 180 degrees from the iPhone's. One is free and the other is closed.

And yet, I don't think the contrast is as clear as that. Freedom means different things to different people. Hence Richard Stallman's quip, "Think free as in free speech, not free beer." While there's no question the code is much more free on Android, I think Steven Levy's point in his piece on the iPad is worth repeating:

While Apple wants to move computing to a curated environment where everything adheres to a carefully honed interface, Google believes that the operating system should be nearly invisible. Good-bye to files, client apps, and onboard storage -- Chrome OS channels users directly into the cloud ...

(continue reading)

OSCON 2010: Open source in a world of new defaults

Registration is now open for the O'Reilly Open Source Convention, July 19-23, Portland, OR.

This year's revolutionary technology frequently becomes the accepted norm a few years down the line. Every so often the revolution is big enough, and a noticeable shift in the technology landscape occurs.

We're at such a point in 2010. At OSCON 2010 in July, we'll be talking about the new environment that developers, businesses and open source projects find themselves in, and how they can navigate and get the best from it.

Cloud by Default

For many years at OSCON we called out "web applications" as a distinct topic. This year it became a useless demarcation, as just about everything is a web application. Cloud computing is in a place similar to web applications a few years ago.

Your next application will probably run in a cloud setting, whether public or private. The cloud brings with it concerns of designing for scalability, replication and failure, presenting new opportunities and techniques. In a few years, these will be second nature, taken for granted. For now, we're learning the ropes, agreeing on standards, pushing forward with developing new tools.

The consequences of the move to the cloud can be felt throughout the OSCON program, but especially in the cloud computing and database tracks. Keynote speaker Stormy Peters will be exploring the challenge that software-as-a-service presents to traditional open source.

One of the interesting spinoffs from cloud computing is that the disciplines of system administration and software development have merged to some extent, as systems management becomes more programmatic and development needs to account for systems architecture. In line with this we've retitled our system administration track Operations and will cover topics such as configuration management, scalability and monitoring.

(continue reading)

Four short links: 31 March 2010

Messaging, Predicting, Visualising, and Patenting

1. ZeroMQ -- bold claim of "Fastest. Messaging. Ever." LGPL, C++ with bindings for many languages, past version 2 already. (via edd on Twitter)
2. Prediction Market News (David Pennock) -- HSX is going to be a real marketplace with real $. The real HSX will of course say goodbye to the virtual specialist and the opening weekend adjust, two facets of the game that make it fun to play, but that create significant amounts of (virtual) wealth out of thin air. The Cantor Gaming group is engaged in other interesting initiatives. They are taking over a sportsbook in Las Vegas and turning it into more of a derivatives exchange with live in-game betting, a step toward my dream of a geek-friendly casino. Interestingly, another company called Veriana Networks is close to launching a competing Hollywood derivatives market called the Trend Exchange.
3. A Pivot Visualization of my Wordpress Blog (Jon Udell) -- using pro-am data exploration tools from Microsoft (Pivot) to work with information from his blog. Contains the scripts he used to do it.
4. Select Committee Report on Patents Bill (PDF) -- New Zealand Government select committee recommends no software patents in NZ. We recommend amending clause 15 to include computer programs among inventions that may not be patented. We received many submissions concerning the patentability of computer programs. Under the Patents Act 1953 computer programs can be patented in New Zealand provided they produce a commercially useful effect [footnote: Under the Patents Act 1953 mathematical algorithms as such are not patentable. They may be patented under the Patents Act when used in a computer, so long as they produce a commercially useful effect.] Open source, or free, software has grown in popularity since the 1980s Protecting software by patenting it is inconsistent with the open source model, and its proponents oppose it. A number of submitters argued that there is no "inventive step" in software development, as "new" software invariably builds on existing software. They felt that computer software should be excluded from patent protection as software patents can stifle innovation and competition, and can be granted for trivial or existing techniques. In general we accept this position.

Why health care is coming to the Open Source convention

This year for the first time, O'Reilly's Open Source convention contains a track on health care IT. The call for participation just went up, soliciting proposals on nine broad areas of technology including health data exchange, mobile devices, and patient-centered care.

One correspondent asked a bit timidly whether it would be all right to submit a proposal if her company didn't use open source software. Definitely! The Open Source convention has always been about a wide range of computing practices that promote openness in various ways. Open source software is a key part of the picture but not the whole picture. Open data, standards, and collaborative knowledge sharing are also key parts of the revolution in today's health care.

This new track is as much a response to urgings from friends and colleagues as it is an O'Reilly initiative. We could use help spreading the word, because the deadline for proposals is tight. In this blog I'll explain why we created the track and why OSCon is a promising venue for trends that will move and shake health care in positive ways.

(continue reading)

Four short links: 26 March 2010

Chrome Extensions in Firefox, AUI Opened, Closing Open Hardware, Fixing Science Metrics

1. Chrome Extensions Manager for Firefox -- lets you run Chrome extensions in Firefox. I don't think, though, that people choose Chrome over Firefox for the extensions (quite the opposite, in fact).
2. Atlassian User Interface -- Javascript HTML UI toolkit, opensourced by Atlassian. (via lachlanhardy on Twitter)
3. Open Source Ethics and Dead End Derivatives -- open source hardware is dealing with the problem of people changing open source designs but not publishing their modified source. Open source software hasn't found an efficient and reproducible mechanism for dealing with this, though I'd love to be shown one. (via bre on Twitter)
4. Let's Make Science Metrics More Scientific (Nature) -- excellent paper about the problem of the metrics for measuring scientific performance are based around papers and citations, but fail to take into account teaching, mentoring, communicating, etc. (via dullhunk on Twitter)

Four short links: 22 March 2010

Trading Software, Learning Programming Languages, Web Security Scanner, Learning as Game

1. Marketcetera -- open source trading platform.
2. Google Code University: Programming Languages -- video-based classes on C++, Python, Java, and Go.
3. Skipfish -- open sourced web application security scanner, from Google.
4. Professor Swaps Grades for XP -- divided class into guilds, awarded XP for achieving various solo, guild, and pickup quests. (via johnny723 on Twitter)

Four short links: 19 March 2010

Load Testing, Chinese Manufacturing, Heroic Forking, and Ubicomp

1. Tsung -- GPLed multi-protocol (HTTP, PostgreSQL, MySQL, WebDAV, SOAP, XMPP) load tester written in Erlang.
2. Myth of China's Manufacturing Prowess -- The latest data shows [...] that the United States is still the largest manufacturer in the world. In 2008, U.S. manufacturing output was $1.8 trillion, compared to $1.4 trillion in China (UN data. China’s data do not separate manufacturing from mining and utilities. So the actual Chinese manufacturing number should be much smaller). Also contains pointers to an interesting discussion of lack of opportunities for college grads in China.
3. OpenSSO and the Value of Open Source -- Oracle are removing all open source downloads and wiki mentions, leaving only the enterprise OpenSSO product on their web site. A Norwegian company has stepped in and will continue the open source project. This is essentially a fork, but for the forces of good. (via normnz on Twitter)
4. The Internet of Things -- 5m video on sensor networks, etc. (via imran on Twitter)

Four short links: 18 March 2010

DIY Newspapers, Saviour Algorithms, Baseline Removal, and Web Scripting

1. Newspaper Club Launches (BBC) -- the uses it has been put to make for good reading: Among the Newspaper Club's first clients were the BBC, Wired UK and Last.fm. Penguin used it to debut a preview of the fifth chapter of The Hitchhiker's Guide to the Galaxy, written by Eoin Colfer.
2. Machine Learning Algorithm with a Capital A -- It’s claimed to be close to the way the brain learns/recognizes patterns and to be a general model of intelligence and it will work for EVERYTHING. This reminded me of a few other things I’ve come across in the past years that claim to be the new Machine Learning algorithm with Capital A, i.e. the algorithm to end all other ML work, which will work on all problems, and so on. Here is a small collection of the three most interesting ones I remembered.
3. fityk -- GPL program for nonlinear fitting of analytical functions (especially peak-shaped) to data (usually experimental data). There are also people using it to remove the baseline from data, or to display data only. (via straup on delicious)
4. Chickenfoot -- Firefox plugin to let you script and manipulate web pages. Useful for automation, like Greasemonkey, but acts on the rendered page and not the HTML source.

Recent Posts

• Report from HIMSS Health IT conference: toward interoperability and openness | by Andy Oram on March 3, 2010
• Living Stories can reinvent the article | by Mac Slocum on February 18, 2010
• Four short links: 10 February 2010 | by Nat Torkington on February 10, 2010
• Four short links: 5 February 2010 | by Nat Torkington on February 5, 2010
• One hundred eighty degrees of freedom: signs of how open platforms are spreading | by Andy Oram on February 4, 2010
• Four short links: 4 February 2010 | by Nat Torkington on February 4, 2010
• What Facebook's HipHop means for developers and businesses | by Mac Slocum on February 3, 2010
• Can open source guide a moon mission? | by Mac Slocum on February 1, 2010
• Four short links: 19 January 2010 | by Nat Torkington on January 19, 2010
• Pew Research asks questions about the Internet in 2020 | by Andy Oram on January 7, 2010
• Four short links: 1 January 2010 | by Nat Torkington on January 1, 2010
• Four short links: 30 November 2009 | by Nat Torkington on November 30, 2009