Google Squared is an Exponential Improvement in Search

One of the things I've learned about Google is that the most amazing things will come out of them with barely a whisper of fanfare. Such is the case with Google Squared, a new Google Labs tool that was released today. What does Google Squared do? It organizes and tables information from searches for you in a way that makes it much more useful.

For example, the first thing I put into Google Squared was [science fiction conventions], and I got back:

Not too bad right off the bat, and by clicking on the X boxes, you can remove columns or rows that don't fit. It works even better for things that are very well defined, like [atomic weights of elements]:

(continue reading)

Four short links: 3 June 2009

Video Chat, NGO Incorp, Smart Grid, and Enterprise Sales Funny

1. Tinychat -- very simple web-based take on videochat. Pro members get higher resolution, more rooms, and privacy. (I like the "free = public, charge for private" business model)
3. One Click Orgs -- One Click Orgs is building a website where groups can quickly create a legal structure and get a simple system for group decisions. We think social enterprises, collectives and activist groups have better things to think about than obscure legal clauses. Still getting built, but a good idea. We're one step closer to Charlie Stross's vision from Accelerando of a twisty maze of cross-shareholding organisations whose bylaws are Python scripts.
4. Trilliant Acquisition Signals Next Phase of Smart Grid -- smart grids rely on networked power meters and consuming devices. Therefore there are possible alliances between powerline broadband and smart meter companies, as this union shows. Finally, a use for broadband power? (via monkchips on Twitter)
5. The Vendor-Client Relationship -- should mandatory watching for everyone in enterprise sales. (via johnclegg on Twitter)

The Economic Crisis and the US Online Job Market

In my previous post, I noted that despite the large decline in total number of job postings, the number Hadoop/MapReduce job postings increased by 49%. What is the current state of the online job market? The financial crisis that began in the Fall of 2008 has had a lasting negative effect on the U.S. online job market. Since late 2008, there have been significantly less jobs posted online.

Using data from SimplyHired^† and a few charts, I'll quickly highlight the impact of the global economic crisis on the U.S. online job market. To quantify the sudden drop in U.S. online job postings, I calculated the average number of job posts per day:

The number of posts declined 49% from Jan/May 2008 to Jan/May 2009. While there has been a downward trend since April 2008, the financial crisis in September 2008 marked the start of even larger reductions. In particular, the relatively small number of job postings in Nov/Dec 2008 has carried over into the first five months of 2009. The sharp seasonal rebound that occurs in Jan/Feb of each year, was practically non-existent in 2009. While some forecasters are seeing signs of a recovery, at least through the first five months of 2009, we haven't detected "green shoots" in the U.S. online job market.

(continue reading)

Mapumental: Time & Scenicness in Maps

MySociety has given us a sneak peak at Mapumental, a map app that lets you pivot on travel-time, "scenicness", and house-price in the London area. Just enter a postal code and if you're looking for a home in the area Mapumental should be very helpful to you. It is an update to a previous foray into temporal maps (you can try it out on the embed in a Radar post of mine).

The map has a slider to control each of the dimensions. The time slider lets you choose how long you want to commute to get to the desired postal code by 9AM. This is a big change from the original map, which let you . The scenic slider lets you determine how nice of a place you want to live in. The housing price map The map above is quite depressing. It shows the areas you could live that will take you less than an hour and a half to get to work, has housing available for 750K GBP and is just barely scenic at level 3. Your options open up a lot if you ignore scenicness for this zipcode.

To make a useful and useable map with this many controls and data points is difficult. The base maps come from Open Street Map. The travel data (rail, bus, ferry) comes from Traveline (National Public Transport Data Repository); it's all based on a Tuesday in October, 2008. The map tiles, sliders and overall UI was inspired built by Stamen Design. MySociety's travel-time maps were pioneered by Chris Lightfoot.

The most unique dataset included in Mapumental is "scenicness". The data was gathered by user votes in the web app Scenic Or Not. As Tom told me, "We have 173,816 1*1 km squares voted on at least 3 times each, by different people. We're 80% of the way to a full dataset. Data comes from Geograph."

They've got more planned for it like making it public, allowing for alternate arrival and departure times, and allowing for multiple destinations to allow for couples. Those are all great additions, but what I would really like to see is either open-sourced code or an API so say that other geographic areas could have this functionality. How would this change the real estate market? If you could see this style of map for any industrialized area would it change the way you think about your quality of life and what it costs. Sounds like a great task for real estate sites Zillow or Trulia, alternately it's something that Walkscore could tackle.

Right now Mapumental is in Beta, but if you want early access Tom Steinberg sent a hint to BoingBoing: "Beta's private at the moment but we're handing out invites in exchange for declarations of love." Send your missives here. To learn even more about the project check out the short video after the jump.

(via Cory @ Boingboing)

Updated: Properly attributed Stamen's role on the project

(continue reading)

Four short links: 2 June 2009

Fonts, Medicine, Healthcare, Project Natal

1. TypeKit -- Jeff Veen's new startup, making typography on the web fail to suck. Every major browser is about to support the ability to link to a font. That means you can write a bit of CSS, include a URL to a font file, and have your page display with the typography you expect. While it’s technically quite easy to link to fonts, it’s legally more nuanced. We’ve been working with foundries to develop a consistent web-only font linking license. We’ve built a technology platform that lets us to host both free and commercial fonts in a way that is incredibly fast, smoothes out differences in how browsers handle type, and offers the level of protection that type designers need without resorting to annoying and ineffective DRM.
2. Talking With Jamie Heywood About PatientsLikeMe (Jon Udell) -- the creator of patientslikeme, a site that provides people with serious conditions a chance to report on the efficacy of their treatment, their unique symptoms, and (if they wish) to connect with the researchers in the drug companies who made the treatments. It's a new closure for the feedback loop of medical research.
3. The Cost Conundrum: What a Texas town can teach us about health care. (New Yorker) -- the lesson is that you tolerate bad ethics, bad business, bad behaviour at your own risk because the rogue you tolerate may become the anchor tenant for a mall of villainy you'll find very hard to dismiss.
4. Microsoft Announces Project Natal -- full-body motion capture for XBox 360, as game controller. I'm keen to see whether having nothing in your hand is as satisfying as having something to hold. Kudos to MSFT for bringing research to market as mainstream entertainment.

Loki's Net

The National Security Risks of Gov 2.0 and the Social Web

Every culture has its Trickster myths because Trickster lives on the edge of what the rest of us perceive as "real." He crosses boundaries so often and with such ease, not to mention panache, that our own boundaries expand because of him. Trickster is “the doorway leading out, the spirit of the road at dusk” (Lewis Hyde) that doesn't belong to any town but is in-between all towns; the province of thieves and spies.

Here's an updated version of an old Trickster tale that I think is particularly relevant to the topic of this post--the national security risks associated with a more open Government in general and social software in particular.

Loki, the Norse God of mischief and mayhem, had taken to the mountains for refuge after angering the other Gods with his latest antics. The first thing he did was build a house with four doors; one on every side so that he could see in all directions. With his Intrusion Detection System in place, Loki spent the rest of his time playing in the water as a salmon, leaping waterfalls and negotiating mountain streams.

One morning, Loki sat by a fire and considered how the gods might capture him. Since he spent much of his time as a fish, Loki grabbed some linen string and fashioned a fishing net of a size and weight sufficient to snare him. Unfortunately, just as he finished, the other Gods rushed in. Loki threw the net into the fire, transformed into a salmon, and swam away. Acting quickly, the Gods extracted the ashes of the net from the fire and, from the remnants, rebuilt Loki’s net, eventually ensnaring him in it.

Like Loki, we construct through our Twitter posts, Facebook Wall entries and LinkedIn profiles our own unique “net” that sets us up for a social engineering exploit, a financial crime, or an act of espionage.

The Trickster archetype aptly frames this discussion about the risks and benefits of bringing Government into a Web 2.0 world because the classic Trickster is neither good nor bad, but encompasses elements of both. Too often, the debate surrounding Gov 2.0 becomes polarizing. Critics are frequently grouped together as Gov 1.0 thinkers struggling against a 2.0 world, while advocates sometimes embrace Gov 2.0 as a holy quest, refusing to acknowledge any significant risks whatsoever.

I cannot emphasize enough that the surest way to slow our progress toward a more technologically open Government is to try to craft this debate in dualistic terms. Indigenous Trickster tales teach us that a more valuable approach is to substitute utility for morality. Loki and Coyote (a famous Trickster in Native American lore) both understand how to trap a fish because they have swum as fish. Hyde writes in his book Trickster Makes This World that “nothing counters cunning like more cunning. Coyote's wits are sharp precisely because he has met other wits.”

There are serious and significant risks associated with Government 2.0 and the use of Social Software from a national security perspective that need to be talked about and addressed. It is a topic that is both complex and far-ranging and deserves much more coverage than I can provide in this post, although I hope to at least start the conversation at a new and edgier level. To give some perspective to the problem, there are 22,000,000 employed by the U.S. government, not counting government contractors. That fact alone makes Gov 2.0 a very significant technological evolution.

There is ample evidence that state and non-state actors are engaged in finding ways to exploit vulnerabilities in the U.S.'s critical infrastructure as well as the Department of Defense's secure (SIPRNET) and non-secure (NIPRNET) networks. Many of these attacks have been well-documented by Inspectors General (IG) and Government Accountability Office (GAO) investigations as well as through Congressional committee testimony by experts. One of the easiest ways for an attacker to gain access to those protected networks is not through the firewall, but through the user. In any secure system, the human element is always the weakest link. As Tim Thomas wrote in his excellent "Cyber-Skepticism" article for IO Sphere, the mind has no firewall but skepticism. The attack vector that best takes advantage of that vulnerability is known as social engineering.

Do you recall how Matthew Broderick's character cracked the password for the DOD computer Joshua in the 1983 movie “War Games?" He studied details about the life of its creator. That's the same strategy that David Kernell used when he allegedly hacked into Governor Palin's Yahoo account, except he had the benefit of a Web 2.0 invention known as Wikipedia.

How did the individuals behind the GhostNet espionage ring manage to entice so many people (1300 computers in 103 countries) to open an infected document which loaded a Chinese trojan named ghostRAT onto their system? They crafted an enticing email and document that was tailor-made for their audience -- supporters and/or employees of the Office of His Holiness the Dalai Lama. It was such an effective social engineering campaign that 30% of the infected computers were in sensitive government offices. And to make matters worse, most anti-virus programs failed to identify the Trojan.

In Cyber Warfare terms, these types of hacks are a part of Computer Network Operations (CNO) known as Computer Network Exploitation (CNE). Today, over 130 countries are developing a cyber warfare capability with CNE as one component.

Social media like Twitter, Facebook, MySpace, LinkedIn, GovLoop, and many others are very attractive venues for CNE by our adversaries because they are easily accessible, target-rich environments that can be exploited with little to no risk under cover of anonymity.

According to a recent study conducted for one of the U.S. Armed Services, 60% of the service members involved in the study have posted enough information on MySpace to make themselves vulnerable to adversary targeting. And these weren’t only young recruits making bad Operations Security (OPSEC) decisions. The 60% group included officers and enlisted troops from Intelligence and Security postings as well as other sensitive positions posting such things as units they have deployed with, new duty stations, personal medical data, job duties, information about training, and pictures of themselves at deployed locations.

In their paper “Social Software and National Security," Mark Drapeau and Linton Wells discuss the use of Twitter by Colleen Graffy, formerly Deputy Assistant Secretary of State for Public Diplomacy, to “impress her personality and message on foreign media prior to arriving in their countries, and after leaving.” As the authors point out, there are positives and negatives to Graffy’s method of using Twitter. One of the negatives that they do not address is that Graffy’s Twitter usage can become a vector for a non-state hacker to exploit with a @colleen_graffy tweet containing a malicious link disguised as a tiny URL. All of a sudden, Graffy‘s public diplomacy 2.0 effort could result in a State Department computer becoming a zombie.

The Open APIs on Twitter and Facebook provide a virtually unlimited resource for building target profiles on employees of sensitive government agencies like the Departments of Defense, State, Justice, Energy, Transportation, and Homeland Security. The Twitter stream, for example, adds a timeline for tracking when you’re at work, where you’re going after work, and what you are doing right now.

Another risk category is disinformation. Twitter received a lot of coverage during the Mumbai terror attacks of November, 2008 for its role in covering the events in real time. Part of what emerged was the potential for terrorists to use Twitter to propagate disinformation about their whereabouts; i.e., to announce a new attack occurring at a wrong address, thus adding chaos and confusion to an already chaotic situation.

Finally, there is the phenomenon of online trust. If you work in a targeted industry, you will be approached, sooner or later, by someone who isn’t who she claims to be for the purpose of gaining and exploiting your trust to further her own nation’s intelligence mission. One of the quickest ways to establish trust online is by finding things you both hold in common. Both Twitter and Facebook postings excel at that discovery effort.

How do you mitigate the risks while enjoying the benefits of Gov 2.0 and the social web? You do it by thinking like your opponent; or like the Trickster. Read your post twice before you hit send; once as you and once as your adversary who is looking to exploit you. If you work for the DOD or a government contractor, start by re-reading your employer's OPSEC guidelines and edit your profile and your posts accordingly. If your office hasn’t created any OPSEC guidelines for social media yet, please let me know. My company GreyLogic is creating training for precisely that purpose. In the meantime, here are five things that you can do right now to reduce your risk profile:

1. Involve your family members. They should understand that by virtue of your employment with a department, agency, or service, their posts are prime fodder for CNE. You can start by having them read this article.

2. Make OPSEC fun by making a game of it. For example, trade Twitter or Facebook aliases among your coworkers and see how much information you can learn about each other by using publicly available search tools. Then draft two or three email topics that would entice that person to take your bait if you were an adversary running a Spear Phishing operation. I promise that you’ll be amazed at the results. In fact, you should do this same exercise with your family members.

3. Be more skeptical about anyone who contacts you as a result of your posting on a social network. See if you can find their Internet footprint by searching on their name and email address. An alias with no Internet history should immediately raise a red flag.

4. Anyone can start a DOJ, DHS, DOE or other government agency community on Ning, LiveJournal, Facebook, etc. Don’t affiliate yourself with any community that you don’t know for sure is an officially sponsored and sanctioned one. Talk about shooting fish in a barrel.

5. Facebook recently reported that 70% of its traffic comes from overseas. Become more cautious about who you friend and who is privy to reading your posts.

In myth, like in life, the Trickster relies on the instincts and appetites of his prey to spring his trap. For those of us in Government or affiliated with Government, we would do well to remember that as we engage with Gov 2.0 on the social web.

Four short links: 1 June 2009

Spymaster, Arsenic, Maps, and Happiness

1. Spymaster -- a faux-spy game on Twitter: Each player becomes a master of a spy ring based upon their Twitter followers list. The more people that follow you and are playing characters in Spymaster, the more powerful your network will be. As a spymaster, you can perform tasks or attack other spymasters on Twitter. With each successful attempt, you will gain virtual currency and points that allow you to grow even stronger. I'm nervous that it's a project of a classified ads company, but intelligent friends appear to be enjoying it, but that may just be be the jaded eye of a world-weary veteran of pyramid schemes and spamalots.
2. Getting Arsenic Out Of Water -- MIT Technology Review piece about the IBM discovery that a chemical used to pattern chips also acts as a membrane to remove arsenic. More stuff that matters. (via roterhund on Twitter)
3. Mapumental -- MySociety folks making maps useful. It's the continuation of time travel maps, where bus, train, tram, tube, and ferry timetables are mashed with real estate prices to show you where you can live for what you can afford and how long a commute you want. A new twist is crowdsourced "how scenic is this area?" data, so you can choose other dimensions for where you might want to live. New dimensions on transportation data and travel planning.
4. What Makes Us Happy? (The Atlantic) -- the real world is a lot more complex than trivial "get happy fast!" self-help books would have you believe. This longitudinal study shows how complex happiness and misery are. Vaillant’s other main interest is the power of relationships. “It is social aptitude,” he writes, “not intellectual brilliance or parental social class, that leads to successful aging.” Warm connections are necessary—and if not found in a mother or father, they can come from siblings, uncles, friends, mentors. The men’s relationships at age 47, he found, predicted late-life adjustment better than any other variable, except defenses. Good sibling relationships seem especially powerful: 93 percent of the men who were thriving at age 65 had been close to a brother or sister when younger. In an interview in the March 2008 newsletter to the Grant Study subjects, Vaillant was asked, “What have you learned from the Grant Study men?” Vaillant’s response: “That the only thing that really matters in life are your relationships to other people.” (via timoreilly on Twitter)

Most Hadoop Jobs Are In California

Given the recent buzz surrounding Hadoop and MapReduce, I was curious if employers were beginning to mention either term in their job postings. Fortunately I have access to a massive job data warehouse dating back to mid-2005. In partnership with SimplyHired and Greenplum, we maintain a data warehouse that contains most of the online job postings in the U.S.

While the percentage of job postings that mention either Hadoop or MapReduce remains miniscule, the number of such postings is growing steadily:

The number of Hadoop/MapReduce job postings (during the Feb/Apr 2009 period) grew 49% compared to 2008. In contrast, the tough economic environment has translated to significantly fewer job postings: the total number of online job postings declined 40% during the same period.

How mainstream is Hadoop? While researching our report on Big Data, we talked to a (database) vendor who jokingly claimed that nobody outside of the West & East coast cared about Hadoop. Analysis of recent job postings seems to support that perspective. During the three most recent months, employers in 18 states posted Hadoop/MapReduce jobs online, but 60% of those were in California. The top 5 states (CA, MD, NY, MA, WA) accounted for 87% of the Hadoop/MapReduce job postings:

Looking at the same period last year, 72% of the job postings were in California, and the top 5 states (CA, WA, TX, PA, VA) accounted for 79%.

Given the presence of large (Google, Yahoo!, Facebook) and small companies (Cloudera, Greenplum, Aster, ...) who are leaders in the use of Hadoop/MapReduce, it's no surprise that at this early stage, a large share of jobs are in California. While the share of California job postings remains high (60%), it's down from 72% last year. As mentioned above, the percentage of job postings that mention either Hadoop or MapReduce remains miniscule, so I caution against reading too much into the geographic distributions. Nevertheless, it's clear that California employers are expressing interest in Hadoop skills ahead of their peers in other states.

Maker Faire Opens Saturday

Maker Faire is here again, our fourth annual event in the Bay Area. Once again, you just won't believe how much there is to see and do at Maker Faire.

Makers were busy today setting up on Friday. In the morning, we had 400 kids visit the fairgrounds for a backstage tour and a chance to spend time with dozens of makers. By the end of the day over most of the makers had arrived. Over 750 came together for beer and a paella dinner outside -- and countless conversations.

This time lapse recording of the day gives you some sense of how all the magic comes together on Friday -- with a view of the rides on the midway rising up from the grass.

I hope to see many of you tomorrow. Bring the whole family. Remember to check makerfaire.com for information on parking and public transit options. We're open 10 am to 8 pm on Saturday and 10 am to 6 pm on Sunday.

Ignite Show: Hillel Cooperman on the Lego Underground

Some people never grow up. Some people wait to have children so that they can become kids again. When Hillel Coopermans's young ones were four he began his family's Lego collection. This led to whole rooms being devoted to the hobby, to eBay auctions, and Leog conventions. In this week's Ignite Show Hillel takes us through the Lego Underground. He delves into Lego CAD and custom Lego pieces. If you need more Lego check out Wired's gallery of Lego projects for this weekend's Maker Faire.

You can also get the Ignite Show on iTunes.

Amazon Hosts TIGER Mapping Data

Last week at Ignite Where Eric Gundersen of Development Seed made a significant announcement for geohackers looking for easy access to open geodata. Amazon will be hosting a copy of TIGER data on EC2 as an EBS (Elastic Block Storage). Eric stated that this happened during the Apps For America contest in 2008 when they need open geo data for their entry Stumble Safely (which maps crime against bars).

Amazon is now hosting allUnited States TIGER Census data in its cloud. We just finished moving 140 gigs of shapefiles of U.S. states, counties, districts, parcels, military areas, and more over to Amazon. This means that you can now load all of this data directly onto one of Amazon’s virtual machines, use the power of the cloud to work with these large data sets, generate output that you can then save on Amazon’s storage, and even use Amazon’s cloud to distribute what you make.

Let me explain how this works. The TIGER data is available as an EBS storeEBS, or Elastic Block Storage, which is essentially a virtual hard drive. Unlike S3, there isn’t a separate API for EBS stores and there are no special limitations. Instead an EBS store appears just like an external hard drive when it’s mounted to an EC2 instance, which is a virtual machine at Amazon. You can hook up this public virtual disk to your virtual machine and work with the data as if it’s local to your virtual machine – it’s that fast.

The TIGER Data is one of the first Public Data Sets to be moved off of S3 and switched to an EBS. By running as an EBS users can mount the EC2 instance as a drive and easily run their processes (like rendering tiles with Mapnik) with the data remotely. If you're a geo-hacker this makes a rich set of Geo data readily available to you without consuming your own storage resources or dealing with the normally slow download process.

I love the idea of Amazon's Public Data Sets. It's an obvious win-win scenario. The public is able to get access to rich data stores at a relatively cheap price and Amazon is able to lure said public onto their service. Smart.

Four short links: 29 May 2009

Meatware Hacks, iPhone Web Stats, Distributed Hash Tables, Richard Feynman Fun

1. Freedom for OS X -- Mac app that disables networking for up to eight hours so you can get work done without Internet distractions. Technology workarounds for meatware bugs. (via Joshua-Michèle Ross).
2. iPhone Casts a Giant Shadow on the Web -- 43% of mobile web traffic is from iPhone users, as measured by "the world's largest purveyor of ads on mobile apps and websites". As I was told today, "more people are spending more time looking at the web through one of these. For how much longer can you afford to ignore it?" (via timoreilly on Twitter)
3. Why you won't be building your killer app on a distributed hash table (Jonathan Ellis) -- locking and sophisticated queries. I'm still trying to figure out where we'll end up with these "let's do something simple in a way that lets us scale horizontally, and then build on top of that" approaches to solving the big data/graph theory problems behind many modern apps.
4. Richard Feynman Interviews at Microsoft -- a bit of fun to start the weekend on. (new URL 20090601)

Recent Posts

• Google I/O in Pictures: Google Culture at Work | by Tim O'Reilly on May 29, 2009
• Google Wave: the Early Days | by Tim O'Reilly on May 28, 2009
• Ignite NYC IV & The First Ignite Film Festival This Monday | by Brady Forrest on May 28, 2009
• Google Wave: What Might Email Look Like If It Were Invented Today? | by Tim O'Reilly on May 28, 2009
• Four short links: 28 May 2009 | by Nat Torkington on May 28, 2009
• New Geo For Devs From Google I/O | by Brady Forrest on May 27, 2009
• FCC discusses broadband: the job is a big one | by Andy Oram on May 27, 2009
• Google I/O keynote, day 1 | by Mike Loukides on May 27, 2009
• Google Web Elements and Google's Iceberg Strategy (Google I/O) | by Tim O'Reilly on May 27, 2009
• Google Bets Big on HTML 5: News from Google I/O | by Tim O'Reilly on May 27, 2009
• Four short links: 27 May 2009 | by Nat Torkington on May 27, 2009
• Geeks Invade Government With Audacious Goals | by Mark Drapeau on May 27, 2009