Government transparency: Using search data to connect with your audience

A couple of weeks ago at Transparency Camp, I gave a talk on using search data to help ensure that the information the organizations in attendance were opening up could be found by the right audiences. It's awesome that organizations like the Sunlight Foundation, Open Congress, and Follow the Money are making details about government actions easily accessible by citizens. And the government itself has made great strides in opening up data with sites such as recovery.gov and data.gov.

But the truth is that most people don't keep a full list of web sites that provide this information handy. Or, actually they do. It's called Google.

When Americans want to know about health care reform, they don't go to opencongress.org and search for "H.R.3200" or H.R.4872". They go to Google and type in "health care reform". One key to making sure that the information you are working so hard to surface makes its way to the citizens who are looking for it? Use free search data to find out the language people are using to refer to that information. At Transparency Camp, I demonstrated a number of these tools, which I'll outline below.

Open Congress and Effective Use of Search Data
But first, does this method of understanding how your audience searches and ensuring your content is visible in search engines really make that much of a difference to the transparency in government and open data initiatives? Just ask OpenCongress.org. (Thanks to David Moore, of the Participatory Politics Foundation and Open Congress for taking the time to talk to me about how Open Congress uses search data to better connect with audiences.)

Open Congress is a non-profit, non-partisan organization that aggregates official government information with conversations about those issues. Their goal is to provide American citizens with visibility into what's happening in Congress. And how to people find them? 70% of their traffic comes from search engines (and 75% of those visitors are from Google). Below is recent data from Google Analytics that shows the distribution of traffic sources over the course of a month.

Open Congress's pages are visible to searchers because they pay attention to what people are searching for. They use free tools from Google, such as Webmaster Tools, Analytics, and Insights to learn about how people are searching and they craft their titles and headlines accordingly. When they had new information about H.R.3590, they checked Google Trends and found that "health care bill" was a trending topic. People were interested in the information they had, but would never find it if it were titled "H.R.3590".

By paying attention to the language of their audience and caring about search visiblity, their overview pages for the three main health care bills have had 1,929,332 views and the pages with the text of those bills have had 1,128,570 views. (You can find more details about the visitor stats for their health care bill pages here.)

Let's see this in action. The site ranks number one on Google for [health care bill text].

(continue reading)

Building Searcher Personas For Greater Customer Engagement and Acquisition

When we want to find more information about something, hear about something interesting from our friends, see a compelling television commercial, or need a local mechanic, chances are the first place we turn is the Google search box. Fifty percent of us in the United States use search engines every day and over 90% of us search every month. These days, your search strategy is your business strategy, whether you realize it or not, because that’s how potential customers are trying to find you. Search is the new yellow pages, 800 number, Sunday circular, card catalog, and cash register. No matter what kind of web site you have--whether it’s a media property like a blog, an ecommerce site, or the online arm of multinational corporation--you want to connect with as many of your potential audience as possible, and organic search can help make that happen.

Effective search acquisition strategy
To effectively use organic search as an acquisition channel, you should ensure that your web site:

• Can be discovered by search engines so the pages can be added to the list the crawler uses to traverse the web.
• Uses site architecture that doesn’t introduce obstacles that make the pages inaccessible to the crawlers.
• Presents all content in a way that’s extractable for search engines to index.
• Is relevant and useful to searchers

Sounds easy, right? The good news is that the path to success is fairly straightforward. But as you might imagine, once you dive into the details, the road starts taking a few hairpin turns. The first three points focus on the web site infrastructure and the last point is all about what’s actually on the site.

(continue reading)

Practical Tips for Government Web Sites (And Everyone Else!) To Improve Their Findability in Search

In an earlier post, I said that key to government opening its data to citizens, being more transparent, and improving the relationship between citizens and government in light of our web 2.0 world was ensuring content on government sites could be easily found in search engines. Architecting sites to be search engine friendly, particularly sites with as much content and legacy code as those the government manages, can be a resource-intensive process that takes careful long-term planning. But

• Assessing who the audience is and what they're searching for
• Ensuring the site architecture is easily crawlable

Use XML Sitemaps
Create XML Sitemaps that list all the pages on the site and submit them to the major search engines.

Why is this important? Many government sites have poor information architecture. Ideally each page of the site should have at least one link to it. This helps users navigate the site and helps search engines find all of the pages. Long term, these sites should revamp their navigational structure so that at least one link exists to every page. Since that may take some time to implement, an XML Sitemap can function in the meantime to provide a list of all pages for search engines to crawl.

Government sites have already made great progress in search by using XML Sitemaps.

The Energy Department's Office of Science and Technology (OSTI) implemented XML Sitemaps protocol with great success. "The first day that Yahoo offered up our material for search, our traffic increased so much that we could not keep up with it,' said Walt Warnick, OSTI's director.

If possible, provide an HTML sitemap as well, which provides a browsable navigation to site visitors. Below is a good example of a browsable HTML sitemap on nih.gov:

Don't block access to content
Make all content available outside of a login, registration form, or other input mechanism. Search engine crawlers can't access content behind a login or registration. If the content requires the visitor to enter an email address or otherwise provide input before accessing it, it won't show up in search results.

(continue reading)

Transforming the Relationship Between Citizens and Government: Making Content Findable Online

Thursday on this blog, Congressman Honda asked, "how can congress take advantage of web 2.0 technologies to transform the relationship between citizens and government?" He noted that "A dramatic shift in perspective is needed before that need can be met. Instead of databases becoming available as a result of Freedom Of Information Act requests, government officials should be required to justify why any public data should not be freely available to the taxpayers who paid for its creation." He asked for input on what web 2.0 features he should add to his website to take advantage of today's online world.

The most important feature government web sites can add isn't really feature at all. But it would absolutely transform the relationship between citizens and government and make an amazing array of public data available. What's this magic feature?

Make government web sites search engine friendly.

How we look for information
Search is the primary navigation point for the web. Often when citizens look for government information, they start at a major search engine. They don't think to themselves, I need some information on vitamins, so I'll just go on over to the Office of Dietary Supplements at https://dietary-supplements.info.nih.gov. And then I need to make sure I'm eating a balanced diet, so I'll just check out https://www.nutrition.gov from the National Agricultural Library. And before I head to the grocery store, I'll make sure I understand how nutrition labels work from the information provided by the Center For Food Safety and Applied Nutrition at https://www.cfsan.fda.gov. Mostly, they go to Google and type in [food labels]. And in some cases, this works perfectly and the information appears.

But when information from government web sites doesn't show up on the first page of results for those searches, the information may as well not exist at all. For instance, an amazing amount of data exists from the U.S. Census Bureau, but it's inaccessible from search engines because it's locked behind JavaScript forms and the content itself doesn't use language that searchers would use. If I search for [98116 census data], results from census.gov are nowhere to be found.

Obstacles to being found in search engines
One problem is that the U.S. Census Bureau pages don't use zip codes to denote regions. They use tract numbers. Even if the pages were written in plain language searchers might use, search engine crawlers couldn't get past the JavaScript forms to access the pages.

Try doing a search using the same terminology as the U.S. Census Bureau, and you start to see the problem with the site's findability. Take [census track 97.02]:

None of those results lead to these handy details:

In addition to being buried behind JavaScript and containing little language people would actually search for, it's hidden in a popup with a URL like this: https://factfinder.census.gov/servlet/IdentifyResultServlet?_mapX=281&_mapY=216&_latitude=&_longitude=&_pageX=442&_pageY=554&_dBy=100&_jsessionId=0001cv7n8rWxjslrmI9aRw5nr-V:134a7lbrs">https://factfinder.census.gov/servlet/IdentifyResultServlet?_mapX=281&_mapY=216&_latitude=&_longitude=&_pageX=442&_pageY=554&_dBy=100&_jsessionId=0001cv7n8rWxjslrmI9aRw5nr-V:134a7lbrs

The server appends a session ID to the end of the URL (the portion beginning with "jessionsId"), which is tied to an individual visitor session and times out after 60 minutes. If I share that URL on a social media site, email, or in this blog post, anyone who tries to visit it just gets a "session as expired" message. It goes without saying that this kind of URL can't be indexed by search engines no matter how sophisticated they become.

(continue reading)

Making Site Architecture Search-Friendly: Lessons From whitehouse.gov

Guest blogger Vanessa Fox is co-chair of the new O'Reilly conference Found: Search Acquisition and Architecture. Find more from Vanessa at ninebyblue.com and janeandrobot.com. Vanessa is also entrepreneur in residence at Ignition Partners, and Features Editor at Search Engine Land.

Yesterday, as President-elect Obama became president Obama, we geeky types filled the web with chatter about change. That change of change.gov becoming whitehouse.gov, that is. The new whitehouse.gov robots.txt file opens everything up to search engines while the previous one had 2400 lines! The site has a blog! The fonts are Mac-friendly! That Obama administration sure is online savvy.

Or is it?

An amazing amount of customer acquisition can come from search (a 2007 Jupiter research study found that 92% of online Americans search monthly and over half search daily). Whitehouse.gov likely doesn't need the kind of visibility that most sites need in search, but when people search for information about today's issues, such as the economy, the Obama administration surely wants the whitehouse.gov pages that explain their position to show up.

The site has a blog, which is awesome, but the title tag, the most important tag on the page, has only the text "blog". Nothing else. Which might help the page rank well for people doing a search for blog, but that's probably not what they're going for. This doesn't just hurt them in search of course. It's also what shows up in the browser tab and bookmarks.

The site runs on IIS 6.0. Does the site developer know about tricky configuration that makes the redirects search engine-friendly?

Search engines are text-based, so they can't read text hidden in images. Some whitehouse.gov pages get around this issue well, by making the text look image-like, but leaving it as text, such as below.

However, other pages have text in images and don't use ALT text to describe them. (This, of course, is an accessibility issue as well, as it keeps screen readers from being able to access the text in the images.) An example of this is the home page, which may be part of why whitehouse.gov doesn't show up on the first page in a search for President Obama.

There are all kinds of technical issues, big and small, that impact whether your site can be found in search results for what you want to be found for. (whitehouse.gov using underscores rather than dashes in URLs, the meta descriptions are the same on every page...) Probably the biggest issue in this case is the lack of 301 redirects between the old site and the new site. When you change domains and move content to the new domain, you don't want to have to rebuild the audience and links all over again. (Not that Obama or whitehouse.gov will have a problem with attracting and audience, but we all can't be president!) When you use a 301 redirect, both visitors and search engines know to replace the old page with the new one.

In the case of change.gov, it's unclear if they intend to maintain the old site. The home page asks people to join them at whitehouse.gov, but all the old pages still exist (even the old home page at https://change.gov/content/home).

And in many cases, the same content exists at both change.gov and whitehouse.org (see, for instance, https://change.gov/agenda/iraq_agenda/ and https://www.whitehouse.gov/agenda/iraq/).

As Matt Cutts, Googler extraordinaire pointed out, give them a few days to relax before worrying so much about SEO. And I certainly think the site is an excellent step towards better communication between the president and the American people. But not everyone has the luxury of having one of the most well-known names and sites in the world, so the technical details are more important for the rest of us.

If you want to know more about technical issues that can keep your site from being found in search and tips for making sure that you don't lose visibility in a site move, join us for the O'Reilly Found conference June 9-11 in Burlingame. And if you're in Mountain View tomorrow night (Thursday, January 22nd), stop by Ooyala from 6pm to 9pm for our webdev/seo meetup, and get all your search questions answered. Hope to see you there! (Macon Phillips and the whitehouse.gov webmasters are welcome, but my guess is that they're a little busy.)

Recent Posts