Time Lapse of Galactic Center of Milky Way rising over Texas Star Party

According to William Castleman: The time-lapse sequence was taken with the simplest equipment that I brought to the star party. I put the Canon EOS-5D (AA screen modified to record hydrogen alpha at 656 nm) with an EF 15mm f/2.8 lens on a weighted tripod. Exposures were 20 seconds at f/2.8 ISO 1600 followed by 40 second interval. Exposures were controlled by an interval timer shutter release (Canon TC80N3). Power was provided by a Hutech EOS203 12v power adapter run off a 12v deep cycle battery. Large jpg files shot in custom white balance were batch processed in Photoshop (levels, curves, contrast, Noise Ninja noise reduction, resize) and assembled in Quicktime Pro. Editing/assembly was with Sony Vegas Movie Studio 9.

[via the Primary Tentacle @ Laughing Squid]

Space Shuttle Atlantis during Solar Transit

In this tightly cropped image, the NASA space shuttle Atlantis is seen in silhouette during solar transit, Tuesday, May 12, 2009, from Florida. This image was made before Atlantis and the crew of STS-125 had grappled the Hubble Space Telescope. Photo Credit: (NASA/Thierry Legault)

Thierry made this image using a solar-filtered Takahashi 5-inch refracting telescope and a Canon 5D Mark II digital camera. Photo Credit: (NASA/Thierry Legault)

You can see more of Thierry's fine work at: www.astrophoto.fr

from nasa hq photostream [via slashdot]

Velocity 2009 - Big Ideas (early registration deadline)

My favorite interview question to ask candidates is: "What happens when you type www.(amazon|google|yahoo).com in your browser and press return?"

While the actual process of serving and rendering a page takes seconds to complete, describing it in real detail can take an hour. A good answer spans every part of the Internet from the client browser & operating system, DNS, through the network, to load balancers, servers, services, storage, down to the operating system & hardware, and all the way back again to the browser. It requires an understanding of TCP/IP, HTTP, & SSL deep enough to describe how connections are managed, how load-balancers work, and how certificates are exchanged and validated... and that's just the first request!

Web Performance & Operations is an emerging discipline which requires incredible breadth, focusing less on specific technologies and more on how the entire system works together. While people often specialize on particular components, great engineers always think of that component in relation to the whole. The best engineers are able to fly to the 50,000 foot view and see the entire system in motion and then zoom in to microscopic levels and examine the tiny movements of an individual part.

John Allspaw recently described this interconnectedness on his blog:

With websites, the introduction of change (for example, a bad database query) can affect (in a bad way) the entire system, not just the component(s) that saw the change. Adding handfuls of milliseconds to a query that’s made often, and you’re now holding page requests up longer. The same thing applies to optimizations as well. Break that [bad] query into two small fast ones, and watch how usage can change all over the system pretty quickly. Databases respond a bit faster, pages get built quicker, which means users click on more links, etc. This second-order effect of optimization is probably pretty familiar to those of us running sites of decent scale.

Working with these systems requires an understanding not only of the way technology interacts, but the way that people do as well. The structure, operation, and development of a website mirrors the organization that creates it, which is why so many people in WebOps focus on understanding and improving management culture & process.

Organizing a conference like Velocity is a wonderful challenge because it requires the same sort of thinking. We focus on the big concepts that everyone needs to know and then go deep into the technologies that change our understanding of the system. We find ways to share the unique experience that can only be gained by operating at scale. We make it safe to share as much of the "Secret Sauce" as we can.

Please join us at Velocity this year, we have an amazing lineup of speakers & participants. Early registration ends on Monday, May 11th at 11:59 PM Pacific. (Radar readers can use "vel09cmb" for an additional 15% discount.)

Importance of Innovation in Finance & BarCampBank

“Progress is not the mere correction of evils. Progress is the constant replacing of the best there is with something still better.” -Edward Filene

Two years ago, when we were organizing the first BarCampBank in the US, many people found it hard to believe that banks & credit unions could a place for meaningful grassroots innovation. Even crazier was the idea of organizing an unconference to begin bringing open source, transparency, identity, and community into the very closed world of banking & finance.

Since then the BarCampBank idea has turned into a movement. There have been over 14 events all over the world, and many of the ideas generated are beginning to turn into action.

To me, the global financial system is a platform that exists to “create more value than it captures”. Tim explained this in his Work on Stuff that Matters post, saying:

“A bank that loans money to a small business sees that business grow, perhaps borrow more money, hire employees who make deposits and take out loans, and so on. The power of this cycle to lift people out of poverty has been demonstrated by microfinance institutions like the Grameen Bank. Grameen is clearly focused on creating more value than they capture; not so the like of Fannie Mae and Freddy Mac, or WaMu, or many of the other failed financial institutions involved in the current financial meltdown.”

There has never been a more important time to bring meaningful innovation into the financial system, and there has never been more opportunity for our community to make it happen.

The next event is occurring this weekend (April 25-26, 2009) on Treasure Island in San Francisco.

After that, the following events are planned:

• BarCampBankVegas is set for May 2, 2009.
• BarCampBankCharleston2 is set for June 13, 2009
• BarCampBankGermany is set for October 23-25, 2009

AT&T; Fiber cuts remind us: Location is a Basket too!

The fiber cuts affecting much of the San Francisco Bay Area this week are similar to the outages in the Middle East last year (radar post), although far more limited in scope and impact.   What I said last year still holds true and is repeated below: 

From an operations perspective these kinds of outages are nothing new, and underscore why having "many eggs in few baskets" is such a problem. I believe we will see similar incidents when we have the first multi-datacenter failures where multiple providers lose significant parts of their infrastructure in a single geographic area.

Remember: Don't put all your eggs in one basket... and Location is a basket too!

To really understand the issue, I recommend Neal Stephenson's incredible (and lengthy) Wired article from 1996 entitled "Mother Earth Mother Board":

[...] It sometimes seems as though every force of nature, every flaw in the human character, and every biological organism on the planet is engaged in a competition to see which can sever the most cables. The Museum of Submarine Telegraphy in Porthcurno, England, has a display of wrecked cables bracketed to a slab of wood. Each is labeled with its cause of failure, some of which sound dramatic, some cryptic, some both: trawler maul, spewed core, intermittent disconnection, strained core, teredo worms, crab's nest, perished core, fish bite, even "spliced by Italians." The teredo worm is like a science fiction creature, a bivalve with a rasp-edged shell that it uses like a buzz saw to cut through wood - or through submarine cables. Cable companies learned the hard way, early on, that it likes to eat gutta-percha, and subsequent cables received a helical wrapping of copper tape to stop it.

[...] There is also the obvious threat of sabotage by a hostile government, but, surprisingly, this almost never happens. When cypherpunk Doug Barnes was researching his Caribbean project, he spent some time looking into this, because it was exactly the kind of threat he was worried about in the case of a data haven. Somewhat to his own surprise and relief, he concluded that it simply wasn't going to happen. "Cutting a submarine cable," Barnes says, "is like starting a nuclear war. It's easy to do, the results are devastating, and as soon as one country does it, all of the others will retaliate."

As the capacity of optical fibers climbs, so does the economic damage caused when the cable is severed. FLAG makes its money by selling capacity to long-distance carriers, who turn around and resell it to end users at rates that are increasingly determined by what the market will bear. If FLAG gets chopped, no calls get through. The carriers' phone calls get routed to FLAG's competitors (other cables or satellites), and FLAG loses the revenue represented by those calls until the cable is repaired. The amount of revenue it loses is a function of how many calls the cable is physically capable of carrying, how close to capacity the cable is running, and what prices the market will bear for calls on the broken cable segment. In other words, a break between Dubai and Bombay might cost FLAG more in revenue loss than a break between Korea and Japan if calls between Dubai and Bombay cost more.

The rule of thumb for calculating revenue loss works like this: for every penny per minute that the long distance market will bear on a particular route, the loss of revenue, should FLAG be severed on that route, is about $3,000 a minute. So if calls on that route are a dime a minute, the damage is $30,000 a minute, and if calls are a dollar a minute, the damage is almost a third of a million dollars for every minute the cable is down. Upcoming advances in fiber bandwidth may push this figure, for some cables, past the million-dollar-a-minute mark. [Link]

(Image source: https://www.flickr.com/photos/mundane_joy/2301368102/)

It's Really Just a Series of Tubes

Molly Wright Steenson hit the Ignite jackpot at Etech this year with her explanation of the steam powered network of pneumatic tubes of the 1800s. If you're someone that, like me, has a somewhat obsessive relationship with Internet Infrastructure, you must watch this talk.

Understanding Web Operations Culture - the Graph & Data Obsession

We’re quite addicted to data pr0n here at Flickr. We’ve got graphs for pretty much everything, and add graphs all of the time.

-John Allspaw, Operations Engineering Manager at Flickr & author of The Art of Capacity Planning

One of the most interesting parts of running a large website is watching the effects of unrelated events affecting user traffic in aggregate. Web traffic is something that companies typically keep very secret, and often the only time engineers can talk about it is late at night, at a bar, and very much off the record.

There are many good reasons for keeping this kind of information confidential, particularly for publicly traded companies with complicated disclosure requirements. There are also downsides, the biggest being that is difficult for peers to learn from each other and compare notes.

John Allspaw recently created a WebOps Visualizations group on Flickr for sharing these kinds of graphs with the confidential information removed. Here’s an example of a traffic drop seen both by Flickr & by Last.FM that coincided with President Obama’s inauguration.

Similar traffic drop on Last.FM seen on the right

Google saw a similar drop as well

Was it because everybody went to Twitter?

Besides being an interesting story, sharing these kinds of graphs help people build better monitoring tools and processes. As just one example: How should the WebOps team respond to this dip in traffic? Is it an outage? The inaguration was a very well known event and so it’s easy to explain the drop in traffic… what happens when a similar drop in traffic occurs? Should the WebOps team be looking at CNN (or trends in twitter) along with everything else?

How do you tell when that unexpected 10% drop in traffic is really just people with something more important to do than browse your site?

(Note: Updated since original posting to add Google & Twitter graphs and annotations, and to switch the Last.FM graphic with an annotated one after I got permission.)

Data Center Power Efficiency

James Hamilton is one of the smartest and most accomplished engineers I know. He now leads Microsoft's Data Center Futures Team, and has been pushing the opportunities in data center efficiency and internet scale services both inside & outside Microsoft. His most recent post explores misconceptions about the Cost of Power in Large-Scale Data Centers:

I’m not sure how many times I’ve read or been told that power is the number one cost in a modern mega-data center, but it has been a frequent refrain. And, like many stories that get told and retold, there is an element of truth to the it. Power is absolutely the fastest growing operational costs of a high-scale service. Except for server hardware costs, power and costs functionally related to power usually do dominate.

However, it turns out that power alone itself isn’t anywhere close to the most significant a cost. Let’s look at this more deeply. If you amortize power distribution and cooling systems infrastructure over 15 years and amortize server costs over 3 years, you can get a fair comparative picture of how server costs compare to infrastructure (power distribution and cooling). But how to compare the capital costs of server, and power and cooling infrastructure with that monthly bill for power?

The approach I took is to convert everything into a monthly charge. [...]

Velocity 2009: Themes, ideas, and call for participation...

Last year's Velocity conference was an incredible success. We expected around 400 people and we ended up maxing out the facility with over 600. This year we're moving the conference to a bigger space and extending it to 3 days to accommodate workshops and longer sessions. Velocity 2009 will be on June 22-24th, 2009 at the Fairmont Hotel in San Jose, CA.

This year's conference will be especially important. I've said many times that Web Performance and Operations is critical to the success of every company that depends on the web. In the current economic situation, it's becoming a matter of survival. The competitive advantage comes from the ability to do two things:

1. Generate more revenue with fewer resources
2. Respond quickly to change

I'm excited to announce that joining Steve Souders & I on this year's program committee are John Allspaw, Artur Bergman, Scott Ruthfield, Eric Schurman, and Mandi Walls.  We've already started working on the program, and have just opened the Call for Participation.

(continue reading)

Major milestone for ProgrammableWeb & "The Web as Platform"

Last week marked an important milestone for the "Web as Platform" as the 1,000 API was added to the ProgrammableWeb registry. John Musser (see: Web2.0 Report) started tracking the first few web service API's back in 2005.

How do these 1000 APIs break down by type? The following chart, derived from our database, shows the the top 15 sectors or markets with the greatest number of competing API providers. As you can see there are already 71 mapping-related APIs alone"

Recent Posts

• DisasterTech: "Decisions for Heroes" on November 1, 2008
• Sprint blocking Cogent network traffic... on October 31, 2008
• Amazon's new EC2 SLA on October 24, 2008
• Incredible images of the Sun on October 15, 2008
• Apple's restrictions mean more jailbreaking & Android adoption on September 23, 2008
• Kaminsky DNS Patch Visualization on August 7, 2008
• The new internet traffic spikes on June 28, 2008
• Video of Rich Wolski's EUCALYPTUS talk at Velocity on June 24, 2008
• Hyperic CloudStatus service dashboard launches at Velocity! on June 23, 2008
• code_swarm - visualizing the life of open source on June 18, 2008