Ride your pony on down to this week’s drinkup at Homestead, 7:30pm. Buy a bowl of peanuts for a buck and throw the shells on the floor. Man, they dont call this the wild west for nothing.

On Sunday, September 27th at 5PM Pacific time (see it in your timezone) we will begin the final move to Rackspace. We are aiming to have less than an hour of website and push unavailability if everything goes according to plan. During this period, public and private clone, fetch, and pull operations will continue to function normally. Once the final data synchronization is complete and the new installation passes a final inspection, we will switch the DNS entries and you’ll be able to start using a new and improved GitHub!

The user impact from the switch should be minimal, but there are a few things to note:

• On your first SSH operation to the new GitHub IP, your SSH client will issue a warning, but will not cause you any hassle or make you change any configuration:
  
  
• After the move, GitHub Pages CNAME users will need to update their DNS entries that reference GitHub by IP. We will announce on this blog and via the @github twitter account when you should update your entries. The new IP address will be 207.97.227.245. Only domains with A records (top-level domains like tekkub.net) need to make this change. Sub-domains should be using CNAME records, which will update automatically. Please contact support if you have any questions.

We appreciate your patience during this procedure and look forward to all the benefits that the new hardware and architecture will make possible!

Both user and repository renaming are temporarily disabled. Sorry about that!

It’s for a good cause. Should be back in a week.

I just wanted to let you all know that ack, a better grep (and one of my favorite tools), is now hosted on GitHub: https://github.com/petdance/ack

Watch the project and start wondering how you ever lived without it.

_   /|
\'o.O'
=(___)=
   U    ack!

(And don’t forget ack.tmbundle, ack.vim, or ack.el while you’re at it.)

We’ve got another URL cleanup for you: ref names in tree browsing URLs.

In The Old Days

You’d take this path:

• https://github.com/ezmobius/redis-rb/tree/master
• https://github.com/ezmobius/redis-rb/tree/308c4e3fb7df212cd4c9f06260bd68a3566cb53c/lib
• https://github.com/ezmobius/redis-rb/blob/308c4e3fb7df212cd4c9f06260bd68a3566cb53c/lib/redis.rb

Gross.

In The New Days

You take this path:

• https://github.com/ezmobius/redis-rb
• https://github.com/ezmobius/redis-rb/tree/master/lib
• https://github.com/ezmobius/redis-rb/blob/master/lib/redis.rb

Much nicer.

In just a few short weeks we will be moving GitHub to a new home at Rackspace. We’re aware of the current stability and performance issues, and we want to let you know what we’re doing about it. After all, we’re GitHub users too! The move to Rackspace will bring about a new backend architecture and a lot more servers, leading to a much improved user experience for everyone. Thanks for sticking with us through our growing pains!

Since we have a highly technical audience, I wanted to share some background on the reasons behind the move, what we’ve been doing to prepare for the big changes ahead, and what kinds of service improvements you can look forward to seeing on the new infrastructure.

As you may know, we’ve had a hosting partnership with Engine Yard since very early in the GitHub story. Engine Yard is dedicated to supporting open source initiatives and saw value in helping us grow the site to foster innovation within the Ruby community. For their tireless support and expertise, we are extremely grateful. We wouldn’t be where we are now without them.

The decision to move hosts is never an easy one. The logistics of migrating a site as large and complex as GitHub are intimidating. The single most important reason we’re undertaking this effort is so that we can give you, our customer, a better experience on GitHub. We’re growing at a rate of over 400 new users and 1000 new repositories every day and these rates are only increasing with time. We need to take drastic action now to put in place the kind of infrastructure that will allow us to provide you with a top-notch user experience.

In making the decision to move hosts, we put together a set of requirements that would be necessary to ensure the viability of our business over the next ten years.

• Price. In order to keep ahead of the traffic curve, we need to have immediate access to affordable, commodity hardware. Within five years, it’s not hard to imagine that a cluster of 100 or more servers will be necessary to keep GitHub running smoothly. To guarantee a sustainable business, this amount of hardware must not be prohibitively expensive.

• Flexibility. We’ve grown to a size where it no longer makes sense to have every server virtualized. The benefits of running bare metal are obvious and have been empirically proven. We need to have the option to run bare metal when it is appropriate to the task at hand. We also need to be able to configure boxes with custom setups. If we need six large hard drives in a certain class of machine, then we must be able to get that. If we need boxes with 32GB or 64GB of RAM, those must be available.

• Capacity. It is undesirable to be the biggest fish in the pond. Our host must have experience with sites that are several orders of magnitude larger than us. We must feel comfortable knowing that all of the scalability requirements that we will encounter over the next ten years will be tractable on an available, battle-tested infrastructure.

• Control. Having direct access (via DRAC or similar) to the actual hardware means we can control every aspect of each server’s setup, from network layout and burn-in tests to operating system and RAID configuration. When system-level problems arise, we must be in a position to fix them without the need for outside intervention. At the end of the day, we should be responsible for as much of our stack as financially feasible.

• Globalization. Our long term plan involves making GitHub faster for our international customers. Our host should have data centers in Europe and Asia so that we don’t have to look outside our primary provider to provision hardware around the world.

• Cloud. On-demand access to a cloud infrastructure will be important to us as we increase the number and variety of low-frequency but long-running jobs that we process. A provider that has a first-class cloud offering would be ideal for keeping latencies low and pricing simple.

• Trust. Our host should be a big-name player in the hosting field with an excellent reputation and multiple recommendations from other large sites. The entire future of our company rests on making GitHub stable, fast, and efficient. It is essential that our host be able to keep up with our exacting standards and provide us with competent service.

After evaluating our options, it became evident that Rackspace was the right choice for our ongoing hosting needs. They meet or exceed every one of our requirements and are the only large provider with a strong offering in both traditional and cloud services. In addition, we’ve arranged a partnership deal with Rackspace that includes discounted hardware that will allow us to bring more machines online faster than would otherwise be possible. It was important to us that this partnership not create any conflict of interest, so we’ll be true paying customers of Rackspace, just with mutually beneficial opportunities that will help keep our plans at their current low prices.

To give you a concrete idea of what this new partnership means to you, consider this: on Engine Yard we currently have the following resources (not including DB and CORAID which are out of our control):

• 10 VMs
• 39 VCPUs
• 54GB RAM

On Rackspace, we’ll be enjoying the following setup:

• 16 physical machines
• 128 physical cores
• 288GB RAM

I think the specifications speak for themselves! Within the new hardware layout, we’re placing a significant importance on high availability and redundancy. On Rackspace, every piece of our infrastructure will have failover. That means two database servers, four web servers, two GitHub Pages instances, two Gem Server instances, two Archive Download instances, distributed Job runners, three pairs of file servers, and plenty more.

Speaking of file servers, the move to Rackspace means we’ll finally be leaving our shared file system behind. We’ve far exceeded the normal IO tolerances of GFS and it has become the source of many of the problems in our stack. It has also prevented us from adding additional hardware to the site for nearly a year. Tremendous kudos go to Chris Wanstrath for his ceaseless and amazing work over the last six months to optimize the site enough that it stays running (hurray for Memcache and Redis!).

Since April I’ve been working on a brand new federated backend architecture that will allow us to store repositories on commodity file servers. When we need more storage capacity, we merely have to add more machines and update a routing table. The file servers expose an RPC interface to the Git repositories that can be accessed from anywhere in the cluster. This will allow us to horizontally and separately scale the frontend, backend, and other pieces of the infrastructure.

There are too many improvements in too many parts of our process and infrastructure to cover in this already lengthy post. I’ll happily dive into the specifics of the new architecture and other logistics in a series of follow up articles over the coming weeks.

Right now we’re putting the finishing touches on the production Rackspace cluster and working on the big repository data migration. I’ll be keeping you updated on the progress and the countdown to the final move. We’re aiming to restrict downtime to the day of the actual move and limit service interruption as much as possible.

It takes a lot of effort to build, maintain, and host a site like GitHub. I’d like to thank Engine Yard for getting us to where we are, Rackspace for helping us get to where we’re going, and you for making GitHub such an amazing project to work on.

As some may have noticed, project root URLs now drop the “/tree/default_branch” suffix.

What was once https://github.com/mojombo/jekyll/tree/master is now https://github.com/mojombo/jekyll.

(Of course “tree/master” will 301 redirect, don’t worry.)

Enjoy!

Our Portland meetup was success – thanks to everyone for coming out!

Tomorrow (September 15th) night I’ll be at the Twisted Spoke in Chicago around 7:30pm.

If you’re a fan of GitHub in Chicago, stop by and say hi!

(Google Maps)

Hot on the heels of my git setup script, user Sirupsen has made a
Starting with Git guide. Okay, that’s a lie, he actually published his guide two days before my script… I’m a slacker, I know, I apologize.

At any rate, thanks for the great guide Sirupsen!

The results have been tallied and the curtains have closed. The 2009 GitHub Contest prize of a Large account and a bottle of 20 year Pappy Van Winkle goes to the Hintly Ensemble team of Daniel Haran (danielharan) and Liang Xiang (xlvector).

Daniel wrote about the contest results on his blog. To find out about how they tackled it, the post points to the winning projects README.

The secret sauce?
More data equals better results.

Congratulations to Daniel and Liang. I hope they can find some way to split the Pappy.

GitHub Community Spirit Prize

A second, secret prize is also being given to Jeremy Barnes (jeremybarnes) for his entry. Jeremys entry came in second place in the contest overall, but we were impressed with his openness during the contest. His results always included all of his source code and he went to great lengths to document how he was approaching the problem in his README. This is in contrast to the scores of entries that only included thier results.txt file and nothing else.

Since Jeremy so completely embraced what I envisoned as the spirit of the contest, we’re giving him a Medium account and a bottle of 23-year old Pappy.

(Glasses and cigars not included)

Congratulations to our two (three) winners, and thanks to everybody that participated!

I’ve been plotting to make the git and GitHub setup process a bit easier for new users, so I spent a small portion of my labor day weekend actually doing some labor (the horror). You can see the results in this repo.

So what is this crap? The goal here was to create a simple script users could run that would set all the config options needed in git, generate a key, upload the public key to GitHub, and encourage the use of strong passphrases and ssh-agent. The script needed to be able to run immediately after install, even for msysgit (windows) users. This ruled out my first choice, ruby, since windows users may not have ruby installed. The script also needed to detect msysgit users and help them set up ssh-agent to run automatically, as msysgit does not do this by default. Mac users don’t need this, as ssh-agent is run and even integrates with the user’s keychain, and many flavors of linux appear to run ssh-agent automatically for the user on login.

If you’ve got a fresh install, or just want to play, give the script a whirl:

$ git clone git://github.com/help/setup.git
$ setup/setup.sh

I’d love to hear feedback, suggestions for other things the script should set, or just pull requests.

As an added bonus, I also made a guide on passphrases and ssh-agent. Everyone should be using a good passphrase on their keys, I recently switched myself.

Whether you use Python, Ruby, Clojure, or good ol’ C, we want to see you at the GitHub Meetup in Portland tomorrow, September 9th at 7pm at Rontoms.

It’s a short, free bus ride (or a short walk) from the Dobuletree (google map). Come mingle with other DVCSers and drink one down in honor of the Octocat.

See you there! (Look for the guy in the Fork You shirt.)

Look at you. Fretting over your busy social calendar. You know its GitHub drinkup Thursday and yet you’ve just heard about the Functional Alcoholics meetup going on the same night! You feel torn. Your palms sweat. Now what are you going to do??

Tell you what. We at GitHub work hard every day to make your life easier, and relieving the weight of your many social burdens is no exception. Instead of making you choose between your favorite late-week, tech-centric drinking events, we’re going to combine them. That’s right. Put them together. In one venue. So. You. Can. Be. At. Both.

Just blew your mind, didn’t I?

The Facts:

Rebaaaaaaaase. I’m always looking for great projects to feature, so don’t hesitate to let me know about yours!

Featured Project

redis is a key-value store that just hit 1.0.0. So what? Well, it’s ridiculously fast, can handle sets and lists along with strings as values, it can asynchronously dump its data, supports sharding on the client side, and plenty more that wouldn’t fit here. It’s easier to think of it as a data structures server instead, or as an aptly named ‘REmote DIctionary Server’. The code is all here on GitHub if you’re a C hacker, and its API is well documented. Bindings are already available in Ruby, Python, PHP, and more so you can play around with it today.

Notably New Projects

clevercss is a much less verbose version of CSS that’s Python backed. This means, of course, it obeys the normal Python indentation rules, which ends up being a good thing. It does allow some magic like pseudo-class shortcuts, grouping shortcuts, constants, and even some basic Color and String methods if you’re feeling daring. If you’re a Pythonista and want write CSS easier, definitely give this a look.

TeXorator is a LaTeX viewer that auto-refreshes the document it’s looking at when changes are made. All you need is OSX and LaTeX installed, and you should be set. It also shows errors that may occur during the document’s generation. This definitely seems like a fun and useful project to hack on, and seeing the idea ported to other operating systems would be neat as well.

spde-examples combines two fantastic projects of the Java world: Scala and Processing. Well technically, Spde does the hard stuff, but this repo contains some seriously cool examples of how it can look. You could check out an example applet here, but you’ll be better served by cloning and running it yourself.

philip-rss is the first RSS reader that I can personally stand to use, mostly because it’s integrated deeply into Firefox. The workflow is simple: click Firefox’s built in RSS reader button to subscribe, and then a new RSS icon button in your lower right hand corner gives you a nifty view of your feeds. The UI is minimalist in a good way: it’s simple and doesn’t get in your way. Check out a more visual tutorial and how to install it here.

Ride your pony on down to this week’s drinkup at Homestead, 7:30pm. Buy a bowl of peanuts for a buck and throw the shells on the floor. Man, they dont call this the wild west for nothing.