Six Apart VP of Services David Jacobs says the TypePad Developer Platform chose GitHub, “as the distribution point for our software, largely because of the social features and because we wanted a distributed group of developers to be able to keep up to date with our tools, and contribute back, very easily.”
and Motion – a Django app powered by the TypePad Developer Platform (docs)
All three projects use GitHub Pages and github-tools to publish their documentation, which is kinda cool.
(Side note: the helper app actually includes some JavaScript I wrote for Baconfile but never released.)
It’s great to see companies embracing open source and even better to see them use GitHub as a way to lower the barrier for contributors. Can’t wait to see how this turns out!
Multi Theft Auto is how you play multiplayer Grand Theft Auto on your computer. It looks pretty sweet and I remember trying it when it first came out a few years ago.
As I’m sure you’re aware, Monday morning’s billing batch processed twice. As soon as we realized this we ran a batch to void the duplicate transactions. We thought this solved the issue, but we were wrong.
Today I got a report that a user’s card was in fact billed twice. Upon further investigation it appears our void batch missed a small handful of users. Specifically, users whose card had declined in a previous batch and were being retried did not have the duplicate charge voided. I have gone through each of these customers by hand and refunded the duplicate transactions.
I’d like to urge everyone who received receipt emails on Monday to double-check with your bank that only one charge has processed through. If the duplicate did post, please don’t hesitate to contact support so I can ensure that a refund has been issued.
I can’t apologize enough for this, we’re really sorry folks.
Monday has come and gone, no one is happier about this than I am. As day 2 is wrapping up, I’d like to post an update for those of you that have been following us.
Before I get to the details, I’d like to remind everyone to open tickets on Tender if you have any issues. For those of you that have open tickets never fear, we’re burning through tickets as fast as we can. When I logged in this morning we had over 120 tickets in the inbox, right now we’re sitting at 35. Unfortunately Tender doesn’t give me stats on the number of tickets opened today, but we’ve dealt with at least 85 plus however many new tickets were opened during the day. I’d bet that’s a few dozen at least. To put that in perspective, before the move on any given stable day we saw maybe a dozen tickets, and on unstable days anywhere from 20 to 40 new tickets on average.
Bugs squashed today
Everyone was busy all day today killing bugs. At one point I estimate mojombo was handling up to 800 problems a minute. Here’s a rundown of the major bug fixes:
502 Bad Gateway errors should, for the most part, be gone. These were hitting gist creation, applying commits in the fork queue and user creation the worst. If you run into this error please open a ticket and detail what you were doing when you encountered it.
Repo under migration errors should also be gone. This included a patch to ensure user paths were generated if they didn’t exist on lookup and a few batch jobs to force repos that had not been created on disk to generate. If you have any new, unpushed repos throwing errors please let us know.
Errors when pushing repos with submodules should be gone now.
A well placed kick to the mail server got outgoing emails working again.
User deletion should work again, I have not personally tested this one because I can’t bear to see users go.
New Pages were not being set up, existing pages were not building on push, and pygments was missing on the Pages server. All of these should be fixed now. If your page has not built, make a new commit to trigger a new build. If you new page hasn’t been set up on the server, open a ticket so we can force it to run.
Outstanding issues
Gist forking still broken.
Zip/tar downloads are failing to generate for a few repos. Please open a ticket and let us know what repo so we can pin this one down.
Connections to ssh.github.com:443 are still not working. We’ve already applied one fix here but it did not work. We are still working on this, firewalled folks, all I can ask is that you be patient.
Some gists made shortly before the lockdown on Sunday did not sync to rackspace.
Gem building is still disabled. We have no ETA on this one, but we’re not afraid to shamelessly plug gemcutter in the mean time.
and now for something completely different
Since the move, we’ve seen over 1000 new users and 2000 new repos. We’ve processed over 800k background jobs, and the background job queues are blazing fast on the new servers. Before the move our low-priority queue (network graph updates, http cloning updates and some other jobs) was backed up for a few weeks. At peak times it would rise to 40k jobs or more. Graph updates were often delayed many hours. We had 25 job runners working nearly full time on the low prio queue. On the new servers we have 40 workers, however they are sitting idle most of the time. Usually only 2 or 3 are active at any given time. The jobs queues, including the low prio queue, have stayed near zero since we cut over the DNS.
HI folks, just wanted to give you all a little feedback on the move again. Get everyone up to speed on the currently known issues.
First and foremost, if you are having any troubles, please don’t hesitate to open a ticket on Tender. If you have an outstanding ticket, please be patient, we’ve not forgotten you. We’re still working through the backlog of tickets to make sure everything gets fixed. I myself took care of roughly 60 tickets early this afternoon, and the inbox is still rising.
The guys spent the better part of the evening together going over the new architecture and making sure everyone is on the same page. I suspect that tomorrow will be devoted to bug fixes and ironing out all the wrinkles from the transition.
At any rate, on to the currently known issues.
Pushing repos that have submodules in subpaths fail to push. We have a fix for this thanks to a user on IRC, it should go out soon.
New user creation throws an error, creates the user in the database but not on disk. Please open a ticket with the username so we can fix this.
New repo creation may fail to create the repo on disk. Again, let us know the repo and we’ll force the creation.
Some repos are throwing “Repo under migration” errors. Let us know what repo so we can re-sync it.
Some repos did not sync after the initial dump, thus showing two week old commits. With these repos you can usually just re-push from your local repo. If you don’t have a repo to push from let us know and we’ll re-sync it for you.
Gist creation was failing, this should be resolved now.
User deletion fails to process now… but why would you ever want to delete your account?
Zip/tar downloads are failing to generate for a few repos. Please open a ticket and let us know what repo.
The Fork Queue may not work or throw errors. If you encounter this you can fetch the repo locally and cherry-pick or merge.
Emails are still not going out. We thought we had this fixed, but it seems to still be happening. If you need to reset an account’s password, please email support@github.com from the account’s primary email so that we can verify ownership and send you a new password over email.
Connections to ssh.github.com:443 are not working. This impacts firewalled users primarily. We’re still working on this one.
Again, thank you everyone for your patience with the fallout from the move. If you have troubles please open tickets on Tender. If you have not yet received a reply to your ticket don’t fear, we will get to your issue as soon as we can.
Your owned and watched repositories in the dashboard sidebar are now ordered by the most recent push and limited to 10. The watched repos sidebar also has a “public / private / sources / forks” filter.
Click “see all” to load all the repositories. We hope you like it – it should speed up the dashboard for everyone, especially those of you watching over 300 repositories!
If you received two email receipts tonight it’s because we accidentally billed you twice. That said, the second charge has already been voided.
In the craziness of migrating hosts we forgot to turn off the cron jobs on our Engine Yard slices. One of those was our nightly billing job so it ran once on EY and again on Rackspace. Fortunately our payment processor, Braintree, has a great API that allowed me to grab all of the customers that were billed twice this morning and void the second charge in about 10 lines of ruby.
This is particularly embarrassing for me, because it could have been easily avoided, but I wanted to be totally honest about what happened and let you know that we took care of it.
Now let’s all get back and enjoy crazy-fast GitHub!!
Thanks to the move to Rackspace, our Pages server’s IP has changed. Pages users that have A records pointing to the EngineYard Pages server need to update their DNS info to point to 207.97.227.245. Users with CNAMEs should update automatically. We will leave the EY Pages server up for a short time, but new Pages will only show up here on the Rackspace Pages server.
Hi folks, if you’re reading this, you’re on the new servers! I wanted to put up a quick update so that everyone knows what to expect going into our first week on Rackspace. Interesting side note, this is also our 500th post. I think someone planned that…
At any rate, all in all the move is progressing nicely. We’ve run into a few issues, but we were expecting something to happen. Damn that Murphy.
We had to drop the records in events table for the time being. These are what show up in your dashboard and profile. We will be bringing this data back in, and new events should be generating as normal.
The gem builder is off for the time being. Gems that have already been built will still be served, but new ones won’t build right now. For those of you that need your rubygem fix, I’d like to shamelessly plug gemcutter
Forks will be giving out warnings on pull telling you to check .git/objects/info/alternates. These are a harmless side-effect of the de-alternates script we ran on repos on Rackspace. Once we’re satisfied that no repos are broken we will patch up these error messages.
New repos are currently being created as non-bare. If you’re running a newer version of git you may get a long warning about pushing to a non-bare repo. You can ignore that warning. We will fix up the currently clothed repos after we’ve fixed the issue causing them not create bare in the first place.
Some new repo jobs did not create on disk while we were trying to fix the bare repo issue. If you created a new repo after the move and pushing fails with an error, please open a ticket so we can ensure your repo is created.
Thanks to our new servers, known bad keys from Debian and Ubuntu are now blacklisted. We estimate that about 1000 keys in our database were impacted by this. If you get authentication errors using keys that worked a day ago, please double-check that they are not on our blacklist. If they are, you should ensure your software is updated and generate new keys. We’ve got a guide to help you out with this.
Some users that are failing key auth are being prompted for git@github.com’s password. You should consider this the same as a Permission denied (publickey) error.
Clone URLs beginning with ssh:// are not functioning currently. This appears to effect users running corkscrew to bypass firewalls and users of pip.
Update ssh cloning should be back now, please let us know if you have any issues.
Re-update I think this only fixed users on port 22, corkscrew users are probably still going to hit errors
Tarball and zip package generation was not working initially. These should work now, please let us know if they don’t.
Repo clone stats are not updating currently, this should be back in a few days.
There you have it. Overall we’re very happy with the progress so far. We’re already seeing marked improvement in site performance. Oh, and if you didn’t notice, there’s some tweaks to the dashboard to make dealing with repos and watches a little easier. You should see some more improvements to the dashboard soon.
This Thursday, September 24th ride your pony on down to this week’s drinkup at Homestead, 7:30pm. Buy a bowl of peanuts for a buck and throw the shells on the floor. Man, they dont call this the wild west for nothing.
On Sunday, September 27th at 5PM Pacific time (see it in your timezone) we will begin the final move to Rackspace. We are aiming to have less than an hour of website and push unavailability if everything goes according to plan. During this period, public and private clone, fetch, and pull operations will continue to function normally. Once the final data synchronization is complete and the new installation passes a final inspection, we will switch the DNS entries and you’ll be able to start using a new and improved GitHub!
The user impact from the switch should be minimal, but there are a few things to note:
On your first SSH operation to the new GitHub IP, your SSH client will issue a warning, but will not cause you any hassle or make you change any configuration:
After the move, GitHub Pages CNAME users will need to update their DNS entries that reference GitHub by IP. We will announce on this blog and via the @github twitter account when you should update your entries. The new IP address will be 207.97.227.245. Only domains with A records (top-level domains like tekkub.net) need to make this change. Sub-domains should be using CNAME records, which will update automatically. Please contact support if you have any questions.
We appreciate your patience during this procedure and look forward to all the benefits that the new hardware and architecture will make possible!
