We appreciate your patience. All SourceForge services are operational at this time with the exception of ViewVC and CVS. Interactive shell service was restored today and we expect ViewVC to return today (Monday), barring complications. We intend to have CVS service online by the end of the week.

Please note that, aside from CVS, our SCM platforms (Subversion, Git, Hg, Bazaar) have remained online. If you’re encountering issues with these platforms, please confirm that you have reset your password. If problems persist, please contact our support team (sfnet_ops@geek.net).

CVS service restore has been time-consuming due to the design of that service. CVS data is housed on direct-attached storage on a small pool of servers. Data restore from backups is in-progress and reload of machines is in-progress as of today (Monday). We intend to have CVS service online by the end of the week.

We have concluded our review of SCM data for the SourceForge Beta SCMs and traditional SCM offering (SVN, Git, Hg, Bzr; excepting CVS, whose data is being restored) and have found no indicator of data tampering. We will be providing further data to projects whose repositories were modified in the attack window (we believe legitimately by their project team) as that they may conduct further validation using the repository access provided through interactive shell service.

Finally, with the rollout of interactive shell service today, we are completing preparations to start the mass rollout of our new Project Web service offering tomorrow (Tuesday). This work will commence with the migration of projects in the A-G unixname range in a Tuesday/Wednesday timeframe.

If you have any questions or concerns, please contact our support team at sfnet_ops@geek.net

Thank you,

The SourceForge Team

As we mentioned yesterday, work is continuing on SCM data validation, and with project web and interactive shell services.

Project Files and Mirrors Update

File Release services came online Tuesday, including the ability to upload new files for download. Data validation of our mirror network has continued and we’re happy to announce that 18 mirrors have been validated, and are synced with new release data. We’re not quite back to full capacity, but we’ll have no trouble handling normal load.

SF.net Beta SCM update

The beta SCM data is hosted on an updated platform, and none of these servers were compromised during this attack. The data, however, was accessible to the attacker. We’ve completed the SCM data validation for the SourceForge Beta and don’t believe there was any tampering. We will be publishing the validation results tomorrow at the same time we cutover this service to new systems with improved security controls.

Non CVS SCM data

We are still working on validating SVN, Hg, Bzr, and Git data on the main sourceforge.net SCM servers. These servers weren’t compromised, but the SCM data was accessible to the attacker. At this time we don’t have any evidence of tampering with SCM data. We will publish the full results of our validation work when the work is complete.

We have also redesigned the platform for these services, and will be pushing out updated configurations and improved security controls. We expect the updates to this service and the results of this validation work to ship later this week. ViewVC (web-based SCM access) will be brought online as we ship the updated SCM servers.

CVS data

CVS servers were compromised, so we are taking extra time with this data. CVS requires the significant validation effort, and its configuration made it harder to get the data we needed to start validation. The good news is that comparisons are running now against backup data.

We still expect this to be one of the last services restored, but are committed to making that happen as quickly as possible. ViewVC (web-based SCM access) for CVS will be brought online as we ship the updated SCM servers.

Project Web

Preparations to roll-out our updated project web offering are also in-progress. Our updated project web service has already been deployed to some projects. This service was not compromised during the attack.

As a precautionary measure, we’ve reloaded our new project web servers and have applied further security controls. We’ll be rolling this service out to all projects starting next week, starting with projects A through G (by UNIX name).

Interactive Shell services will be brought online with this new project web launch and should be available again as soon as individual projects are migrated over to the new system. In the mean time, project web content may be managed via SCP, SFTP or rsync over SSH.

We appreciate your support and will continue our efforts over the coming days!

We love all the projects at SourceForge and, like mom, we’d never play favorites. We’ll admit though, we just adore February’s project of the month, eGuideDog. Under development since 2006, this amazing and frequently-updated project creates free software for the blind.

“The TTS engine is a basic software for applications with voice. WebSpeech is a Javascript library for web developers to write pages with voice. I also released two games based on WebSpeech for the blind. WebAnywhere, which we help Jeffrey P. Bigham develop, is for blind users to access websites without a desktop screen reader, and Gradint is a blind-friendly language learning system,” explains Project Leader Cameron Wong.

Wong does the majority of software development himself and devotes about 50 hours of his time each month to the project, and he’s quick to credit key contributor Silas Brown for his “significant suggestions, help, and encouragement.” Brown is also one of the voices behind open source speech synthesizer eSpeak.

Run, don’t walk over to eGuideDog’s Project of the Month page to learn more about what Wong is doing and consider pitching in to help. Even if all you can spare is a couple of hours a month, it’s still valuable to such a worthwhile project.

File upload services

The shell servers used to provide this service were accessed illegitimately and have now been reloaded, with enhanced security controls.

File upload capability (SFTP, SCP, rsync over SSH) has been restored. Web based updates should work again immediately. SSH host keys have been updated and new fingerprints have been published to Site Docs and Site Status.

SSH host key fingerprints: https://sourceforge.net/apps/trac/sourceforge/wiki/SSH%20key%20fingerprints

User SSH key data

User SSH public key data was accessible during this attack. Users may wish to generate new SSH keys on a precautionary basis, though it is generally accepted that the exposure of public key data does not compromise the private key.

As a further precaution, we have processed all user SSH key data on file and have cleared SSH keys for users when we found anything extra data, e.g. private key data, or even junk text. Users whose keys were cleared will be notified by email and will need to generate and upload new keys.

SSH key generation instructions: https://sourceforge.net/apps/trac/sourceforge/wiki/SSH%20keys

File download services

Servers with write access to our file download data had been accessed illegitimately. As a precautionary measure we have done a complete reload of our master download mirror and controls around these servers have been enhanced.

We have identified modifications and uploads of files in the download service (File Release System) which occurred during the attack window by checking both timestamps and stored checksums. At this time we have no reason to believe any files were released or modified as part of this attack.

Projects with files added/modified during the attack window have been notified by email and while we believe these adds/changes are legitimate, we are asking these projects to double check our validation.

As a further precaution we are also in the midst of a full validation all downloadable files on all download mirrors. We have no reason to believe any tampering occurred with our download mirrors, and service will remain online as we complete our validation. Mirrors will be updated with new file releases on a server-by-server basis as we complete this validation work.

Other services

SCM data validation is in-progress. CVS service and ViewVC (web browsing of SCM) service remain offline pending completion of data validation activities.

Preparations to roll-out our updated project web offering are also in-progress. Interactive shell service is presently offline and will be restored in the same context as project web service restoration activities.

As we’ve previously announced, SourceForge.net has been the target of a directed attack. We have completed the first round of analysis, and have a much more solid picture of what happened, the extent of the impact, our plan to reduce future risk of attack. We’re still working hard on fixing things, but we wanted to share what we know with the community.

We discovered the attack on Wednesday, and have been working hard to get things back in order since then. While several boxes were compromised we believe we caught things before the attack escalated beyond its first stages.

Our early assessment of which services and hosts were impacted, and the choice to disable CVS, ishell, file uploads, and project web updates appears to have prevented any further escalation of the attack or any data corruption activities.

We expect to continue work on validating data through the weekend, and begin restoring services early next week. There is a lot of data to be validated and these tests will take some time to run.  We’ll provide more timeline information as we have more information.

We recognize that we could get services back online faster if we cut corners on data validtion. We know downtime causes serious inconveniences for some of you. But given the negative consequences of corrupted data, we feel it’s vital to take the time to validate everything that could potentially have been touched.

Attack Description

The general course of the attack was pretty standard. There was a root privilege escalation on one of our platforms which permitted exposure of credentials that were then used to access machines with externally-facing SSH. Our network partitioning prevented escalation to other zones of our network.

This is the point where we found the attack, locked down servers, and began work on analysis and response.

Immediate Response

Our first action response included many of the standard steps:

* analysis of the attack and log files on the compromised servers
* methodically checking all other services and servers for exploits
* further network lockdown and updating of server credentials

Service shutdown

Once we knew the attack was present, we locked down the impacted hosts, so that we could reduce the risk of escalation, from those servers to other hosts, and prevent possible data gathering activities.

This strategy resulted in service downtime for:

* CVS Hosting
* ViewVC
* New Release upload capability
* ProjectWeb/shell

Password invalidation

Our analysis uncovered (among other things) a hacked SSH daemon, which was modified to do password capture. We don’t have reason to the attacker was successful in collecting passwords. But, the presence of this daemon and server level access to one-way hashed, and encrypted, password data led us to take the precautionary measure of invalidating all SourceForge user account passwords. Users have been asked to recover account access by email.

Data Validation

It’s better to be safe than sorry, so we’ve decided to perform a comprehensive validation of project data from file releases, to SCM commits. We will compare data agains pre-attack backups, and will identify changed and added. We will review that data, and will will also refer anything suspicious to individual project teams for further assessment as needed.

The validation work is a precaution, because while we don’t have evidence of any data tampering, we’d much prefer to burn a bunch of CPU cycles verifying everything than to discover later that some extra special trickery lead to some undetected badness.

Service Restoration

Now that most of the analysis is done, we’ve started the next stage of our efforts, which includes the obvious work of restoring compromised boxes from bare metal, and implementing a number of new controls to reduce likelihood of future attack.

We will of course also be updating the credentials which reside on these hosts and performed quite a few steps to further lock down access to these machines.

We are in process of bringing services back one by one, as data validation is completed, and we get the newly configured hosts online. We expect that data validation will progress through the weekend, and we’ll really start getting swinging on service restoration early next week.

File Release Services

Many folks have suggested that the most likely motivation for an attack against sourceforge would be to corrupt project releases.

We’ve found no evidence of this, but are taking extrodinary care to make sure that we don’t somehow distribute corrupted release files.

We are performing validation of data against stored hashes, backups, and additional data copies.

We expect to restore these services first, as soon as data validation is completed.

Project Web

One attack vector that impacts our services directly is the shared project web space. So, let’s talk about that in a bit more detail.

Sourceforge.net has been around a long time, and security decisions made a decade ago are now being reassessed. In most cases past decisions were made around the general principle that we trust open source developers to work together, play nice, and generally do the right thing. Services were rolled out based on widespread trust for the developer community. And that philosophy served us well.

But in the years since then, we’ve evolved from hundreds of sf.net users to millions, and in many cases it’s time to re-asses the balance between widespread trust and security. Project Web is a prime example of this, and we’ve been working at a deliberate pace to isolate project web space, and have begun rolling out the new “secure project web” service to many of our projects.

This new secure project web includes a new security model that moves us away from shared hosting while preserving the scalability we need for mass hosting.

Because of this attack we’ll be accelerating the rollout of Secure Project Web services as part of the process of bringing the project web service back online. This will allow us to provide both improved functionality, and better secruity.

CVS

CVS service is one of SourceForge.net’s oldest services and, due to limitations in CVS itself, cannot readily live on our scalable network storage solution. Validation of this data is going to require several days and we anticipate that this service will be restored sometime in the later part of week.

We are also considering the end-of-life of the CVS service and hope to have user support in migrating CVS users to Subversion in coming months. Subversion generally provides parity to CVS commands, and many of our users have made this transition successfully in the past.

From SVN, projects can move to Git if desired.

Looking forward

We are very much committed to the ongoing process of improving our security, and we will continue making behind the scenes improvements to our infrastructure on a regular basis. This isn’t a one time event, it’s a process, and we’re going to stay fully engaged over the long term.

I’d like to end with a more personal note, I’ve been working with our Ops team a lot this week, and I think we can all say that the patience and support that we’ve received from the community has been the best part of a very bad week.

Thanks again for all the support and encouragement.