The November Roundtable featured a demonstration of the current state of a new multi-factor authentication (MFA) system which Infra hopes to bring live soon The system is working from end to end, but a lot of automation needs to be completed before …
Danny Angus became VP-Infra in 2025, succeeding Myrle Krantz
What is your technical background?
I have no relevant qualification. I'm basically self taught; From books initially, then the internet. Participating in ASF projects has been a very important part of my learning, and that continues today.
The Infrastructure team was busy at Community over Code in Minneapolis this month. The team gave a track of presentations, had a host of valuable hallway conversations, and even hosted the first-ever Games Night at an ASF convention …
At the July Roundtable, Daniel Gruno presented on the catalog server, now in development: catalog-test.apache.org/lookup/project. The plan is to move all metadata that the Foundation and projects use into one place for reference and research:
Discussions at the June Roundtable covered, among other things:
Misuse of GitHub Actions. Projects are using, and sometimes force-pushing, tags, rather than SHAs. While there are GHA linters that can help identify problems, we should move actively to encourage projects to stop …
There was no roundtable in May. The next roundtable will be Wednesday, June 4, 2025, 1700 UTC. The main topic will be a presentation on JFrog and Artifactory, a universal binary repository manager that enables teams to store, manage, and organize software …
The next Infra roundtable will be Wednesday, April 2, 2025, 1700 UTC. This will be an "ask us anything" day, so feel free to bring any infrastructure issue that you find interesting/puzzling/scary.
The February roundtable took place Wednesday, February 5, 2025. The main topic was creation, review and management of custom GitHub Actions that projects can use in their compile and build processes, and the self-serve way to add custom GitHub Actions to the GHA Allowlist.
The January roundtable took place Wednesday, January 8, 2025. The main topic was a deep dive into asf.yaml, the Swiss Army Knife of ASF build utilities.
The next version of .asf.yaml addresses a number of headaches with the first version, such …
The December roundtable did not take place due to presenter illness. We have pushed its topic to next month. That roundtable will be Wednesday, January 8, 2025, 1700 UTC. The main topic will be a deep dive into asf.yaml, the Swiss Army …
Infra held two Roundtables this month. The November 6 meeting was at 1700 UTC (more convenient for folks in Europe and North America) and the meeting on the 13th was at 1000 UTC in hopes that ASF folks in Asia …
We are seeing a sharp increase in 'phishing attempts', fake emails pretending to come from various ASF groups and committees. Infra has some anti-phishing advice to help you avoid falling for these fake messages.
Chis Thistlethwaite, who has been an important part of the Infrastructure team for almost nine years, is moving on to another project. His last day of work at The ASF is October 11, 2024.
The Qbot assistant in the ASF Slack workspace is continuing to evolve. The latest addition is Lauren, a member of the imaginary ASF Human Resources department. Lauren occasionally appears in Infra team conversations, and we thought it would be fair …
25 years ago, it was somewhat difficult for people to put personal content onto the web, so the ASF provided that feature for our committers. It was first known as people.apache.org and later as home.apache.org. Many ASF committers have made use of the service, and it …
The Infrastructure team's Qbot is a Slack assistant that can simplify both important and frivolous tasks on Slack channels in the the-asf workspace. It can support requests in channels for PMCs, ASF committees, the ASF Board and …
A change to website creation and builds for projects
Many ASF projects, and the Infrastructure team, have been using the ASF-pelican template and service to build and deploy their websites. Infra has decided to deprecate its customized buildsite offerings in favor of those …
The dashboard, at infra-reports.apache.org, provides a collection of reports on the overall health and activity of the infrastructure at the ASF. These reports can be helpful in understanding the status of all the ASF 'under the hood' resources …
Due to misconfigurations in their builds, some projects have been using unsupportable numbers of GitHub Actions. As part of fixing this situation, Infra has added a 'resource use' section to the policy on GitHub Actions. This …
More than 50 people posted responses to our 2023 'year in review' survey. Here is a summary of the responses to questions and the comments added in each section.
We're grateful to those who took the time to fill out the survey. Without feedback like this, the Infra team can …
Someone who responded to our annual survey (see below) suggested that we start a newsletter to share developments and other news. The suggester thought we should send it to the dev@ list of every PMC and PPMC, but we are going to start with a more limited distribution …
Committers, contributors, and ASF Members have a wealth of information about every aspect of software and community development. When we share our knowledge, its reach multiplies and our effectiveness increases.
For the infrastructure that supports all our efforts at the ASF, here are some ways to share what you know …
Gradle, a longtime leader in developer productivity and build analytics, has become a platinum-level sponsor of the ASF. The ASF instance of Develocity, formerly Gradle Enterprise, is at https://develocity.apache.org.
But what is it?
Develocity is a suite of acceleration and analytics …
We're announcing a new look for selfserve.apache.org today! It's a completely updated site, featuring a new layout which we hope will suit everyone better. The new update also cleans up a some tech debt we've had with the old site, so adding new features or integrations …
I'd like to thank everyone who took time to take the survey. This is the first time that I know of that we've tried something like this, and feedback is very important. I'd also like to take a second to clarify my tone for this article. I'm presenting as …
This afternoon, we rolled out two new Shared Jenkins Nodes, jenkins-win-azr-7 and 8. They are both in rotation, using the labels Windows and Windows-Docker. The second label was put in place as the older nodes can't run Docker Desktop. I've also tried setting these up by cloning disks instead of …
Today, the Infrastructure Team took the step to disable public signups to ASF Jira
This was not done lightly. The main reason for this is the amount of spam and spam accounts that are created every day, it has
made managing Jira a big effort. In addition, Infra are planning …
Just moved posts over from blogs.apache.org/infra. New posts all go through the infrastructure-website repo and it should be as easy as posting some markdown.
At around 09:11 UTC on Tuesday, July 12th 2022, the primary mailing list server (colloquially known as Hermes) at The Apache Software Foundation suffered a fatal breakdown and became unresponsive.
The Infrastructure team (Infra) was immediately notified and, in cooperation with our data center provider, attempted …
Since before the start of COVID-19, the Infrastructure team had one open staff position. We have been able to fill it this year with a strong addition to the team, Chris Wells.
Where do you live?
Birch Run, Michigan, USA. I was born in Flint, Michigan, but I have moved …
As we head into April and the middle of the spring of 2022, we are pleased to announce that we have migrated our writable git repository service, gitbox.apache.org, to a new location, a new cluster of hardware bits, and a new platform. The migration took around 35 minutes …
The Infrastructure
team is constantly looking for ways to do its work more quickly while
maintaining the ASF standards of reliability, security, and
almost-continuous availability. The more the team can speed up and
improve standard processes that almost every project goes through,
the better it is for the whole community …
Today, ASF Infra launched new features for projects wishing to update their notification schemes for git/github activity on their own.
Via the .asf.yaml file, projects can now define notification schemes for their repositories, including targets for commit emails, GitHub events, and Jira notification options.
To better provide our millions of users with downloads, the Apache Infrastructure Team has been restructuring the way downloads work for our main distribution channels over the past few weeks. For users, this will largely go unnoticed, and for projects likely the same; but we …
**Note**: Until June, 2023, the ASF supported using Apache Roller for project blogs. That support has ended. Please visit Project Blog for more information about blogs for projects.
The Infrastructure team (Infra) works behind the scenes to make it possible for Apache's galaxy of committers to do the cool stuff they do, and for the open-source world to get, use, and rely on applications Apache projects produce. Infra supports additions to code repositories, a constant stream of conversation …
On August 31st 2019, the machine hosting our subversion-to-git mirrors and synchronization process for GitHub suffered a catastrophic drive error due to a power failure at our data center in Virginia. The power failure was, unfortunately, of such a nature, that recovering the disk data was not possible …
Over the past few days we have implemented rate limiting on selected services across the ASF.
As our foundation grows, so do the number of users and robots utilizing our services. In order to accommodate as many as possible with what resources we have, we have opted to implement rate-limiting …
**Note**: Until June, 2023, the ASF supported using Apache Roller for project blogs. That support has ended. Please visit Project Blog for more information about blogs for projects.
We've updated blogs.a.o to the latest version of Roller, 5.2.2!!
[IF YOUR PROJECT DOES NOT HAVE GIT REPOSITORIES ON GIT-WIP-US PLEASE DISREGARD THIS POST]
Hello Apache projects,
I am writing to you because you may have git repositories on the git-wip-us server, which is slated to be decommissioned in the coming months. All repositories will be moved to the new …
UPDATE: We have received enough applicants at this time. Thank you all for your interest.
The Apache Software Foundation (ASF) seeks to fill an Infrastructure Systems Administrator position. You will be responsible for working with the existing technical infrastructure team. The ASF manages a world-wide network of open source software …
When it comes to [Jenkins], it has long been known that [polling must die].
While we could go and create post commit hooks in all the ASF hosted Git repositories, that is something that realistically is just creating an added maintenance burden.
UPDATE: We have received enough applicants at this time. Thank you all for your interest.
The Apache Software Foundation (ASF) seeks to fill an Infrastructure Systems Administrator/Architect position. You will be responsible for working with the existing technical infrastructure team, and VP of Infrastructure at the Apache Software Foundation …
As people have noticed, our JIRA instance (arguably the largest public instance in the world) has been suffering from a yet unknown issue as of late. We are reasonably sure that this is related to specific queries being made against the instance (possibly automated queries from scrapers), but have yet …
My name is Daniel Takamori and I'm so happy to be joining the Infra team here at Apache. I'm from Oregon in the United States and really enjoy the rain. While at Oregon State University I studied mathematics and physics with a lean towards error correcting codes and mathematical modelling …
The replacement buildbot master is now live. The CMS service and the ci.apache.org website have been restored. The project CI builds are mostly working but builds that upload docs, snapshots etc. to the buildmaster for publishing are likely to fail at the upload …
For the past few months the Infrastructure team have been working extremely hard to re-design, implement and manage changes to the email service architecture. Today we are proud to announce that phase 1 of this has been completed, and has been running for several days now.
As some of you may have already encountered, certain services within Apache appear to have broken SSL support. While the cert is still valid, there is a part of the cert that both Microsoft and Google have stopped accepting as valid. We are working on fixing this and will use …
If you have worked on a web site for an Apache project, you've probably come across the fact that everything has to be in Subversion for web sites. The reason for this has been the desire to have a unified standard for publishing web site contents across all projects. The …
Travis-CI is a distributed continuous integration platform that integrates well with projects on GitHub. As many of our projects are taking advantage of our GitHub integration, they're also making use of Travis-CI for testing of inbound patches.
Travis CI offers a free account for open source projects, with a built …
Infra no longer offers the Jira Service Desk. You can manage most standard service requests yourself through the Self-Serve service.
As part of our ongoing efforts to streamline our service offerings, and to make it easier to interact with the Infrastructure team we are launching an instance of JIRA Service …
One of these changes is to our infrastructure, and is a move from manually managed and maintained web servers towards re-deployable, configuration managed servers that tend to themselves and rarely, if ever, require manual intervention. As such, we have started moving towards no longer …
Please note than on Thursday 15th at 20:00 UTC the Infrastructure team will be taking the read/write git repositories offline. We expect that this migration to last about 4 hours.
During the outage the service will be migrated from an old host to a new one. We …
On Wednesday December 3rd the main US host for the ASF subversion service fails resulting in loss of service. This loss of subversion service prevent committers from submitting any changes, and whilst we have an EU mirror it is read-only and does not allow for any changes to be …
The primary master machine that hosts the Apache Software Foundation's subversion repositories is currently undergoing some emergency maintenance due to disk errors.
We do not currently have an ETA on when this will be fixed.
In the meantime, there will be no access to commit to SVN.
The read-only mirror …
**Note**: We no longer use the MoinMoin service. Projects can create a wiki in the ASF Confluence Wiki.
In recent months we have become increasingly aware of a slowing down of our MoinMoin wiki service. We have attributed this, at least in part, due to the way MoinMoin stores some …
The ASF Infrastructure team is pleased to announce the availability of a new code signing service for Java, Windows and Android applications. This service is available to any Apache project to use to sign their releases. Traditionally, Apache projects have shipped source code. The code tarballs are signed with a …
The ASF Infrastructure team is happy to announce that you can now set up jobs on builds.apache.org to listen for pull requests to github.com/apache repositories, build that pull request’s changes, and then comment on the pull request with the build’s results. This is done …
Apache committers are granted shell access to a host known as either people.apache.org or minotaur. As you may know, there has been a two year grace period in which we have advertised the upcoming change away from password logins to SSH key only.
For a very long time now we have allowed committers to send email from their @apache.org email address from any host. 10 years ago this was less of an issue than it is today. In the current world of mass spam and junk flying around, mail server providers are …
So Tuesday morning we got a report in IRC that a committer was trying to get a release out and could not deploy. Shortly after a Nexus issue was reported in Jira INFRA-8321. A few hours later another issue INFRA-8322 related to Nexus was opened. So far, nothing …
A couple of weeks ago, Apache's Infrastructure team added a new feature to our Jenkins server, builds.apache.org to help deal with the at times overwhelming queues of builds waiting for an executor. While this has been improved dramatically by the increase in workers generously provided by Yahoo! on …
As the Apache Software Foundation (ASF) has grown, the infrastructure required to support its diverse set of projects has grown as well. To care for the infrastructure that the ASF depends on, the foundation has hired several contractors to supplement the dedicated cadre of volunteers who help maintain the ASFs …
We are pleased to announce that we have a new status page for our infrastructure and the ASF as a whole.
Where we have previously been focused on reporting the up/down status of our services, we have now begun to look a bit more at the broader picture of …
Good news! We've finally laid the necessary groundwork to extend the bypassing of committer emails sent from their apache.org addresses, from commit lists to now all Apache mailing lists. This feature was activated earlier today and represents a significant benefit for cross-collaboration between Apache mailing lists for committers, relieving …
Note: This page, including the update note, were correct when written, but are now out of date. A description of the situation related to DMARC and DKIM as of November, 2025 is available on this wiki page: cwiki.apache.org/confluence/display/INFRA/DMARC+and+DKIM+for+ASF+mail.
Overview: During the afternoon of May 6th we began experiencing delays in mail delivery of 1-2 hours. Initial efforts at remediation seemed to clear this up but on the morning of May 7th the problem worsened and we proactively disabled mail service to deal with the failure. This outage affected …
23 may 2014 the old monitoring system "nagios" was put to sleep, and "circonus" was given production status.
The new monitoring system is sponsored by circonus and most of the monitoring as well as the central database runs on www.circonus.com. The infrastructure team have built and deployed logic …
During the afternoon of May 6th we began experiencing delays in mail delivery of 1-2 hours. Initial efforts at remediation seemed to clear this up but on the morning of May 7th the problem worsened and we proactively disabled mail service to deal with the failure. The underlying hardware suffered …
What we've learned about the heartbleed incident is that it is hard, in the sense of perhaps only viable to a well-funded blackhat operation, to steal a private certificate and key from a vulnerable service. Nevertheless, the central role Apache projects play in the modern software development world …
**Note**: Projects and the ASF itself used the Apache Content Management System from 2010 to 2021. It is no longer available.
Links to suggestions for setting up a project website, and website guidelines, are available under "PMC resources" in the general Infrastructure documentation page.
After a few weeks of hard work and mind-boggling debugging, we are pleased to announce tighter and smarter integration between GitHub and the Apache Software Foundation's infrastructure.
These new features mean a much higher level of replication and retention of what goes on on GitHub, which in turns both help …
Note: As of May, 2024, Apache Paste is no longer available.
Today, the Apache Infrastructure team launched https://paste.apache.org, a new ASF-driven site for posting snippets, scripts, logging output, configurations and much more and sharing them with the world.
Since out last update over a year ago, the Infra Team has expanded by another NINE (9) members!
Congrats and our warmest thanks go to:
Niklas Gustavsson - (ngn) Jeremy Thomerson - (jrthomerson) Mark Struberg - (struberg) Eric Evans - (eevans) Brandon Williams - (brandonwilliams) Mohammad Nour El-Din - (mnour) David Nalley - (ke4qqq) Yang Shih-Ching - (imacat …
**Note**: This service is no longer available from Infra.
Daniel Gruno has recently developed a comments system for Apache projects to use. The purpose of the system is to enable public commentary on project webpages and is already in production use in the httpd and trafficserver projects. This new system …
**Note**: Projects and the ASF itself used the Apache Content Management System from 2010 to 2021. It is no longer available.
Links to suggestions for setting up a project website, and website guidelines, are available under "PMC resources" in the general Infrastructure documentation page.
Last week, internal audit activity discovered that the access logs of some committer-only Apache services contained passwords but had been available to every Apache committer.
The problem
The httpd logs of several ASF services are aggregated and archived on minotaur.apache.org. Minotaur is also people.apache.org, the shell …
**Note**: Projects and the ASF itself used the Apache Content Management System from 2010 to 2021. It is no longer available.
Links to suggestions for setting up a project website, and website guidelines, are available under "PMC resources" in the general Infrastructure documentation page.
**Note**: Projects and the ASF itself used the Apache Content Management System from 2010 to 2021. It is no longer available.
Links to suggestions for setting up a project website, and website guidelines, are available under "PMC resources" in the general Infrastructure documentation page.
Well, some are not exactly new faces, but since our last blog update of new infra members in 2009 , we have conned with promises of fame, fortune and beer the following new additions to the infra team:
The Infrastructure Team has just enabled a new feature to control your SpamAssassin Threshold for your apache.org account. The default score for user delivery has always remained at 10, but with this new feature you can lower that score to anything you want. Many people with older accounts …
The infrastructure team are pleased to announce the availability of id.apache.org the new password management tool for all ASF committers and members. This new service will allow users to:
Reset forgotten LDAP passwords themselves, no need to contact the Infra team anymore.
As of approximately 03:00 (UTC) today the infrastructure team have enabled a password policy for all LDAP accounts.
This policy has been implemented at the LDAP infrastructure level and will affect all users. It has been deployed using OpenLDAP's password policy schema, and overlay.
**Note**: Projects and the ASF itself used the Apache Content Management System from 2010 to 2021. It is no longer available.
Links to suggestions for setting up a project website, and website guidelines, are available under "PMC resources" in the general Infrastructure documentation page.
We know we have projects that use reviewboard externally to the ASF, we also have some projects using codereview.appspot.com and we also have some projects using Fisheye/Clover externally.
Well, due to popular request, we now have an internal ReviewBoard running on https://reviews.apache.org !!
Yesterday, the main ASF SVN code repository passed the 1 million commit mark. Shortly thereafter one of the ASF members enquired as to how he could best grab the SVN log entries for all of these commits. As always, there were a bunch of useful replies, but they were all …
Today we've brought s.apache.org online. It's a URL shortening service that's limited to Apache committers- the people who write all that Apache software! One of the main reasons we're providing this service is to allow committers to use shortened links whose provenance is known to be a trusted …
Apache.org services recently suffered a direct, targeted attack against our infrastructure, specifically the server hosting our issue-tracking software.
The Apache Software Foundation uses a donated instance of Atlassian JIRA as an issue tracker for our projects. Among other projects, the ASF Infrastructure Team uses it to track issues and …
Here at the ASF we have a subversion setup with all our projects code in one repository, with each of those projects having their own style of trunk/branches/tags/site etc.. This works well for us, but did present us with some initial problems when setting up our Buildbot …
The infrastructure team have now completed the next stage of the planned LDAP migration. We have migrated our old SVN authorisation file, and POSIX groups into LDAP data. SVN access control is now managed using these groups.
This means to change access the Subversion repositories is now as simple as …
Tonight we enabled a pair of Intel X25-M's to serve as l2arc cache for the zfs array which contains all of our svn repositories. Over the next few hours as these SSD's start serving files from cache, the responsiveness and overall performance of svn on eris (our master US-based server …
In the first week of October we started getting reports of performance issues, mainly connection timeouts, on all of our services hosted at https://issues.apache.org/. On further inspection we noticed a huge amount of "Browser disconnect" errors in the error log right at the beginning of the ssl …
Last week we posted about the security breach that caused us to temporarily suspend some services. All services
have now been restored. We have analyzed the events that led to the breach, and continued to work on improving the security of our systems.
This is a short overview of what happened on Friday August 28 2009
to the apache.org services. A more detailed post will come at a later
time after we complete the audit of all machines involved.
On August 27th, starting at
about 18:00 UTC an account used for …
One of the more common issues committers face at Apache is in trying to send mail from their apache.org account. We've just made that process a whole lot easier by setting up an SSL-enabled, smtp-auth based mail submission service on people.apache.org port 465; which is compatible with …
**Note**: Until June, 2023, the ASF supported using Apache Roller for project blogs. That support has ended. Please visit Project Blog for more information about blogs for projects.
Previously, to be able to preview a draft post by any Roller Blog, one had to be a member user of that …
The ASF Infrastructure Team will be upgrading the Confluence instance powering https://cwiki.apache.org from Confluence 2.2.9 to Confluence 2.10.3 on July 11 at 0400 UTC, or July 10 at 2100 PST. The migration is expected to take several hours.
Earlier this week the Infrastructure team rolled out phase one of the planned LDAP services.
We are using LDAP for authentication of shell accounts. For now this is the extent of the implementation, however the next phase should follow this quite quickly.
Git is a new version control system that has been getting increasingly popular during the past few years. Many Apache contributors have also expressed interested in using Git for working with Apache codebases. While the canonical location of all Apache source code is our Subversion repository, we also want to …
Established today, we now have a dedicated mailing list to talk about and work out all things to do with our build services. Currently infrastructure provides projects with use of Hudson, Continuum, Gump and now we have another option in Buildbot. Buildbot is a new service here at Apache Infrastructure …
Last week (at ApacheCon 2009 EU) I gave a session talk about "Subversion on-the-fly Replication" and how we (ASF) deployed such an setup last year with in the Apache Software Foundation. So check out the slides if you are interested in how it works, why you should do it, what …
As of this afternoon whilst at ApacheCon Europe 2009, we have gotten our initial LDAP platform in place ready for testing. This will allow us to move to a centralized AAA system.
In preparation for upgrading Subversion to the latest version (1.6.0), we are running an svn dump on svn.apache.org. This will chew up enough disk IO to be noticeable to svn users. We expect the dump to finish sometime during this weekend.
**Note**: Until June, 2023, the ASF supported using Apache Roller for project blogs. That support has ended. Please visit Project Blog for more information about blogs for projects.
Roller was installed today for the use of ASF Projects. Those committers with personal blogs should instead add their feed to planet …