Review Foundation infrastructure and rethink how we manage it.

[joesepi]

This was a task in the post bootstrap project board. It seems to make sense to bump this up to an issue and use it to track infrastructure needs.

[joesepi]

Starting to capture infrastructure requests: #285

[mcollina]

One of the things that we talked about a lot in the Node.js project is if the Foundation could add some staff to help in the maintainance of our infrastructure. I’m not 100% sure what we could do in this regard, but it would help the project a lot.

[mhdawson]

I've proposed asking for help on the infrastructure side a few times in the past, but there was some opposition/concern. I do think this is a good topic to discuss though as it's far from a solved problem.

[mhdawson]

@rvagg, @Trott since they have been involved in some of the past discussion.

[Trott]

One of the things that we talked about a lot in the Node.js project is if the Foundation could add some staff to help in the maintainance of our infrastructure. I’m not 100% sure what we could do in this regard, but it would help the project a lot.

There was a lot of ambivalence about this. @rvagg can probably articulate it better than anyone else, I think.

[dmethvin]

I'd be interested in specifics about the concerns. In the jQuery Foundation we did pay some people for software development and support, which did create problems. I would much prefer that we find some companies that would be willing to have their employees donate time. Hopefully we could determine some value prop for them that made sense.

[rvagg]

I can't find links to the previous incarnations of this discussion, perhaps it was in meetings or other places than the build repo issue tracker. This was one comment I made recently about resourcing capacity: nodejs/build#1154 (comment), it's a related point but not my central argument.

The central argument goes something like this: introduce staff paid from a central pool and you'll reduce incentive for individuals to volunteer as there's a clear multi-tier status thing going on where a paid individual obviously has official blessing while a volunteer can never aspire to have the same kind of ownership of their work. We're already suffering from a multi-tier access problem in Build which is limiting volunteer incentives, paid staff will accelerate that significantly without recourse. And for companies the incentive to pay people to contribute directly (like IBM and Microsoft do now) is reduced, particularly if they are already paying into a central pool that is going to be paying individuals. They'll just cost-shift to the central pool and we'll end up poorer because those companies are no longer spending in addition to foundation membership. Then the member companies that don't contribute in addition (like certain large ones that don't contribute anything to the unglamorous and non-PR generating activities) will get to say they contribute to this work while doing nothing more than they already do and avoiding the community pressure to contribute more that would otherwise apply. All the while the central pool of fund now has a stone tied around its neck that it can never escape from and will likely increase over time, because scope creep is real.

My main concern is the former though, I'm convinced this is the easiest way to watch volunteer effort and genuine community dry up in our most unglamorous areas. The history of community volunteer effort suffering once paid staff are introduced is clear well beyond open source.

But, I'm also sick of being the only one arguing this point, it's clearly not convincing people so I'm not going to continue pushing this point. If you want to try it then I'll at least enjoy it for its empirical potential to test opposing theories about open source community dynamics.

[Trott]

I agree with @rvagg that paying for this will have unintended negative consequences. But I also agree that the current situation cries out for improvement and I don't have any great ideas on how to do that.

[dylans]

For what it’s worth, with Dojo we had the strategy that ongoing regular infrastructure was volunteer driven (mostly sponsored by SitePen), but we had a few major migrations over the past 15 years (scary to think it’s been that long) that were more like a full-time job for 1-3 months and for those spikes the foundation funded it because we had high priorities that couldn’t wait for sponsor availability. This allowed us to not replace or stifle volunteers, but add to the effort for a very short duration.

[Trott]

For what it’s worth, with Dojo we had the strategy that ongoing regular infrastructure was volunteer driven (mostly sponsored by SitePen), but we had a few major migrations over the past 15 years (scary to think it’s been that long) that were more like a full-time job for 1-3 months and for those spikes the foundation funded it because we had high priorities that couldn’t wait for sponsor availability. This allowed us to not replace or stifle volunteers, but add to the effort for a very short duration.

Temporary supplement of paid staff to get over migration humps or to just clean things up might be an idea worth considering.

[rvagg]

Temporary supplement of paid staff to get over migration humps or to just clean things up might be an idea worth considering.

Tightly scoped work for long-standing TODO items that we just can't do with volunteer efforts might be worth considering too. As long as it doesn't become something common enough that volunteers will be thinking "if I just hold out long enough on <big thing that I'm the most likely candidate to do> they'll pay me to do this!".

But even with that, the ones I can think of that we have right now are mostly oustanding because they are very sensitive and we're not willing to give access to the resources to just anyone, so the pool of people available is very small. One example is the logging on nodejs.org that we need to switch to pulling from cloudflare so we can properly let cloudflare do its job. To do this work you really need access to the crown jewels. That's me, Michael, Joao.

[rvagg]

Here's an adjacent thought: we discussed with GitHub recently about migrating some of our CI to GitHub Actions (I believe we can talk about this publicly now they've announced the beta?). From the feature set they've described that they are working on, I can imagine replacing a very large chunk (maybe all) of our CI infra to GitHub Actions. Having such a standardised interface would throw open the path to contribution. No more battling with Jenkins and worrying about an entirely separate permissions system.

They offered to give us early access and I've asked for it but haven't heard anything back yet. There's likely a very large up-front investment but it's potentially a path to solving a lot of our infra technical debt and people-bandwidth problems.

[wesleytodd]

@rvagg I was able to get Express beta access by talking with @chrispat. Also, after that @bnb ping'd me on this twitter thread, and @bdougie has said he would hook up our other orgs. Maybe they can also help node out with this.

PS. GH actions and CI are working great for me so far. I have a repo where I am testing them out before we go and open it up on the express repo to replace travis. But assuming that continues to go well we will be moving to it across all the express repos in the near future.

[mhdawson]

We will need to figure out OS/platform aspects of using GitHub actions, but exploring GitHub actions might be a good candidate for temporary help.

The other aspect mentioned was encouraging companies to contribute people time. I prefer this approach as well but I think we need something to encourage this more as only a few companies have stepped up.

[mgol]

@rvagg

introduce staff paid from a central pool and you'll reduce incentive for individuals to volunteer as there's a clear multi-tier status thing going on

I get what you're saying & I observed the same in some OSS projects. That said, infrastructure is a bit special here. Access to the project infrastructure often gives a lot of power; how to recruit volunteers when the first thing you'd have to do is give them access to highly privileged services? When you pay someone, at least you have some accountability.

In case of jQuery, for example, we had an awesome team that managed our infra but they mostly moved on (which they have every right to) and don't work on it anymore. Paid stuff can't detract existing volunteers when there are none and recruiting new ones is hard due to reasons above. How would you proceed in such a case?

[rvagg]

That said, infrastructure is a bit special here. Access to the project infrastructure often gives a lot of power

Yep, that's one of our key challenges. The primary way we've tackled it is to view corporate employment as a easier gate. If you have an employment contract with an IBM then your behaviour is tied to IBM, even legally, so it's easier to gate trust. We also want to do it in other ways to prevent giving all of this over to large corporates (and tbh, IBM is our only major corporate directly involved, Microsoft is involved by proxy and there are contractual relationships that help). We still don't have good mechanism for establishing strong trust relationships with people outside of existing corporate legal boundaries; other than history (me and some others).

One way we want to tackle this, on our TODO list, is to break up our access levels much more so that the crown jewels are not required to access relatively straightforward things. Right now we invest a lot in our primary web server, releases, logs, and much more. This means that anything that goes near that server must be strongly gated and this means that simpler things, like managing the nodejs.org content is awkward! We've automated most of it but it's far from optimal when we want to empower the website team who are not involved in infra at all. So, if we can get to it, atomised access controls, less all-in-one-place arrangements. But this is another one of those large jobs that requires someone like me or Michael to take most of the burden of. I'm hoping GitHub Actions will be a bit of a forcing function here to help us break things up more and help us reset our relationship to our infra.

[bdougie]

Hi everyone, late to the conversation. But I am happy to get any Project access to Actions during the beta period (general launch is 13 November 2019). Happy to take pings in a Twitter DM or email to bdougie@github.com to avoid blowing up this issue.

[bnb]

Also late, but also more than happy to help put in time and effort to move in the direction of GitHub CI if that's something we're interested in. I don't doubt that there's a bunch of things that could be shared between projects, and that it could be a key outcome from this org. Plus, with the addition of direct action imports that are written in JavaScript... it kinda makes a lot of sense for the OpenJS Foundation to be using 😅

[joesepi]

Related #333 per @jorydotcom

[jorydotcom]

Cross-posting this comment from #333 to improve visibility:

Per our discussion in the CPC meetings, I'm posting a link to the Draft "Infra Services Menu" which outlines the vendored services that the OpenJS Foundation currently supports from an IT perspective. This list is not intended to be immutable; it is just meant to communicate clearly what any OpenJS Foundation project will get in terms of "out-of-the-box" infra support from Foundation IT / program staff.

This list is based in large part on the initial work we've conducted in creating an 'infrastructure inventory' for OpenJS Foundation projects. That work is not entirely complete, but is expected to be complete as we get through project onboarding.

The ask is as follows: please take a look at the draft services menu and let me or @brianwarner know if there are any clarifications, asks, etc. Also, if we (you, me) haven't chatted about your project's infra setup let's do that soon!

[joesepi]

Closing after having landed Infrastructure Menu in CPC repo.