Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Flatten project, scope TSC, big changes to structure. #59

Closed
wants to merge 3 commits into from

Conversation

mikeal
Copy link
Contributor

@mikeal mikeal commented Mar 8, 2016

Based on the recent TSC meeting and some of the feedback I've gotten on recent issues I've logged I created this initial draft of a new structure.

This does a few critical things:

  • Sets the boundary for "long term project hosting" under the TSC to projects that are connected to Core development. However, it makes it clear that projects that are connected to Core development may stretch far beyond the needs of Core (evangelism, website, documentation).
  • Renames the "Incubator" to "Mentorship" in order to reduce many false assumptions from Apache's Incubator. This still reserves the ability for the TSC to intervene in projects outside of the above scope, like Express, without assuming the TSC will host them long term. The issue of where to host these projects long term is a question for another day -- there are many options we can explore.
  • Flattens all projects and Working Groups under the TSC. The line between what should and should not be under the CTC was always a bit difficult to draw. Flattening makes autonomy more clear and allows the TSC to focus on improving communication and collaboration between the different parts (which the CTC has never done much of because it's so focused on the needs of direct Core development).

There's some very good things this accomplishes.

  • It solves the "blessing projects" problem. By not hosting projects next to Core we can avoid making them "official."
  • It makes the TSC focus on some core problems we're experiencing at scale.
    • Automating reporting, communication and transparency for projects and working groups.
    • Creating and promoting contribution and governance policies for use both within the TSC and in the ecosystem.
    • Creating measurements and tools for evaluating project health. First for use within the Foundation but conceptually for use outside it later on.

The good thing about flattening this stuff and staying away from a hierarchy is that we do a better and more explicit job of groups being service providers. This allows group to be more effective in this role within the project but also with the broader community.

What is not yet resolved is the question of who is on the TSC. Proportional representation is difficult, anything as simple as "one rep per project" ignores the disparity in contributors and scope between projects.

This document also isn't enough, there's a lot of other process points in other now irrelevant documents.

And finally, because this is a huge change that effects every group and project it's going to need to reach a consensus among all the sub projects. @nodejs/tsc @nodejs/ctc


If your project is interested in joining the Node.js Foundation please read the [Project Lifecycle.md](./Project-Lifecycle.md) documentation.
# TSC
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is already a # The Node.js Foundation TSC above... this could probably be left out.

@joshgav
Copy link
Contributor

joshgav commented Mar 9, 2016

Can we more explicitly define "Project" and "Working Group"? A quick read of Project-Lifecycle.md indicates that a Project can have WGs, while WGs cannot have (sub-)WGs. Are there other differences?

We could rationalize all top-level things as Working Groups, drop the "Project" moniker, and then provide optional guidance for subcommittees for WGs which want them.

Re current CTC WGs, perhaps they should be evaluated one by one and either become a proper top-level WG or remain a subcommittee of Core. Some of them could perhaps be closed or consolidated in this process - for example post-mortem and tracing, intl and i18n, Addon API and nan, etc.

It would be helpful if all of this was documented in one place, probably this nodejs/TSC repo, rather than duplicated/shared with nodejs/node. The nodejs.org website should also be synced with info in this repo; many people start there.

Thanks!

@mikeal
Copy link
Contributor Author

mikeal commented Mar 9, 2016

@joshgav first off, Project Lifecycle will need to be completely re-written if this is passed.

The main difference between a project and a working group is that a project has a definable code base it works on and a working group is broader in scope and probably only using a repo for communication and tracking purposes.

@joshgav
Copy link
Contributor

joshgav commented Mar 9, 2016

@mikeal From your explanation I see why a WG isn't necessarily a project; but perhaps a project is by definition a WG? I'm trying to rationalize this as much for myself as for others :).

The Node project could be a collection of WGs governed by the TSC. WGs could include any or all of

  • code,
  • an issue tracker/discussion forum,
  • a chief Technical Committee
  • subcomittees

@joshgav
Copy link
Contributor

joshgav commented Mar 9, 2016

For the sake of discussion, here's an idea of what the collection of WGs and subcommittees might look like:

  • Core -> Build, Testing, Benchmarking, Streams, HTTP
  • Futures -> Roadmap, Promises, Docker
  • Diagnostics -> Tracing, Post-Mortem
  • Modules -> nan, Addon API, npm
  • Security
  • Docs
  • i18n
  • Evangelism -> Website
  • Inclusivity

@Fishrock123
Copy link
Contributor

@joshgav I don't think that would work well.

  • Futures -> Roadmap, Promises, Docker

Roadmap-related things are nothing if not associated to core.

  • Diagnostics -> Tracing, Post-Mortem
  • Modules -> nan, Addon API, npm

Likewise

  • Evangelism -> Website

Those two definitely do not directly overlap like that.

@mikeal
Copy link
Contributor Author

mikeal commented Mar 9, 2016

I think we're missing the point a little.

The basic idea here is to limit the scope of the TSC (with the exception of mentoring other projects) to coordinating sub-groups that are integrated into Core development in some way. Almost all of these groups have a broader set of responsibilities beyond Core development but some part of them needs to be integrated nonetheless.

Consolidating groups is a poor plan for the simple reason that many of these groups are already huge and consolidating them would be a poor plan being that some of them are literally the result of them delegating responsibilities in order to scale.

Measuring the links between groups and their connections to each other and Core is incredibly difficult and actually changes over time. I'm convinced it's actually impossible to maintain a description of these connections, we attempted to once six months ago and ended up with a crazy web which has only gotten crazier since then.

Instead, all I think anyone can expect the TSC and these groups to do is find better ways of communicating the work they are already doing. These groups operate in the open but the volume of work is impossible to understand without better reporting and communication. If we have better communication all the people working in these groups have the ability to understand as much as they need to understand about each sub-group.

This was working a lot better and tighter when we could rely on a lot of common members across groups, that doesn't seem to work anymore, and even people who spend a considerable amount of time trying to understand what is going on across the org can't do it. If people like me and @rvagg can't figure this out then we have to assume that absolutely nobody in the community has an understanding of this.

@nebrius
Copy link
Contributor

nebrius commented Mar 9, 2016

From your explanation I see why a WG isn't necessarily a project; but perhaps a project is by definition a WG? I'm trying to rationalize this as much for myself as for others :).

This is an imperfect and unofficial definition, to be sure, but I think of projects as producing a software product that can be used by developers inside and outside of the Node.js project. Conversely, I view WGs as organizations where people discuss topics related to one or more projects in order to influence how projects create their products.

@mikeal
Copy link
Contributor Author

mikeal commented Mar 9, 2016

Feedback from @jasnell in CTC meeting: Several of these are subsets of Core development and shouldn't be put under the TSC directly.

Response from @mikeal was that those groups look more like teams than Working Groups and that these were chartered as WGs because there's a perception that if you want to use another repo to organize work it needs to be a WG.

Positive response from @jasnell that we should maybe just make these teams and be clearer that nodejs org members can use multiple repos freely to coordinate.

@joshgav
Copy link
Contributor

joshgav commented Mar 10, 2016

A thought: What if the TSC and CTC reunited into a chief TC (TSC), and the role defined for the TSC now became a Project Management WG under the chief TC?

Since @mikeal proposes (comment) that we define the domain of the TSC as projects relevant to Core, the domain of the CTC and TSC mostly overlaps; but they serve different roles in this domain. At Microsoft we might call these "dev" and "program management (PM)." CTC members need to focus on engineering, TSC on cross-team coordination and project management. A reunited CTC/TSC would manage all aspects of core, delegating PM duties to the PM workgroup. All other workgroups would then report to the united TSC and CTC.

Possibly, rather than a sub-WG, the chief TC could be composed of both project management people (TSC) and core devs (CTC).

Thoughts?

@Fishrock123
Copy link
Contributor

Just as a note, someone in Q/A for the TSC meeting today brought up that why this is even necessary may not be apparent without actually listening to previous TSC meetings. We should probably express somewhere in written form here on our concerns about adding new projects.

@williamkapke
Copy link
Contributor

@Fishrock123 I am that someone. Does it anonymize us on YouTube? Maybe I'll go back to using #node-dev then :/

Here is more details on what I was referring to:

Node.js Foundation Technical Steering Committee (TSC) Meeting 2016-02-18
Almost all of this video contains the buildup of the problem that this is trying to be addressed.
https://www.youtube.com/watch?v=T6wFkZgnNEQ

2016-03-03 Node.js Foundation Technical Steering Committee (TSC) Meeting
The audio starts here 15:23 with @mikeal talking about bringing new projects in and the topic takes up most of the time of the meeting (until around 39:20)

Interesting points:

2016-03-09 Node.js Foundation Core Technical Committee (CTC) Meeting
@mikeal Kicks off the topic by giving a good summary of what and why 12:38 to 15:50. The topic continues (again largest topic of the meeting) until 28:46

Interesting points:

  • 20:05 @mikeal talks about trying to drawing "something like an org chart- and it looked like a big web of a mess"
  • 22:25 "There isn't as much overlap [between the TSC & CTC] anymore"

@mikeal
Copy link
Contributor Author

mikeal commented Mar 15, 2016

I made some updates to this based on the feedback I got last week.

  • Reduce the number of projects and working groups. Many of these were just teams that had a repo they were using for organization.
  • Reduce process for creating teams. It was somewhat telling that one of the most active "working groups" was never even chartered (LTS). Rather than run through the whole chartering process this just lets project create teams and repos at will without extraneous process. The only reason to charter a WG or new project is to actually delegate and remove responsibility from another group.

@Fishrock123
Copy link
Contributor

I am that someone. Does it anonymize us on YouTube? Maybe I'll go back to using #node-dev then :/

@williamkapke No, but it's not always apparent what usernames are across services or if people want to be mentioned. :)

@mikeal
Copy link
Contributor Author

mikeal commented Mar 16, 2016

Here's a basic drawing of the latest proposal.

Obviously libuv and express aren't "teams" but the rest is accurate.

screen shot 2016-03-15 at 5 20 07 pm

@orangemocha
Copy link
Contributor

One concern I have with this PR is that it bundles two sets of important changes:

  1. The decision to focus the TSC on Core
  2. The organization restructuring to accomplish that mission

The new organizational structure makes sense to me, once we have decided to focus on Core. But the fundamental decision of what the TSC mission should be is something that we have been trying to figure out for quite some time now, and that in my opinion still needs to be resolved first. (I am also noting that this is a 180-degree turn from the proposal of just a few weeks ago).

The end result might be the same, but I would prefer a separate PR about the TSC mission, so that we can have a focused discussion about it and finally reach a consensus.

@mikeal
Copy link
Contributor Author

mikeal commented Mar 16, 2016

(I am also noting that this is a 180-degree turn from the proposal of just a few weeks ago).

Sure, but that proposal wasn't accepted because nobody was comfortable with that direction :)

One concern I have with this PR is that it bundles two sets of important changes

The TSC can discuss each of these points in isolation but I don't see how it benefits the discussion to bring the entire proposal back to first principals because most people have a hard time visualizing the actual impact of the first discussion without the second point -- that's why they are bundled.

In particular, both decisions have an impact on the CTC and without the structural proposal I don't think people who aren't active in the TSC meetings will be able to understand the impact.

@mikeal
Copy link
Contributor Author

mikeal commented Mar 16, 2016

The decision to focus the TSC on Core

So, maybe the language needs to be clearer, but this doesn't focus the TSC "on Core" but on integrating various groups into Core development. Perhaps that can be made clearer.

@joshgav
Copy link
Contributor

joshgav commented Mar 16, 2016

Since the task of Build is to build Core, should it be part of Core? Or what's the reasoning behind it being separate?

nan seems to be part of a larger domain for core extensibility and the module subsystem. For example, that domain might include nodejs/api#10 and nodejs/node-eps#3.

@mikeal
Copy link
Contributor Author

mikeal commented Mar 16, 2016

@joshgav roughly 90% of what Build builds is Node.js. They also handle the hosting of the website, micro-sites, etc. They also have build and test resources for nan and libuv. The Build WG manages a lot of infrastructure and operates essentially independently already so it makes sense to retain this level of autonomy.

nan seems to be part of a larger domain for core extensibility and the module subsystem. For example, that domain might include nodejs/api#10 and nodejs/node-eps#3.

Even if an addon api or new core native API were to happen we would still need to maintain nan, it's highly depended on and needs to remain independent. It also doesn't make sense to put alternatives to nan under nan so those should remain in Core.

@joshgav
Copy link
Contributor

joshgav commented Mar 16, 2016

@mikeal

It also doesn't make sense to put alternatives to nan under nan so those should remain in Core.

Actually I was thinking they all belong together under extensibility, modules or some such. Which will raise the question of whether that should be under Core :)

@nebrius
Copy link
Contributor

nebrius commented Mar 17, 2016

Do I remember correctly that intl does website work as well? If so, should they be top level?

@nebrius
Copy link
Contributor

nebrius commented Mar 17, 2016

I could get behind the idea of a "modules" or "native modules" top level group, of which NAN is one team under them.

@mikeal
Copy link
Contributor Author

mikeal commented Mar 17, 2016

Do I remember correctly that intl does website work as well? If so, should they be top level?

There's i18n, which is the localization communities, and Intl which is a group working on the Intl support in core :)

Any project or working group can create teams at their discretion. Any org member
can create a new repo for organizing this kind of effort without a charter or other
process. Examples of Core teams are: Long Term Support, Cryto, Roadmap, Benchmark,
Streams, Testing.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

needs some definite of what responsibility teams are allowed to take and when

@Fishrock123
Copy link
Contributor

Who gets TSC seats from this new structure then, if anyone?

@mikeal
Copy link
Contributor Author

mikeal commented Mar 17, 2016

Who gets TSC seats from this new structure then, if anyone?

I left that for the TSC to figure out after you've agreed to the scope. The scope sort of dictates the kind of representation you'd want to see.

WG vs. Project

No difference really, WG is just an easier term to refer to a slate of responsibility when it's larger than a code base.

@williamkapke
Copy link
Contributor

@mikeal, I think the drawing you created provides tremendous context for the proposed change(s). I think it would really help the discussion to have a drawing for the EXISTING structure. Basically, a before and after.

I think it would also serve this thread well to edit the OP to include the chart(s) and give that visual right away.

these goals and eventually allow them to graduate the incubation phase.
* Core
* Website
* nan
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

NAN

@Fishrock123
Copy link
Contributor

I think I've brought this up in the meetings but:

  1. While I'm not against restructuring if necessary, I'd rather not do so again so soon.
  2. I'm not sure how putting more communicative work on the TSC will actually improve org-wide communication in the short term.
  3. Having the possibility to still take projects on in a longer term basis may still be desired, but definitely should not be the default or something you can just apply for upfront.

@rvagg
Copy link
Member

rvagg commented Mar 23, 2016

While I don't mind the majority of the way this is structured now I'm pretty sure there isn't a mood for doing yet another big pivot. The question of scope is an important one but I'd much prefer to frame in terms of a consultation within the Foundation. The TSC has an obvious role in core, we have (and should have), a lot of latitude to do what we think is needed there and we have the trust of the board on that. Beyond core, as @Fishrock123 says, there may be situations where limiting our scope hinders our ability to do real good for the Node ecosystem. But, I think it's completely appropriate for those actions to trigger a conversation with the Board (in particular, but also the executive) regarding broader strategy.

This is the theme I'm going to be pushing next week at the in-person Board meeting. I don't think it's in our interests to act on this here before that discussion and limit our options prematurely. I expect that the Board will be asking us to limit our options somewhat because I don't think they have any stomach for the Foundation to become the next Apache Foundation, but I agree with that and would like to encourage us to think about expansion in terms of strategy where one arm of the Foundation can propose to the rest a course of action but which requires agreement from the other arms to execute on that.

All centered around a "mission", which is kind of terrible in the by-laws, which is why I'm proposing:

The Node.js Foundation exists to increase the adoption of Node.js.
This goal is achieved by:

  1. Maintaining the health of the core Node.js project
  2. Fostering and encouraging a healthy Node.js open source ecosystem
  3. Fostering and encouraging a healthy commercial ecosystem of training, products and services for Node.js

@mikeal
Copy link
Contributor Author

mikeal commented Mar 23, 2016

@rvagg can you work that up into a PR or a new Issue that can be tagged for the agenda tomorrow and potentially passed by the TSC?

@mikeal
Copy link
Contributor Author

mikeal commented Mar 23, 2016

I'm going to close this.

I wrote up what appeared to be the consensus a few meetings back. It is now clear that it is not the consensus, and that there may not be a consensus to this level of detail.

I'll leave it to Rod to write up a less detailed statement that might reach a consensus.

@mikeal mikeal closed this Mar 23, 2016
@williamkapke
Copy link
Contributor

@mikeal you closed #54 and #58 in favor of this thread... which is now closed too. Should you reopen them?

@rvagg:

While I don't mind the majority of the way this is structured now I'm pretty sure there isn't a mood for doing yet another big pivot.

What is the current structure of the Technical arm of the Foundation? I, as an outsider that has followed pretty closely over the years, can't figure it out.

@mikeal
Copy link
Contributor Author

mikeal commented Mar 23, 2016

@williamkapke nah, leave them closed, we should defer these topics to whatever Rod ends up logging.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

10 participants