[WIP][ENH] improve guidelines #28

Remi-Gau · 2024-02-06T16:20:25Z

related to some discussions from the last steering group meeting
to discuss with @cmaumet

Remi-Gau · 2024-02-19T09:40:13Z

docs/guide.md

+1. Communicate with the BIDS maintainers to make your BEP official.
+  This entails registering the BEP with a number on


From discussion with Camille:

It is unclear what is in the internal process or criteria that says that a BEP can get a number.

from the POV of the contributor what would make them going from opening an issue to asking for an official number.

ACTION: maintainers and steering to clarify this

We never formalized this, but from my perspective, BEP leads need to:

open an issue on the spec repo and announce their intent, asking for community (/maintainer/steering) feedback

wait until such feedback comes (e.g., in the form of "we have something similar already, see here", or in the form of "this seems to be a niche case, are you sure that enough people care?", or in the form of "wonderful! We've been needing this!")

get started with a google doc and formalize who are the beps and contributors (if any yet)

Once these three steps are fulfilled, I'd be happy to give the BEP a number and ask the leads to please announce the initiative far and wide.

Thanks @sappelhoff for fleshing this out! I do think this is important and we should get consensus (at minima amongst maintainers + steering?) and make those explicit. Maybe a point to discuss at the next steering/maintainer meetings?

One question for me here is step 2 "wait until such feedback comes" --> should we commit that someone on BIDS governance (a maintainer / a steering member / someone else) will give feedback? i.e. what happens if no feedback comes?

i.e. what happens if no feedback comes?

there have been situations in the past where not a lot of feedback came. I think one example is the suggested BEP that came our of the earlier "cancelled" MIDS BEP:

BIDS-BodyPart extracted from BEP025 bids-specification#1569

there is just not a lot happening there and I could understand if the current lead of that "pre-BEP" would be kind of frustrated.

I don't really have a solution for it. Sometimes people also suggest ideas that don't really resonate with anyone, and then, no interaction may also be an answer in itself? But we should figure out more respectful ways of dealing with this.

we should figure out more respectful ways of dealing with this.

I agree this needs to be dealt with, but I would not frame it in terms of what is respectful or not, but rather what ensures that the work done is retrievable if interest returns and is formalized in a way that we retain what was learned (e.g., after all the work we realized it was not the best way of solving something).

I think having zombie BEPs is the worst (least respectful) option. As long as the BEP lifecycle is clear, if a BEP needs to be called "cancelled", I think that's okay (preserving it FAIR and provenance traced).

There are two topics in this thread, on 1) perhaps a this can be something like:

Open an initial “issue” on the BIDS specification repository [issues page] to gauge interest in your potential BEP, and to collect feedback by more community members and BIDS maintainers. Tag several BIDS community members with relevant expertise to comment, and ask whether they can provide feedback of the following form:
(a) We have something similar already, see here.
(b) This seems to be a niche case, are you sure that enough people care?
(c) Wonderful, we've been needing this!
(d) I do not have sufficient experience/time to comment on this topic now (perhaps tag someone else instead)

on 2) I would prefer 'shelving' rather than 'cancelling' zombie BEPs. This makes it clear that if community interest resurfaces or new leads emerge, they can be taken of the shelve.

Remi-Gau · 2024-02-19T09:51:01Z

docs/guide.md

+## Developping the BIDS extension proposal
+
+1. Make sure you are familiar with [the BIDS specification]https://bids-specification.readthedocs.io/en/latest/)
+   and the [bids-examples](https://github.com/bids-standard/bids-examples).


NOTE: landing page of mkdocs version of the example is confusing as it talks about a repo.

Remi-Gau · 2024-02-19T10:04:22Z

docs/guide.md


-8. Share the draft (remember to
+1. Share the draft (remember to


Is this supposed to be a google draft or to refer to the pull request?

Remi-Gau · 2024-02-19T10:05:35Z

docs/guide.md


-10. Help to merge the extension into the main specification (this will require
+1. Help to merge the extension into the main specification (this will require


When are people supposed to move from google doc to start using markdown / github to open a pull request ?

Should they submit only markdown?
Or should they to a certain extend take care of the schema aspect of things + mkdocs macro?

I would argue that

people should be allowed to develop BEP directly as a PR without going through google doc

it might be recommended to work on BEP while preparing the PR reflecting current state in google doc at any point in time: this would allow to

see how "intrusive" BEP is -- is it easy to represent it in current schema's model?

see if it is "typical", as if e.g. adding new datatype does not require overloading some common entities present in a similar class etc

validate sample datasets using new bids-validator which could use the schema from BEP

Given that the governance mentions a google doc step, I think it is hard for now to say that people can skip this.

https://bids.neuroimaging.io/governance.html#b-standard-decision-making-process-overview

Starting to think we may have to amend the governance too on the next election to clarify our process.

I totally agree with @Remi-Gau's initial question. In a private conversation that ended up in #29, I had mentioned this problem and suggested two scenarios:

When the Google Docs document becomes unmanageable by browsers (this happens when there are sufficient numbers of contributors and comments in the history). The BEP Leads could evaluate whether this moment has arrived every month since the start of the Google Doc.

When the Google Docs has not tracked new changes (changes, not comments) for a period of NN months (I'd say 3 or 4?).

Here, it would also clearly state how the migration from GDocs to GH should happen:

Block the GDocs so that no new changes can happen during migration and afterwards.

Whatever it is decided about Remi's questions above: https://github.com/bids-standard/bids-extensions/pull/28/files#r1494306610

re @Remi-Gau 's last comment on inter-dependency with governance principles:

governance principle is already "buggy" since BEP migration from google doc to PR already includes not only changes to markdown but also to the schema (yaml files)

I have now submitted Remove (now outdated/incomplete) going from Google doc to markdown file bids-website#373 to remove "markdown" specific, keeping overall "Google doc --> PR"

I will provide suggestion below on that as well

more thought is needed indeed to how adjust governance and instructions here. I think keeping BEPs hostages of Google drive can be a disadvantage/unnecessary burden in a number of cases since preparing PR right away encourages desired qualities (review of schema for needed changes which requires better familiarization with BIDS internals, development of example etc)

Remi-Gau · 2024-02-19T10:06:27Z

docs/guide.md

   converting the proposal to Markdown and submitting a Pull Request at
   [the BIDS specification repository])

-11. Create example datasets.


Give more specific info on opening PR on the bids examples.

I am not sure why this change. We do talk about PR against bids-specification above, so why not to talk about examples? hence I would have returned back (can't add suggestion on deleted line)

1. Create a Pull Request with example dataset(s) for [bids-examples] repository. Those examples should pass BIDS Validator when using modified BIDS Specification schema provided in the aforementioned Pull Request for [the BIDS specification repository].

for more information, see https://pre-commit.ci

oesteban

I think this is a very necessary effort, thanks @Remi-Gau

What I'm missing is a direct and clear mapping of the governance's "Key definitions" to these stages.

oesteban · 2024-02-28T14:42:16Z

docs/guide.md

+1. Make sure you are familiar with [the BIDS specification](https://bids-specification.readthedocs.io/en/latest/)
+   and the [bids-examples](https://github.com/bids-standard/bids-examples).


IMHO, someone who gets to this point without already being very familiar with BIDS is starting from a bad place. Familiarizing with BIDS should be mentioned before even opening an issue to gauge the interest.

@oesteban - I agree with your point and I think this line was moved down but we had not decided where it should be moved exactly yet.

Though to give a bit more context, I believe that the rationale here was to try and reduce the burden for new contributors who are not yet fully familiar with BIDS but who would like to contribute. Requesting as one of the first steps to be familiar with the whole of the BIDS spec and with the BIDS examples seems (to me at least) a bit too high and may be intimidating.

I see value in proposing pathways where people can more easily post an issue to inquire about a BEP they would like to start. This means we can have an open discussion earlier rather than later.

Thanks @cmaumet for writing what was in my head.

Requesting as one of the first steps to be familiar with the whole of the BIDS spec and with the BIDS examples seems (to me at least) a bit too high and may be intimidating.

For much I agree with this, its implementation places a large burden on the BIDS maintainers who (implicitly, and this should be explicit in the bylaws) end up being those facilitating actors who get BEPs to move forward. The same way we need to go through training before scanning people or animals, proposing a BEPs seems to me that it requires a deep understanding of BIDS.

People who are not deeply familiar but still have energy and time to contribute can definitely deliver relevant improvements by proposing new examples and contributing to the starter kit and other documents where the entry barriers are lower.

The problem with the spec is that it is a binding document (and with the backward compatibility principle, mistakes are costly). Entering a BEP without the sufficient familiarity with BIDS is a recipe to waste everyone's time (starting with the BEP leads themselves, which goes against their motivation to contribute in first place) at best, and have longstanding consequences at worst.

One problem (and I'm going a bit off-topic) is that BEPs are only submitted to review when they are "almost finished". I think the process of starting a BEP should be revised and perhaps follow an approach like bids-standard/bids-specification#1602, where a 1-page document stating problems, solutions tried, etc. is first to be voted. Only then the BEP moves forward if that vision gathers sufficient support from the community.

To properly write such a "BEP-in-brief" document, the future leads will need a deep understanding of BIDS. If that is not the case, the BIDS SC will identify issues in the proposal and request changes before things move forward.

IMHO, BEPs are not suffering from lack of initiative or motivation, but rather, lack of a clear vision (the path is found by creating many examples and convincing others, instead of following a clear target) and procedural (or lack of clarity thereof) issues.

Hi @oesteban! Thanks for sharing this and I can see that @sappelhoff agrees 👍 I fear that if we require new BEP leads to know all of BIDS we will end up with a smaller and smaller set of people who will be able to contribute to BIDS in the future. We will also possibly miss contributions from domain experts (who are not necessarily BIDS experts but know the target domain well).

But I think we can probably agree on a way forward :)

I concur that the role of the maintainers here should be made explicit and this must be done in agreement with the maintainers. Similarly, I think steering has a role to play.

I do like your proposal of a one page submission and of clarifying the process of how we gauge interest from the community (not sure if a vote is best but agree we should be explicit) before BEP leads get a BEP number and go on with developing a full BEP. This would also help making it more concrete what is meant by "Make sure you are familiar with the BIDS specification and the bids-examples." (i.e. different people may have very different interpretation of what this entails).

I fear that if we require new BEP leads to know all of BIDS we will end up with a smaller and smaller set of people who will be able to contribute to BIDS in the future.

IMHO, you shouldn't fear this for two reasons:

First is that initiating a BEP with a vision that is not aligned with BIDS is bound to fail. Often, BEPs get stuck for this, and this effectively reduces the number of people who will be able to contribute. This problem is already here, and this solution could prevent it from happening so unfortunately often.

The second reason is that it is not the same to contribute than it is to propose a BEP. If the BEP has solid foundations and the BEP leaders understand the BIDS' vision (this means not just the specs but also how things work and how to engage people), then the potential number of contributors only grow. This is because BEP leads must engage experts and other views who may not be familiar with BIDS directly, but can share their expertise.

This understanding opposes the suggestion that BIDS can expect a domain expert to come and lead a BEP on top of their already overloaded duties, which I feel is unrealistic. I believe the role of the BEP lead and BEP contributor should be addressed differently. In other words, we really need to decouple what is BIDS-dependent and what is domain-dependant.

Without any strong guidelines, perhaps something like the following? This is similar as noted for Python setting expectations

'To support relatively new BIDS contributors, it is strongly recommended that BEP leads who are relatively new to BIDS seek a current BIDS contributor as a co-BEP lead. Please be mindful of the burden on BIDS maintainers and consider that burn-out is common in open source due to a misunderstanding of what users, contributors, and maintainers expect from each other.'

Thanks for the specific suggestion @dorahermes. I think it captures the idea well. The intention I tried to shape is not deterring experts without BIDS experience in leading BEPs. It is rather that we should avoid letting a BEP start if its lead does not have someone with a "BIDS compass". Without such a role, the effort is bound to fail and dissolve.

On the contrary, if a BEP has a strongly opinionated (even biased) compass, then the BEP review process should identify and correct for the problems resulting from those biases. Without a compass, this checkpoint is hard to even reach.

oesteban · 2024-02-28T14:50:36Z

docs/guide.md


-10. Help to merge the extension into the main specification (this will require
+1. Help to merge the extension into the main specification (this will require


I totally agree with @Remi-Gau's initial question. In a private conversation that ended up in #29, I had mentioned this problem and suggested two scenarios:

When the Google Docs document becomes unmanageable by browsers (this happens when there are sufficient numbers of contributors and comments in the history). The BEP Leads could evaluate whether this moment has arrived every month since the start of the Google Doc.

When the Google Docs has not tracked new changes (changes, not comments) for a period of NN months (I'd say 3 or 4?).

Here, it would also clearly state how the migration from GDocs to GH should happen:

Block the GDocs so that no new changes can happen during migration and afterwards.

Whatever it is decided about Remi's questions above: https://github.com/bids-standard/bids-extensions/pull/28/files#r1494306610

Remi-Gau

Need to make sure that the guidelines are compatible with the governance.

https://bids.neuroimaging.io/governance.html#b-standard-decision-making-process-overview

francopestilli · 2024-03-05T21:34:54Z

My two cents regarding this process is that there are the following needs:
-- Establish a clear timeline. I would suggest we consider not allowing starting a BEP creation without a committed set of checkpoints with the steering committee and an explicit timeline.
-- Flag GitHub and Google Docs at the beginning of new BEPs. If the start of a BEP is made on Google Docs, there must be a cross-link to a GitHub discussion also with an explicit statement that a BEPXX was started on Topic Y and using the Google Doc Z. It is critical to cross-link GitHub and Docs so that work does not get too far into the weeds before feedback is given by people representing different perspectives.

docs/guide.md

yarikoptic · 2024-03-21T14:00:25Z

docs/guide.md

+1. Help to merge the extension into the main specification (this will require
   converting the proposal to Markdown and submitting a Pull Request at
   [the BIDS specification repository])


Suggested change

1. Help to merge the extension into the main specification (this will require

converting the proposal to Markdown and submitting a Pull Request at

[the BIDS specification repository])

1. Help to merge the extension into the main specification.

This will require converting the proposal to changes in [the BIDS specification repository] (schema YAML files, text Markdown files, etc) and submitting a Pull Request.

yarikoptic · 2024-03-21T14:06:30Z

docs/guide.md

-11. Create example datasets.
-
-12. If necessary, contribute a pull request to the
+1. If necessary, contribute a pull request to the


Given that now all validation rules should be encoded in the schema, I am not sure even if this rule remains pertinent. I would make it a little clear(er) that most likely this is not needed

Suggested change

1. If necessary, contribute a pull request to the

1. If necessary (should rarely be necessary), contribute a pull request to the

Perhaps this should say how (and/or who) the need for changes in the validator is determined, as opposed to grading the rarity of the event.

could you suggest wording to express your idea?

done above (I think GH will not reorganize the timeline of my review) or below (if the review order is changed)

yarikoptic · 2024-03-21T14:07:20Z

docs/guide.md

+
+
+


unnecessary, not pertinent changes

Suggested change

(all 3 should be deleted, diff seems to show only 1... dunno)

oesteban · 2024-03-25T09:13:13Z

docs/guide.md

+1. If necessary, contribute a pull request to the
      [BIDS Validator repository](https://github.com/bids-standard/bids-validator)
      as well to incorporate the extension.


@yarikoptic something along these lines (will probably require making consistent with the above line about examples):

Suggested change

1. If necessary, contribute a pull request to the

[BIDS Validator repository](https://github.com/bids-standard/bids-validator)

as well to incorporate the extension.

1. Submit at least one example dataset to the

[BIDS-examples](https://github.com/bids-standard/bids-examples) repository.

All uploaded examples MUST successfully pass the BIDS-Validator.

If the BIDS-Validator requires modifications beyond the updates to the schema

done within the BEP, contribute a pull request to the

[BIDS Validator repository](https://github.com/bids-standard/bids-validator).

If the BEP does not include examples, or there is uncertainty about whether

the BIDS-Validator will properly operate on more complex cases, at least

one maintainer of the BIDS-Validator project MUST accept the proposed

changes or request changes to the BIDS-Validator before acceptance.

Thanks for taking a stab at it! "explicit better than implicit" so overall I like the "spirit" of presentation here but IMHO it is too verbose here.
Quite often verbose presentation of details leads to duplication and thus outdated instructions, inconsistencies (e.g. governance doc says one, another one says another) etc. So IMHO it is better to keep it more succinct and to the point, not unlike prior instructions/steps. Then each individual component (e.g. bids-examples or bids-validator) repositories would be a location for ultimate instructions (relates on acceptance of PRs). Moreover BEP might not need a new dataset but might require changes/extensions to existing ones so I might be better to account for that... what about middle ground (modifying your proposed one):

Suggested change

1. If necessary, contribute a pull request to the

[BIDS Validator repository](https://github.com/bids-standard/bids-validator)

as well to incorporate the extension.

1. Submit a pull request (e.g. with an example dataset) reflecting changes proposed by BEP to the

[BIDS-examples](https://github.com/bids-standard/bids-examples) repository.

All additions or changes to example datasets MUST successfully pass the BIDS-Validator using BEP schema.

If the BIDS-Validator requires modifications beyond the updates to the schema

done within the BEP, contribute a pull request to the

[BIDS Validator repository](https://github.com/bids-standard/bids-validator).

oesteban · 2024-03-25T09:14:11Z

docs/guide.md

-11. Create example datasets.
-
-12. If necessary, contribute a pull request to the
+1. If necessary, contribute a pull request to the
      [BIDS Validator repository](https://github.com/bids-standard/bids-validator)
      as well to incorporate the extension.


Heads up - this sentence seems cut off. I would suggest a change, but I don't really know what this is meaning to say:

Suggested change

as well to incorporate the extension.

as well to incorporate the extension.

oesteban · 2024-03-25T09:15:29Z

docs/guide.md

-11. Create example datasets.
-
-12. If necessary, contribute a pull request to the
+1. If necessary, contribute a pull request to the


done above (I think GH will not reorganize the timeline of my review) or below (if the review order is changed)

Remi-Gau · 2024-06-21T09:57:09Z

Note that most if not all of this repo will be transferred to the new BIDS website:

bids website repo: https://github.com/bids-standard/bids-website/tree/main
pages from the bids-extension repo as rendered in the website: https://bids-website.readthedocs.io/en/latest/extensions/docs/index.html

If the move is confirmed / community approved, I would suggest closing this PR and consider opening a new one on the bids-website repo.

Remi-Gau · 2024-09-25T13:30:25Z

Content of this repo has been centralized into the new BIDS website.

Will close this PR soon as it has should now target the new BIDS website: will probably do so when a working group to refine the BEP process is formed.

Remi-Gau added 2 commits February 6, 2024 17:03

improve guidelines

f1b6f2a

reorder steps

21b5d96

Remi-Gau commented Feb 19, 2024

View reviewed changes

link to examples

d9d26ea

Remi-Gau commented Feb 19, 2024

View reviewed changes

list of steps to develop BEP

c02a16a

Remi-Gau commented Feb 19, 2024

View reviewed changes

sappelhoff mentioned this pull request Feb 27, 2024

add technical committee #29

Closed

Remi-Gau and others added 2 commits February 28, 2024 11:28

Merge branch 'main' into guidelines

6ea4d10

[pre-commit.ci] auto fixes from pre-commit.com hooks

f092c4d

for more information, see https://pre-commit.ci

Remi-Gau marked this pull request as ready for review February 28, 2024 10:29

Remi-Gau marked this pull request as draft February 28, 2024 10:29

oesteban reviewed Feb 28, 2024

View reviewed changes

Remi-Gau commented Feb 28, 2024

View reviewed changes

sappelhoff mentioned this pull request Feb 29, 2024

[ENH] add advisory group member table bids-standard/bids-website#272

Merged

yarikoptic reviewed Mar 21, 2024

View reviewed changes

docs/guide.md Outdated Show resolved Hide resolved

yarikoptic reviewed Mar 21, 2024

View reviewed changes

docs/guide.md Outdated Show resolved Hide resolved

Two minor typo fixes

02bca73

yarikoptic reviewed Mar 21, 2024

View reviewed changes

oesteban reviewed Mar 25, 2024

View reviewed changes

Remi-Gau added the wontfix This will not be worked on label Sep 25, 2024

		1. Communicate with the BIDS maintainers to make your BEP official.
		This entails registering the BEP with a number on


		8. Share the draft (remember to
		1. Share the draft (remember to


		10. Help to merge the extension into the main specification (this will require
		1. Help to merge the extension into the main specification (this will require

		1. Make sure you are familiar with [the BIDS specification](https://bids-specification.readthedocs.io/en/latest/)
		and the [bids-examples](https://github.com/bids-standard/bids-examples).

	1. If necessary, contribute a pull request to the
	1. If necessary (should rarely be necessary), contribute a pull request to the

-. If necessary, contribute a pull request to the
-      [BIDS Validator repository](https://github.com/bids-standard/bids-validator)
-      as well to incorporate the extension.
+.  Submit at least one example dataset to the
+    [BIDS-examples](https://github.com/bids-standard/bids-examples) repository.
+    All uploaded examples MUST successfully pass the BIDS-Validator.
+    If the BIDS-Validator requires modifications beyond the updates to the schema
+    done within the BEP, contribute a pull request to the
+    [BIDS Validator repository](https://github.com/bids-standard/bids-validator).
+    If the BEP does not include examples, or there is uncertainty about whether
+    the BIDS-Validator will properly operate on more complex cases, at least
+    one maintainer of the BIDS-Validator project MUST accept the proposed
+    changes or request changes to the BIDS-Validator before acceptance.

	as well to incorporate the extension.
	as well to incorporate the extension.

[WIP][ENH] improve guidelines #28

Are you sure you want to change the base?

[WIP][ENH] improve guidelines #28

Conversation

Remi-Gau commented Feb 6, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

oesteban Feb 29, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

oesteban left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

oesteban Mar 18, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Remi-Gau left a comment

Choose a reason for hiding this comment

francopestilli commented Mar 5, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yarikoptic Mar 21, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Remi-Gau commented Jun 21, 2024

Remi-Gau commented Sep 25, 2024

oesteban Feb 29, 2024 •

edited

Loading

oesteban Mar 18, 2024 •

edited

Loading

yarikoptic Mar 21, 2024 •

edited

Loading