Review guidelines

“Reviews allow for discussion of proposed changes and help ensure that the changes meet the repository’s contributing guidelines and other quality standards.“ (from GitHub documentation)

How do I process own PRs

• Decide if my PR is (a) ready to review or (b) I just need to complex testing of my contribution or (c) I want to share my work-in-progress and ask someone for help:
  • (a) -> make it as regular PR
  • (b,c) -> make it as Draft PR
• Make PR title reasonably short and sufficiently self-explainable so just after reading it anyone can get a basic idea of what it is about.
• Write short summary of your work, link issue which it fixes or closes, eventually what is missing to be done as a checkpoint-list.
• Add a label to classify the PR if it is fix or enhancement, logger to another part of code so reviewers can simpler take it according to their domain of expertise...
• Add milestone where it shall fit so we can put it to timeline, mainly distinguish if it goes to closes bugfix release 1.0.x or feature release 1.x.0

General principles

• We value everyone putting time and effort into PyTorch Lightning, but not every change should be included.
• We encourage everyone to open issues describing what they want to do before going off and doing it.
• Every feature we add takes attention and maintenance, but most importantly- might break something else.
• Discuss the tradeoffs in the PR or associated issue.

How do I review a PR?

1. Review the PR Description

• Follow the steps from self submitting PR above ^^.
• Make sure that the PR title summarizes the change (not "fixes bug #2168").
• Make sure there's an issue associated with this PR. No issue associated? Look for the relevant issue or ask the author to create one.
• Make sure PR has the right labels (bugfix, docs, tests, enhancements, etc.).
• Make sure PR description includes:
  • A detailed summary of what changed
  • The motivation for the change

2. Make sure the PR's scope makes sense.

• For new features or enhancements- ask yourself if we want this change in the product?
• Refactors should always be in separate PRs than changes.
• Every PR should change/fix exactly one thing. Otherwise, ask the author to split into smaller PRs.
• Any code changes (may be skipped form typos in docs) both features and bug-fixes shall be cover by tests existing or newly added ones, If there is a bugfix and all tests were passing on master before and the PR does not add a new test to reveal the issue, ask the author to add it... and add label "waiting for author".

3. Verify the code is correct and readable.

• Code is read way more often than it is written or modified, so readability is always important. You should aim to understand every changed line. If you don't understand it, it's likely someone else won't either. Ask questions and comments for things you don't understand.
• If you don’t think a variable name accurately represents what’s in it, suggest a better one.
• Preferably use GH suggestion tooling which makes clear what and a=how you recommend to change.
• When something has to change in the code, explain what should be changed and more importantly, why it should be changed (it is always useful to include links to external documentation/articles that provide more detailed information on the subject).

4. Verify the changes are covered by tests.

• Are there sufficient tests to ensure this change doesn’t break in the future?
• Read the tests and make sure they will break if the code does. The code in the tests is just as important as the code in the product.
• No tests? Ask the author to add them or add them yourself.
• Make sure error scenarios and edge cases are well handled.
  • Would this work for a single node? multi-node?
  • CPU? GPU? TPU?
  • Anything with spawn needs to be serializable. Spawn is basically like: a car is driving down a single lane- this is the main process. When spawn is called, that car gets copied into 8 other cars (8 GPUs). That car then goes into its own lane. The car is trained on that lane, but the original car isn’t... so consider what happens in those scenarios.
  • DDP is like the same analogy but instead of copying the car 8 times, we copy it 7 times and the main car then joins those 7 to train like all.
  • Make sure this works with 16-bit precision.

5. Check if the change is mentioned in CHANGELOG

• Any bugfix, feature, substantial change in code behaviour shall be included
• No need for typos, docs, and CI edits
• [this is testing suggestion] make 2 line spacing between all items in "active" section (unreleased changes), this shall prevent collisions in git, later before release we format the active section at once

Approving PRs

If you checked all the above and everything is looking good, you may approve the PR in your review or request changes. As a rule of thumb:

• Approve: if you are confident that the issue or feature is fully addressed and the content can be merged as it is now (no extra conditions in comments).
• Comment: if there are any doubts (in your understanding, in the general direction, etc.) but you want to express your opinion about a specific point.
• Request changes: if you are sure that the actual change will break something or has obvious issues.

Remember that if you approve a PR and it reaches the minimum number of required approvals and given that all tests pass, the branch will be automatically merged into master.

Reviewers are our friends and partners!

• Be nice and constructive, remember your fist PR and what helped you / made you motivated to continue contributing.
• Use suggestions for simple edits when you know what it should be instead of leaving a long description (you can still add a short justification of your proposal) so the author can simply accept it.