Adds current contents of readme.com docs to repo and adds notes on ho…

…w to sync. (pantsbuild#15810)

Per discussion, this is the first in a series of docs updates that will allow us to manage Pants' canonical docs from inside our repo. The markdown files are a copy of `v2.12` from `readme.com`.

Once accepted, these docs will be synced up to `readme.com` as `v2.13`. I will then create new versions of this PR against the 2.12 and 2.11 release branches, containing the docs from those versions on readme.com.

Addresses pantsbuild#15811

docs/.gitignore

@@ -0,0 +1,2 @@
+node_modules
+

docs/NOTES.md

@@ -0,0 +1,64 @@
+# Docs process
+
+Pants currently hosts documentation at Readme.com, and we use a combination of their `rdme` tool to sync handwritten markdown docs, and a custom `generate-docs` script to update Pants' reference documentation.
+
+Currently the rdme process is manual, until we bed down the process, at which point we'll add it to CI.
+
+The motivation for in-repo docs is covered [on this Google doc](https://docs.google.com/document/d/1bZE8PlF9oRzcPQz4-JUFr5vfD0LFHH4V3Nj2k221CFM/view)
+
+## Versions
+
+Readme expects every version of the docs to correspond to a semver release. Our convention is as follows:
+
+* A version on readme.com corresponds to a pants release (e.g. pants `v2.11` has docs `v2.11`)
+* The current development (`main` branch) docs are kept in a readme.com version that will reflect the next version of Pants (e.g. if the most recent release branch is `v2.97`, then `main`'s docs should be synced to `v2.98`).
+
+
+# Using `rdme` (general notes)
+
+## Setup
+
+### Install `node`
+
+```
+brew install node
+```
+
+### Install `rdme`
+
+From the `docs` directory,
+
+```
+npm install rdme
+```
+
+### Log in.
+
+```
+npx rdme login --2fa
+```
+
+(the `--2fa` flag makes `rdme` prompt for 2fa codes, which is necessary if you have TOTP 2fa set up on your account)
+
+
+## When cutting a new release branch
+
+Create a fork of the most recent docs branch, and mark it as `beta`, for example:
+
+```
+npx rdme versions:create --version=v2.98 --fork="v2.97" --main=false --beta=true --isPublic=true
+```
+
+will create a new docs version, `2.98` based on a copy of the docs from version `2.97`. 
+
+
+## Sync docs changes up to `readme.com`
+
+Docs markdown files are stored in the `markdown` directory. `rdme` does not do bidirectional sync, so any changes made on readme.com itself _will be deleted_.
+
+Make sure you apply any changes from readme.com locally before syncing up.
+
+```
+npx rdme docs markdown --version v2.98
+```
+

docs/markdown/Contributions/contributor-overview.md

@@ -0,0 +1,166 @@
+---
+title: "Contribution overview"
+slug: "contributor-overview"
+excerpt: "The flow for making changes to Pants."
+hidden: false
+createdAt: "2020-05-16T22:54:21.867Z"
+updatedAt: "2022-06-04T12:23:09.473Z"
+---
+We welcome contributions of all types: from fixing typos to bug fixes to new features. For further questions about any of the below, please refer to the [community overview](doc:community).
+
+> 👍 Help wanted: identifying bad error messages
+> 
+> We strive—but sometimes fail—to make every error message easy to understand and to give insight into what went wrong and how to fix it.
+> 
+> If you ever encounter a confusing or mediocre error message, we would love your help to identify the error message. Please open a [GitHub issue](https://github.com/pantsbuild/pants/issues) with the original Pants command, the error message, and what you found confusing or think could be improved.
+> 
+> (If you'd be interested in then changing the code, we'd be happy to point you in the right direction!)
+
+Documentation Fixes
+-------------------
+
+Pants uses [readme.com](https://readme.com/) for the documentation at [pantsbuild.org](https://pantsbuild.org). On any of docs pages, click "Suggest Edits" at the top right corner to be dropped into the editor interface. You will be asked to log in or sign up to readme.com, which only asks for your email address and a name.
+
+When ready, click "Submit Suggested Edits" at the top right corner in order to send them to us for review.
+
+Pants's tech stack
+------------------
+
+Most of Pants is written in Python 3. The majority of contributions touch this Python codebase.
+
+We rely on several Python features that you will want to acquaint yourself with:
+
+- [Type hints and MyPy](https://mypy.readthedocs.io/en/stable/)
+- [Dataclasses](https://realpython.com/python-data-classes/)
+- [`async`/`await` coroutines](https://www.python.org/dev/peps/pep-0492)
+  - We do _not_ use `asyncio`. The scheduler is implemented in Rust. We only use `async` coroutines.
+- [Decorators](https://realpython.com/primer-on-python-decorators/)
+- [Comprehensions](https://www.geeksforgeeks.org/comprehensions-in-python/)
+
+Pants's engine is written in Rust. See [Developing Rust](doc:contributions-rust) for a guide on making changes to the internals of Pants's engine.
+
+First, share your plan
+----------------------
+
+Before investing your time into a code change, it helps to share your interest. This will allow us to give you initial feedback that will save you time, such as pointing you to related code.
+
+To share your plan, please either open a [GitHub issue](https://github.com/pantsbuild/pants/issues) or message us on [Slack](doc:getting-help#slack) (you can start with the #general channel). Briefly describe the change you'd like to make, including a motivation for the change.
+
+If we do not respond within 24 business hours, please gently ping us by commenting "ping" on your GitHub issue or messaging on Slack asking if someone could please take a look.
+
+> 📘 Tip: Can you split out any "prework"?
+> 
+> If your change is big, such as adding a new feature, it can help to split it up into multiple pull requests. This makes it easier for us to review and to get passing CI.
+> 
+> This is a reason we encourage you to share your plan with us - we can help you to scope out if it would make sense to split into multiple PRs.
+
+Design docs
+-----------
+
+Changes that substantially impact the user experience, APIs, design or implementation, may benefit from a design doc that serves as a basis for discussion. 
+
+We store our design docs in [this Google Drive folder](https://drive.google.com/drive/folders/1LtA1EVPvalmfQ5AIDOqGRR3LV86_qCRZ). If you want to write a design doc, [let us know](https://www.pantsbuild.org/docs/getting-help) and if necessary we can give you write access to that folder.
+
+We don't currently have any guidelines on the structure or format of design docs, so write those as you see fit.
+
+Developing your change
+----------------------
+
+To begin, [set up Pants on your local machine](doc:contributor-setup).
+
+To run a test, run:
+
+```bash
+$ ./pants test src/python/pants/util/frozendict_test.py
+```
+
+Periodically, you will want to run MyPy and the autoformatters and linters:
+
+```bash
+# Format un-committed changes
+$ ./pants --changed-since=HEAD fmt
+
+# Run the pre-commit checks, including `check` and `lint`
+$ build-support/githooks/pre-commit
+```
+
+See our [Style guide](doc:style-guide) for some Python conventions we follow.
+
+> 📘 You can share works in progress!
+> 
+> You do not need to fully finish your change before asking for feedback. We'd be eager to help you while iterating.
+> 
+> If doing this, please open your pull request as a "Draft" and prefix your PR title with "WIP". Then, comment on the PR asking for feedback and/or post a link to the PR in [Slack](doc:community).
+
+Opening a pull request
+----------------------
+
+When opening a pull request, start by providing a concise and descriptive title. It's okay if you aren't sure what to put - we can help you to reword it. 
+
+Good titles:
+
+- Fix typo in `strutil.py`
+- Add Thrift code generator for Python
+- Fix crash when running `test` with Python 3.9
+
+Bad titles:
+
+- Fix bug
+- Fix #8313
+- Add support for Thrift code generation by first adding the file `codegen.py`, then hooking it up, and finally adding tests
+
+Then, include a description. You can use the default template if you'd like, or use a normal description instead. Link to any corresponding GitHub issues.
+
+Finally—if you have the permissions—add exactly one of the following labels to your PR. Otherwise, a maintainer will do this for you:
+
+- `category:new feature` for new features
+- `category:user api change` for changes that affect how end-users interact with Pants
+- `category:plugin api change` for changes that affect how plugin authors interact with Pants internals
+- `category:performance` for changes focused on improving performance
+- `category:bugfix` for bugfixes
+- `category:documentation` for documentation changes, including logging and help messages
+- `category:internal` for miscellaneous, internal-facing changes
+
+Pick the first of these that applies to your change. I.e., if you have modified the user API in a change that also improves performance, use `category:user api change`.
+
+These labels are used to generate the changelist for each release.
+
+> 📘 Tip: Review your own PR
+> 
+> It is often helpful to other reviewers if you proactively review your own code. Specifically, add comments to parts where you want extra attention.
+> 
+> For example:
+> 
+> - "Do you know of a better way to do this? This felt clunky to write."
+> - "This was really tricky to figure out because there are so many edge cases. I'd appreciate extra attention here, please."
+> - "Note that I did not use a dataclass here because I do not want any of the methods like `**eq**` to be generated."
+
+> 📘 FYI: we squash merge
+> 
+> This means that the final commit message will come from your PR description, rather than your commit messages. 
+> 
+> Good commit messages are still very helpful for people reviewing your code; but, your PR description is what will show up in the changelog.
+
+### CI
+
+We use GitHub Actions for CI. Look at the "Checks" tab of your PR.
+
+> 📘 Flaky tests?
+> 
+> We unfortunately have some flaky tests. If CI fails and you believe it is not related to your change, please comment about the failure so that a maintainer may investigate and restart CI for you.
+> 
+> Alternatively, you can push an empty commit with `git commit --allow-empty` to force CI to restart. Although we encourage you to still point out the flake to us.
+
+### Review feedback
+
+One or more reviewers will leave feedback. If you are confused by any of the feedback, please do not be afraid to ask for clarification!
+
+If we do not respond within 24 business hours, please gently ping us by commenting "ping" on your pull request or messaging on Slack asking if someone could please take a look.
+
+Once one or more reviewers have approved—and CI goes green—a reviewer will merge your change.
+
+> 📘 When will your change be released?
+> 
+> Your change will be included in the next weekly dev release, which usually happens every Friday or Monday. If you fixed a bug, your change may also be cherry-picked into a release candidate from the prior release series.
+> 
+> See [Release strategy](doc:release-strategy).

docs/markdown/Contributions/development.md

@@ -0,0 +1,14 @@
+---
+title: "Development"
+slug: "development"
+excerpt: "How to make code changes to Pants."
+hidden: false
+createdAt: "2020-07-23T21:02:22.190Z"
+updatedAt: "2021-11-09T22:03:40.187Z"
+---
+* [Setting up Pants](doc:contributor-setup)
+* [Style guide](doc:style-guide)
+* [Developing Rust](doc:contributions-rust)
+* [Internal Architecture](doc:internal-rules-architecture)
+* [Debugging and benchmarking](doc:contributions-debugging)
+* [Running Pants from sources](doc:running-pants-from-sources)

docs/markdown/Contributions/development/contributions-debugging.md

@@ -0,0 +1,86 @@
+---
+title: "Debugging and benchmarking"
+slug: "contributions-debugging"
+excerpt: "Some techniques to figure out why Pants is behaving the way it is."
+hidden: false
+createdAt: "2020-09-04T23:43:34.260Z"
+updatedAt: "2022-03-09T16:40:50.789Z"
+---
+[block:api-header]
+{
+  "title": "Benchmarking with `hyperfine`"
+}
+[/block]
+We use `hyperfine` to benchmark, especially comparing before and after to see the impact of a change: https://github.com/sharkdp/hyperfine.
+
+When benchmarking, you must decide if you care about cold cache performance vs. warm cache (or both). If cold, use `--no-pantsd --no-local-cache`. If warm, use hyperfine's option `--warmup=1`.
+
+For example:
+
+```
+❯ hyperfine --warmup=1 --runs=5 './pants list ::`
+❯ hyperfine --runs=5 './pants --no-pantsd --no-local-cache lint ::'
+```
+[block:api-header]
+{
+  "title": "Profiling with py-spy"
+}
+[/block]
+`py-spy` is a profiling sampler which can also be used to compare the impact of a change before and after: https://github.com/benfred/py-spy.
+
+To profile with `py-spy`:
+
+1. Activate Pants' development venv
+    * `source ~/.cache/pants/pants_dev_deps/<your platform dir>/bin/activate`
+2. Add Pants' code to Python's path
+    * `export PYTHONPATH=src/pants:$PYTHONPATH`
+3. Run Pants with `py-spy` (be sure to disable `pantsd`)
+    * `py-spy record --subprocesses -- python -m pants.bin.pants_loader --no-pantsd <pants args>`
+
+The default output is a flamegraph. `py-spy` can also output speedscope (https://github.com/jlfwong/speedscope) JSON with the `--format speedscope` flag. The resulting file can be uploaded to https://www.speedscope.app/ which provides a per-process, interactive, detailed UI.
+
+Additionally, to profile the Rust code the `--native` flag can be passed to `py-spy` as well. The resulting output will contain frames from Pants Rust code.
+[block:api-header]
+{
+  "title": "Identifying the impact of Python's GIL (on macOS)"
+}
+[/block]
+
+[block:embed]
+{
+  "html": "<iframe class=\"embedly-embed\" src=\"//cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FzALr3zFIQJo%3Ffeature%3Doembed&display_name=YouTube&url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DzALr3zFIQJo&image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FzALr3zFIQJo%2Fhqdefault.jpg&key=f2aa6fc3595946d0afc3d76cbbd25dc3&type=text%2Fhtml&schema=youtube\" width=\"640\" height=\"480\" scrolling=\"no\" title=\"YouTube embed\" frameborder=\"0\" allow=\"autoplay; fullscreen\" allowfullscreen=\"true\"></iframe>",
+  "url": "https://www.youtube.com/watch?v=zALr3zFIQJo",
+  "title": "Identifying contention on the Python GIL in Rust from macOS",
+  "favicon": "https://www.youtube.com/s/desktop/c9a10b09/img/favicon.ico",
+  "image": "https://i.ytimg.com/vi/zALr3zFIQJo/hqdefault.jpg"
+}
+[/block]
+
+[block:api-header]
+{
+  "title": "Obtaining Full Thread Backtraces"
+}
+[/block]
+Pants runs as a Python program that calls into a native Rust library. In debugging locking and deadlock issues, it is useful to capture dumps of the thread stacks in order to figure out where a deadlock may be occurring.
+
+One-time setup:
+
+1. Ensure that gdb is installed.
+    * Ubuntu: `sudo apt install gdb`
+2. Ensure that the kernel is configured to allow debuggers to attach to processes that are not in the same parent/child process hierarchy.
+   * `echo 0 | sudo tee /proc/sys/kernel/yama/ptrace_scope`
+   * To make the change permanent, add a file to /etc/sysctl.d named `99-ptrace.conf` with contents `kernel.yama.ptrace_scope = 0`. **Note: This is a security exposure if you are not normally debugging processes across the process hierarchy.**
+3. Ensure that the debug info for your system Python binary is installed.
+   * Ubuntu: `sudo apt install python3-dbg`
+
+Dumping thread stacks:
+
+1. Find the pants binary (which may include pantsd if pantsd is enabled).
+    * Run: `ps -ef | grep pants`
+2. Invoke gdb with the python binary and the process ID:
+    * Run: `gdb /path/to/python/binary PROCESS_ID`
+3. Enable logging to write the thread dump to `gdb.txt`: `set logging on`
+4. Dump all thread backtraces: `thread apply all bt`
+5. If you use pyenv to mange your Python install, a gdb script will exist in the same directory as the Python binary. Source it into gdb:
+    * `source ~/.pyenv/versions/3.8.5/bin/python3.8-gdb.py` (if using version 3.8.5)
+6. Dump all Python stacks: `thread apply all py-bt`