Add initial staging docs

fixes #1014

CHANGES/1014.doc

@@ -0,0 +1 @@
+Added initial staging docs.

doc_requirements.txt

@@ -15,4 +15,7 @@ sphinxcontrib-jquery
 sphinxcontrib-openapi
 mistune<4.0.0
 Jinja2<3.2
+
+# Unified docs
+pulp-docs @ git+https://github.com/pulp/pulp-docs@main
 # Extra requirements

pyproject.toml

@@ -53,6 +53,7 @@ ignore = [
     "dev_requirements.txt",
     "doc_requirements.txt",
     "docs/**",
+    "staging_docs/**",
     "template_config.yml",
     ".pep8speaks.yml",
     ".ci/**",

staging_docs/dev/guides/bindings.md

@@ -0,0 +1,39 @@
+# Client Bindings
+
+## Python Client
+
+The [pulp-deb-client package](https://pypi.org/project/pulp-deb-client/) on PyPI provides bindings for all `pulp_deb` [REST API][1] endpoints.
+It is currently published daily and with every RC.
+
+The [pulpcore-client package](https://pypi.org/project/pulpcore-client) on PyPI provides bindings for all `pulpcore` [REST API][2] endpoints.
+It is currently published daily and with every RC.
+
+## Ruby Client
+
+The [pulp_deb_client Ruby Gem](https://rubygems.org/gems/pulp_deb_client) on rubygems.org provides bindings for all `pulp_deb` [REST API][1] endpoints.
+It is currently published daily and with every RC.
+
+The [pulpcore_client Ruby Gem](https://rubygems.org/gems/pulpcore_client) on rubygems.org provides bindings for all `pulpcore` [REST API][2] endpoints.
+It is currently published daily and with every RC.
+
+## Client in a Language of Your Choice
+
+A client can be generated using Pulp's OpenAPI schema and any of the available [OpenAPI generators](https://openapi-generator.tech/docs/generators).
+
+Generating a client is a two step process:
+
+1. Download the OpenAPI schema for pulpcore and all installed plugins:
+   ```bash
+   curl -o api.json http://<pulp-hostname>:24817/pulp/api/v3/docs/api.json
+   ```
+   The OpenAPI schema for a specific plugin can be downloaded by specifying the plugin's module name as a GET parameter.
+   For example for `pulp_deb` only endpoints use a query like this:
+   ```bash
+   curl -o api.json http://<pulp-hostname>:24817/pulp/api/v3/docs/api.json?plugin=pulp_deb
+   ```
+2. Generate a client using OpenAPI generator:
+   The schema can then be used as input to the `openapi-generator-cli`.
+   See [try OpenAPI generator](https://openapi-generator.tech/#try) for documentation on getting started.
+
+[1]: https://staging-docs.pulpproject.org/pulp_deb/restapi/
+[2]: https://staging-docs.pulpproject.org/pulpcore/restapi/

staging_docs/dev/guides/plugin_maintenance.md

@@ -0,0 +1,102 @@
+# Plugin Maintenance
+
+This part of the documentation is intended as an aid to current and future plugin maintainers.
+
+## Plugin Version Semantics
+
+Release version strings of the `pulp_deb` plugin use the following format: `X.Y.Z<beta_tag?>`
+
+A `X` version release, signifies one or more of the following:
+
+- There are major new features.
+- There has been a major overhaul of existing features.
+- The plugin has entered a new stage of its development.
+- The plugin is compatible with a new pulpcore `X` version.
+
+!!! note
+    A `X` version release, is more of a high level communication, than something with a detailed technical definition.
+    It is up to the judgement of plugin maintainers when a new `X` version is warranted.
+
+
+A `Y` version release, signifies one or more of the following:
+
+- This release contains new features.
+- This release may require a newer version of pulpcore (or some other dependency) than the `Y-1` release branch did.
+- A `Y` version release is given its own release branch.
+
+A `Z` version release, signifies the following:
+
+- A `Z` version release may contain only bugfixes (semantic versioning).
+- A `Z` version change is cherry-picked to the relevant `Y` version release branch.
+
+Pulp plugins follow a "release whenever" policy, this means both `Y` and `Z` releases may happen whenever there are relevant changes, and as the need arises.
+
+## Plugin Release Steps
+
+!!! note
+    The [pulpcore release guide](https://pulp.plan.io/projects/pulp/wiki/Release_Guide) wiki article is now badly out of date, but may still be of interest as a reference.
+
+
+At the time of this writing the release process required the following steps:
+
+1. (optional) Run the "CI Update" action and review and merge any resulting PRs.
+2. (`Y` version release only) Run the "Create New Release Branch" action.
+3. Run the "Release Pipeline" action from the relevant release branch and merge any resulting PRs.
+4. Check that each of the following has been released:
+   * The [pulp_deb python package](https://pypi.org/project/pulp-deb/) on pypi.org.
+   * The [pulp-deb-client package](https://pypi.org/project/pulp-deb-client) on pypi.org.
+   * The [pulp_deb_client Ruby Gem](https://rubygems.org/gems/pulp_deb_client) on rubygems.org.
+   * This `pulp_deb` documentation.
+5. (`Y` version release only) Update the `ci_update_branches` variable in the `template_config.yml` file in the [pulp_deb source repository](https://github.com/pulp/pulp_deb).
+6. (optional) Run the "changelog update" action to add any release branch changelog commits to the main branch.
+   If you skip this, it will be picked up by the nightly CI.
+7. Post a release announcement under [Pulp community forum announcements](https://discourse.pulpproject.org/c/announcements/6).
+   Hint: Simply copy and past from past release announcements making adjustments as necessary.
+
+
+## Using the Plugin Template
+
+For existing plugins, the [Pulp plugin template](https://github.com/pulp/plugin_template) is used to keep the plugin CI up to date and synchronized across all plugins.
+Applying the latest plugin template is primarily performed via GitHub actions pipeline.
+This pipeline is periodically run or can be triggered manually (for example before a new `Y` release).
+The pipeline will open a PR against every branch listed in the `ci_update_branches` variable of the `template_config.yml` file of the `pulp_deb` source repository.
+
+It is also possible (and sometimes useful) to run the plugin template locally.
+Make sure you clone the `plugin_template` repository in the same folder as the `pulp_deb` repository.
+You can then issue the below template commands within the root of the plugin template repository to apply changes.
+
+!!! important
+    The following commands are usually applied via the GitHub actions pipeline, which is the preferred way for creating the needed PRs.
+
+```none
+./plugin-template --generate-config pulp_deb
+./plugin-template --github pulp_deb
+```
+
+!!! important
+    The following plugin template commands are primarily intended for creating the skeleton for a new plugin.
+    It sometimes makes sense to apply them to an existing plugin, but the applied changes will clash with the existing plugin content, and need to be manually evaluated.
+
+```none
+./plugin-template --docs pulp_deb
+./plugin-template --bootstrap pulp_deb
+```
+
+## Building the docs and API docs
+
+The content for the [pulp_deb REST API documentation](https://staging-docs.pulpproject.org/pulp_deb/changes/) is extracted via API call to a running pulp instance.
+It will contain various docstrings from the plugin code as deployed to that instance.
+As a result, building the docs can only be done via the CI pipelines or a full fledged development environment.
+
+This can be done within a `pulp3-source-*` vagrant box from the `pulp_installer` repository, that has the `pulp_deb` plugin installed.
+Within such a box run the following commands:
+
+```none
+cd /home/vagrant/devel/pulp_deb/docs/
+make html
+```
+
+You can now find the built documentation at `docs/_build/html/index.html` within your local `pulp_deb` repository.
+You can also find the API doc contents within `docs/_static/api.json`.
+
+You can open the locally built documentation in a browser, but you will not be able to view the API docs, since those make use of an external service, that obviously has no access to your local build.

staging_docs/user/guides/_SUMMARY.md

@@ -0,0 +1,10 @@
+[//]: Learn more here about the syntax:
+[//]: https://staging-docs.pulpproject.org/pulp-docs/docs/dev/reference/markdown-cheatsheet/#ordering-files
+
+* [Setting up API tooling](api_tooling.md)
+* [Repository Synchronization](sync.md)
+* [Package Uploads](upload.md)
+* [Publish Repositories](publish.md)
+* [Signing Service Creation](signing_service.md)
+* [Advanced Copy](advanced_copy.md)
+* [Configuring Checksums](checksums.md)

staging_docs/user/guides/advanced_copy.md

@@ -0,0 +1,66 @@
+# Advanced Copy
+
+You can copy packages between Pulp repositories, *along with all associated structure content* using the advanced copy API.
+
+For example, let us first set up an example repository with some test content:
+
+```bash
+NAME='advanced-copy-example'
+REMOTE_OPTIONS=(
+  --url=https://fixtures.pulpproject.org/debian/
+  --distribution=ragnarok
+  --distribution=ginnungagap
+  --architecture=ppc64
+  --architecture=armeb
+  --policy=on_demand
+)
+pulp deb remote create --name=${NAME} ${REMOTE_OPTIONS[@]}
+pulp deb repository create --name=${NAME}-src --remote=${NAME}
+pulp deb repository sync --name=${NAME}-src
+```
+
+Now let us copy just one package from this src repository to a different target repository using the advanced copy API:
+
+```bash
+NAME='advanced-copy-example'
+PULP_URL='http://localhost:5001'
+SRC_VERSION_HREF=$(pulp deb repository show --name=${NAME}-src | jq -r '.latest_version_href')
+TARGET_REPO_HREF=$(pulp deb repository create --name=${NAME}-target | jq -r '.pulp_href')
+PACKAGE_HREF=$(
+  pulp deb repository content --type=package list --repository=${NAME}-src \
+  | jq -r '.[0].pulp_href'
+)
+echo "\
+[
+  {
+    \"source_repo_version\": \"${SRC_VERSION_HREF}\",
+    \"dest_repo\": \"${TARGET_REPO_HREF}\",
+    \"content\": [
+      \"${PACKAGE_HREF}\"
+    ]
+  }
+]" > copy_config.json
+http post ${PULP_URL}/pulp/api/v3/deb/copy/ config:=@copy_config.json
+```
+
+After the copy task has completed, check the new repository version to see we do not just have `deb.package` type content, but also structure content of type `deb.package_release_component` and others:
+
+```bash
+pulp deb repository version show --repository=${NAME}-target
+```
+
+Our example `copy_config.json` from the above example might look like this:
+
+```json
+[
+  {
+    "source_repo_version": "/pulp/api/v3/repositories/deb/apt/018a600b-06d1-71a8-ac7f-6275e8ed7fd7/versions/1/",
+    "dest_repo": "/pulp/api/v3/repositories/deb/apt/018a600b-a719-75ea-b7b8-4916176496ba/",
+    "content": [
+      "/pulp/api/v3/content/deb/packages/018a2c87-43d7-7013-a40f-db1ca6e2f367/"
+    ]
+  }
+]
+```
+
+This example could easily be extended with extra values in the `"content"` list to copy several packages at once.

staging_docs/user/guides/api_tooling.md

@@ -0,0 +1,75 @@
+# Setting up API tooling
+
+In order to use Pulp we must be able to talk to the Pulp API.
+Usage examples throughout this documentation depend on the API tooling described below.
+
+
+## Setting up Pulp CLI for pulp_deb
+
+!!! tip
+    We recommend installing Pulp CLI in a virtual python environment.
+
+
+Pulp CLI for `pulp_deb` can be installed via the [pulp-cli-deb python package](https://pypi.org/project/pulp-cli-deb/), on any host that can reach the REST API of your Pulp instance:
+
+```none
+pip install pulp-cli-deb
+```
+
+This will also pull in the [pulp-cli python package](https://pypi.org/project/pulp-cli/) as a dependency.
+The `pulp-cli` package contains the core of Pulp CLI, as well as the subcommands for many other Pulp content plugins like `pulp_file` and `pulp_rpm`.
+
+Once you have installed Pulp CLI, you will need to configure it, so that it can talk to the REST API of your Pulp instance:
+
+```none
+pulp config create --help  # List a description of available config options
+pulp config create -e  # Open a generated default config file for editing
+```
+
+Make sure you set the `base_url`, the `api_root` and any API credentials.
+The default location for the CLI config file is `~/.config/pulp/cli.toml`.
+
+To test if Pulp CLI can reach the Pulp API use:
+
+```none
+pulp status
+```
+
+!!! note
+    The status API endpoint does not require authentication, so this will work even if the configured API credentials are incorrect.
+
+
+For more information see the [pulp-cli documentation](https://staging-docs.pulpproject.org/pulp-cli/docs/user/).
+To open bug reports or feature requests against `pulp-cli-deb`, see the [pulp-cli-deb issue tracker](https://github.com/pulp/pulp-cli-deb/issues).
+
+To start using the CLI commands for `pulp_deb`, consult:
+
+```none
+pulp deb --help
+```
+
+
+## Setting up httpie and jq
+
+Where there are feature gaps in Pulp CLI, or for advanced scripting scenarios, it may be preferrable to interact with the `Pulp REST API ` directly.
+Examples within this documentation will use the [httpie](https://httpie.io/) utility for this purpose.
+httpie can be installed via `pip`, so you could install it alongside Pulp CLI in a virtual python environment using:
+
+```none
+pip install pulp-cli-deb httpie
+```
+
+You will need to configure your Pulp API credentials, or else specify them with each call to `http`.
+For example, you can adjust the following `.netrc` example as needed:
+
+```none
+machine localhost
+login admin
+password password
+```
+
+For alternative methods, please consult `http --help`.
+
+Both in combination with Pulp CLI as well as httpie, we recommend and make use of [jq](https://jqlang.github.io/jq/) for parsing API responses.
+`jq` can normally be installed via your preferred package manager, for example using `sudo apt-get install jq` for APT based systems.
+Being a simple command-line JSON parser, `jq` requires no additional configuration.

staging_docs/user/guides/checksums.md

@@ -0,0 +1,46 @@
+# Configuring Checksums
+
+What checksum algorithems are available on any given Pulp instance, is controlled by the pulpcore `ALLOWED_CONTENT_CHECKSUMS` setting. If enabled, `pulp_deb` will make use of MD5, SHA-1, SHA-256, and SHA-512. SHA-256 is required and cannot be disabled.
+
+!!! important
+    For compliance reasons, MD5 and SHA-1 have been disabled by default since pulpcore 3.11.
+
+!!! note
+    It is almost universal practice within the Debian ecosystem to make use of MD5 in particular.
+    If your Pulp instance is configured to disallow the MD5 checksum algorithm, `pulp_deb` will log a warning with every sync as well as every use of the APT publisher.
+    You can disable these warnings, by changing the `FORBIDDEN_CHECKSUM_WARNINGS` setting to `False`.
+
+
+!!! warning
+    While the APT publisher respects the `ALLOWED_CONTENT_CHECKSUMS` setting, the verbatim publisher will mirror an exact copy of the synced upstream metadata.
+    By its very nature, the verbatim publisher  does not respect the `ALLOWED_CONTENT_CHECKSUMS` setting.
+    If your policy prohibits the presence of certain checksum types, either do not use the verbatim publisher, or only synchronize repositories that are already compliant with your policy.
+
+
+## Enabling MD5 and/or SHA1
+
+In order to opt in to MD5 and/or SHA1, the values `md5` and/or `sha1` must be added to the list configured for the `ALLOWED_CONTENT_CHECKSUMS` setting. In addition a `pulpcore-manager` command must be run to generate any missing checksums within the DB for the settings change to take effect.
+
+See the [pulpcore configuration documentation](https://staging-docs.pulpproject.org/pulpcore/docs/admin/guides/configure-pulp/) for more information on how to apply settings. Look for the section on `ALLOWED_CONTENT_CHECKSUMS` in the [pulpcore settings documentation](https://staging-docs.pulpproject.org/pulpcore/docs/admin/learn/settings/#pulp-settings) in particular.
+
+Once you have updated your configuration file you will need to halt your Pulp instance, and then run the following command to ensure your DB is made consistent with your `ALLOWED_CONTENT_CHECKSUMS` setting:
+
+```none
+pulpcore-manager handle-artifact-checksums
+```
+
+!!! note
+    Missing checksums will need to be recalculated for all your artifacts which can take some time.
+    If you have a missmatch between your configured `ALLOWED_CONTENT_CHECKSUMS` setting and the checksums present in the DB, Pulp will simply refuse to start.
+
+
+## Security Implications
+
+APT repository metadata files (release files and package indecies) will typically include several checksums for all referenced files.
+During synchronization, `pulp_deb` will check the downloaded files against the provided checksums.
+Together with a valid (and trusted) release file signature this will guarantee the integrity of the synchronized repository.
+During publication, the APT publisher will include any supported (and permitted) checksumtypes in any metadata files it publishes.
+
+While the MD5 and SHA-1 algorithms are no longer considered secure, `pulp_deb` *requires* the presence of SHA-256 checksums to function, and is therefore *never* dependent on unsecure algorithms for integrity checking.
+MD5 and SHA-1 are checked and used *in addition* to the other algorithms.
+This is consistent with widespread usage within the larger Debian ecosystem.