Skip to content

Add documentation for external pypi #65

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 6 commits into from
Mar 25, 2022
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions doc/source/guidelines/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -18,3 +18,4 @@ functionalities such as logging, data transfer, and application APIs.
logging
service_abstraction
test_practices
private_packaging
79 changes: 79 additions & 0 deletions doc/source/guidelines/private_packaging.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,79 @@
.. _private_dependencies:


Hosting Private Dependencies
============================
There will be cases in which it is necessary to host and pull packages that are
not ready to be hosted to the public `PyPI`_. For example, if a PyAnsys library
requires auto-generated gRPC interface files from an as-of-yet private feature
or service, this package should be hosted on a private PyPI repository.

Ansys has a private repository at `PyAnsys PyPI`_, and access is controlled via

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@akaszynski what are other companies doing which might have similar scenarios?

Copy link
Contributor Author

@akaszynski akaszynski Mar 25, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Desire

There is a huge push for packages to be hosted in GitHub packages, and I think the drive is to do that:
https://github.saobby.my.eu.orgmunity/t/pypi-compatible-github-package-registry/14615/103

Solutions I've seen in the wild

A common solution has been to use public ADO for hosting both universal artifacts and Python packages. Another alternative within GitHub is to use a repository for that:
https://github.com/astariul/github-hosted-pypi

While this solution works, it's non-optimal as you have to git push and we would still have to track a PAT as pulling would require access to a private repository.

Other alternatives include

In the end we chose ADO to:

  • Avoid the security risk external pulling from internal, having to create a new internal registry.
  • Avoid creating our own private PyPI registry thereby opening up a new attack surface.
  • Reuse a tried and true Microsoft backed service.

a secret PAT, specified in the GitHub secret ``PYANSYS_PYPI_PRIVATE_PAT`` which
is only available to repositories within the `PyAnsys`_.

.. note::
Forked GitHub repositories do not have access to GitHub secrets. This is
designed to protect against PRs that could potentially scrape tokens from
our CI/CD.


Upload
------
Packages can be uploaded to the private repository with the following short
bash script. If you are operating out of a GitHub CI pipeline, email the
PyAnsys Core team at pyansys.core@ansys.com for the PAT,
``PYANSYS_PYPI_PRIVATE_PAT``.

Assuming you are already in a Python repository containing your wheels and/or
source distribution within the ``dist`` directory:

.. code::

pip install build twine pip -U

REPOSITORY_URL="https://pkgs.dev.azure.com/pyansys/_packaging/pyansys/pypi/upload"
python -m twine upload dist/* \
-p $PYANSYS_PYPI_PRIVATE_PAT \
-u PAT \
--repository-url $REPOSITORY-URL

Alternatively, you can use environment variables instead of CLI arguments for twine.

.. code::

export TWINE_USERNAME=PAT
export TWINE_PASSWORD=$PYANSYS_PYPI_PRIVATE_PAT
export TWINE_REPOSITORY_URL="https://pkgs.dev.azure.com/pyansys/_packaging/pyansys/pypi/upload"

python -m twine upload dist/*


Download
--------
To download the Python package from the `PyAnsys PyPI`_, use the following:

.. code::

INDEX_URL=https://$PYANSYS_PYPI_PRIVATE_PAT@pkgs.dev.azure.com/pyansys/_packaging/pyansys/pypi/simple/
pip install ansys-<product/tool>-<library> --index-url $INDEX_URL --no-dependencies

.. warning::
Take care to always use the ``--index-url`` switch rather than the
``--extra-index-url`` switch. As noted in the `pip Documentation`_, the
``--index-url`` switch changes the Python Package Index, and forces ``pip``
to only use packages from that package index.

Our package index uses PyPI upstream, and therefore other users cannot
inject packages from PyPI that would supersede our packages, even if they
are of a higher version.

This is not the case if you use ``--extra-index-url``, which adds rather
than replaces the default package index. For security do not use
``--extra-index-url``.


.. _PyPI: https://pypi.org/
.. _PyAnsys PyPI: https://pkgs.dev.azure.com/pyansys/_packaging/pyansys/pypi
.. _PyAnsys: https://github.com/pyansys
.. _pip Documentation: https://pip.pypa.io/en/stable/cli/pip_install/