Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add examples extension #54

Merged
merged 20 commits into from
Jun 1, 2023
Merged

Add examples extension #54

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
20 commits
Select commit Hold shift + click to select a range
bd8cdd4
Initial stripped down extension
May 24, 2023
90a6555
Make it more obvious that this is examples for salt analytics (change…
May 25, 2023
4d3e418
Add initial requirements setup
May 30, 2023
8fffed2
initial requirements setup
May 30, 2023
a9c2bbf
Add examples tests and add pre-commit step for combining examples req…
May 31, 2023
79c2b01
Run pre-commit
May 31, 2023
8387e07
Use a set, order doesn't matter
May 31, 2023
9187f23
We don't support Python < 3.7, and we need to ensure no accidental es…
May 31, 2023
5edbc55
Fix example plugins test, we switched to an underscore for consistency
May 31, 2023
5165a0c
Move the examples tests to be inside the examples subdirectory
May 31, 2023
62b3b86
Avoid `Duplicate module named 'tests'`
May 31, 2023
391b08c
It's examples, not extras
May 31, 2023
0b4cfdb
Remove the old example plugins test
May 31, 2023
ff7cabc
Don't hardcode the tests directory in `ci.yml`
May 31, 2023
9948dd7
Add example pipeline instructions
May 31, 2023
aa7139e
Skip the machine learning tests on windows, they are unable to load t…
May 31, 2023
d31f374
Pin to a lower `importlib-metadata` requirement
May 31, 2023
999cccd
Make pytestmark always a multiline list and move mypy configuration t…
Jun 1, 2023
7d6cf16
Switch to testsuite string value instead of boolean value for nicer a…
Jun 1, 2023
b99ec4e
Changelog
Jun 1, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 0 additions & 1 deletion .dockerignore
View file
Open in desktop

This file was deleted.

67 changes: 42 additions & 25 deletions .github/workflows/ci.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -85,14 +85,22 @@ jobs:
max-parallel: 10
matrix:
include:
- {"salt-version": "3005.0", "python-version": "3.7", "source": "pypi"}
- {"salt-version": "3005.0", "python-version": "3.8", "source": "pypi"}
- {"salt-version": "3005.0", "python-version": "3.9", "source": "pypi"}
- {"salt-version": "3006.0", "python-version": "3.7", "source": "pypi"}
- {"salt-version": "3006.0", "python-version": "3.8", "source": "pypi"}
- {"salt-version": "3006.0", "python-version": "3.9", "source": "pypi"}
- {"salt-version": "3006.0", "python-version": "3.10", "source": "pypi"}
- {"salt-version": "3006.x", "python-version": "3.10", "source": "onedir"}
- {"salt-version": "3005.0", "python-version": "3.7", "testsuite": "core", "source": "pypi"}
- {"salt-version": "3005.0", "python-version": "3.7", "testsuite": "examples", "source": "pypi"}
- {"salt-version": "3005.0", "python-version": "3.8", "testsuite": "core", "source": "pypi"}
- {"salt-version": "3005.0", "python-version": "3.8", "testsuite": "examples", "source": "pypi"}
- {"salt-version": "3005.0", "python-version": "3.9", "testsuite": "core", "source": "pypi"}
- {"salt-version": "3005.0", "python-version": "3.9", "testsuite": "examples", "source": "pypi"}
- {"salt-version": "3006.0", "python-version": "3.7", "testsuite": "core", "source": "pypi"}
- {"salt-version": "3006.0", "python-version": "3.7", "testsuite": "examples", "source": "pypi"}
- {"salt-version": "3006.0", "python-version": "3.8", "testsuite": "core", "source": "pypi"}
- {"salt-version": "3006.0", "python-version": "3.8", "testsuite": "examples", "source": "pypi"}
- {"salt-version": "3006.0", "python-version": "3.9", "testsuite": "core", "source": "pypi"}
- {"salt-version": "3006.0", "python-version": "3.9", "testsuite": "examples", "source": "pypi"}
- {"salt-version": "3006.0", "python-version": "3.10", "testsuite": "core", "source": "pypi"}
- {"salt-version": "3006.0", "python-version": "3.10", "testsuite": "examples", "source": "pypi"}
- {"salt-version": "3006.x", "python-version": "3.10", "testsuite": "core", "source": "onedir"}
- {"salt-version": "3006.x", "python-version": "3.10", "testsuite": "examples", "source": "onedir"}

steps:
- uses: actions/checkout@v3
Expand Down Expand Up @@ -121,20 +129,20 @@ jobs:
env:
SALT_REQUIREMENT: salt~=${{ matrix.salt-version }}
run: |
nox --force-color -e tests-${{ matrix.source == 'onedir' && 'onedir' || '3' }} --install-only
nox --force-color -e tests${{ matrix.testsuite == 'examples' && '-examples' || '' }}-${{ matrix.source == 'onedir' && 'onedir' || '3' }} --install-only

- name: Test
env:
SALT_REQUIREMENT: salt~=${{ matrix.salt-version }}
SKIP_REQUIREMENTS_INSTALL: YES
run: |
nox --force-color -e tests-${{ matrix.source == 'onedir' && 'onedir' || '3' }} -- -vv tests/
nox --force-color -e tests${{ matrix.testsuite == 'examples' && '-examples' || '' }}-${{ matrix.source == 'onedir' && 'onedir' || '3' }} -- -vv

- name: Upload Logs
if: always()
uses: actions/upload-artifact@main
with:
name: runtests-${{ runner.os }}-${{ matrix.python-version }}-salt-${{ matrix.salt-version }}-${{ matrix.source }}.log
name: runtests-${{ runner.os }}-${{ matrix.python-version }}-salt-${{ matrix.salt-version }}-${{ matrix.source }}${{ matrix.testsuite == 'examples' && '-examples' || '' }}.log
path: artifacts/runtests-*.log


Expand All @@ -150,11 +158,16 @@ jobs:
max-parallel: 10
matrix:
include:
- {"salt-version": "3005.0", "python-version": "3.8", "source": "pypi"}
- {"salt-version": "3006.0", "python-version": "3.8", "source": "pypi"}
- {"salt-version": "3006.0", "python-version": "3.9", "source": "pypi"}
- {"salt-version": "3006.0", "python-version": "3.10", "source": "pypi"}
- {"salt-version": "3006.x", "python-version": "3.10", "source": "onedir"}
- {"salt-version": "3005.0", "python-version": "3.8", "testsuite": "core", "source": "pypi"}
- {"salt-version": "3005.0", "python-version": "3.8", "testsuite": "examples", "source": "pypi"}
- {"salt-version": "3006.0", "python-version": "3.8", "testsuite": "core", "source": "pypi"}
- {"salt-version": "3006.0", "python-version": "3.8", "testsuite": "examples", "source": "pypi"}
- {"salt-version": "3006.0", "python-version": "3.9", "testsuite": "core", "source": "pypi"}
- {"salt-version": "3006.0", "python-version": "3.9", "testsuite": "examples", "source": "pypi"}
- {"salt-version": "3006.0", "python-version": "3.10", "testsuite": "core", "source": "pypi"}
- {"salt-version": "3006.0", "python-version": "3.10", "testsuite": "examples", "source": "pypi"}
- {"salt-version": "3006.x", "python-version": "3.10", "testsuite": "core", "source": "onedir"}
- {"salt-version": "3006.x", "python-version": "3.10", "testsuite": "examples", "source": "onedir"}

steps:
- uses: actions/checkout@v3
Expand Down Expand Up @@ -195,7 +208,7 @@ jobs:
- name: Define test session
shell: bash
id: define-test-session
run: echo "test-session=tests-${{ matrix.source == 'onedir' && 'onedir' || '3' }}" >> "$GITHUB_OUTPUT"
run: echo "test-session=tests${{ matrix.testsuite == 'examples' && '-examples' || '' }}-${{ matrix.source == 'onedir' && 'onedir' || '3' }}" >> "$GITHUB_OUTPUT"

- name: Install Test Requirements
shell: bash
Expand Down Expand Up @@ -230,7 +243,7 @@ jobs:
if: always()
uses: actions/upload-artifact@main
with:
name: runtests-${{ runner.os }}-${{ matrix.python-version }}-salt-${{ matrix.salt-version }}-${{ matrix.source }}.log
name: runtests-${{ runner.os }}-${{ matrix.python-version }}-salt-${{ matrix.salt-version }}-${{ matrix.source }}${{ matrix.testsuite == 'examples' && '-examples' || '' }}.log
path: artifacts/runtests-*.log


Expand All @@ -246,10 +259,14 @@ jobs:
max-parallel: 6
matrix:
include:
- {"salt-version": "3005.0", "python-version": "3.9", "source": "pypi"}
- {"salt-version": "3006.0", "python-version": "3.9", "source": "pypi"}
- {"salt-version": "3006.0", "python-version": "3.10", "source": "pypi"}
- {"salt-version": "3006.x", "python-version": "3.10", "source": "onedir"}
- {"salt-version": "3005.0", "python-version": "3.9", "testsuite": "core", "source": "pypi"}
- {"salt-version": "3005.0", "python-version": "3.9", "testsuite": "examples", "source": "pypi"}
- {"salt-version": "3006.0", "python-version": "3.9", "testsuite": "core", "source": "pypi"}
- {"salt-version": "3006.0", "python-version": "3.9", "testsuite": "examples", "source": "pypi"}
- {"salt-version": "3006.0", "python-version": "3.10", "testsuite": "core", "source": "pypi"}
- {"salt-version": "3006.0", "python-version": "3.10", "testsuite": "examples", "source": "pypi"}
- {"salt-version": "3006.x", "python-version": "3.10", "testsuite": "core", "source": "onedir"}
- {"salt-version": "3006.x", "python-version": "3.10", "testsuite": "examples", "source": "onedir"}

steps:
- uses: actions/checkout@v3
Expand Down Expand Up @@ -278,20 +295,20 @@ jobs:
env:
SALT_REQUIREMENT: salt~=${{ matrix.salt-version }}
run: |
nox --force-color -e tests-${{ matrix.source == 'onedir' && 'onedir' || '3' }} --install-only
nox --force-color -e tests${{ matrix.testsuite == 'examples' && '-examples' || '' }}-${{ matrix.source == 'onedir' && 'onedir' || '3' }} --install-only

- name: Test
env:
SALT_REQUIREMENT: salt~=${{ matrix.salt-version }}
SKIP_REQUIREMENTS_INSTALL: YES
run: |
nox --force-color -e tests-${{ matrix.source == 'onedir' && 'onedir' || '3' }} -- -vv tests/
nox --force-color -e tests${{ matrix.testsuite == 'examples' && '-examples' || '' }}-${{ matrix.source == 'onedir' && 'onedir' || '3' }} -- -vv

- name: Upload Logs
if: always()
uses: actions/upload-artifact@main
with:
name: runtests-${{ runner.os }}-${{ matrix.python-version }}-salt-${{ matrix.salt-version }}-${{ matrix.source }}.log
name: runtests-${{ runner.os }}-${{ matrix.python-version }}-salt-${{ matrix.salt-version }}-${{ matrix.source }}${{ matrix.testsuite == 'examples' && '-examples' || '' }}.log
path: artifacts/runtests-*.log

build:
Expand Down
9 changes: 8 additions & 1 deletion .pre-commit-config.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -69,6 +69,13 @@ repos:
- pyyaml==6.0
- jinja2==3.1.2
- packaging==23.0
- id: tools
alias: examples-requirements
name: Collect examples requirements
files: "^examples/requirements/"
args:
- pre-commit
- examples-requirements

# ----- Code Formatting and Analysis ---------------------------------------------------------->
- repo: https://github.com/charliermarsh/ruff-pre-commit
Expand Down Expand Up @@ -100,7 +107,7 @@ repos:
hooks:
- id: mypy
name: Run mypy against the code base
files: ^(src/|tests/).*\.py$
files: ^(src/|tests/|examples/).*\.py$
args: []
additional_dependencies:
- types-attrs
Expand Down
1 change: 1 addition & 0 deletions MANIFEST.in
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
recursive-exclude examples *
7 changes: 7 additions & 0 deletions README.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -86,6 +86,7 @@ For more detailed information, refer to `CONTRIBUTING`_.
.. _salt: https://github.com/saltstack/salt
.. _engine: https://docs.saltproject.io/en/latest/topics/engines/index.html
.. _CONTRIBUTING: https://github.com/saltstack/salt-analytics-framework/blob/main/CONTRIBUTING.md
.. _examples: https://github.com/saltstack/salt-analytics-framework/blob/main/examples

..
include-ends-here
Expand All @@ -94,3 +95,9 @@ Documentation
=============

The full documentation can be seen `here <https://salt-analytics-framework.readthedocs.io>`_.


Examples
========

Some examples of custom pipelines are provided. You can find them at `examples`_.
1 change: 1 addition & 0 deletions changelog/54.feature.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Add a Jupyter notebook processor that allows running parameterized notebooks using `papermill`
Loading