Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

OCPP E2E tests #959

Merged
merged 56 commits into from
Nov 26, 2024
Merged

OCPP E2E tests #959

merged 56 commits into from
Nov 26, 2024

Conversation

Pietfried
Copy link
Contributor

@Pietfried Pietfried commented Nov 12, 2024 •
edited
Loading

Describe your changes

This PR adds OCPP1.6 and OCPP2.0.1 E2E tests to everest-core.

TODOs:

  • Include tests in CI

Issue ticket number and link

This issue summarizes todos to clean up the ocpp tests with subsequent PRs

Checklist before requesting a review

  • I have performed a self-review of my code
  • I have made corresponding changes to the documentation
  • I read the contribution documentation and made sure that my changes meet its requirements

Pietfried and others added 18 commits November 8, 2024 07:33
@Pietfried
Initially adding ocpp integration tests into seperate testing directo…
440c250 
…ry ocpp_tests/

Signed-off-by: Piet Gömpel <pietgoempel@gmail.com>
@hikinggrass
Install everest-testing with Python3_EXECUTABLE to respect venv
c672c2d 
Also install it in editable mode

Signed-off-by: Kai-Uwe Hermann <kai-uwe.hermann@pionix.de>
@hikinggrass
Use a proper temporary directory in signed firmware downloader for si…
ebeeff2 
…gnature validation

This also fixes OCPP 1.6 ocpp_compliance_tests.py:test_signed_update_firmware

Signed-off-by: Kai-Uwe Hermann <kai-uwe.hermann@pionix.de>
@hikinggrass
Merge remote-tracking branch 'origin/main' into feature/ocpp-integrat…
c4d48a5 
…ion-tests

Signed-off-by: Kai-Uwe Hermann <kai-uwe.hermann@pionix.de>
@hikinggrass
Update everest-utils rev to use update everest-testing
be0da67 
Signed-off-by: Kai-Uwe Hermann <kai-uwe.hermann@pionix.de>
@hikinggrass
WIP OCPP Tests in ci
19a39f7 
Signed-off-by: Kai-Uwe Hermann <kai-uwe.hermann@pionix.de>
@hikinggrass
Depend on ci job
5d81f88 
Signed-off-by: Kai-Uwe Hermann <kai-uwe.hermann@pionix.de>
@hikinggrass
Add missing run_ocpp_tests script
96b770e 
Signed-off-by: Kai-Uwe Hermann <kai-uwe.hermann@pionix.de>
@hikinggrass
Move ocpp-tests job to its own workflow depending on the build workflow
c251c54 
Signed-off-by: Kai-Uwe Hermann <kai-uwe.hermann@pionix.de>
@hikinggrass
Get buildkit tag from earlier step
657b0a7 
Signed-off-by: Kai-Uwe Hermann <kai-uwe.hermann@pionix.de>
@hikinggrass
Move ocpp-tests job back to build_and_test workflow temporarily
6342aa6 
Signed-off-by: Kai-Uwe Hermann <kai-uwe.hermann@pionix.de>
@hikinggrass
ocpp-tests needs ci job
7be00a8 
Signed-off-by: Kai-Uwe Hermann <kai-uwe.hermann@pionix.de>
@hikinggrass
Fix docker image tag input
22c716f 
Signed-off-by: Kai-Uwe Hermann <kai-uwe.hermann@pionix.de>
@hikinggrass
Properly get tag from output
dd409e7 
Signed-off-by: Kai-Uwe Hermann <kai-uwe.hermann@pionix.de>
@hikinggrass
Prefix tag
31db428 
Signed-off-by: Kai-Uwe Hermann <kai-uwe.hermann@pionix.de>
@hikinggrass
Copy scripts
9841b66 
Signed-off-by: Kai-Uwe Hermann <kai-uwe.hermann@pionix.de>
@hikinggrass
Always download wheels
b1a1580 
Signed-off-by: Kai-Uwe Hermann <kai-uwe.hermann@pionix.de>
@Pietfried
Added README for ocpp tests and changed directory to be specified to …
2f993d5 
…be the everest install directory

Signed-off-by: Piet Gömpel <pietgoempel@gmail.com>
@Pietfried Pietfried mentioned this pull request Nov 12, 2024
Open
6 tasks
Pietfried and others added 11 commits November 12, 2024 12:05
@Pietfried
Merge branch 'main' into feature/ocpp-integration-tests
025857d
@hikinggrass
Install requirements for ocpp integration tests
a7a45f7 
Signed-off-by: Kai-Uwe Hermann <kai-uwe.hermann@pionix.de>
@hikinggrass
Fix runs-on
dd8f38e 
Signed-off-by: Kai-Uwe Hermann <kai-uwe.hermann@pionix.de>
@Pietfried
Fix unused imports
2a82c2f 
Signed-off-by: Piet Gömpel <pietgoempel@gmail.com>
@Pietfried
Ran black formatter
0301363 
Signed-off-by: Piet Gömpel <pietgoempel@gmail.com>
@hikinggrass
Add create_ocpp_tests_image script
bf9b6cc 
Signed-off-by: Kai-Uwe Hermann <kai-uwe.hermann@pionix.de>
@hikinggrass
Install iso15118 wheel
3eefd2b 
Signed-off-by: Kai-Uwe Hermann <kai-uwe.hermann@pionix.de>
@Pietfried
Fixed some codacy remarks
251ce15 
Signed-off-by: Piet Gömpel <pietgoempel@gmail.com>
@hikinggrass
Fixed path to OCPP 1.6 and 2.0.1 testsets
8ef308b 
Signed-off-by: Kai-Uwe Hermann <kai-uwe.hermann@pionix.de>
@hikinggrass
Pin websockets to 13.1
547cab9 
Signed-off-by: Kai-Uwe Hermann <kai-uwe.hermann@pionix.de>
@Pietfried
Fix some codacy issues
307540e 
Signed-off-by: Piet Gömpel <pietgoempel@gmail.com>
@hikinggrass
codacy fixes
1f5ed0b 
Signed-off-by: Kai-Uwe Hermann <kai-uwe.hermann@pionix.de>
@hikinggrass
attempt at fixing codacy markdown issues
63fcdca 
Signed-off-by: Kai-Uwe Hermann <kai-uwe.hermann@pionix.de>
@hikinggrass
attempt to fix codacy issues
98ae4b4 
Signed-off-by: Kai-Uwe Hermann <kai-uwe.hermann@pionix.de>
@hikinggrass
codacy fix
be2190a 
Signed-off-by: Kai-Uwe Hermann <kai-uwe.hermann@pionix.de>
@Pietfried Pietfried marked this pull request as ready for review November 15, 2024 13:58
@maaikez
Copy link
Contributor

maaikez commented Nov 18, 2024

  • Please add to the readme to also install pytest-xdist, pytest-timeout and pytest-html.
  • Please add to the readme :
cd everest-core/
cmake -S . -B build -DBUILD_TESTING=ON
cmake --build build --target install --parallel 12
. build/venv/bin/activate
cmake --build build --target everestpy_pip_install_dist
cmake --build build --target everest-testing_pip_install_dist
cmake --build build --target iso15118_pip_install_dist
python3 -m pip install aiofile>=3.7.4
python3 -m pip install netifaces>=0.11.0
cd tests
python3 -m pytest -rA --everest-prefix ../build/dist/ core_tests/startup_tests.py

@maaikez
Copy link
Contributor

maaikez commented Nov 18, 2024

  • Please add to the readme to also install pytest-xdist, pytest-timeout and pytest-html.

    • Please add to the readme :
cd everest-core/
cmake -S . -B build -DBUILD_TESTING=ON
cmake --build build --target install --parallel 12
. build/venv/bin/activate
cmake --build build --target everestpy_pip_install_dist
cmake --build build --target everest-testing_pip_install_dist
cmake --build build --target iso15118_pip_install_dist
python3 -m pip install aiofile>=3.7.4
python3 -m pip install netifaces>=0.11.0
cd tests
python3 -m pytest -rA --everest-prefix ../build/dist/ core_tests/startup_tests.py

Piet pointed me to the requirements.txt in tests/ocpp, where the first point is already 'done' (those packages are in there)

@maaikez
Copy link
Contributor

maaikez commented Nov 18, 2024

I would like to have an sh script to run all tests and that it is not needed to go to different folders. Would that be an option?

maaikez
maaikez reviewed Nov 18, 2024
View reviewed changes
tests/ocpp_tests/run-testing.sh
echo "Running $PARALLEL_TESTS tests in parallel"

# run all tests in parallel
"$PYTHON_INTERPRETER" -m pytest -d --tx "$PARALLEL_TESTS"*popen//python="$PYTHON_INTERPRETER" -rA --junitxml=result.xml --html=report.html --self-contained-html --max-worker-restart=0 --timeout=300 --dist loadgroup test_sets/ocpp16/*.py test_sets/ocpp201/*.py --everest-prefix "$EVEREST_CORE_DIR/build/dist"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is nice to be able to run all tests in parallel. Is it possible to add this as an environment variable so you can choose how many processors are used? Instead of always using all of them for the tests.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you can already run e.g.

./run-testing.sh 4

to run with 4 workers.

maaikez
maaikez requested changes Nov 18, 2024
View reviewed changes
tests/ocpp_tests/README.md Outdated
If you run the test cases individually,
make sure to have all required certificates and configs
for the test cases installed using the
convenience scripts inside [test_sets/everest/aux](test_sets/everest-aux/)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is not clear to me where this everest-aux directory is. It is not in test_sets.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it is in tests/ocpp_tests/test_sets/everest-aux . The path was wrong ;)

@maaikez
Copy link
Contributor

maaikez commented Nov 18, 2024

Looks good, only one thing in the documentation and some other remarks, but maybe those are good for the clean-up issue

@Pietfried
Updated test README
f804a48 
Signed-off-by: Piet Gömpel <pietgoempel@gmail.com>
@Pietfried Pietfried requested a review from maaikez November 19, 2024 08:27
maaikez
maaikez reviewed Nov 19, 2024
View reviewed changes
tests/ocpp_tests/README.md
@@ -54,7 +62,7 @@ specified in `test_sets/ocpp201/remote_control.py`.
If you run the test cases individually,
make sure to have all required certificates and configs
for the test cases installed using the
convenience scripts inside [test_sets/everest/aux](test_sets/everest-aux/)
convenience scripts inside [test_sets/everest-aux](test_sets/everest-aux/)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Isn't the everest-aux directory in another repo (everest-framework or something)? I could not find it in the test_sets directory of everest-core.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Its here: https://github.com/EVerest/everest-core/tree/feature/ocpp-integration-tests/tests/ocpp_tests/test_sets/everest-aux

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it's even in the pull request :D. Oops.

Might have searched for everest_aux or something indeed.

hikinggrass
hikinggrass requested changes Nov 19, 2024
View reviewed changes
dependencies.yaml Outdated
@@ -86,7 +86,7 @@ ext-mbedtls:
# everest-testing and ev-dev-tools
everest-utils:
git: https://github.com/EVerest/everest-utils.git
git_tag: v0.4.2
git_tag: 9c7a4928251f47ffb7f8197f6cba8596fd15d242
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this needs to point to released version of everest-utils not a commit on a branch

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

EVerest/everest-utils#167

@Pietfried Pietfried mentioned this pull request Nov 19, 2024
Merged
4 tasks
@maaikez
Copy link
Contributor

maaikez commented Nov 20, 2024

One more addition to the readme: can you add there that you have to start mosquitto?

@maaikez
Copy link
Contributor

maaikez commented Nov 20, 2024

Is it possible to also add the california pricing and display message tests by the way?
https://github.com/PionixInternal/ocpp-testing/pull/56

@Pietfried
Copy link
Contributor Author

Is it possible to also add the california pricing and display message tests by the way? PionixInternal/ocpp-testing#56

Sure you can add them here. In case they cause issues we can also add them in a subsequent PR.

hikinggrass and others added 2 commits November 25, 2024 14:15
@hikinggrass
Bump everest-utils to 0.4.3
3121548 
Signed-off-by: Kai-Uwe Hermann <kai-uwe.hermann@pionix.de>
@Pietfried
Merge branch 'main' into feature/ocpp-integration-tests
42d0b54
@Pietfried Pietfried changed the title OCPP Integration tests OCPP E2E tests Nov 26, 2024
@Pietfried Pietfried merged commit 81b366d into main Nov 26, 2024
11 of 12 checks passed
@Pietfried Pietfried deleted the feature/ocpp-integration-tests branch November 26, 2024 08:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@hikinggrass hikinggrass hikinggrass approved these changes

@maaikez maaikez maaikez approved these changes

@corneliusclaussen corneliusclaussen corneliusclaussen approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@Pietfried @maaikez @hikinggrass @corneliusclaussen