Skip to content

Commit

Permalink
run-test-plan.py: Run in parallel where possible
Browse files Browse the repository at this point in the history 
Use python's asyncio / await / async to run modules within one test plan
in parallel if they don't have an alias, and to run plans in parallel
where they use no / different aliases.

The parallelism of tests within a plan should speed up the CIBA tests
and OIDC test modules where we use dcr can hence can proceed without
any alias. Unfortunately the CIBA tests had to have parallelism
disabled because Authlete's CIBA simulated authentication device
doesn't seem to cope with multiple parallel.

Reduces the time to run oidcc-basic-certification-test-plan from 313
seconds to 63 seconds on my local machine - most of which comes from
the test that sleeps for 30 seconds before reusing an auth code.

The ability to run plans with different aliases in parallel means we
can run some of the FAPI tests in parallel with other tests. There may
be further potential to speed this up by tweaking the aliases and
redirect urls we use in (say) half of the FAPI tests.

There was some hassle with python modules/node, I tried switching away
from alpine for the reasons given here, where we were seeing issues
installing the aiohttp module:

https://pythonspeed.com/articles/alpine-docker-python/

but ending up switching back because the version of node (10) in Debian
buster is ancient and doesn't support some of the crypto Fillip's
client needs. Luckily the aiohttp module is one of the ones that can
relatively easily be made to work on alpine.

We add a retrying http client, as it seems either the parallelisation
or something about the asyncio http client ends up giving weird http
errors - I think we're running into the same bug as here:

aio-libs/aiohttp#4581

Switching to https://www.python-httpx.org might be an option to avoid
that, but that's still in beta.

part of #783

# Conflicts:
#	.gitlab-ci.yml
#	.gitlab-ci/run-tests.sh
#	scripts/run-test-plan.py
#	test/Dockerfile
  • Loading branch information
@jogu
jogu committed Nov 21, 2021
1 parent abe1fe2 commit 120e393
Show file tree
Hide file tree
Showing 6 changed files with 334 additions and 168 deletions.
5 changes: 3 additions & 2 deletions .gitlab-ci.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -522,7 +522,7 @@ local_test:

function install_test_dependencies() {
echo "Installing extra dependencies"
apk add -U openssh-client git nodejs npm
apk add openssh-client git gcc musl-dev bash
eval $(ssh-agent -s)
echo "$SSH_PRIVATE_KEY" | ssh-add -
cd ..
Expand All @@ -532,10 +532,11 @@ local_test:

function set_up_for_running_test_plan() {
install_test_dependencies
pip install requests
pip install aiohttp aiofiles aiohttp-retry
}

function run_client_test_plan() {
apk add nodejs npm
echo "Running automated tests against $CONFORMANCE_SERVER"
../conformance-suite/.gitlab-ci/run-tests.sh --client-tests-only
}
Expand Down
27 changes: 13 additions & 14 deletions .gitlab-ci/local-provider-oidcc-conformance-config.json
View file
Original file line number Diff line number Diff line change
@@ -1,5 +1,4 @@
{
"alias": "local-oidcc",
"description": "oidc-provider OIDC",
"server": {
"discoveryUrl": "https://oidcc-provider:3000/.well-known/openid-configuration"
Expand Down Expand Up @@ -54,7 +53,7 @@
},
{
"task": "Verify Complete",
"match": "*/test/a/local-oidcc/callback*",
"match": "*/test/*/callback*",
"commands": [
[
"wait",
Expand Down Expand Up @@ -88,14 +87,14 @@
[
"wait",
"contains",
"/test/a/local-oidcc/post",
"/test/*/post",
10
]
]
},
{
"task": "Verify Complete",
"match": "*/test/a/local-oidcc/post*"
"match": "*/test/*/post*"
}
]
}
Expand Down Expand Up @@ -155,7 +154,7 @@
},
{
"task": "Verify Complete",
"match": "*/test/a/local-oidcc/callback*",
"match": "*/test/*/callback*",
"commands": [
[
"wait",
Expand Down Expand Up @@ -223,7 +222,7 @@
},
{
"task": "Verify Complete",
"match": "*/test/a/local-oidcc/callback*",
"match": "*/test/*/callback*",
"commands": [
[
"wait",
Expand Down Expand Up @@ -450,7 +449,7 @@
},
{
"task": "Verify Complete",
"match": "*/test/a/local-oidcc/callback*",
"match": "*/test/*/callback*",
"commands": [
[
"wait",
Expand Down Expand Up @@ -529,7 +528,7 @@
},
{
"task": "Verify Complete",
"match": "*/test/a/local-oidcc/callback*",
"match": "*/test/*/callback*",
"commands": [
[
"wait",
Expand Down Expand Up @@ -608,7 +607,7 @@
},
{
"task": "Verify Complete",
"match": "*/test/a/local-oidcc/callback*",
"match": "*/test/*/callback*",
"commands": [
[
"wait",
Expand Down Expand Up @@ -687,7 +686,7 @@
},
{
"task": "Verify Complete",
"match": "*/test/a/local-oidcc/callback*",
"match": "*/test/*/callback*",
"commands": [
[
"wait",
Expand Down Expand Up @@ -766,7 +765,7 @@
},
{
"task": "Verify Complete",
"match": "*/test/a/local-oidcc/callback*",
"match": "*/test/*/callback*",
"commands": [
[
"wait",
Expand Down Expand Up @@ -845,7 +844,7 @@
},
{
"task": "Verify Complete",
"match": "*/test/a/local-oidcc/callback*",
"match": "*/test/*/callback*",
"commands": [
[
"wait",
Expand Down Expand Up @@ -935,7 +934,7 @@
},
{
"task": "Verify Complete",
"match": "*/test/a/local-oidcc/callback*",
"match": "*/test/*/callback*",
"commands": [
[
"wait",
Expand Down Expand Up @@ -1025,7 +1024,7 @@
},
{
"task": "Verify Complete",
"match": "*/test/a/local-oidcc/callback*",
"match": "*/test/*/callback*",
"commands": [
[
"wait",
Expand Down
8 changes: 5 additions & 3 deletions .gitlab-ci/run-tests.sh
View file
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
#!/bin/sh
#!/bin/bash

set -e
cleanup() {
Expand Down Expand Up @@ -36,8 +36,8 @@ EXPECTED_FAILURES_FILE="../conformance-suite/.gitlab-ci/expected-failures-server
EXPECTED_SKIPS_FILE="../conformance-suite/.gitlab-ci/expected-skips-server.json|../conformance-suite/.gitlab-ci/expected-skips-ciba.json|../conformance-suite/.gitlab-ci/expected-skips-client.json"

makeClientTest() {
. node-client-setup.sh
. node-core-client-setup.sh
. ./node-client-setup.sh
. ./node-core-client-setup.sh

#BRAZIL (note brazil_client_scope is not a variant but is used to pass scopes to tests. brazil_client_scope values use - instead of space)
#TESTS="${TESTS} fapi1-advanced-final-client-test-plan[client_auth_type=private_key_jwt][fapi_profile=openbanking_brazil][fapi_auth_request_method=by_value][fapi_response_mode=plain_response][fapi_jarm_type=oidc][brazil_client_scope=openid-payments] automated-brazil-client-test.json"
Expand Down Expand Up @@ -336,6 +336,7 @@ elif [ "$#" -eq 1 ] && [ "$1" = "--client-tests-only" ]; then
TESTS="${TESTS} --expected-skips-file ${EXPECTED_SKIPS_FILE}"
TESTS="${TESTS} --show-untested-test-modules client"
TESTS="${TESTS} --export-dir ../conformance-suite"
TESTS="${TESTS} --no-parallel" # there seemed to be a lot of "Server disconnected" failures trying to run these in parallel
elif [ "$#" -eq 1 ] && [ "$1" = "--server-tests-only" ]; then
echo "Run server tests"
makeServerTest
Expand All @@ -355,6 +356,7 @@ elif [ "$#" -eq 1 ] && [ "$1" = "--ciba-tests-only" ]; then
TESTS="${TESTS} --expected-skips-file ${EXPECTED_SKIPS_FILE}"
TESTS="${TESTS} --show-untested-test-modules ciba"
TESTS="${TESTS} --export-dir ../conformance-suite"
TESTS="${TESTS} --no-parallel" # the authlete authentication device simulator doesn't seem to support parallel authorizations
elif [ "$#" -eq 1 ] && [ "$1" = "--local-provider-tests" ]; then
echo "Run local provider tests"
makeLocalProviderTests
Expand Down
Loading

0 comments on commit 120e393

Please sign in to comment.