Skip to content

Use bashunit for e2e tests #3160

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 28 commits into from
Jun 19, 2024
Merged

Use bashunit for e2e tests #3160

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
28 commits
Select commit Hold shift + click to select a range
75a8fca
Use bashunit for e2e tests
staabm Jun 15, 2024
2c588d7
fix
staabm Jun 15, 2024
1328bc9
Update test.sh
staabm Jun 15, 2024
80e5419
Create test.sh
staabm Jun 15, 2024
995d940
Update test.sh
staabm Jun 15, 2024
33fb5b6
Update test.sh
staabm Jun 15, 2024
e516557
Update test.sh
staabm Jun 15, 2024
da1d6d1
Update test.sh
staabm Jun 15, 2024
08e23b8
Update test.sh
staabm Jun 15, 2024
6731637
Update test.sh
staabm Jun 15, 2024
e6f8b03
Update test.sh
staabm Jun 15, 2024
f4f5a3c
Update e2e-tests.yml
staabm Jun 15, 2024
357137b
install bashunit in CI
staabm Jun 15, 2024
1e78af9
Discard changes to .github/workflows/e2e-tests.yml
staabm Jun 16, 2024
06b377f
added assert.sh wrapper script
staabm Jun 16, 2024
ec5d625
cleanup
staabm Jun 16, 2024
eb0b893
Update e2e-tests.yml
staabm Jun 16, 2024
4b69f45
Update assert.sh
staabm Jun 16, 2024
4cb78ea
Update e2e-tests.yml
staabm Jun 16, 2024
67d6f64
Update assert.sh
staabm Jun 16, 2024
1d27a21
fix
staabm Jun 16, 2024
0c740f7
debug
staabm Jun 16, 2024
e175098
fix
staabm Jun 16, 2024
fe5ccbe
fix
staabm Jun 16, 2024
7d37b27
fix
staabm Jun 16, 2024
51fe9c5
use equals assertion
staabm Jun 16, 2024
17479ba
try bashunit beta
staabm Jun 17, 2024
93ef01a
fix typo
staabm Jun 17, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 10 additions & 7 deletions .github/workflows/e2e-tests.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -194,26 +194,26 @@ jobs:
patch -b data/TraitOne.php < TraitOne.patch
OUTPUT=$(../../bin/phpstan analyze --no-progress --level 8 --error-format raw data/ || true)
echo "$OUTPUT"
[ $(echo "$OUTPUT" | wc -l) -eq 1 ]
grep 'Method TraitsCachingIssue\\TestClassUsingTrait::doBar() should return stdClass but returns Exception.' <<< "$OUTPUT"
../bashunit -a line_count 1 "$OUTPUT"
../bashunit -a contains 'Method TraitsCachingIssue\TestClassUsingTrait::doBar() should return stdClass but returns Exception.' "$OUTPUT"
- script: |
cd e2e/trait-caching
../../bin/phpstan analyze --no-progress --level 8 --error-format raw data/
patch -b data/TraitTwo.php < TraitTwo.patch
OUTPUT=$(../../bin/phpstan analyze --no-progress --level 8 --error-format raw data/ || true)
echo "$OUTPUT"
[ $(echo "$OUTPUT" | wc -l) -eq 1 ]
grep 'Method class@anonymous/TestClassUsingTrait.php:20::doBar() should return stdClass but returns Exception.' <<< "$OUTPUT"
../bashunit -a line_count 1 "$OUTPUT"
../bashunit -a contains 'Method class@anonymous/TestClassUsingTrait.php:20::doBar() should return stdClass but returns Exception.' "$OUTPUT"
- script: |
cd e2e/trait-caching
../../bin/phpstan analyze --no-progress --level 8 --error-format raw data/
patch -b data/TraitOne.php < TraitOne.patch
patch -b data/TraitTwo.php < TraitTwo.patch
OUTPUT=$(../../bin/phpstan analyze --no-progress --level 8 --error-format raw data/ || true)
echo "$OUTPUT"
[ $(echo "$OUTPUT" | wc -l) -eq 2 ]
grep 'Method TraitsCachingIssue\\TestClassUsingTrait::doBar() should return stdClass but returns Exception.' <<< "$OUTPUT"
grep 'Method class@anonymous/TestClassUsingTrait.php:20::doBar() should return stdClass but returns Exception.' <<< "$OUTPUT"
../bashunit -a line_count 2 "$OUTPUT"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would like to reproduce it locally, so I cloned your fork (git@github.com:Chemaclass/staabm-phpstan-src.git), checkout your bashunit branch, and I get this error when running composer install - which I think it shouldn't happen as it is a virgin start (I also set php 8.1), so I guess there is a problem on the deps? Did you face the same issue, @staabm ? Any idea how to find a work around to this? 340425921-5b9fabbb-1df8-4442-bddc-29454deabaae

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See the README.

If you encounter dependency problem, try using export COMPOSER_ROOT_VERSION=1.11.x-dev

Copy link
Contributor Author

@staabm staabm Jun 17, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think I can reproduce on the CLI outside of phpstan repo with:

OUTPUT="one\ntwo\nthree"
➜  echo $OUTPUT
one
two
three

➜ ./bashunit -a line_count 3 "$OUTPUT"

✗ Failed: Main::exec assert
    Expected 'one\ntwo\nthree'
    to contain number of lines equal to '3'

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, in such a case the bug seems to be in the assert_line_count where it might consider that as 1 line and not as 3. I think that's the direction of the 🐛

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hmm .. the unit test regarding it looked good:
https://github.com/TypedDevs/bashunit/pull/259/files#diff-8c5ca8e706f9d9fb330cda7953d707bb4180b68f55fdff87ec30ad3f8e179ad6R339-R341

I don't know bash good enough, but I had a feeling the problem is related to the assertion via args passing

Copy link
Contributor

@Chemaclass Chemaclass Jun 18, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can reproduce it, yes. This is a bashunit issue, I will work on it today 🏗️
UPDATE: It took me a couple of hours... but finally I got something out: TypedDevs/bashunit#265

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Alright, TypedDevs/bashunit#265 is merged and we are getting closer to the light!
Can you try again to reproduce it with the latest beta, @staabm ?

Copy link
Contributor

@Chemaclass Chemaclass Jun 18, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With the latest error, I think we might have found a major problem wrt multiline strings mmmm I have an idea, but it might take a while... let me see if I can create a draft for this today 🏗️

UPDATE: This might be the fix for the multiline string bug: TypedDevs/bashunit#266

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same drill: TypedDevs/bashunit#266 is merged, can you try the latest beta again, @staabm? 🤞🏼

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Chemaclass all e2e are green now. thank you very much for your efforts <3.

../bashunit -a contains 'Method TraitsCachingIssue\TestClassUsingTrait::doBar() should return stdClass but returns Exception.' "$OUTPUT"
../bashunit -a contains 'Method class@anonymous/TestClassUsingTrait.php:20::doBar() should return stdClass but returns Exception.' "$OUTPUT"

steps:
- name: "Checkout"
Expand All @@ -233,6 +233,9 @@ jobs:
- name: "Patch PHPStan"
run: "patch src/Analyser/Error.php < e2e/PHPStanErrorPatch.patch"

- name: "Install bashunit"
run: "curl -s https://bashunit.typeddevs.com/install.sh | bash -s e2e/ beta"

- name: "Test"
run: "${{ matrix.script }}"

Expand Down
1 change: 1 addition & 0 deletions .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,3 +9,4 @@
/tests/.phpunit.result.cache
/tests/PHPStan/Reflection/data/golden/
tmp/.memory_limit
e2e/bashunit
Loading