Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use bashunit for e2e tests #3160

Merged
merged 28 commits into from
Jun 19, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 10 additions & 7 deletions .github/workflows/e2e-tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -194,26 +194,26 @@ jobs:
patch -b data/TraitOne.php < TraitOne.patch
OUTPUT=$(../../bin/phpstan analyze --no-progress --level 8 --error-format raw data/ || true)
echo "$OUTPUT"
[ $(echo "$OUTPUT" | wc -l) -eq 1 ]
grep 'Method TraitsCachingIssue\\TestClassUsingTrait::doBar() should return stdClass but returns Exception.' <<< "$OUTPUT"
../bashunit -a line_count 1 "$OUTPUT"
../bashunit -a contains 'Method TraitsCachingIssue\TestClassUsingTrait::doBar() should return stdClass but returns Exception.' "$OUTPUT"
- script: |
cd e2e/trait-caching
../../bin/phpstan analyze --no-progress --level 8 --error-format raw data/
patch -b data/TraitTwo.php < TraitTwo.patch
OUTPUT=$(../../bin/phpstan analyze --no-progress --level 8 --error-format raw data/ || true)
echo "$OUTPUT"
[ $(echo "$OUTPUT" | wc -l) -eq 1 ]
grep 'Method class@anonymous/TestClassUsingTrait.php:20::doBar() should return stdClass but returns Exception.' <<< "$OUTPUT"
../bashunit -a line_count 1 "$OUTPUT"
../bashunit -a contains 'Method class@anonymous/TestClassUsingTrait.php:20::doBar() should return stdClass but returns Exception.' "$OUTPUT"
- script: |
cd e2e/trait-caching
../../bin/phpstan analyze --no-progress --level 8 --error-format raw data/
patch -b data/TraitOne.php < TraitOne.patch
patch -b data/TraitTwo.php < TraitTwo.patch
OUTPUT=$(../../bin/phpstan analyze --no-progress --level 8 --error-format raw data/ || true)
echo "$OUTPUT"
[ $(echo "$OUTPUT" | wc -l) -eq 2 ]
grep 'Method TraitsCachingIssue\\TestClassUsingTrait::doBar() should return stdClass but returns Exception.' <<< "$OUTPUT"
grep 'Method class@anonymous/TestClassUsingTrait.php:20::doBar() should return stdClass but returns Exception.' <<< "$OUTPUT"
../bashunit -a line_count 2 "$OUTPUT"
staabm marked this conversation as resolved.
Show resolved Hide resolved
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would like to reproduce it locally, so I cloned your fork (git@github.com:Chemaclass/staabm-phpstan-src.git), checkout your bashunit branch, and I get this error when running composer install - which I think it shouldn't happen as it is a virgin start (I also set php 8.1), so I guess there is a problem on the deps? Did you face the same issue, @staabm ? Any idea how to find a work around to this? 340425921-5b9fabbb-1df8-4442-bddc-29454deabaae

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See the README.

If you encounter dependency problem, try using export COMPOSER_ROOT_VERSION=1.11.x-dev

Copy link
Contributor Author

@staabm staabm Jun 17, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think I can reproduce on the CLI outside of phpstan repo with:

OUTPUT="one\ntwo\nthree"
➜  echo $OUTPUT
one
two
three

➜ ./bashunit -a line_count 3 "$OUTPUT"

✗ Failed: Main::exec assert
    Expected 'one\ntwo\nthree'
    to contain number of lines equal to '3'

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, in such a case the bug seems to be in the assert_line_count where it might consider that as 1 line and not as 3. I think that's the direction of the 🐛

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hmm .. the unit test regarding it looked good:
https://github.com/TypedDevs/bashunit/pull/259/files#diff-8c5ca8e706f9d9fb330cda7953d707bb4180b68f55fdff87ec30ad3f8e179ad6R339-R341

I don't know bash good enough, but I had a feeling the problem is related to the assertion via args passing

Copy link
Contributor

@Chemaclass Chemaclass Jun 18, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can reproduce it, yes. This is a bashunit issue, I will work on it today 🏗️
UPDATE: It took me a couple of hours... but finally I got something out: TypedDevs/bashunit#265

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Alright, TypedDevs/bashunit#265 is merged and we are getting closer to the light!
Can you try again to reproduce it with the latest beta, @staabm ?

Copy link
Contributor

@Chemaclass Chemaclass Jun 18, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With the latest error, I think we might have found a major problem wrt multiline strings mmmm I have an idea, but it might take a while... let me see if I can create a draft for this today 🏗️

UPDATE: This might be the fix for the multiline string bug: TypedDevs/bashunit#266

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same drill: TypedDevs/bashunit#266 is merged, can you try the latest beta again, @staabm? 🤞🏼

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Chemaclass all e2e are green now. thank you very much for your efforts <3.

../bashunit -a contains 'Method TraitsCachingIssue\TestClassUsingTrait::doBar() should return stdClass but returns Exception.' "$OUTPUT"
../bashunit -a contains 'Method class@anonymous/TestClassUsingTrait.php:20::doBar() should return stdClass but returns Exception.' "$OUTPUT"

steps:
- name: "Checkout"
Expand All @@ -233,6 +233,9 @@ jobs:
- name: "Patch PHPStan"
run: "patch src/Analyser/Error.php < e2e/PHPStanErrorPatch.patch"

- name: "Install bashunit"
run: "curl -s https://bashunit.typeddevs.com/install.sh | bash -s e2e/ beta"

- name: "Test"
run: "${{ matrix.script }}"

Expand Down
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -9,3 +9,4 @@
/tests/.phpunit.result.cache
/tests/PHPStan/Reflection/data/golden/
tmp/.memory_limit
e2e/bashunit
Loading