Failing tests

[domenkozar]

We should fix all the tests and accept zero-test policy now with devenv-nixpkgs

As of this commit we have failing tests:

• tests: it just fails without any output, needs investigation
• examples/terraform: due to How to use "permittedInsecurePackages"? #792
• examples/phoenix: no idea how to fix it /cc @sevillaarvin
• examples/rubyonrails: cc @bobvanderlinden

To run an example: PATH=$PWD/result/bin:$PATH ./result/bin/devenv-run-tests --only rubyonrails examples

[totoroot]

For both examples/phoenix and examples/rubyonrails, the failing PostgreSQL connection seems to be the culprit.

I'm working on fixing those in #1094.

[totoroot]

I'm not sure why, but when running PATH=$PWD/result/bin:$PATH ./result/bin/devenv-run-tests --only rubyonrails examples for a second time, tests are passing just fine 🤔

The first time, tests are failing and I'm getting a bunch of error logs like this:

[postgres	] initdb: error: could not open file "<...>/devenv/examples/rubyonrails/.devenv.nRzxd7tCwYbf/state/postgres/base/5/3118": No such file or directory

@bobvanderlinden fixed the test for rubyonrails in #1089.

#1094 should fix the test for phoenix.

[totoroot]

Hmm, tests for rubyonrails and phoenix are still failing, even after 'fixes' got merged. The database connection can't be established for both.

[bobvanderlinden]

Hmm, tests for rubyonrails and phoenix are still failing, even after 'fixes' got merged. The database connection can't be established for both.

At least for rubyonrails: the test isn't deterministic. It depends on something from the system that isn't in devenv/envvar. Maybe a leftover process that isn't killed after running the test?

It seems that once it succeeds, it'll keep succeeding. Need to dig into this again (third time 😅) and maybe make sure it succeeds after a reboot.

[domenkozar]

What I noticed with rubyonrails, if I delete blog then when running devenv test there won't be any logs from the processes at all.

I think what's happening is that when processes start, they invoke enterShell and we're also invoking it to run the tests.

Something makes the process script crash, resulting in no PostgreSQL running.

[domenkozar]

❯ devenv up
warning: creating lock file '/examples/rubyonrails/devenv.lock'
• Building processes ...
• Using Cachix: devenv
warning: creating lock file '/examples/rubyonrails/devenv.lock'
✔ Building processes in 1.3s.
• Starting processes ...
• Building shell ...
warning: creating lock file '/examples/rubyonrails/devenv.lock'
✔ Building shell in 1.0s.
/home/domen/dev/cachix/devenv/examples/rubyonrails/blog /home/domen/dev/cachix/devenv/examples/rubyonrails
Bundle complete! 14 Gemfile dependencies, 81 gems now installed.
Bundled gems are installed into `/home/domen/dev/cachix/devenv/examples/rubyonrails/.devenv/state/.bundle`
/home/domen/dev/cachix/devenv/examples/rubyonrails
24-04-08 09:25:25.562 FTL Unable to create configuration directory for process compose error="could not create any of the following paths: /.config, /etc/xdg"

[domenkozar]

What I noticed with rubyonrails, if I delete blog then when running devenv test there won't be any logs from the processes at all.

I think what's happening is that when processes start, they invoke enterShell and we're also invoking it to run the tests.

Something makes the process script crash, resulting in no PostgreSQL running.

Yeah that's it, if I run two concurrent devenv shell it crashes.

[domenkozar]

Here's the long term plan, devenv test should use unix sockets and then we can query if processes started up.

Short term, we should just periodically check if $PID is there.

[sevillaarvin]

I see. So it's launching multiple devenv shells huh. That is a bit tricky.

• .test.sh depends on devenv up
• devenv up depends on enterShell
• enterShell executed by multiple devenv shells (not yet sure why)

One consequence of this is that while .test.sh is already running, a devenv shell's enterShell is still executing code, i.e., generating a new project, which leads to timeout when the project is not ready yet after the default 15 seconds timeout.

I found that increasing the timeout to wait_for_port 4000 60 can work around it somewhat.