sockets still blocked when --disable-socket is not used

[orionlee]

I tried to integrate pytest-socket in my project.
It works mostly fine for the cases that I want sockets disabled by passing --disable-socket in the option.

However, when I want to run the entire test suite normally with network, i.e., without passing --disable-socket, somehow the plugin incorrectly blocks sockets, raising error:

>       raise SocketBlockedError()
E       pytest_socket.SocketBlockedError: A test tried to use socket.socket.

An example from CI results:
https://github.com/orionlee/lightkurve/runs/2612809173?check_suite_focus=true

The project in question uses a variety of popular scientific computing and astronomy packages.

Unfortunately, I cannot yet reproduce it in an isolated setup. I'm still submitting this issue to see if you have any inklings on what's going on.

What I've tried so far:

• Running individual test functions (rather than test suite/file) works.
• Running a singlet test file, e.g., test_interact.py, produces the socket blocked error.
• I also tried to disable some other pytest plugins, specifically pytest-remotedata and https://github.com/astropy/pytest-doctestplus. It has no impact either.

Environments:

• pytest-socket version: 0.4.0
• OS: both Windows on Linux
• python: 3.7, 3.8
• pytest: 6.2.3
• pytest plugins used:
  plugins: cov-2.10.1, doctestplus-0.8.0, remotedata-0.3.2, forked-1.3.0, socket-0.4.0, xdist-2.2.1

[miketheman]

Hi @orionlee !

Thanks for the detailed report - I'm not certain pytest-socket will play nice with pytest-remotedata - as we both want to control the behavior of socket.*. I haven't heard of that plugin until now, so it's entirely possible that there's conflict between the behavior of --disable-socket and whatever remote data is doing.

You mentioned that you tried disabling that plugin, can you share how? From the CI results it looks like this is the commit that's being run - and that's passing --disable-socket globally - unless I'm misreading something?

[orionlee]

1. My understanding is that pytest-remotedata does not control sockets. It really is a test filter trick so that for tests marked with @pytest.mark.remote_data are skipped by default, unless --remote-data is passed as an argument to pytest.
   Typically you'd mark a test requiring network access with @pytest.mark.remote_data so they are skipped by default. But it doesn't really care if the tests require network or not.
   E.g., technically one can use it to distinguish short and long tests. If one marks all the long-running tests with remote_data, pytest would by default run the short regression test.

2. Regarding the CI results: while --disable-socket is passed to many runs, the specific run, pytest-linux (3.8-remote-data), does not have --disable-socket. It ran with the following command:

poetry run pytest --remote-data --doctest-modules

The run is intended to be a normal full test that uses networks. But somehow many tests failed with pytest_socket.SocketBlockedError. So pytest_socket still intercepted the tests, despite --disable-socket was not passed in the command line. As an example, see line 109 of the run's log.

Regarding disable pytest-remotedata: I did it with a brute force way (to be sure it is not there). In my local environment, I temporarily uninstalled the plugin from my conda environment, and did not pass --remote-data in invoking pytest.

[miketheman]

Hey @orionlee !

I've changed some of the core internals of the plugin - most importantly that we no longer use an autouse fixture, rather leverage the setup step of pytest to configure each test.

This ought to be released in 0.50.0 soon, and I hope you try it out!

Closing this issue, feel free to reopen if this doesn't work out!