testsuite require network access by default

[jonassmedegaard]

The test file tests/test_doctests.py fails in a build environment without network access.

Some build environments - including Debian autobuilders - explicitly disable network to make builds more deterministic.

Please rely on network access only optionally.
One elegant way to do that is to borrow the logic used in Perl: Check environment variable EXTENDED_TESTING and if not set to a non-empty value then skip all such "extended testing".

Thanks for considering.

[mikeboers]

I'm pretty sure a few more will also fail, because we pull from the FFmpeg FATE suite for samples in a few places.

[jonassmedegaard]

Quoting Mike Boers (2020-04-16 15:23:16)

I'm pretty sure a few more will also fail, because we pull from the FFmpeg FATE suite for samples in a few places.

Right, that file was just the tip of the iceberg. These files require network during testsuite initialization: tests/test_doctests.py tests/test_timeout.py These tests seemingly require network when checked: TestAudioFifo.test_data \ TestCodecContext.test_parse \ TestEncoding.test_encoding_aac \ TestEncoding.test_encoding_dnxhd \ TestEncoding.test_encoding_dvvideo \ TestEncoding.test_encoding_h264 \ TestEncoding.test_encoding_mjpeg \ TestEncoding.test_encoding_mp2 \ TestEncoding.test_encoding_mpeg1video \ TestEncoding.test_encoding_pcm_s24le \ TestEncoding.test_encoding_png \ TestEncoding.test_encoding_tiff \ TestContainers.test_context_manager \ TestDecode.test_decode_audio_sample_count \ TestDecode.test_decoded_motion_vectors \ TestDecode.test_decoded_motion_vectors_no_flag \ TestDecode.test_decoded_time_base \ TestDecode.test_decoded_video_frame_count \ TestBasicAudioEncoding.test_audio_transcode \ TestAudioProbe.test_container_probing \ TestAudioProbe.test_stream_probing \ TestDataProbe.test_container_probing \ TestDataProbe.test_stream_probing \ TestSubtitleProbe.test_container_probing \ TestSubtitleProbe.test_stream_probing \ TestAudioFifo.test_data \ TestCodecContext.test_parse \ TestEncoding.test_encoding_aac \ TestEncoding.test_encoding_dnxhd \ TestEncoding.test_encoding_dvvideo \ TestEncoding.test_encoding_h264 \ TestEncoding.test_encoding_mjpeg \ TestEncoding.test_encoding_mp2 \ TestEncoding.test_encoding_mpeg1video \ TestEncoding.test_encoding_mpeg4 \ TestEncoding.test_encoding_pcm_s24le \ TestEncoding.test_encoding_png \ TestEncoding.test_encoding_tiff \ TestContainers.test_context_manager \ TestDecode.test_decode_audio_sample_count \ TestDecode.test_decoded_motion_vectors \ TestDecode.test_decoded_motion_vectors_no_flag \ TestDecode.test_decoded_time_base \ TestDecode.test_decoded_video_frame_count \ TestBasicAudioEncoding.test_audio_transcode \ TestAudioProbe.test_container_probing \ TestAudioProbe.test_stream_probing \ TestDataProbe.test_container_probing \ TestDataProbe.test_stream_probing \ TestSubtitleProbe.test_container_probing \ TestSubtitleProbe.test_stream_probing \ TestVideoProbe.test_container_probing \ TestVideoProbe.test_stream_probing \ TestPythonIO.test_reading \ TestPythonIO.test_reading_no_seek \ TestSeek.test_decode_half \ TestSeek.test_deprecated_stream_seek \ TestSeek.test_seek_end \ TestSeek.test_seek_float \ TestSeek.test_seek_int64 \ TestSeek.test_seek_middle \ TestSeek.test_seek_start \ TestSeek.test_stream_seek \ TestStreams.test_selection \ TestStreams.test_stream_tuples \ TestSubtitle.test_movtext \ TestSubtitle.test_vobsub ...and it is an incomplete list - this command still fails: ```bash cd '/build/python-av-7.0.1~git/.pybuild/cpython3_3.8/build'; python3.8 -m pytest --ignore=tests/test_doctests.py --ignore=tests/test_timeout.py -k 'not TestAudioFifo.test_data and not TestCodecContext.test_parse and not TestEncoding.test_encoding_aac and not TestEncoding.test_encoding_dnxhd and not TestEncoding.test_encoding_dvvideo and not TestEncoding.test_encoding_h264 and not TestEncoding.test_encoding_mjpeg and not TestEncoding.test_encoding_mp2 and not TestEncoding.test_encoding_mpeg1video and not TestEncoding.test_encoding_pcm_s24le and not TestEncoding.test_encoding_png and not TestEncoding.test_encoding_tiff and not TestContainers.test_context_manager and not TestDecode.test_decode_audio_sample_count and not TestDecode.test_decoded_motion_vectors and not TestDecode.test_decoded_motion_vectors_no_flag and not TestDecode.test_decoded_time_base and not TestDecode.test_decoded_video_frame_count and not TestBasicAudioEncoding.test_audio_transcode and not TestAudioProbe.test_container_probing and not TestAudioProbe.test_stream_probing and not TestDataProbe.test_container_probing and not TestDataProbe.test_stream_probing and not TestSubtitleProbe.test_container_probing and not TestSubtitleProbe.test_stream_probing and not TestAudioFifo.test_data and not TestCodecContext.test_parse and not TestEncoding.test_encoding_aac and not TestEncoding.test_encoding_dnxhd and not TestEncoding.test_encoding_dvvideo and not TestEncoding.test_encoding_h264 and not TestEncoding.test_encoding_mjpeg and not TestEncoding.test_encoding_mp2 and not TestEncoding.test_encoding_mpeg1video and not TestEncoding.test_encoding_mpeg4 and not TestEncoding.test_encoding_pcm_s24le and not TestEncoding.test_encoding_png and not TestEncoding.test_encoding_tiff and not TestContainers.test_context_manager and not TestDecode.test_decode_audio_sample_count and not TestDecode.test_decoded_motion_vectors and not TestDecode.test_decoded_motion_vectors_no_flag and not TestDecode.test_decoded_time_base and not TestDecode.test_decoded_video_frame_count and not TestBasicAudioEncoding.test_audio_transcode and not TestAudioProbe.test_container_probing and not TestAudioProbe.test_stream_probing and not TestDataProbe.test_container_probing and not TestDataProbe.test_stream_probing and not TestSubtitleProbe.test_container_probing and not TestSubtitleProbe.test_stream_probing and not TestVideoProbe.test_container_probing and not TestVideoProbe.test_stream_probing and not TestPythonIO.test_reading and not TestPythonIO.test_reading_no_seek and not TestSeek.test_decode_half and not TestSeek.test_deprecated_stream_seek and not TestSeek.test_seek_end and not TestSeek.test_seek_float and not TestSeek.test_seek_int64 and not TestSeek.test_seek_middle and not TestSeek.test_seek_start and not TestSeek.test_stream_seek and not TestStreams.test_selection and not TestStreams.test_stream_tuples and not TestSubtitle.test_movtext and not TestSubtitle.test_vobsub' ``` -- * Jonas Smedegaard - idealist & Internet-arkitekt * Tlf.: +45 40843136 Website: http://dr.jones.dk/ [x] quote me freely [ ] ask before reusing [ ] keep private

[mikeboers]

Is there a stage in the build environment in which network access is allowed to fetch the assets for the tests?

[jonassmedegaard]

On Debian autobuilders, no.
But there is the option of including the test assets in a package - either shipping them with this package or separately packaged.
Depending on what really need testing, a more elegant option is, I guess, to not use ffmpeg test assets but tell ffmpeg to generate test data and use that (as I guess what needs testing is not internal routines of ffmpeg but interfacing with those)

[jonassmedegaard]

This seems to be the most minimal workaround for now:

cd '/build/python-av-7.0.1~git/.pybuild/cpython3_3.8/build'; python3.8 -m pytest --ignore=tests/test_doctests.py --ignore=tests/test_timeout.py -k '-test_audio_transcode -test_container_probing -test_context_manager -test_data -test_decode_audio_sample_count -test_decoded_motion_vectors -test_decoded_time_base -test_decoded_video_frame_count -test_decode_half -test_deprecated_stream_seek -test_encoding -test_movtext -test_parse -test_reading -test_seek_end -test_seek_float -test_seek_int64 -test_seek_middle -test_seek_start -test_selection -test_stream_probing -test_stream_seek -test_stream_tuples -test_vobsub'

[jlaine]

Hi @jonassmedegaard ! Are you planning on packaging PyAV for Debian? That would be awesome news!

[jonassmedegaard]

Quoting Jeremy Lainé (2020-04-22 09:51:04)

Hi @jonassmedegaard ! Are you planning on packaging `PyAV` for Debian? That would be awesome news!

Yes, done already and now just awaiting ftpmaster approval - it is listed as "python-av" in https://ftp-master.debian.org/new.html

…

-- * Jonas Smedegaard - idealist & Internet-arkitekt * Tlf.: +45 40843136 Website: http://dr.jones.dk/ [x] quote me freely [ ] ask before reusing [ ] keep private

[jlaine]

@jonassmedegaard I can't remember, does buildd accept network tests which only rely on the loopback interface?

[jonassmedegaard]

Quoting Jeremy Lainé (2020-04-30 16:42:50)

@jonassmedegaard I can't remember, does `buildd` accept network tests which only rely on the loopback interface?

I think it does, yes - but am not certain - try ask on irc in #debian-devel

…

-- * Jonas Smedegaard - idealist & Internet-arkitekt * Tlf.: +45 40843136 Website: http://dr.jones.dk/ [x] quote me freely [ ] ask before reusing [ ] keep private

[jlaine]

@jonassmedegaard how are you handling this in Debian nowadays?

[jonassmedegaard]

During build (where network access is forbidden and blocked) I test like this:

pytest --ignore=tests/test_doctests.py --ignore=tests/test_timeout.py -k 'not test_audio_transcode and not test_codec_tag and not test_container_probing and not test_context_manager and not test_data and not test_decode_audio_sample_count and not test_decoded_motion_vectors and not test_decoded_time_base and not test_decoded_video_frame_count and not test_decode_half and not test_deprecated_stream_seek and not test_encoding and not test_movtext and not test_parse and not test_reading and not test_seek_end and not test_seek_float and not test_seek_int64 and not test_seek_middle and not test_seek_start and not test_selection and not test_stream_probing and not test_stream_seek and not test_stream_tuples and not test_vobsub and not test_reformat_pixel_format_align'

For continuous integration tests I do same, and then run another test flagged as "flaky" and "needs-internet", where I omit only test_reformat_pixel_format_align (which consistently fails on PowerPC).

[jonassmedegaard]

if you want to know the exact tests I omit then it is easier to read here.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[jlaine]

Lets keep this alive

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[jonassmedegaard]

la la la

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[emilazy]

Still relevant for us.