Remove access to public s3 buckets

[leshikus]

Is your feature request related to a problem? Please describe.
This is a compliance issue which comes from a principle of least privilege.

[leshikus]

Thinking of PR for test_io.py I've come to the following question.

Is it important for test_read_parquet_s3 that we use 1) s3, 2) generic file system? Or can we just use https URL as read_csv argument? @YarShev @anmyachev

[leshikus]

This feature requires an additional PR for test_io.py and test_io_exp.py - reopening

[anmyachev]

Thinking of PR for test_io.py I've come to the following question.

Is it important for test_read_parquet_s3 that we use 1) s3, 2) generic file system? Or can we just use https URL as read_csv argument? @YarShev @anmyachev

I think we should keep using s3.
However, let's try to transfer the files to the Modin repository, if they are small, then put them on the local S3 server and read from it.

For example: test_to_parquet_s3, which uses s3_resource and s3_storage_options fixtures.

[leshikus]

My take on this.

1. While having tests for multiple protocols is potentially a good thing.

2. I think this is out of scope of this particular feature request.

This test does not test how modin implementation handles s3 protocol. It copies file using s3fs (which is not a part of modin). Modin implementation then works with a file object.

            fs = s3fs.S3FileSystem(anon=True)
            with fs.open(dataset_url, "rb") as file_obj:
                eval_io("read_parquet", path=file_obj, engine=engine)

I cannot say if other tests do not test s3 protocol, needs further investigation.

3. Adding an s3 server will impact CI complexity and stability. I don't think the decision like this should be taken lightly.

4. Moving some test csv files to modin repo is a good thing due to the same reason. Less servers involved in testing means better CI stability.

[anmyachev]

Adding an s3 server will impact CI complexity and stability. I don't think the decision like this should be taken lightly.

We are already setting up the s3 server in Modin's tests:

modin/modin/conftest.py

Line 574 in 992eff5

def s3_base(worker_id, monkeysession):

[anmyachev]

I think this is out of scope of this particular feature request.

Agree

[anmyachev]

This test does not test how modin implementation handles s3 protocol. It copies file using s3fs (which is not a part of modin). Modin implementation then works with a file object.

This is actually an implementation of how Modin handles the s3 protocol - using s3fs library. We need to be sure that s3fs is used correctly.