TST: Create compressed salary testing data

[dhimmel]

Create compressed versions of the salary dataset for testing #14576.

Rename salary.table.csv to salaries.tsv because the dataset is tab rather than comma delimited. Remove the word table because it's implied by the extension. Rename salary.table.gz to salaries.tsv.gz, since compressed files should append to not strip the original extension.

Created new files by running the following commands:

cd pandas/io/tests/parser/data
bzip2 --keep salaries.tsv
xz --keep salaries.tsv
zip salaries.tsv.zip salaries.tsv

There will be CI testing failures since the modified files are not yet in master or on amazon S3. @jorisvandenbossche how do we deal with this?

[jreback]

don't use tsv for extension, use csv

[dhimmel]

don't use tsv for extension, use csv

Why is that? The table isn't comma separated. This leads to issues like GitHub failing to render the table:

[jreback]

because you are changing something that is standard

if you wish to do that make a new PR but to be honest it's pretty non standard

[dhimmel]

because you are changing something that is standard

@jreback hmm. Presently, the CSV standard is being violated as RFC 4180 specifies:

Within the header and each record, there may be one or more fields, separated by commas.

However, salary.table.csv does meet the TSV standard:

Fields in a record are separated from each other by a tab character.

So I'm happy to save this for another pull request, but I thought it would be most efficient to address it here... especially since:

1. we're already standardizing the file names in other ways -- like removing .table and adding the extension to proceed .gz.
2. changing the name of testing files creates temporary testing failures... so might as well do two birds with one stone?

Just let me know what's the best way forward.

[dhimmel]

Tagging @jreback @sinhrks @jorisvandenbossche -- let me know what to do next here, so I can continue with #14576.

[jorisvandenbossche]

@dhimmel

• let's keep the files as .csv for now. Although those files meet the tsv standard, this standard is certainly not followed in all fields, and also in the pandas codebase we use the .csv file extension for all text files at the moment, no matter the delimiter that is used.
• standardizing the file names in the other ways is certainly fine! (the compression suffixes and leaving out the 'table')

For the CI failures, those are indeed not avoidable. But since it doesn't change code, only the file names, it's OK if you can confirm those specific tests pass locally.

[codecov-io]

Current coverage is 85.27% (diff: 100%)

Merging #14587 into master will decrease coverage by <.01%

@@             master     #14587   diff @@
==========================================
  Files           140        140          
  Lines         50693      50693          
  Methods           0          0          
  Messages          0          0          
  Branches          0          0          
==========================================
- Hits          43230      43229     -1   
- Misses         7463       7464     +1   
  Partials          0          0          

Powered by Codecov. Last update 7a2bcb6...759d48d

[dhimmel]

let's keep the files as .csv for now.

Done in 24341b5.

For the CI failures, those are indeed not avoidable. But since it doesn't change code, only the file names, it's OK if you can confirm those specific tests pass locally.

@jorisvandenbossche, here're the errors I get when I run nosetests pandas locally in the development environment (full log):

ERROR: test_url_gz (pandas.io.tests.parser.test_network.TestUrlGz)
ERROR: test_url_gz_infer (pandas.io.tests.parser.test_network.TestUrlGz)
ERROR: test_url (pandas.io.tests.parser.test_parsers.TestCParserHighMemory)
ERROR: test_url (pandas.io.tests.parser.test_parsers.TestCParserLowMemory)
ERROR: test_url (pandas.io.tests.parser.test_parsers.TestPythonParser)

It appears to me that all errors are from remote URLs not resolving... but how do I "confirm these specific tests pass locally". Should I locally change the URLs?

[jorisvandenbossche]

It appears to me that all errors are from remote URLs not resolving... but how do I "confirm these specific tests pass locally". Should I locally change the URLs?

Ah, yes, of course you have locally the same problem .. No problem, it's ok like this.

[jorisvandenbossche]

@dhimmel Thanks!

[jorisvandenbossche]

I made a small correction: ed21736

[dhimmel]

I made a small correction: ed21736

Yeah I should have more explicit that https://s3.amazonaws.com/pandas-test/salary.table.gz would have to be renamed on S3.

So in the longer term, here's what I'm thinking. First, is the S3 test actually valuable in addition to the GitHub URL test?

• If yes, then we should add the four compressed variants of salaries.csv to S3 using the new filename.
• If no, then we should only test the GitHub URLs, thereby removing the S3 (non-automated) dependency. As of 272669b, Refactor compression code to expand URL support #14576 removes the S3 test.

So if the S3 tests are valuable, then I'd suggest uploading salaries.csv* to S3 and I'll add those tests to #14576 accordingly.

[jorisvandenbossche]

@dhimmel that sounds right (the S3 here is just an url (not an S3 test), so github url is just as fine)

[dhimmel]

@dhimmel that sounds right (the S3 here is just an url (not an S3 test), so github url is just as fine)

@jorisvandenbossche, based on how the reading from URL works, we actually will need to also test reading compressed files from S3 URLs. See bff0f3e

So would it be possible to upload salaries.csv, salaries.csv.bz2, salaries.csv.gz, salaries.csv.xz, and salaries.csv.zip to the S3?

[jorisvandenbossche]

@jreback Do you know who has access to the amazon data?

@dhimmel Are you that it is not just an url in this case? (the link you gave does not lead to visible code ("We went looking everywhere, but couldn’t find those commits"), probably because you had to force push after a rebase or so)

[dhimmel]

probably because you had to force push after a rebase or so

Yep I rebased. See c042391. The point is that a different implementation is used to read URLs that are S3.

[jorisvandenbossche]

Yes, but I think this url is not seen as a 'S3 url', but just as a plain url (if you enter this url in a browser, you just get to download the file). A S3 url would more look like "s3://pandas-test/somefile.csv". What you link to is in the s3.py file, not in the main parsing file.

[jorisvandenbossche]

For example:

In [14]: pd.io.common._is_s3_url('https://s3.amazonaws.com/pandas-test/salary.table.gz')
Out[14]: False

In [15]: pd.io.common._is_s3_url('s3://pandas-test/salary.table.gz')
Out[15]: True