Unblock supported compression libs in pytables #16196
Conversation
Codecov Report
@@ Coverage Diff @@
## master #16196 +/- ##
==========================================
- Coverage 90.39% 90.37% -0.02%
==========================================
Files 161 161
Lines 50863 50863
==========================================
- Hits 45977 45967 -10
- Misses 4886 4896 +10
Continue to review full report at Codecov.
|
pandas/tests/io/test_pytables.py
Outdated
| @@ -736,6 +736,114 @@ def test_put_compression_blosc(self): | |||
| store.put('c', df, format='table', complib='blosc') | |||
| tm.assert_frame_equal(store['c'], df) | |||
|
|
|||
| def test_complibs(self): | |||
| # Test data | |||
| data = [{'ID': '7082-GTZ-572490', | |||
There was a problem hiding this comment.
don't specify data like this, just use tm.makeDataFrame
pandas/tests/io/test_pytables.py
Outdated
| all_tests = [(lib, lvl) for lib in all_complibs for lvl in all_levels] | ||
|
|
||
| for (lib, lvl) in all_tests: | ||
| with ensure_clean_store(self.path, complib=lib, |
There was a problem hiding this comment.
I would rather you use .to_hdf and .read_hdf here
|
this is going to need a release note for 0.20.1 (but doesn't exist as of yet, soon). |
pandas/tests/io/test_pytables.py
Outdated
| @@ -736,6 +736,20 @@ def test_put_compression_blosc(self): | |||
| store.put('c', df, format='table', complib='blosc') | |||
| tm.assert_frame_equal(store['c'], df) | |||
|
|
|||
| def test_complibs(self): | |||
| df = tm.makeDataFrame() | |||
|
|
|||
There was a problem hiding this comment.
can you add a test that validates that complevel=0 turns off compression?
There was a problem hiding this comment.
How do I best do that? There is the h5dump utility that can tell me if its compressed or I can compare the size of the files perhaps?
There was a problem hiding this comment.
look in the meta data of the file
lots of _ attributes that define things like this
pytables docs might be helpful
pandas/tests/io/test_pytables.py
Outdated
| @@ -736,6 +736,20 @@ def test_put_compression_blosc(self): | |||
| store.put('c', df, format='table', complib='blosc') | |||
| tm.assert_frame_equal(store['c'], df) | |||
|
|
|||
| def test_complibs(self): | |||
| df = tm.makeDataFrame() | |||
There was a problem hiding this comment.
add a comment referencing the issue
pandas/tests/io/test_pytables.py
Outdated
| @@ -4942,7 +4956,7 @@ def test_invalid_complib(self): | |||
| columns=list('ABCDE')) | |||
| with ensure_clean_path(self.path) as path: | |||
| pytest.raises(ValueError, df.to_hdf, path, | |||
There was a problem hiding this comment.
this needs to have be
with pytest.raises(ValueError):
...
|
It turned out that not all configurations has lzo compression available, so I removed those tests for those scenarios. Is this looking good? The whatsnew file for 20.1 is still not in existence? |
|
@linebp put this in 0.20.2 Enhancements (0.20.1 doesn't exist and was a combine release with 0.20.0). ping on green. |
pandas/core/generic.py
Outdated
| complib : {'zlib', 'lzo', 'bzip2', 'blosc', None}, default None | ||
| Specifies the compression library to be used. | ||
| Additional compressors for Blosc are supported (default if no | ||
| compressor specified: 'blosc:blosclz'): |
There was a problem hiding this comment.
add as of 0.20.2 these addtional compressors are available.
pandas/io/pytables.py
Outdated
| complib : {'zlib', 'lzo', 'bzip2', 'blosc', None}, default None | ||
| Specifies the compression library to be used. | ||
| Additional compressors for Blosc are supported (default if no | ||
| compressor specified: 'blosc:blosclz'): |
|
Its all green now. |
|
thanks! |
(cherry picked from commit 541e8e8)
(cherry picked from commit 541e8e8)
git diff upstream/master --name-only -- '*.py' | flake8 --diff