ENH/BUG: ignore line comments in CSV files GH2685

[holocronweaver]

closes #2685

I have added the ability for both the C and Python CSV parsers to ignore commented lines (i.e., lines beginning with a comment character). Currently the C parser preserves commented lines as empty lines (all NaN), while the Python parser ignores them all together.

In addition, I fixed a small related problem with the CSV format sniffer in the Python parser.

I plan to finish up this work by ignoring empty lines as per #4466.

[cpcloud]

no need for parens here is not binds tighter than and

[cpcloud]

Can you add a test and release notes? thx!

[holocronweaver]

Sorry, I am new to pandas dev. I am guessing a unit test for commented lines in a CSV file is what you have in mind?

[cpcloud]

Yep!

[holocronweaver]

Where should the test be created? There does not seem to be a particular file for parsers. Maybe test_frame since parsers return frames?

[cpcloud]

check out pandas/tests/test_parsers.py

[holocronweaver]

Sorry, missed the tests folder in pandas/io.

Having trouble setting up the test to expect different output for C and Python parsers. The tests seem to lock the parser engine and ignore the engine parameter in read_csv, causing my test to fail. The Python parser omits empty lines, while the C parser does not. In #4466 I propose making the behavior the same by making the C parser follow the Python parser behavior. Perhaps I should just go ahead and implement my suggestion? Or is there a method to query the current engine from within a test?

[jreback]

it normally goes thru 3 different version of the parser if you put your test in ParserTests, python parsing, c parsing, and i think low memory c parsing. You can put a test in say PythonParsing if you only want to have it run on that. Best prob to put it in the main test class if you want to have similar behavior in all parsers.

if you step thru it it call read_csv (and not pd.read_csv), which sets the engine depending on the iteration. I think you can set engine='python' in any event when you call read_csv to specify locally

[jreback]

@holocronweaver how's this coming along?

[holocronweaver]

Almost done, though temporarily delayed due to work. I will try to get this finished up tomorrow if possible. Worst case would be next weekend.

[jreback]

gr8
ping when ready

[jreback]

@holocronweaver how's this coming along?

[holocronweaver]

@jreback It is basically done, but I need time to test and debug. I am currently finishing a GSoC project which ends next week, so I will have a bit of free time again and will try to push this as soon as I get a chance.

[jreback]

@holocronweaver perfect...pls ping back when to take a look

[jreback]

@holocronweaver how's this coming?

[jreback]

@holocronweaver ping!

[jreback]

@holocronweaver going to be able to rebase this in the next couple of days?

[holocronweaver]

@jreback Sorry, have been very busy at work. Will be at least another week, though I will try to get it done sooner. Apologies again for the long delay.

[jreback]

@holocronweaver ok...let us know

[jreback]

@holocronweaver can are to rebase this?

[holocronweaver]

@jreback Sure, when I get back from holiday travels.

[jreback]

@holocronweaver progress on this?

[holocronweaver]

@jreback No, but it is on my TODO list. Crunch time is preventing anything extracurricular.

[jreback]

@holocronweaver update?

[jreback]

@holocronweaver update on this?

[jreback]

closing in favor of #7470

[jreback]

see here: https://github.com/pydata/pandas/pull/7470/files

try skip_blank_lines=False (is the original behavior)

[amanshei]

Thanks!!

On Sat, Dec 27, 2014 at 6:27 PM, jreback notifications@github.com wrote:

see here: https://github.com/pydata/pandas/pull/7470/files

try skip_blank_lines=False (is the original behavior)

—
Reply to this email directly or view it on GitHub
#4505 (comment).