Ignore comment lines in read_csv parsing

[mdmueller]

This is the first part of #7470.
closes #2685

[jreback]

@AmrAS1 pls squash, otherwise looks ok

@cpcloud @jorisvandenbossche ?

[jorisvandenbossche]

can you use double backtck quotes? (read_csv) for the functions read_csv and read_table (for parameter names single is OK)

[jorisvandenbossche]

not familiar with the c parser code, so can't comment on that, but added some doc comments.

@AmrAS1 You updated the io.rst explanation of the parameters, but I think you forgot to also update the parameter explanations in the docstring (it was in your previous PR I think, so probably just have to copy it)

[mdmueller]

@jorisvandenbossche - Oops, I'll update that now. Let me know if there are any other doc problems.

[jorisvandenbossche]

looking good what the docs is concerned.

Maybe you could also add a test for the combination of using comments with skiprows and header? (or are there already?)

[mdmueller]

Ok, updated with a skiprows/header test

[jorisvandenbossche]

Shouldn't you also test it seperate? Because by providing skiprows, you don't test if header=1 is skipping the commented lines automatically (only testing that skiprows does remove also the commented lines)? So for example also read_csv(StringIO(data), comment='#', header=1) should yield the same result?

[mdmueller]

Oops, I think I meant to have a row between X,Y,Z and A,B,C with skiprows=4. I'll fix that and add a couple separate tests.

[mdmueller]

Good thing you caught this--I just found that PythonParser wasn't behaving as expected! I currently have it adjusting self.line_pos when passing through skiprows, must've put that in before I decided on the correct functionality. I'll fix this.

[jreback]

@jorisvandenbossche ok?

[jreback]

@AmrAS1 thanks!

[mdmueller]

@jreback - You're welcome!

[jreback]

#7470 can do for 0.15.0 as its an API change

[jreback]

give a check on the docs when they are built (shortly): http://pandas-docs.github.io/pandas-docs-travis/io.html#comments and verify how it looks..thxs

[mdmueller]

Sure thing

[mdmueller]

Weird...there's one section in which ipython outputs this:

In [15]: data = 'a,b,c\n# commented line\n1,2,3\n#another comment\n4,5,6'

In [16]: print(data)
a,b,c
1,2,3
4,5,6

# commented line
#another comment

The rest of the output looks fine.

[jorisvandenbossche]

Indeed strange. I suppose this is an issue with the processing in the ipython directive. If you want to look into it, the code is in https://github.com/pydata/pandas/tree/master/doc/sphinxext/ipython_sphinxext (but which is a copy from ipython itself), or you can open an issue at ipython.