Support with_projection and with_columns in ipc and parquet reader in eager mode

[illumination-k]

Support with_projection and with_columns in IPC and parquet reader to select columns in eager mode.

related to #1569

[ritchie46]

Looks very good @illumination-k 👍

Do you think you can also expose it in the python API? If you want, I can do it as well.

[ghuls]

Maybe also remove duplicates after sorting:
https://github.com/jorgecarleitao/arrow2/blob/3caa20421f8a6950dec6e2ad1c54125ef3a896ab/src/io/ipc/read/reader.rs#L236

[illumination-k]

Sure, we can remove duplicates, but It seems that it would be more helpful to return an error if there is a duplicate.

It's not good to get an error in select C, B, but wouldn't it be better to get an error in select B, B?

[ritchie46]

I prefer returning an error indeed. The API expects unique columns.

[ghuls]

I plan to use it with projections of 100k columns

let mut prj = Vec::with_capacity(cols.len());

[illumination-k]

Looks very good @illumination-k 👍

Do you think you can also expose it in the python API? If you want, I can do it as well.

I don't understand much about the python API, but can I just rewrite the following part?

polars/py-polars/src/dataframe.rs

Lines 175 to 201 in a2bbd4d

	#[staticmethod]
	#[cfg(feature = "parquet")]
	pub fn read_parquet(py_f: PyObject, stop_after_n_rows: Option<usize>) -> PyResult<Self> {
	use EitherRustPythonFile::*;
	let result = match get_either_file(py_f, false)? {
	Py(f) => {
	let buf = f.as_buffer();
	ParquetReader::new(buf)
	.with_stop_after_n_rows(stop_after_n_rows)
	.finish()
	}
	Rust(f) => ParquetReader::new(f)
	.with_stop_after_n_rows(stop_after_n_rows)
	.finish(),
	};
	let df = result.map_err(PyPolarsEr::from)?;
	Ok(PyDataFrame::new(df))
	}
	#[staticmethod]
	#[cfg(feature = "ipc")]
	pub fn read_ipc(py_f: PyObject) -> PyResult<Self> {
	let file = get_file_like(py_f, false)?;
	let df = IpcReader::new(file).finish().map_err(PyPolarsEr::from)?;
	Ok(PyDataFrame::new(df))
	}

polars/py-polars/polars/eager/frame.py

Line 487 in a2bbd4d

def read_parquet(

polars/py-polars/polars/eager/frame.py

Line 506 in a2bbd4d

def read_ipc(file: Union[str, BinaryIO]) -> "DataFrame":

[illumination-k]

@ritchie46
I successfully ran pl.read_ipc and pl.read_parquet without pyarrow and added test. In my opinion, some development instructions for Python would have been very much appreciated, eg., how to test, how to format, how to set up virtualenv etc...

[ritchie46]

@ritchie46 I successfully ran pl.read_ipc and pl.read_parquet without pyarrow and added test. In my opinion, some development instructions for Python would have been very much appreciated, eg., how to test, how to format, how to set up virtualenv etc...

Thanks. It looks solid. Great work!

Maybe you could add what you missed in the cobtrubution guide?

[illumination-k]

Oh I'm really sorry, I missed the last part... It is sufficient for me.

[ritchie46]

Very nice work @illumination-k. Thanks a lot!