Dynamic reads

[robinbernon]

Adding ability to read a subset of parquet data dynamically.

[alecmocatta]

Looking good overall, thanks @robinbernon! Would you be able to fix a couple of other minor issues so that the linting step in the tests passes? e.g. https://dev.azure.com/alecmocatta/amadeus/_build/results?buildId=1591&view=logs&jobId=4420cb5a-3e60-5d7c-f139-f152148f0805&j=6471ded8-2f96-5fe2-f49c-aa706746e11a&t=0c335986-3956-5fa5-937e-2ddc0500a0f1

[alecmocatta]

What's the reason for this?

[robinbernon]

Was getting some strange issues due to double struct use in hashlink lib.rs. Found this fix was necessary to stop the given issue when compiling.

[alecmocatta]

Sorry I realise I overlooked when you messaged about this. This would have a perf regression. Looks like it might be a simple fix to hashlink, would you be able to make that PR? We can patch until your PR is merged and a new release published:

[patch.crates-io]
hashlink = { version = "0.6", git = "...", branch = "..." }

[robinbernon]

Have made the fix for this and submitted a PR: kyren/hashlink#7

[PsiACE]

Would you like to evaluate ritelinked ? This is a version derived from hashlink . We use griddle to reduce the tail delay and have solved the serialization problem.

[robinbernon]

Yeh sure, happy to have a look :) Do you have any example code available for a common use case you might encounter that wouldn't work directly with LinkedHashMap or BTreeMap?

[PsiACE]

@robinbernon You can directly replace hashlink with ritelinked without any additional changes, and the APIs of the two are compatible. In fact, ritelinked is a hashlink that only provides LinkedHashMap and LinkedHashSet, we just did some work to make it more usable .

[alecmocatta]

How come this is removed?

[robinbernon]

Sorry have put it back now, currently experiencing an issue with the predicate derives - Have got the specific issue highlighted in the following branch: https://github.com/robinbernon/amadeus/blob/temp2/amadeus-derive/src/lib.rs#L279-L290.

Pull request has been modified.

[alecmocatta]

The helpers here unfortunately break amadeus on stable. Ideally they impl serde_closure::traits::FnMut (which work on stable) rather than std::ops::FnMut (which don't yet). Implementing that is currently pretty rough

amadeus/amadeus-serde/src/json.rs

Lines 45 to 51 in 7f071aa

	FnMutNamed! {
	pub type Closure<P, Row, E> = |self|partition=> P| -> Output<P, Row, E>
	where
	P: Partition,
	Row: SerdeData,
	E: 'static
	{

I've been meaning to write another proc macro for serde_closure to make this easier but haven't had a chance yet.

The other alternative is to make helpers nightly-only, and hide it from the docs. What do you think?

[robinbernon]

The helpers here unfortunately break amadeus on stable. Ideally they impl serde_closure::traits::FnMut (which work on stable) rather than std::ops::FnMut (which don't yet). Implementing that is currently pretty rough

amadeus/amadeus-serde/src/json.rs

Lines 45 to 51 in 7f071aa

	FnMutNamed! {
	pub type Closure<P, Row, E> = |self|partition=> P| -> Output<P, Row, E>
	where
	P: Partition,
	Row: SerdeData,
	E: 'static
	{

I've been meaning to write another proc macro for serde_closure to make this easier but haven't had a chance yet.
The other alternative is to make helpers nightly-only, and hide it from the docs. What do you think?

Issue with impl serde closure instead of normal closure is that it would mean all adapter operations for parallel streams would have to change to using serde closures to support the change. There is currently a noticeable lack of intellisense when working with macros even on the best available IDE's - due to this I'd personally prefer to keep the standard closures in parallel streams as I like having intellisense when working on the various adapter operations etc before moving them into the required serde macros for distributed processing. Any chaance I can include the helpers here as a nightly feature then?

[alecmocatta]

it would mean all adapter operations for parallel streams would have to change to using serde closures to support the change

Oh right, I see the issue. Yes agreed it's strongly desirable to keep std::ops::Fn* on parallel streams. Given this, if we do merge these helpers they will need to be behind a cfg(nightly).