Avro examples not usable, how to use Avro?

[Samrose-Ahmed]

The Avro examples/documentation seems incorrect. Looks like it references private crates/functions. How would one read/write Avro?

[jorgecarleitao]

Hey!

Could you compile with --features io_avro? We document that certain modules require feature flags.

The issue here is that atm there is no way in Rust to have no default features and at the same time automatically pass --features to the example (i.e. cargo.toml does not allow specifying which features are required for each example)

[Samrose-Ahmed]

Hi I did compile with io_avro.

If I use the code:

use arrow2::io::avro::avro_schema::file::Block;
use arrow2::io::avro::avro_schema::read_async::{block_stream, decompress_block, read_metadata};
use arrow2::io::avro::read::{deserialize, infer_schema};

I get error

no `avro_schema` in `io::avro`

Also

function `infer_schema` is private

[jorgecarleitao]

just to check, is this the current release or the latest main? I am being unable to reproduce this in both cases.

For example, check https://github.com/DataEngineeringLabs/arrow2-benches/tree/latest, where a Cargo.toml with the latest main and the benches containing the reader work as intended?

[Samrose-Ahmed]

I saw this on arrow2 version 0.13.0 and 0.13.1 (which are latest based on crates.io). The link you mentioned seems to be using arrow2 version=0.6?

[jorgecarleitao]

the main is still using 0.6, yes; I meant the branch "latest".

[jorgecarleitao]

ohh, I now realize that I forgot to push the change locally. Really sorry about that :( Could you check the branch "latest" now?

[Samrose-Ahmed]

Apologies if I'm misunderstanding but I don't see any branch called "latest" on https://github.com/jorgecarleitao/arrow2/branches ?

[Samrose-Ahmed]

Changing my Cargo.toml to use the git repository instead of Cargo v 0.13.1 (git = "https://github.com/jorgecarleitao/arrow2") seems to have fixed this issue. I guess the docs were for the git main not the 0.13.1 version.

[FireArrow]

There are similar problems with the parquet examples, which are written for latest main, but doesn't work on 0.13.1 (the latest release as of writing this). This caused me more confusion than I want to admit. What about splitting the examples and the actual development so the examples can be merged as part of creating a release? Or at the very least mention the difference somehow?

[jorgecarleitao]

Note that the examples are present in the released artifact for a specific version, https://docs.rs/crate/arrow2/latest/source/examples/ . We also tag every released version, so e.g. https://github.com/jorgecarleitao/arrow2/tree/v0.13.0 provides the examples for the released version. If I'm looking into the examples in the branch main, I would expect to get the examples compatible with main?

@Samrose-Ahmed glad that the issue is addressed. Note that I was not pointing to this repository in the comment above - https://github.com/DataEngineeringLabs/arrow2-benches/tree/latest

[FireArrow]

What you are saying makes sense. But when trying this lib out the first place you look at is the guide linked in the readme which seems to be following main, not the release the readme was written for. Is there a way to view different branches on github.io, or something like that?

[jorgecarleitao]

Ahhh, I see. I created #1233 to try to improve the situation.

Essentially, after the PR, we will have:

• https://jorgecarleitao.github.io/arrow2/main/guide
• https://jorgecarleitao.github.io/arrow2/main/docs
• https://jorgecarleitao.github.io/arrow2/{tag}/guide
• https://jorgecarleitao.github.io/arrow2/{tag}/docs

where {tag} is a released version. I have to investigate if I can retroactively generate the old tags, but it seems unlikely.

[jorgecarleitao]

@Samrose-Ahmed, @FireArrow, do you believe #1233 closes this issue?

[Samrose-Ahmed]

Yes, thank you for the update.