GH-31621: [JS] Fix Union null bitmaps

[trxcllnt]

This PR fixes Union null handling, and re-enables the disabled SparseUnion test.

The format doc says:

Unlike other data types, unions do not have their own validity bitmap.

Therefore we need to remove null masks from union types and allow them to delegate validity to their children.

Also fixes #37063 for good measure.

• Closes: [JS] Allow appending null on sparse union and map builders #31621
• Closes: [JavaScript] Vector includes will return false for first element #37063
• Closes [Integration][JavaScript] Add null type integration test #24123
• Closes [Integration][JS] Fix map type to allow non-standard field names #17168

[github-actions]

⚠️ GitHub issue #31621 has been automatically assigned in GitHub to PR creator.

[pitrou]

@trxcllnt Does this allow removing the skip here?

arrow/dev/archery/archery/integration/datagen.py

Lines 1713 to 1715 in b9453a2

	generate_unions_case()
	.skip_category('C#')
	.skip_category('JS'),

[trxcllnt]

@pitrou yes, it should. I'll enable the integration tests now.

[trxcllnt]

The integration test is failing because the unions test has duplicate field names. I have a fix for it locally, but it involves renaming the duplicated sparse and dense fields to sparse_{1,2} and dense_{1,2}.

[trxcllnt]

From the logs, it seems like the C# implementation may not be aligning everything to 4-byte boundaries. Is there any way to get the files C# produces to confirm?

################# FAILURES #################
FAILED TEST: primitive C# producing,  JS consuming
FAILED TEST: null C# producing,  JS consuming
FAILED TEST: decimal C# producing,  JS consuming
FAILED TEST: datetime C# producing,  JS consuming
FAILED TEST: nested C# producing,  JS consuming
FAILED TEST: recursive_nested C# producing,  JS consuming

[kou]

@eerhardt Could you take a look at this?

[eerhardt]

@trxcllnt - can you run the integration tests locally? The arrow file should be written to the location that --arrow-file specifies when the test runs.

[trxcllnt]

@eerhardt I've only ever compiled and run the JS, C++, and Java. Don't really wanna install mono, is there a way to do it via docker?

[eerhardt]

Don't really wanna install mono

You don't need to install mono. .NET runs on Linux and Mac. https://dot.net/download

is there a way to do it via docker?

Yes, this is how the integration tests run in CI. You can execute the same commands as what CI does:

arrow/.github/workflows/integration.yml

Lines 85 to 95 in 2981942

	- name: Setup Archery
	run: pip install -e dev/archery[docker]
	- name: Execute Docker Build
	env:
	ARCHERY_DOCKER_USER: ${{ secrets.DOCKERHUB_USER }}
	ARCHERY_DOCKER_PASSWORD: ${{ secrets.DOCKERHUB_TOKEN }}
	run: >
	archery docker run \
	-e ARCHERY_DEFAULT_BRANCH=${{ github.event.repository.default_branch }} \
	-e ARCHERY_INTEGRATION_WITH_RUST=1 \
	conda-integration

You should be able to speed it up by commenting out the languages you don't need to build here:

arrow/docker-compose.yml

Lines 1709 to 1715 in 2981942

	["/arrow/ci/scripts/rust_build.sh /arrow /build &&
	/arrow/ci/scripts/cpp_build.sh /arrow /build &&
	/arrow/ci/scripts/csharp_build.sh /arrow /build &&
	/arrow/ci/scripts/go_build.sh /arrow &&
	/arrow/ci/scripts/java_build.sh /arrow /build &&
	/arrow/ci/scripts/js_build.sh /arrow /build &&
	/arrow/ci/scripts/integration_arrow.sh /arrow /build"]

and here:

arrow/ci/scripts/integration_arrow.sh

Lines 28 to 40 in 2981942

	archery integration \
	--run-flight \
	--with-cpp=1 \
	--with-csharp=1 \
	--with-java=1 \
	--with-js=1 \
	--with-go=1 \
	--gold-dirs=$gold_dir/0.14.1 \
	--gold-dirs=$gold_dir/0.17.1 \
	--gold-dirs=$gold_dir/1.0.0-bigendian \
	--gold-dirs=$gold_dir/1.0.0-littleendian \
	--gold-dirs=$gold_dir/2.0.0-compression \
	--gold-dirs=$gold_dir/4.0.0-shareddict \

[conbench-apache-arrow]

After merging your PR, Conbench analyzed the 6 benchmarking runs that have been run so far on merge-commit b957847.

There were no benchmark performance regressions. 🎉

The full Conbench report has more details. It also includes information about possible false positives for unstable benchmarks that are known to sometimes produce them.