Merge upstream 4.0.4

CHANGELOG.md

@@ -2,6 +2,20 @@
 
 ## v4.0
 
+### v4.0.4
+- feat(arrow): GeoArrowLoader (#2796)
+- fix(draco): revert --format=esm for the worker (#2795)
+- feat(arrow): Support GeoJSON output from Geo ArrowLoader (#2794)
+- chore: refactor geospatial exmaple to use hooks (#2793)
+- chore(arrow): triangulateOnWorker plumbing (#2788)
+- feat(website): add GeoParquet datasets (#2786)
+- fix(tile-converter): 'finalizing conversion' added (#2787)
+- feat(GeoArrow): getBinaryGeometriesFromArrow enhancement (#2785)
+- chore(Arrow): add test cases for multipolygon with holes (#2782)
+- chore(worker-utils): worker simplification
+- chore: Add license headers (#2784)
+- chore: Upgrade to docusaurus 3 (#2757)
+
 ### v4.0.3
 
 - feat(tile-converter): estimation of time remaining (#2774)

docs/arrowjs/README.md

@@ -2,30 +2,28 @@
 
 ## Why does loaders.gl provide an Arrow JS API Reference?
 
-loaders.gl is designed to output parsed tables and meshes in binary columnar format (whenever the parsed data structure allows). Binary columnar tables are a compact and efficient representation that is easy to work with analytically in JavaScript and to seamlessly upload to GPUs (via e.g. WebGL or WebGPU) for ultra-performance rendering and computation.
+> The idea is that this documentation should at some point be contributed back to the Apache Arrow project/repository, but so far this has not happened.
 
-While loaders.gl is great for loading data into binary columnar tables, it does not provide much support for working with those tables. Instead, the intention is that the application will use complementary libraries like Apache Arrow for that purpose. Apache Arrow defines both a file format and a powerful "data frame" API (i.e. a data science optimized, Python `pandas`-style API) for working with binary columnar data.
+loaders.gl is designed to output parsed tables and meshes in binary columnar format (whenever the parsed data structure allows). Binary columnar tables are a compact and efficient representation that is easy to work with analytically in JavaScript and to seamlessly upload to GPUs (via e.g. WebGL or WebGPU) for ultra-performance rendering and computation.
 
-While the Apache Arrow JS library itself is excellent, the [reference documentation for the Apache Arrow JavaScript bindings](https://arrow.apache.org/docs/js/) is unfortunately rather thin. It can therefore be challenging to get up to speed on the Arrow JS API, which is why this documentation is provided in loaders.gl.
+While loaders.gl can load data into binary columnar tables, it only provides limited support for working with binary tables. The intention is that the application should be able to use complementary libraries like Apache Arrow JS.
 
-> Perhaps this documentation could at some point be contributed back to the Apache Arrow project, but so far this has not happened.
+While the Apache Arrow JS library itself is excellent, the [reference documentation for the JavaScript bindings](https://arrow.apache.org/docs/js/) is unfortunately rather thin. It can therefore be challenging to get up to speed on the Arrow JS API, which is why this documentation is provided in loaders.gl.
 
 ## About Apache Arrow JS
 
-The Apache Arrow JavaScript API is designed to help applications tap into the full power of working with binary columnar data in the Apache Arrow format. Arrow JS has a rich set of classes that supports use cases such as batched loading and writing, as well performing data frame operations on Arrow encoded data, including applying filters, iterating over tables, etc.
+The Apache Arrow JavaScript API is designed to help applications work with binary columnar data in the Apache Arrow format. Arrow JS offers a core set of classes that supports use cases such as batched loading and writing, column and row access, schemas etc.
 
 ## Getting Started
 
 To install and start coding with Apache Arrow JS bindings, see the [Getting Started](/docs/arrowjs/get-started).
 
-
 ## About Apache Arrow
 
 Apache Arrow is a performance-optimized binary columnar memory layout specification for encoding vectors and table-like containers of flat and nested data. The Arrow spec is design to eliminate memory copies and aligns columnar data in memory to minimize cache misses and take advantage of the latest SIMD (Single input multiple data) and GPU operations on modern processors.
 
 Apache Arrow is emerging as the standard for large in-memory columnar data (Spark, Pandas, Drill, Graphistry, ...). By standardizing on a common binary interchange format, big data systems can reduce the costs and friction associated with cross-system communication.
 
-
 ## Resources
 
 There are some excellent resources available that can help you quickly get a feel for what capabilities the Arrow JS API offers:

docs/arrowjs/api-reference/data.md

@@ -68,17 +68,17 @@ Convenience methods for creating Data instances for each of the Arrow Vector typ
 
 ### `Data.Utf8<T extends Utf8>`(type: T, offset: Number, length: Number, nullCount: Number, nullBitmap: NullBuffer, valueOffsets: ValueOffsetsBuffer, data: Uint8Array) : Data
 
-### `Data.List<T extends List>`(type: T, offset: Number, length: Number, nullCount: Number, nullBitmap: NullBuffer, valueOffsets: ValueOffsetsBuffer, child: Data<T['valueType']> | Vector<T['valueType']>) : Data
+### `Data.List<T extends List>`(type: T, offset: Number, length: Number, nullCount: Number, nullBitmap: NullBuffer, valueOffsets: ValueOffsetsBuffer, child: `Data<T['valueType']> | Vector<T['valueType']>`) : Data
 
 ### `Data.FixedSizeList<T extends FixedSizeList>`(type: T, offset: Number, length: Number, nullCount: Number, nullBitmap: NullBuffer, child: Data | Vector) : Data
 
 ### `Data.Struct<T extends Struct>`(type: T, offset: Number, length: Number, nullCount: Number, nullBitmap: NullBuffer, children: (Data | Vector)[]) : Data
 
-### `Data.Map<T extends Map_>`(type: T, offset: Number, length: Number, nullCount: Number, nullBitmap: NullBuffer, children: (Data | Vector)[]) : Data
+### `Data.Map<T extends Map_>`(type: T, offset: Number, length: Number, nullCount: Number, nullBitmap: NullBuffer, children: `(Data | Vector)[])` : Data
 
-### `Data.Union<T extends SparseUnion>`(type: T, offset: Number, length: Number, nullCount: Number, nullBitmap: NullBuffer, typeIds: TypeIdsBuffer, children: (Data | Vector)[]) : Data
+### `Data.Union<T extends SparseUnion>`(type: T, offset: Number, length: Number, nullCount: Number, nullBitmap: NullBuffer, typeIds: TypeIdsBuffer, children: `(Data | Vector)[])` : Data
 
-### `Data.Union<T extends DenseUnion>`(type: T, offset: Number, length: Number, nullCount: Number, nullBitmap: NullBuffer, typeIds: TypeIdsBuffer, valueOffsets: ValueOffsetsBuffer, children: (Data | Vector)[]) : Data
+### `Data.Union<T extends DenseUnion>`(type: T, offset: Number, length: Number, nullCount: Number, nullBitmap: NullBuffer, typeIds: TypeIdsBuffer, valueOffsets: ValueOffsetsBuffer, children: `(Data | Vector)[])` : Data
 }
 
 

docs/developer-guide/concepts/javascript-apis.md

@@ -13,7 +13,7 @@ Over the years, a number of different JavaScript APIs have emerged. Depending on
 
 loaders.gl standardizes on the `fetch()` API. The result of a `fetch` operation is a `Response` object which can be passed to many loaders.gl functions, meaning that the application can call `fetch()` itself to fully control the requests.
 
-:::information
+:::info
 The `fetch()` API emerged in browsers, but is now also supported natively on Node, starting with Node.js v18.
 For older Node.js versions, `@loaders.gl/polyfills` installs polyfills of `fetch`, `Response` and `Headers` classes.
 :::

docs/docs-sidebar.json

@@ -68,6 +68,8 @@
     "label": "Formats",
     "items": [
       "formats/README",
+      "modules/arrow/formats/arrow",
+      "modules/arrow/formats/geoarrow",
       "modules/bson/formats/bson",
       "modules/csv/formats/csv",
       "modules/pcd/formats/pcd",

docs/modules/arrow/README.md

@@ -1,8 +1,10 @@
 # Overview
 
 ![arrow-logo](./images/apache-arrow-small.png)
+ 
+![apache-logo](../../images/logos/apache-logo.png)
 
-The `@loaders.gl/arrow` module handles [Apache Arrow](https://arrow.apache.org/), an emerging standard for large in-memory columnar data.
+The `@loaders.gl/arrow` module provides support for the [Apache Arrow](/docs/modules/arrow/formats/arrow) and [GeoArrow](/docs/modules/arrow/formats/geoarrow) formats. 
 
 ## Installation
 

docs/modules/arrow/formats/arrow.md

@@ -1,5 +1,21 @@
 # Apache Arrow
 
 ![arrow-logo](../images/apache-arrow-small.png)
+ 
+![apache-logo](../../../images/logos/apache-logo.png)
+
+- *[`@loaders.gl/arrow`](/docs/modules/arrow)* - loaders.gl implementation
+- *[Apache Arrow](https://arrow.apache.org/)* - A specification for large in-memory columnar data.
+- *[ArrowJS](https://arrow.apache.org/docs/js)* - official documentation on ArrowJS API.
+- *[ArrowJS](/docs/arrowjs)* - loaders.gl documentation on ArrowJS API.
+
+The Apache Arrow project specifies a language-independent binary columnar memory format. It enables zero-copy shared memory and streaming messaging, interprocess communication, and is supported by many programming languages and data libraries.
+
+This Apache Arrow specification supports encoding vectors and table-like containers of flat and nested data. 
+
+The Arrow spec is performance-optimized to eliminate memory copies and aligns columnar data in memory to minimize cache misses and take advantage of the latest SIMD (Single input multiple data) and GPU operations on modern processors.
+
+Apache Arrow is emerging as a de-facto standard for large in-memory columnar data (Spark, Pandas, Drill, ...). 
+
+By standardizing on a common binary interchange format, big data systems can reduce the costs and friction associated with cross-system communication.
 
-For more information, see [ArrowJS](/docs/arrowjs) documentation.

docs/modules/arrow/formats/geoarrow.md

@@ -0,0 +1,43 @@
+# GeoArrow
+
+![arrow-logo](../images/apache-arrow-small.png)
+ 
+![apache-logo](../../../images/logos/apache-logo.png)
+
+- *[`@loaders.gl/arrow`](/docs/modules/arrow)* - loaders.gl implementation
+- *[GeoArrow Specification](https://github.com/geoarrow/geoarrow)
+- *[Apache Arrow](https://arrow.apache.org/)* - A specification for large in-memory columnar data.
+- *[ArrowJS](/docs/arrowjs)* - loaders.gl documentation on ArrowJS API.
+
+## Overview
+
+GeoArrow is a specification for storing geospatial data in Apache Arrow memory layout. It ensures geospatial tools can interoperate and leverage the growing Apache Arrow ecosystem.
+
+GeoArrow enables each row in an Arrow table to represent a feature as defined by the OGC Simple Feature Access standard (i.e. Point, LineString, Polygon, MultiPoint, MultiLineString, MultiPolygon, and GeometryCollection). 
+
+Aside from geometry, simple features can also have additional standard Arrow columns that provide additional non-spatial attributes for the feature.
+
+Geospatial tabular data where one or more columns contains feature geometries and remaining columns define feature attributes. The GeoArrow specification defines how such vector features (geometries) can be stored in Arrow (and Arrow-compatible) data structures.
+
+Note that GeoArrow is not a separate format from Apache Arrow rather, the GeoArrow specification simply describes additional conventions for metadata and layout of geospatial data. This means that a valid GeoArrow file is always a valid Arrow file. This is done through [Arrow extension type](https://arrow.apache.org/docs/format/Columnar.html#extension-types) definitions that ensure type-level metadata (e.g., CRS) is propagated when used in Arrow implementations.
+
+
+## Relationship with GeoParquet
+
+The [GeoParquet specification](https://github.com/opengeospatial/geoparquet) is closely related to GeoArrow. Notable differences:
+
+- GeoParquet is a file-level metadata specification
+- GeoArrow is a field-level metadata and memory layout specification
+
+## Geometry Types
+
+| Geometry type              | Read | Write | Description |
+| -------------------------- | ---- | ----- | ----------- |
+| `geoarrow.multipolygon`    | ✅    | ❌     |             |
+| `geoarrow.polygon`         | ✅    | ❌     |             |
+| `geoarrow.multipoint`      | ✅    | ❌     |             |
+| `geoarrow.point`           | ✅    | ❌     |             |
+| `geoarrow.multilinestring` | ✅    | ❌     |             |
+| `geoarrow.linestring`      | ✅    | ❌     |             |
+| `geoarrow.wkb`             | ❌    | ❌     |             |
+| `geoarrow.wkt`             | ❌    | ❌     |             |

docs/modules/core/api-reference/fetch-file.md

@@ -24,9 +24,9 @@ Returns:
   - `json()`: Promise.String` - Loads the file and decodes it into JSON.
   - `body` : ReadableStream` - A stream that can be used to incrementally read the contents of the file.
 
-::info
+:::info
 Use of `fetchFile` is optional. loaders.gl `parse()` function can be used with data loaded via any mechanism the application prefers, e.g. directly using `fetch`, `XMLHttpRequest` etc.
-::
+:::
 
 ## Node.js local file system support
 
@@ -39,10 +39,10 @@ The `fetchFile()` function supports fetching data from the local file system und
 `fetchFile` will delegate any url that starts with `http://` `https://` or `data://`
 to the built-in `fetch` function. Other URLs will be interpreted as local files.
 
-::caution
+:::caution
 loaders.gl v4.0+ applications no longer need to install the `@loaders.gl/polyfills` 
 module to get `fetch` support under Node.js v18+.
-::
+:::
 
 ## Path prefix resolution
 

docs/modules/core/api-reference/parse-sync.md

@@ -1,8 +1,8 @@
 # parseSync
 
-::caution
+:::caution
 Synchronous parsing is not supported by all loaders. Refer to the documentation for each loader.
-::
+:::
 
 :::caution
 When calling parse from a loader to invoke a sub-loader, do not use this function. Use `parseSyncWithContext` counterparts in `@loaders.gl/loader-utils``

docs/modules/gltf/api-reference/gltf-scenegraph.md

@@ -74,7 +74,7 @@ const gltfBuffer = encodeSync(gltfBuilder.gltf, GLTFWriter);
 
 Contains all data of gltf including the json part and the binary chunk. The binary chunk is generated by `createBinChunk()` method
 
-### sourceBuffers: Array<ArrayBuffer | { arrayBuffer?: ArrayBuffer; buffer?: ArrayBuffer; byteOffset: number; byteLength: number}>
+### `sourceBuffers: Array<ArrayBuffer | { arrayBuffer?: ArrayBuffer; buffer?: ArrayBuffer; byteOffset: number; byteLength: number}>`
 
 Accumulates buffers of resources added by `addMesh()`, `addImage()` etc. These buffers should be compiled before encoding using `createBinaryChunk()`
 
@@ -197,46 +197,46 @@ Also adds the same extensionName to the `extensionsRemoved`.
 
 Set the default scene which is to be displayed at load time
 
-### addScene(arguments: {nodeIndices: number[]}): number;
+### `addScene(arguments: {nodeIndices: number[]}): number`
 
 Add a scene to the json part
 
-### addNode(arguments: {meshIndex: number}): number;
+### `addNode(arguments: {meshIndex: number}): number;`
 
 Add a node to the json part
 
-### addMesh(arguments: {attributes: object, indices: object, material: number, mode = 4})
+### `addMesh(arguments: {attributes: object, indices: object, material: number, mode = 4})`
 
-### addPointCloud(attributes)
+### `addPointCloud(attributes)`
 
-### addBufferView(buffer)
+### `addBufferView(buffer)`
 
 Add one untyped source buffer, create a matching glTF `bufferView`, and return its index
 
 > The binary data will not be added to the gltf buffer until `createBinChunk()` is called.
 
-### addTexture(arguments: {imageIndex: number}): number;
+### `addTexture(arguments: {imageIndex: number}): number;`
 
 Add a texture to the json part
 
-### addMaterial(pbrMaterialInfo: object): number;
+### `addMaterial(pbrMaterialInfo: object): number;`
 
 Add a material to the json part
 
-### addAccessor(bufferViewIndex, accessor)
+### `addAccessor(bufferViewIndex, accessor)`
 
 Adds an accessor to a bufferView
 
 > The binary data will not be added to the gltf buffer until `createBinChunk()` is called.
 
-### addImage(imageData, mimeType)
+### `addImage(imageData, mimeType)`
 
 Adds a binary image. Builds glTF "JSON metadata" and saves buffer reference
 Buffer will be copied into BIN chunk during "pack"
 
 > The binary data will not be added to the gltf buffer until `createBinChunk()` is called.
 
-### createBinChunk()
+### `createBinChunk()`
 
 Packs any pending binary data into the first binary glTF buffer.
 

docs/modules/loader-utils/api-reference/request-scheduler.md

@@ -35,7 +35,7 @@ if (requestToken) {
 - `throttleRequests`?: boolean;
 - `maxRequests`?: number;
 
-### scheduleRequest(handle: any, getPriority?: () => number): Promise<{done: () => any)}>;
+### `scheduleRequest(handle: any, getPriority?: () => number): Promise<{done: () => any)}>`
 
 Called by an application that wants to issue a request, without having it deeply queued by the browser
 

docs/modules/math/api-reference/gl-type.md

@@ -24,7 +24,7 @@ var size = GLType.getSizeInBytes(GL.BYTE);
 
 ## Static Methods
 
-### GLType.fromTypedArray(typedArray: Typed Array | Function) : string
+### GLType.fromTypedArray(typedArray: Typed Array \| Function) : string
 
 Returns the size, in bytes, of the corresponding datatype.
 
@@ -38,7 +38,7 @@ Throws
 
 - glType is not a valid value.
 
-Gets the {@link ComponentDatatype} for the provided TypedArray instance.
+Gets the component data type for the provided TypedArray instance.
 
 - array The typed array.
 
@@ -64,7 +64,7 @@ Equivalent to `GLType.getArrayType(glType).BYTES_PER_ELEMENT`.
 
 Returns `true` if `glType` is a valid WebGL data type.
 
-### static GLType.createTypedArray(glType : number, buffer : ArrayBuffer [, byteOffset : number [, length : number]]) : TypedArray
+### `static GLType.createTypedArray(glType : number, buffer : ArrayBuffer [, byteOffset : number [, length : number]]) : TypedArray`
 
 Creates a typed view of an array of bytes.
 

docs/modules/mvt/api-reference/tilejson-loader.md

@@ -1,6 +1,10 @@
-# PMTilesLoader
+# TileJSONLoader
 
-The `PMTilesLoader` parses header/metadata from a pmtiles archive
+<p class="badges">
+  <img src="https://img.shields.io/badge/From-v4.0-blue.svg?style=flat-square" alt="From-v4.0" />
+</p>
+
+The `TileJSONLoader` parses header/metadata from a pmtiles archive
 
 | Loader                | Characteristic                                     |
 | --------------------- | -------------------------------------------------- |

docs/modules/pmtiles/api-reference/pmtiles-source.md

@@ -1,5 +1,9 @@
 # PMTilesSource 🆕
 
+<p class="badges">
+  <img src="https://img.shields.io/badge/From-v4.0-blue.svg?style=flat-square" alt="From-v4.0" />
+</p>
+
 The `PMTilesSource` reads individual tiles from a PMTiles archive file.
 
 | Loader                | Characteristic                                |

docs/modules/textures/api-reference/load-image-array.md

@@ -46,7 +46,7 @@ the `getUrl` callback will be called for each image with the following parameter
 
 Note: In addition to these values, all `options` passed in to `loadImageArray` are also available in the `getUrl` method.
 
-### loadImageArray(count : Number | String, getUrl : ({index}) => String, options? : Object) : image[] | image[][]
+### loadImageArray(count : Number | String, getUrl : `({index}) => String`, options? : Object) : `image[] | image[][]`
 
 Parameters:
 

docs/modules/textures/api-reference/load-image-cube.md

@@ -48,7 +48,7 @@ The following fields will be supplied as named parameters to the `getUrl` functi
 
 Note: In addition to these values, all `options` passed in to `loadImageCube` are also available in the `getUrl` method.
 
-### loadImageCube(getUrl : ({face, direction, index}) => String, options? : Object) : Object
+### loadImageCube(getUrl : `({face, direction, index}) => String`, options? : Object) : Object
 
 Loads and image cube, i.e. 6 images keyed by WebGL face constants (see table).
 

docs/modules/wkt/api-reference/hex-wkb-loader.md

@@ -1,12 +1,11 @@
 # HexWKBLoader 🆕 🚧
 
-
-![ogc-logo](../../../images/logos/ogc-logo-60.png)
-
 <p class="badges">
   <img src="https://img.shields.io/badge/From-v2.2-blue.svg?style=flat-square" alt="From-v2.2" />
 </p>
 
+![ogc-logo](../../../images/logos/ogc-logo-60.png)
+
 Loader for hex encoded [Well-known binary][wkb] format for representation of geometry.
 
 [wkb]: https://en.wikipedia.org/wiki/Well-known_text_representation_of_geometry#Well-known_binary

docs/modules/wkt/api-reference/twkb-loader.md

@@ -1,8 +1,7 @@
 # TWKBLoader 🆕 🚧
 
-
 <p class="badges">
-  <img src="https://img.shields.io/badge/From-v2.2-blue.svg?style=flat-square" alt="From-v2.2" />
+  <img src="https://img.shields.io/badge/From-v4.0-blue.svg?style=flat-square" alt="From-v4.0" />
 </p>
 
 Loader for the [Well-known binary][wkb] format for representation of geometry.