Skip to content

Commit

Permalink
docs: Fix website, upgrade to docusaurus 3.5 (#3139)
Browse files Browse the repository at this point in the history
  • Loading branch information
ibgreen authored Oct 18, 2024
1 parent 264e413 commit 7c3a294
Show file tree
Hide file tree
Showing 93 changed files with 18,683 additions and 23,677 deletions.
2 changes: 1 addition & 1 deletion .github/workflows/test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,7 @@ jobs:
- name: Build website
run: |
cd website ; yarn install
cd website; yarn build
- name: Coveralls
uses: coverallsapp/github-action@master
Expand Down
3 changes: 3 additions & 0 deletions docs/modules/3d-tiles/api-reference/tile-3d-writer.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
# Tiles3DWriter

TBA
73 changes: 48 additions & 25 deletions docs/roadmap.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,50 @@
# Roadmap

loaders.gl is developed under open governance by multiple contributors working with their own priorities.
While it is hard to predict what work will be completed and when, current development tracks and aspirations include:
loaders.gl is developed under open governance by multiple contributors working with their own priorities. This page aims to give information about upcoming releases and directions.

## v4.4 (in development)

loaders.gl v4.4 will focus on cloud-native, binary data.
A number of modules will expose "ArrowLoaders" will return binary data in the Apache Arrow and Apache GeoArrow formats.

While no loader support has been removed, the flavor of the loaders.gl framework is changing.

**Apache Arrow as a core format**

- Many new loaders now return binary data in the Apache Arrow format.
- This aligns with parallel efforts in companion libraries like deck.gl (as well as the ecosystem at large) to work with zero-copy, compact binary data instead of bloated, deserialized javascript data structures.
- Binary columnar data brings in an order of magnitude better memory usage and improved load/processing performance on big datasets.
- The Apache Arrow JS library is now a central dependency of loaders.gl.

**Improved `DataSource` APIs**

- The `Source` and `DataSource` APIs have matured and are now easier to work with.
- Consule the upgrade guide for migration details.

### Per-module changes

- **`@loaders.gl/csv`**

- `CSVArrowLoader` - New CSV loader that returns Apache Arrow tables.

- **`@loaders.gl/parquet`**

- `ParquetArrowLoader` now returns Apache Arrow tables and leverages the high-performance `parquet-wasm` library.
- The v4 Parquet loader is still available as `ParquetJSONLoader`

- **`@loaders.gl/schema-utils`**

- New module for working with and converting Apache Arrow data.

- **`@loaders.gl/gis`**
- Now provides support for working Apache GeoArrow data.

### Upgrading to v4.4

- The `Source` and `DataSource` APIs have matured leading to some minor breaking changes.
- TBA...

---

## v5.0

Expand All @@ -11,27 +54,7 @@ While it is hard to predict what work will be completed and when, current develo
- More comprehensive support for `options.shape` to control the output format of loaders.
- `ffmpeg` WASM integration for `@loaders.gl/video`

## v4.x

- `TableTileSource` - tile adapter for WMS (support binary in/out)
- `Tile3DSource` - tile source for 3D Tiles
- Better examples, integrate examples into pages, improve drag and drop for testing.
- Better schema support in all loaders
- Option to return Apache Arrow from all loaders
- user controlled logging
- `BasisLoader` - WebGPU / luma v9 compatibility

"Completionist tasks"

- ArcGISImageServerSource
- ArcGISFeatureServerSource
- WFSSource
- WMTSSource
- ...

## v4.0 ✅
**Single output format per loader**

- EcmaScript module support ✅
- More gLTF Extensions: `EXT_mesh_features` and `EXT_structural_metadata` for 1.1 3D Tiles attributes support ✅
- More comprehensive support for `options.shape` to control the output format of loaders. ✅
- Node v18 support. ✅
- The `shape` option that was introduced in loaders.gl v3 to allow loaders to return different data formats is now deprecated and will be removed in many places.
- Instead, applications can use utilities in the `@loaders.gl/schema-utils` and `@loaders.gl/gis` modules to convert for Apache Arrow and Apache GeoArrow to more traditional (but less efficient) JavaScript formats.
29 changes: 2 additions & 27 deletions docs/upgrade-guide.md
Original file line number Diff line number Diff line change
@@ -1,33 +1,8 @@
# Upgrade Guide

## Upgrading to v5.0 (In development)
## Upgrading to v4.3

loaders.gl v5 is a major re-imagination of the library the optimizes loaders.gl for big cloud-native, binary data.

**Apache Arrow as the core format**

- Most loaders now return binary data in the Apache Arrow format.
- This aligns with parallel efforts in companion libraries like deck.gl to work with zero-copy, compact binary data instead of bloated, deserialized javascript data structures.
- Binary columnar data can bring in an order of magnitude better memory usage and load/processing performance on big datasets.

**Single output format per loader**

- The `shape` option that was introduced in loaders.gl v3 to allow loaders to return different data formats is now deprecated and removed in many places.
- Instead, applications can use utilities in the `@loaders.gl/schema-utils` and `@loaders.gl/gis` modules to convert for Apache Arrow and Apache GeoArrow to more traditional (but less efficient) JavaScript formats.

**Apache Arrow JS**

- The Apache Arrow JS library is now a central dependency.
- This does add to the bundle size, especially for the script versions of the loaders where code reuse between bundles is harder.
- The Apache Arrow JS library is powerful but can be hard to work with as it can handle very complex binary data schemas. loaders.gl provides its own set of Apache Arrow docs and a suite of utility functions making it easier

**Improved `DataSource` APIs**

- The `Source` and `DataSource` APIs have matured leading to some minor breaking changes.

## Upgrading to v4.3 (In development)

**Breaking Changes**
**Minor Breaking Changes**

- `loaders.gl/core` - Improved loader type inference: `load()`, `parse()`, etc now infer correct option and return types even when an array of loaders is passed. This can lead to previously undetected type errors now being reported and breaking your build, requiring you to fix the errors.
- `loaders.gl/core` - internal tile index in `Source` classes has changed from `tileIndex.zoom` to `tileIndex.z`. Should not impact most applications.
Expand Down
38 changes: 2 additions & 36 deletions docs/whats-new.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -4,41 +4,7 @@

Target Release Date: Q1 2025

loaders.gl v4.4 will focus on cloud-native, binary data.
A number of modules will expose "ArrowLoaders" will return binary data in the Apache Arrow and Apache GeoArrow formats.

While no loader support has been removed, the flavor of the loaders.gl framework is changing.

**Apache Arrow as a core format**
- Many new loaders now return binary data in the Apache Arrow format.
- This aligns with parallel efforts in companion libraries like deck.gl (as well as the ecosystem at large) to work with zero-copy, compact binary data instead of bloated, deserialized javascript data structures.
- Binary columnar data brings in an order of magnitude better memory usage and improved load/processing performance on big datasets.
- The Apache Arrow JS library is now a central dependency of loaders.gl.

**Improved `DataSource` APIs**
- The `Source` and `DataSource` APIs have matured and are now easier to work with.
- Consule the upgrade guide for migration details.

**Single output format per loader**
- `<LoaderOptions>.shape` is deprecated.
- This will simplify loader implementations.
- Applications benefit from more predictable return types for loaders.
- When needed, applications can now instead use utilities in the `@loaders.gl/schema-utils` and `@loaders.gl/gis` modules to convert Apache Arrow and Apache GeoArrow to more traditional (but less efficient) JavaScript formats.

Per-module changes

- **`@loaders.gl/csv`**
- `CSVArrowLoader` - New CSV loader that returns Apache Arrow tables.

- **`@loaders.gl/parquet`**
- `ParquetArrowLoader` now returns Apache Arrow tables and leverages the high-performance `parquet-wasm` library.
- The v4 Parquet loader is still available as `ParquetJSONLoader`

- **`@loaders.gl/schema-utils`**
- New module for working with and converting Apache Arrow data.

- **`@loaders.gl/gis`**
- Now provides support for working Apache GeoArrow data.
See the [v4.4 Roadmap](/docs/roadmap) for more details.

## v4.3

Expand Down Expand Up @@ -70,7 +36,7 @@ A minor release that includes:

**@loaders.gl/3d-tiles**

- **3D Tiles 1.1 Writing** [`Tile3DWriter`](/docs/modules/3d-tiles/api-reference/tile3d-writer) now supports writing 3DTiles 1.1
- **3D Tiles 1.1 Writing** [`Tile3DWriter`](/docs/modules/3d-tiles/api-reference/tile-3d-writer) now supports writing 3DTiles 1.1
- **3D Tiles 1.1 Extensions** `EXT_mesh_features` and `EXT_structural_metadata` are now supported.

**@loaders.gl/tile-converter**
Expand Down
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
Original file line number Diff line number Diff line change
Expand Up @@ -9,16 +9,16 @@
"start-local": "webpack-dev-server --env.local --progress --hot --open"
},
"dependencies": {
"react": "^16.3.0",
"react-dom": "^16.3.0",
"@probe.gl/bench": "^4.0.2",
"@probe.gl/react-bench": "^4.0.2"
"@probe.gl/react-bench": "^4.0.2",
"react": "^16.3.0",
"react-dom": "^16.3.0"
},
"devDependencies": {
"@loaders.gl/core": "4.4.0-alpha.0",
"@babel/cli": "^7.0.0",
"@babel/core": "^7.4.0",
"@babel/preset-react": "^7.0.0",
"@loaders.gl/core": "4.4.0-alpha.0",
"babel-loader": "^8.0.5",
"babel-plugin-inline-import": "^3.0.0",
"css-loader": "^2.1.1",
Expand Down
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
1 change: 1 addition & 0 deletions modules/arrow/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -60,6 +60,7 @@
"@loaders.gl/gis": "4.4.0-alpha.0",
"@loaders.gl/loader-utils": "4.4.0-alpha.0",
"@loaders.gl/schema": "4.4.0-alpha.0",
"@loaders.gl/schema-utils": "4.4.0-alpha.0",
"@loaders.gl/wkt": "4.4.0-alpha.0",
"@loaders.gl/worker-utils": "4.4.0-alpha.0",
"@math.gl/polygon": "^4.1.0",
Expand Down
1 change: 1 addition & 0 deletions modules/draco/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -59,6 +59,7 @@
"dependencies": {
"@loaders.gl/loader-utils": "4.4.0-alpha.0",
"@loaders.gl/schema": "4.4.0-alpha.0",
"@loaders.gl/schema-utils": "4.4.0-alpha.0",
"@loaders.gl/worker-utils": "4.4.0-alpha.0",
"draco3d": "1.5.7"
},
Expand Down
1 change: 1 addition & 0 deletions modules/gis/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,7 @@
"dependencies": {
"@loaders.gl/loader-utils": "4.4.0-alpha.0",
"@loaders.gl/schema": "4.4.0-alpha.0",
"@loaders.gl/schema-utils": "4.4.0-alpha.0",
"@mapbox/vector-tile": "^1.3.1",
"@math.gl/polygon": "^4.1.0",
"pbf": "^3.2.1"
Expand Down
1 change: 1 addition & 0 deletions modules/las/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,7 @@
"dependencies": {
"@loaders.gl/loader-utils": "4.4.0-alpha.0",
"@loaders.gl/schema": "4.4.0-alpha.0",
"@loaders.gl/schema-utils": "4.4.0-alpha.0",
"laz-perf": "^0.0.6"
},
"peerDependencies": {
Expand Down
3 changes: 2 additions & 1 deletion modules/obj/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,8 @@
},
"dependencies": {
"@loaders.gl/loader-utils": "4.4.0-alpha.0",
"@loaders.gl/schema": "4.4.0-alpha.0"
"@loaders.gl/schema": "4.4.0-alpha.0",
"@loaders.gl/schema-utils": "4.4.0-alpha.0"
},
"peerDependencies": {
"@loaders.gl/core": "4.4.0-alpha.0"
Expand Down
3 changes: 2 additions & 1 deletion modules/pcd/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,8 @@
},
"dependencies": {
"@loaders.gl/loader-utils": "4.4.0-alpha.0",
"@loaders.gl/schema": "4.4.0-alpha.0"
"@loaders.gl/schema": "4.4.0-alpha.0",
"@loaders.gl/schema-utils": "4.4.0-alpha.0"
},
"peerDependencies": {
"@loaders.gl/core": "4.4.0-alpha.0"
Expand Down
3 changes: 2 additions & 1 deletion modules/ply/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,8 @@
},
"dependencies": {
"@loaders.gl/loader-utils": "4.4.0-alpha.0",
"@loaders.gl/schema": "4.4.0-alpha.0"
"@loaders.gl/schema": "4.4.0-alpha.0",
"@loaders.gl/schema-utils": "4.4.0-alpha.0"
},
"peerDependencies": {
"@loaders.gl/core": "4.4.0-alpha.0"
Expand Down
66 changes: 66 additions & 0 deletions modules/schema/src/deprecated/mesh-utils.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,66 @@
// Mesh category utilities
// TODO - move to mesh category module, or to math.gl/geometry module
import {MeshAttributes} from '../categories/category-mesh';
import {TypedArray} from '../types/types';

type TypedArrays = {[key: string]: TypedArray};

/**
* Holds an axis aligned bounding box
* TODO - make sure AxisAlignedBoundingBox in math.gl/culling understands this format (or change this format)
*/
type BoundingBox = [[number, number, number], [number, number, number]];

/**
* Get number of vertices in mesh
* @param attributes
* @deprecated Import from shema-utils
*/
export function getMeshSize(attributes: TypedArrays): number {
let size = 0;
for (const attributeName in attributes) {
const attribute = attributes[attributeName];
if (ArrayBuffer.isView(attribute)) {
// @ts-ignore DataView doesn't have BYTES_PER_ELEMENT
size += attribute.byteLength * attribute.BYTES_PER_ELEMENT;
}
}
return size;
}

/**
* Get the (axis aligned) bounding box of a mesh
* @param attributes
* @returns array of two vectors representing the axis aligned bounding box
* @deprecated Import from shema-utils
*/
// eslint-disable-next-line complexity
export function getMeshBoundingBox(attributes: MeshAttributes): BoundingBox {
let minX = Infinity;
let minY = Infinity;
let minZ = Infinity;
let maxX = -Infinity;
let maxY = -Infinity;
let maxZ = -Infinity;

const positions = attributes.POSITION ? attributes.POSITION.value : [];
const len = positions && positions.length;

for (let i = 0; i < len; i += 3) {
const x = positions[i];
const y = positions[i + 1];
const z = positions[i + 2];

minX = x < minX ? x : minX;
minY = y < minY ? y : minY;
minZ = z < minZ ? z : minZ;

maxX = x > maxX ? x : maxX;
maxY = y > maxY ? y : maxY;
maxZ = z > maxZ ? z : maxZ;
}
return [
[minX, minY, minZ],
[maxX, maxY, maxZ]
];
}
6 changes: 6 additions & 0 deletions modules/schema/src/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -143,3 +143,9 @@ export type {
// FixedSizeList,
// Struct
// } from './lib/table/arrow-api';

// DEPRECATED

// MESH CATEGORY

export {getMeshSize, getMeshBoundingBox} from './deprecated/mesh-utils';
2 changes: 1 addition & 1 deletion modules/tiles/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -51,7 +51,7 @@
"@probe.gl/stats": "^4.0.2"
},
"devDependencies": {
"@deck.gl/core": "^8.9.0"
"@deck.gl/core": "^9.0.33"
},
"peerDependencies": {
"@loaders.gl/core": "4.4.0-alpha.0"
Expand Down
21 changes: 15 additions & 6 deletions package.json
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,9 @@
"KML"
],
"workspaces": [
"modules/*"
"modules/*",
"examples",
"website"
],
"scripts": {
"bootstrap": "ocular-bootstrap",
Expand Down Expand Up @@ -62,11 +64,18 @@
"resolutions": {
"apache-arrow": "^17.0.0",
"typescript": "^5.6.0",
"@cmfcmf/docusaurus-search-local": "~1.1.0",
"@docusaurus/module-type-aliases": "~3.3.2",
"@docusaurus/plugin-client-redirects": "~3.3.2",
"@docusaurus/plugin-content-docs": "~3.3.2",
"@docusaurus/tsconfig": "~3.3.2"
"@loaders.gl/core": "^4.4.0-alpha.0",
"@loaders.gl/draco": "^4.4.0-alpha.0",
"@loaders.gl/geopackage": "^4.4.0-alpha.0",
"@loaders.gl/i3s": "^4.4.0-alpha.0",
"@loaders.gl/las": "^4.4.0-alpha.0",
"@loaders.gl/loader-utils": "^4.4.0-alpha.0",
"@loaders.gl/obj": "^4.4.0-alpha.0",
"@loaders.gl/ply": "^4.4.0-alpha.0",
"@loaders.gl/schema": "^4.4.0-alpha.0",
"@loaders.gl/schema-utils": "^4.4.0-alpha.0",
"@loaders.gl/textures": "^4.4.0-alpha.0",
"@loaders.gl/wms": "^4.4.0-alpha.0"
},
"volta": {
"node": "20.18.0",
Expand Down
Loading

0 comments on commit 7c3a294

Please sign in to comment.