Passing binary data to the Accessors - GeoParquet/GeoArrow

[janneskruseMapular]

Hi! I was recently trying to catch up on the awesome work of Kyle Barron (e.g. here: https://observablehq.com/@kylebarron/geoparquet-on-the-web). However, I was unable to get it to run with Deck.gl.

What I am trying to do is, to render GeoArrow or Geoparquet converted to GeoArrow as a layer with deck. I was reading the documentation on how to pass binary data to Deck gl. and WebGl in general:
https://developer.mozilla.org/en-US/docs/Web/API/WebGLRenderingContext/vertexAttribPointer
https://deck.gl/docs/api-reference/core/layer#basic-properties
https://deck.gl/docs/developer-guide/performance

And so I think I would need to pass the data to the accessors via the attributes on the data object like so for coordinates (size 2: x, y) and colors (size 3: r,g,b):

  const data = [{
    length: numPoints, // jsTable.numRows,
    attributes: {
      getPosition: {value: flatCoordinateArray, size: 2 },
      getFillColor: { value: colorAttribute, size: 3 }
    },
  }];

If I do so, by either using real data as GeoArrow or creating random data, I get all my data points rendered at origin (0,0). Does anyone have an idea what could cause that and how to solve it?
Thanks a lot in advance!

This is my code for the map component:

import React, { use, useEffect, useRef, useState } from "react";
import {Map, useControl} from 'react-map-gl';
import {MapboxOverlay} from '@deck.gl/mapbox';
import {DeckProps, MapView} from '@deck.gl/core';
import {ScatterplotLayer} from '@deck.gl/layers';
import 'mapbox-gl/dist/mapbox-gl.css';
import wasmInit, { readParquet } from "parquet-wasm";
import { readGeoParquet } from "@geoarrow/geoparquet-wasm";
import { Table, tableFromIPC } from "apache-arrow";
import { fromArrow, names } from 'arquero';
import * as d3 from 'd3';

const MAPBOX_ACCESS_TOKEN = process.env.NEXT_PUBLIC_MAPBOX_ACCESS_TOKEN;

const Mapstyles = {
  dark: "mapbox://styles/donpatronstudios/clwkuqnvu00w101r0g1wpcs2s",
  satellite: "mapbox://styles/donpatronstudios/clwnx0fxo00w801nygohw9lq0",
  "satellite-dark":"mapbox://styles/donpatronstudios/clgdxjsmv000j01p4t48jmcgs"
}; 

function DeckGLOverlay(props: DeckProps) {
  const overlay = useControl<MapboxOverlay>(() => new MapboxOverlay(props));
  overlay.setProps(props);
  return null;
}

function getTooltip({object}) {
  return object && object.message;
}


async function fetchData() {
  try {

    const PARQUET_WASM_VERSION = "0.6.1";
    const PARQUET_WASM_CDN_URL = `https://cdn.jsdelivr.net/npm/parquet-wasm@${PARQUET_WASM_VERSION}/esm/parquet_wasm_bg.wasm`;
    // const wasmFile = await fetch("/index_bg.wasm");
    // console.log("Wasm file", wasmFile);
    const wasmFile = await fetch(PARQUET_WASM_CDN_URL);
    await wasmInit(wasmFile);
    // // Fetch Feather file
    const resp = await fetch("/data/2019-01-01_performance_mobile_tiles_centroids_brotli.parquet");
    if (!resp.ok) {
      throw new Error(`Failed to fetch Parquet file: ${resp.statusText}`);
    }
    // convert the response to an Arrow Table
    // const arrayBuffer = await resp.arrayBuffer();
    const parquetBytes = new Uint8Array(await resp.arrayBuffer());
    // GeoParquet encoded
    // const wasmTable = readGeoParquet(new Uint8Array(arrayBuffer));
    const wasmTable = readParquet(parquetBytes, {
      batchSize: Math.pow(2, 31)
    });
    const jsTable = tableFromIPC(wasmTable.intoIPCStream());
    console.log("Arrow table", jsTable);

    // let dt = fromArrow(jsTable);
    // console.log("Arquero table", dt);
    const geometryColumn = jsTable.getChild('geometry');
    let flatCoordinateArray = geometryColumn.getChildAt(0).data[0].values;
    const colorsColumn = jsTable.getChild('colors');
    const flatColorArray = colorsColumn.getChildAt(0).data[0].values;
    console.log("Geometry column", flatCoordinateArray);
    console.log("Colors column", flatColorArray);


    const metricMapping = {
        "Download speed (kbps)": "avg_d_kbps",
        "Upload speed (kbps)": "avg_u_kbps",
        "Latency (ms)": "avg_lat_ms"
    }

    const form = {
      metric: "Download speed (kbps)",
      lowerBound: "1000",
      upperBound: "30000"
    };

    const colorScale = (values, lowerBound, upperBound) => {
      const linearScale = d3.scaleLinear()
        .domain([parseFloat(lowerBound), parseFloat(upperBound)])
        .range([0, 1]);

      const outputArray = new Float32Array(values.length * 3);
      for (let i = 0; i < values.length; ++i) {
        const value = values[i];
        const color = d3.color(d3.interpolateViridis(linearScale(value)));
        outputArray[i * 3] = color.r / 255;
        outputArray[i * 3 + 1] = color.g / 255;
        outputArray[i * 3 + 2] = color.b / 255;
      }

      return outputArray;
    };

    const getColorAttribute = () => {
      const colName = metricMapping[form.metric];
      const column = jsTable.getChild(colName);
      return colorScale(column.data[0].values, form.lowerBound, form.upperBound);
    };
    let colorAttribute = getColorAttribute();

    console.log("Color attribute", colorAttribute);


    // Number of points you want to generate
    const numPoints = 100;

    // Generate random coordinates
    // Assuming these coordinates are latitude and longitude
    // Latitude range: -90 to 90, Longitude range: -180 to 180
    flatCoordinateArray = new Array(numPoints * 2).fill(0).map(() => {
      return Math.random() > 0.5
        ? Math.random() * 180 // Longitude
        : Math.random() * -180; // Latitude
    });

    // Generate random RGB colors
    colorAttribute = new Array(numPoints * 3).fill(0).map(() => {
      return Math.floor(Math.random() * 256); // RGB component
    });

    console.log("Coordinates", flatCoordinateArray);
    console.log("Colors", colorAttribute);


    // Create the data object for the ScatterplotLayer (size: 2 for x, y and size: 3 for RGB)
    const data = [{
      length: numPoints, // jsTable.numRows,
      attributes: {
        getPosition: {value: flatCoordinateArray, size: 2 },
        getFillColor: { value: colorAttribute, size: 3 }
      },
    }];

    return data;
  }
  catch (error) {
    console.error("Error fetching data", error);
  }
}

function InteractiveMap(): JSX.Element {
  const [layers, setLayers] = useState([
    new ScatterplotLayer({
      id: 'deckgl-circle',
      data: [
        {position: [0.45, 51.47]}
      ],
      getPosition: d => d.position,
      getFillColor: [255, 0, 0, 100],
      getRadius: 1000,
      beforeId: 'waterway-label' // In interleaved mode render the layer under map labels
    })
  ]);

  useEffect(() => {
    fetchData().then(data => {
      setLayers([
        new ScatterplotLayer({
          id: 'internet-speeds',
          data: data,
          // getNormal: [0, 0, 1],
          getRadius: 100,
          radiusMinPixels: 20,
          // beforeId: 'waterway-label', // In interleaved mode render the layer under map labels
          pickable: true,
        })
      ]);
    });
  }, []);

  useEffect(() => {
    console.log("Layers", layers);
  }
  , [layers]);

  return (
    <Map
      style={{  height: "100vh" }} // width: "100%",
      initialViewState={{
        longitude: 0.45,
        latitude: 51.47,
        zoom: 4
      }}
      mapStyle= "mapbox://styles/mapbox/dark-v9" //"https://basemaps.cartocdn.com/gl/positron-nolabels-gl-style/style.json" // // {Mapstyles.dark}
      mapboxAccessToken={MAPBOX_ACCESS_TOKEN}
    >
      <DeckGLOverlay layers={layers} interleaved />
    </Map>
  );
}

const InteractiveMapMemo = React.memo(InteractiveMap);
export default InteractiveMapMemo;

[ibgreen]

If you want to see some source code, kepler.gl 3.0 has a GeoArrowLayer implementation, and also the GeoArrowLayers from lonboard are staged in (but not yet published from) deck.gl-community/modules/arrow-layers.

[janneskruseMapular]

Perfect, thank you! I'll have a look at that

[janzkyle]

@janneskruseMapular have you found a source code containing geoparquet samples? I only found samples with arrow/feather as the data source but can't find any that has geoparquet.

[janneskruseMapular]

@janzkyle I think you would need to use WASM in a similar way I used it above. There is this cool package: https://github.com/geoarrow/deck.gl-layers from Kyle Barron, maybe you can steel from it. However, I could not get it to run so far for the points layer (internet data) I tried.

[ibgreen]

FWIW, I started an effort to move @kylebarron 's deck.gl-layers into the deck.gl-community repo, as a first step to incubating them towards eventual inclusion as a core deck.gl layer package.

• https://github.com/visgl/deck.gl-community/tree/master/modules/arrow-layers
• https://visgl.github.io/deck.gl-community/docs/modules/arrow-layers

Unfortunately the new module still needs a bit more work before being usable, so I am basically just suggesting that if people are trying to get these layers working, there is an opportunity to collaborate on that in the deck.gl-community repo.

[kylebarron]

However, I could not get it to run so far for the points layer (internet data) I tried.

You need to ensure you're using native geoarrow types, not WKB-encoded geometries.

I only found samples with arrow/feather as the data source but can't find any that has geoparquet.

You would need a GeoParquet implementation that is able to parse the WKB-encoded geometries to native GeoArrow coordinates. There's a pure-JS implementation in loaders.gl that @ibgreen has worked on, or you can look at my rust-based WebAssembly implementation. The latest version isn't currently published to NPM but you can run npm run build here to build it yourself.

Unfortunately the new module still needs a bit more work before being usable, so I am basically just suggesting that if people are trying to get these layers working, there is an opportunity to collaborate on that in the deck.gl-community repo.

I haven't had time recently to put into the JS bindings (but on the bright side they seem to be relatively stable), but if anyone is interested in updating deck.gl-community, I could review PRs.