Make hafas-client work in browser / webpack

[yu-re-ka]

hafas-client has a lot of dependencies on node builtins that are not easily filled on browser platforms. my fork currently has some modifications to make this work, but sacrifices nodejs support and features.

It would be great to have both nodejs and browser (with some bundler like webpack) working out of the box.

[derhuerst]

related: #56 (comment)

[derhuerst]

I would like to ask what your plans are:

• Almost all hafas-client profiles won't work in the browser, because their respective HAFAS endpoints don't enable CORS. If you want to run a proxy server anyways, why not run hafas-rest-api@5, which adds CORS and caching out-of-the-box?
• If you want to run hafas-client in browser-like environments without cross-origin restrictions (e.g. Cloudflare Workers, react-native, Deno), we indeed need a way to automatically shim the Node builtins used by hafas-client!

[derhuerst]

It has been a while since I last tried to run hafas-client in the browser; I have just tried it using hafas-client@6.0.1.

CORS proxy

I have modified node_modules/hafas-client/lib/request.js to use a locally-running warp-cors instance:

--- a/node_modules/hafas-client/lib/request.js
+++ b/node_modules/hafas-client/lib/request.js
@@ -156,7 +156,10 @@ const request = async (ctx, userAgent, reqData) => {
 	}
 
 	const reqId = randomBytes(3).toString('hex')
-	const url = profile.endpoint + '?' + stringify(req.query)
+	let url = new URL('http://localhost:3030')
+	url.pathname = profile.endpoint
+	url.search = '?' + stringify(req.query)
+	url = url.href
 	const fetchReq = new Request(url, req)
 	profile.logRequest(ctx, fetchReq, reqId)

Webpack setup

{
  "private": true,
  "name": "hafas-client-web",
  "version": "1.0.0",
  "type": "module",
  "main": "index.js",
  "devDependencies": {
    "assert": "^2.0.0",
    "browserify-zlib": "^0.2.0",
    "buffer": "^6.0.3",
    "crypto-browserify": "^3.12.0",
    "stream-browserify": "^3.0.0",
    "util": "^0.12.5",
    "webpack": "^5.75.0",
    "webpack-cli": "^5.0.1"
  },
  "dependencies": {
    "hafas-client": "^6.0.1"
  }
}

// webpack.config.js
import webpack from 'webpack'
import {createRequire} from 'node:module'
const require = createRequire(import.meta.url)
import {join as pathJoin, dirname} from 'node:path'

export default {
	plugins: [
		new webpack.DefinePlugin({
			'process.env.DEBUG': JSON.stringify(undefined),
			'process.env.NODE_DEBUG': JSON.stringify(undefined),
		}),
	],
	resolve: {
		fallback: {
			// Node builtins
			assert: require.resolve('assert/'),
			buffer: require.resolve('buffer/'),
			crypto: require.resolve('crypto-browserify'),
			stream: require.resolve('stream-browserify'),
			util: require.resolve('util/'),
			zlib: require.resolve('browserify-zlib'),
		},
	},
	experiments: {
		topLevelAwait: true,
	},
}

// index.js
import {createClient} from 'hafas-client'
import {profile} from 'hafas-client/p/db/index.js'

const client = createClient(profile, 'hafas-client bundling experiment')

console.log(await client.locations('hbf'))

npm install
webpack build -o dist --mode development ./index.js
npx serve dist

necessary modifications in hafas-client

The generated bundle doesn't run in a browser as-is:

1. The global Buffer does not exist; This can be fixed easily by importing from node:buffer.
2. The $HTTP_PROXY/$HTTPS_PROXY/$LOCAL_ADDRESS https.Agents can't be bundled by webpack. My workaround was to remove them.
3. Most profiles load ./base.json via module.createRequire(import.meta.url)('./base.json'), which can't be bundled by webpack. I used the dynamic import() API, which is experimental in Node.

I ended up with these modifications:

--- a/node_modules/hafas-client/p/db/index.js
+++ b/node_modules/hafas-client/p/db/index.js
@@ -1,8 +1,3 @@
-// todo: use import assertions once they're supported by Node.js & ESLint
-// https://github.com/tc39/proposal-import-assertions
-import {createRequire} from 'module'
-const require = createRequire(import.meta.url)
-
 import trim from 'lodash/trim.js'
 import uniqBy from 'lodash/uniqBy.js'
 import slugg from 'slugg'
@@ -19,7 +14,7 @@ import {parseLocation as _parseLocation} from '../../parse/location.js'
 import {formatStation as _formatStation} from '../../format/station.js'
 import {bike} from '../../format/filters.js'
 
-const baseProfile = require('./base.json')
+const {default: baseProfile} = await import('./base.json', {assert: {type: 'json'}})
 import {products} from './products.js'
 import {formatLoyaltyCard} from './loyalty-cards.js'
 import {ageGroup, ageGroupFromAge} from './ageGroup.js'

--- a/node_modules/hafas-client/lib/request.js
+++ b/node_modules/hafas-client/lib/request.js
@@ -1,48 +1,11 @@
-import ProxyAgent from 'https-proxy-agent'
-import {isIP} from 'net'
-import {Agent as HttpsAgent} from 'https'
-import roundRobin from '@derhuerst/round-robin-scheduler'
 import {randomBytes} from 'crypto'
 import createHash from 'create-hash'
+import {Buffer} from 'buffer'
 import {stringify} from 'qs'
 import {Request, fetch} from 'cross-fetch'
 import {parse as parseContentType} from 'content-type'
 import {HafasError, byErrorCode} from './errors.js'
 
-const proxyAddress = process.env.HTTPS_PROXY || process.env.HTTP_PROXY || null
-const localAddresses = process.env.LOCAL_ADDRESS || null
-
-if (proxyAddress && localAddresses) {
-	console.error('Both env vars HTTPS_PROXY/HTTP_PROXY and LOCAL_ADDRESS are not supported.')
-	process.exit(1)
-}
-
-const plainAgent = new HttpsAgent({
-	keepAlive: true,
-})
-let getAgent = () => plainAgent
-
-if (proxyAddress) {
-	// todo: this doesn't honor `keepAlive: true`
-	// related:
-	// - https://github.com/TooTallNate/node-https-proxy-agent/pull/112
-	// - https://github.com/TooTallNate/node-agent-base/issues/5
-	const agent = new ProxyAgent(proxyAddress)
-	getAgent = () => agent
-} else if (localAddresses) {
-	const agents = process.env.LOCAL_ADDRESS.split(',')
-	.map((addr) => {
-		const family = isIP(addr)
-		if (family === 0) throw new Error('invalid local address:' + addr)
-		return new HttpsAgent({
-			localAddress: addr, family,
-			keepAlive: true,
-		})
-	})
-	const pool = roundRobin(agents)
-	getAgent = () => pool.get()
-}
-
 const id = randomBytes(3).toString('hex')
 const randomizeUserAgent = (userAgent) => {
 	let ua = userAgent
@@ -114,7 +77,6 @@ const request = async (ctx, userAgent, reqData) => {
 	})
 
 	const req = profile.transformReq(ctx, {
-		agent: getAgent(),
 		method: 'post',
 		// todo: CORS? referrer policy?
 		body: JSON.stringify(rawReqBody),

proposed modifications

• let lib/request.js import Buffer
• move the Agent-related stuff into a separate file, so that it can be shimmed to null using the webpack config
• add instructions to the docs on how to configure webpack when bundling hafas-client
• adopt dynamic import() once it is marked as stable in Node

What do you think?

[yu-re-ka]

I would like to ask what your plans are:

* Almost all `hafas-client` profiles won't work in the browser, because their respective HAFAS endpoints don't enable [CORS](https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS). If you want to run a proxy server anyways, why not run [`hafas-rest-api@5`](https://github.com/public-transport/hafas-rest-api/tree/5), which adds CORS and caching out-of-the-box?

* If you want to run `hafas-client` in browser-like environments without cross-origin restrictions (e.g. Cloudflare Workers, react-native, Deno), we indeed need a way to automatically shim the Node builtins used by `hafas-client`!

I run hafas-client in the browser for trainsear.ch with a CORS proxy. The CORS proxy is just a dumb http server (in my case httproxide but really it could be any http server / reverse proxy).
There are many reasons why I prefer run hafas-client in the client rather than on the server. The hafas response format is actually not too bad for being transmitted over a network connection. Especially when Polyline parsing is enabled, the hafas-client parsed version of the response is gigantic compared to the data returned by hafas.
But it also means there is no additional versioning between the hafas-client api and the code that uses the data needed, since they are always delivered as a bundle.

[yu-re-ka]

The proposed modifications look good, that would improve my experience and hopefully make my fork unnecessary at one point

[derhuerst]

The hafas response format is actually not too bad for being transmitted over a network connection. Especially when Polyline parsing is enabled, the hafas-client parsed version of the response is gigantic compared to the data returned by hafas.

Using gzip with default settings, the hafas-client-formatted version is about 2x the size, so I see your point.

[derhuerst]

• let lib/request.js import Buffer
• move the Agent-related stuff into a separate file, so that it can be shimmed to null using the webpack config
• add instructions to the docs on how to configure webpack when bundling hafas-client
• adopt dynamic import() once it is marked as stable in Node

The proposed modifications look good […].

In c2a71b0, I have done the 1st task. Whoever wants to get started on this can work on the 2nd and 3rd task right away. With the 4th task (dynamic import()), I'd like to wait until it (hopefully) has become stable.

[derhuerst]

I have published c2a71b0 as hafas-client@6.0.2.

[yu-re-ka]

I fear the import Buffer from 'node:buffer' version does not work with webpack at all.
Assuming import Buffer from 'buffer' does not work on nodejs, best way forward for that part that I now see is removing the import again and then using webpack's ProvidePlugin to add it to the global scope.