Webpack and other bundler support

[callmehiphop]

Seems like a lot of users want to use webpack with a number of the nodejs client libraries.

Threads where this comes up:

• Add support for angular nodejs-dialogflow#390
• Webpack compatibility nodejs-dialogflow#16
• Can I use this package in react-native mobile app ? nodejs-speech#367
• Running cloud trace with webpack cloud-trace-nodejs#774
• vuejs module  nodejs-dialogflow#551

Threads demonstrating use of fallback true:

• using fallback=true with speech.

[alexander-fenster]

There are short-term and long-term plans for this.

The short-term solution is to make google-gax webpack'able by replacing all the gRPC specifics with grpc-fallback, which is, essentially, sending serialized protobuf to HTTP/1 endpoints. Our two interns, @LibanOdowa and @ramya-ramalingam, are working on this project. We expect it to be done some time this June/July.

The long-term solution is to use grpc-web library, which (unfortunately) is not compatible to grpc / @grpc/grpc-js in terms of the interface it provides, so plugging it into gax would not be that easy. @eoogbe is working on making grpc-web work for our use cases.

[guybedford]

We've been able to support Webpack builds of google cloud libraries in ncc through the use of static analysis in our asset relocation loader (https://github.com/zeit/webpack-asset-relocator-loader), which should support both client and server use cases. We've been fixing bugs as we find them on the grpc libraries, and need to support this anyway.

If it would be useful to provide some examples of browser workflows with this approach we could gladly assist with this.

[bcoe]

@guybedford 👋 would love to work with you to start adding some web examples over time.

[guybedford]

@bcoe more than happy to share how we achieved this with ncc, and possibly even make our approach available to other webpack users, but it does involve an asset analysis running against all source code to determine the ".proto" references.

Having a fallback JSON-loading / browser bundling approach seems to make sense though architecturally for this project. Perhaps a simple "browser": { "./loader.js": "./loader-browser" } mapping in the package.json could fork between the loader for Node.js and browsers, switching between the binary and JSON formats appropriately. I think that's a well-enough established pattern to work for users, and might simplify the detections and code inclusions.

[bcoe]

@alexander-fenster is there a thread you can link to here that demonstrates how to use the grpc-fallback, I know you documented this in another thread.

[alexander-fenster]

@bcoe For using it with Node.js and service account, just pass fallback: true in the client constructor and it will go on the fallback route without gRPC and will use require() to read the JSON proto file:

const fooClient = new client.FooClient({fallback: true});

To use in non-Node setting, one cannot use service account and will need to use OAuth2 workflow. It can be done using the following code. It really depends on how your application works (if it's a website, or an Electron app, or something else), so I'm just showing the general idea of performing the OAuth2 workflow leaving the specifics behind (e.g. how to redirect the user to auth URI, how to extract the auth code from the query string, etc.)

The parameters client_id and client_secret should be obtained from Google Cloud Console.

const client = require('@google-cloud/foo'); // any GAPIC library

const auth = require('google-auth-library');
const oauth2client = new auth.OAuth2Client(client_id, client_secret, callback_uri);
const authUrl = oauth2client.generateAuthUrl({
  access_type: 'offline',
  scope: client.FooClient.scopes
});
// redirect user to authUrl and wait for them coming back to callback_uri

// in callback_uri handler, get the auth code from query string and obtain a token:
const tokenResponse = await oauth2client.getToken(code);
oauth2client.setCredentials(tokenResponse.tokens);

// now use this oauth2client!
const fooClient = new client.FooClient({ auth: oauth2client }); // <-- auth passed here

If the library detects it's running in a browser (has a window global object), it will follow the fallback route automatically (no need to pass fallback: true).

[ddittoPersevere]

I am very new to development, so please forgive me if I'm overlooking the answer to this...
I am using the Google Drive api as a database of sorts to send json from an Electron application to the Drive, and then I need to retrieve that json in a web application. I'm having to do this due to the Electron application being used offline.
Since the web application uses webpack (using React and Redux), it won't let me pull the data from the Drive, without running the script that pulls the data independently (command line).

ERROR in ./node_modules/google-auth-library/build/src/auth/googleauth.js
Module not found: Error: Can't resolve 'child_process' in 'C:\Users\SSC05\Desktop\Class\indecision\pid_web\node_modules\google-auth-library\build\src\auth'
 @ ./node_modules/google-auth-library/build/src/auth/googleauth.js 16:24-48
 @ ./node_modules/google-auth-library/build/src/index.js
 @ ./node_modules/googleapis-common/build/src/index.js
 @ ./node_modules/googleapis/build/src/googleapis.js
 @ ./node_modules/googleapis/build/src/index.js
 @ ./src/scripts/dataPull.js
 @ ./src/routers/AppRouter.js
 @ ./src/app.js

ERROR in ./node_modules/https-proxy-agent/index.js
Module not found: Error: Can't resolve 'net' in 'C:\Users\SSC05\Desktop\Class\indecision\pid_web\node_modules\https-proxy-agent'
 @ ./node_modules/https-proxy-agent/index.js 5:10-24
 @ ./node_modules/gaxios/build/src/gaxios.js
 @ ./node_modules/gaxios/build/src/index.js
 @ ./node_modules/google-auth-library/build/src/transporters.js
 @ ./node_modules/google-auth-library/build/src/index.js
 @ ./node_modules/googleapis-common/build/src/index.js
 @ ./node_modules/googleapis/build/src/googleapis.js
 @ ./node_modules/googleapis/build/src/index.js
 @ ./src/scripts/dataPull.js
 @ ./src/routers/AppRouter.js
 @ ./src/app.js

ERROR in ./src/scripts/dataPull.js
Module not found: Error: Can't resolve 'readline' in 'C:\Users\SSC05\Desktop\Class\indecision\pid_web\src\scripts'
 @ ./src/scripts/dataPull.js 3:15-34
 @ ./src/routers/AppRouter.js
 @ ./src/app.js

ERROR in ./node_modules/https-proxy-agent/index.js
Module not found: Error: Can't resolve 'tls' in 'C:\Users\SSC05\Desktop\Class\indecision\pid_web\node_modules\https-proxy-agent'
 @ ./node_modules/https-proxy-agent/index.js 6:10-24
 @ ./node_modules/gaxios/build/src/gaxios.js
 @ ./node_modules/gaxios/build/src/index.js
 @ ./node_modules/google-auth-library/build/src/transporters.js
 @ ./node_modules/google-auth-library/build/src/index.js
 @ ./node_modules/googleapis-common/build/src/index.js
 @ ./node_modules/googleapis/build/src/googleapis.js
 @ ./node_modules/googleapis/build/src/index.js
 @ ./src/scripts/dataPull.js
 @ ./src/routers/AppRouter.js
 @ ./src/app.js

This, of course, isn't going to work for the web application, because it needs to pull data from the Drive every time the site is visited.
I know this is a very specific situation, but I'm just not sure where to implement the 'fallback-route', or if that is even what needs to occur.
Again, please forgive me if this is basic knowledge.

[alexander-fenster]

Hi @ddittoPersevere, thank you for asking! It's possible to use the Drive library in your web application if you prepare a separate Webpack bundle by following these steps. Please let us know if anything is unclear!

[alexander-fenster]

@en4letto If you pass {fallback: true}, the execution should go to node_modules/google-gax/build/src/fallback.js. There is just one fetch call there, could you add some console.log statements to see what kind of fetch is being called there? Should be window.fetch, not the one from node-fetch.

[en4letto]

@alexander-fenster thanks for the insight!

I did some additional checks, here's what I found:
As you said the problematic fetch lies in the node_modules/google-gax/build/src/fallback.js, the code is:

const fetch = isbrowser_1.isBrowser()
                    ? // eslint-disable-next-line no-undef
                        window.fetch
                    : nodeFetch;

I've added some logs and the code correctly assigns the nodeFetch variable to the fetch one.

I found an issue about webpack and node-fetch and, as suggested in this comment, I tried to change
const nodeFetch = require("node-fetch");
to
const nodeFetch = require("node-fetch").default;.

WIth this edit the code works perfectly!
It seems that the problem is related to webpack and node-fetch, so the speech library is not directly involved.

Do you happen to have some suggestions on how to possibly apply this fix when bundling the code with webpack?

Thanks again!

[alexander-fenster]

Thank you for these details! I will test if adding .default does not break any of the scenarios covered by CI, and can just release google-gax with this fix. If we are lucky it will happen early next week and so you won't need to make any hacks to temporary work around this.

Having said that - @en4letto can you tell us more about the environment you're running your code in? If this is an express app that runs with Node, why do you need to webpack it? Fallback mode was initially supposed to help run the code in browsers (where gRPC is not available), so I just want to understand your use case and maybe make some recommendations.

[en4letto]

@alexander-fenster I am setting up a production environment in GCP. I'm using the managed instance groups, and I wanted the lowest startup time possible for the instances inside the node group.
The deploy is managed via Gitlab CI and the build job (npm install and webpack bundling) is done by a "service" instance (a simple compute engine VM outside of the groups).
I didn't want to move the node_modules folder between different VMs and I found a way to bundle the whole express app in a single JS file with webpack so that the instances inside the managed group only have to get this JS bundle and execute it with node.

Hope this is clear! If you have some recommendations and suggestions I'll be happy to know them, as Node.js is not my specialty :)

[denniskribl]

Hey @alexander-fenster @en4letto,

is there an issue which I can follow for the google-gax node-fetch thing? Had no luck finding the right one. Trying to use the @google-cloud/recaptcha-enterprise package with the fallback option. We are bundling our node lambdas with webpack and running into this error.

Thanks in advance :)

[gmcdev]

@alexander-fenster confirming that I run across this fetch problem in an Electron environment, as you noted above

[MrHash]

I was generally able to workaround webpack issues using { fallback: true } in client constructors with target: 'node' & { resolve: { mainFields: ['main'] } } (https://webpack.js.org/configuration/resolve/#resolvemainfields) in the webpack config. This also seems to work for other libs using gax.

[alexander-fenster]

The fix is coming to all the libraries that will stop using fs and will start using require to load the JSONs instead. It should help all of you folks who use webpack with target: 'node'. Stay tuned :)

[ghost]

Any updates on this? I need to use @google-cloud/trace-agent in my cloudfunctions bundled via esbuild.

[alexander-fenster]

@s-dschro any update on what exactly? :) This issue is a collection of various problems related to webpack and/or other bundlers. What is the exact error you have with @google-cloud/trace-agent?

[ghost]

@alexander-fenster googleapis/cloud-trace-nodejs#1393

[alexander-fenster]

Thanks @s-dschro, I'll try to route that issue as appropriate.

[JJPell]

@bcoe For using it with Node.js and service account, just pass fallback: true in the client constructor and it will go on the fallback route without gRPC and will use require() to read the JSON proto file:

const fooClient = new client.FooClient({fallback: true});

To use in non-Node setting, one cannot use service account and will need to use OAuth2 workflow. It can be done using the following code. It really depends on how your application works (if it's a website, or an Electron app, or something else), so I'm just showing the general idea of performing the OAuth2 workflow leaving the specifics behind (e.g. how to redirect the user to auth URI, how to extract the auth code from the query string, etc.)

The parameters client_id and client_secret should be obtained from Google Cloud Console.

const client = require('@google-cloud/foo'); // any GAPIC library

const auth = require('google-auth-library');
const oauth2client = new auth.OAuth2Client(client_id, client_secret, callback_uri);
const authUrl = oauth2client.generateAuthUrl({
  access_type: 'offline',
  scope: client.FooClient.scopes
});
// redirect user to authUrl and wait for them coming back to callback_uri

// in callback_uri handler, get the auth code from query string and obtain a token:
const tokenResponse = await oauth2client.getToken(code);
oauth2client.setCredentials(tokenResponse.tokens);

// now use this oauth2client!
const fooClient = new client.FooClient({ auth: oauth2client }); // <-- auth passed here

If the library detects it's running in a browser (has a window global object), it will follow the fallback route automatically (no need to pass fallback: true).

@alexander-fenster When trying the non-Node sample provided within a React app, I get this error message:

Module not found: Can't resolve 'http2' in ...

Any ideas what could be causing this?

[alexander-fenster]

@JJPell http2 only exists in Node.js and is used by gRPC transport. In React app, try using fallback: true when you create an instance of the client, and set your bundler config to ignore http2.

[Stevemoretz]

Hi I'm trying to use this package or https://github.com/google/google-api-javascript-client for using google tag manager on react-native, react-native has fetch and no window object, anyone has a working build configuration for it?

[ErwinThompsonF]

Hi I'm trying to use google cloud secret manager but this error keeps showing up whenever I try to import the SecretManagerServiceClient I tried following the documentation here but still can't make it run I also tried adding fallback: true to the constructor but it keep showing this error

There are multiple errors that are similar to this error message so I didn't copy them but essentially all error are coming from tunnel-agent

./node_modules/tunnel-agent/index.js:6:12-28 - Error: Module not found: Error: Can't resolve 'https' in '/[file path]/node_modules/tunnel-agent'

BREAKING CHANGE: webpack < 5 used to include polyfills for node.js core modules by default.
This is no longer the case. Verify if you need this module and configure a polyfill for it.

If you want to include a polyfill, you need to:
        - add a fallback 'resolve.fallback: { "https": require.resolve("https-browserify") }'
        - install 'https-browserify'
If you don't want to include a polyfill, you can use an empty module like this:
        resolve.fallback: { "https": false }

sample code

async accessSecret() {
        const client = new SecretManagerServiceClient({ fallback: true });
        const [secret] = await client.accessSecretVersion({
            name: 'projects/my-project/secrets/my-secret',
        });

        // The secret payload is a base64-encoded string.sa
        const payload = secret.payload.data.toString();

        console.log(`The secret payload is: ${payload}`);
        console.log('test')
    }

btw I am using angular 14.2.12 and using google cloud secret manager version 5.2.0

Any help would be much appreciated thank you.