How to use/bundle the web3 package correctly? #224
Comments
|
Hi @rhlsthrm , this sound like your project setup triggered the Serverless native packaging for some reason, which should not be. With version In your Notice the missing BTW: Since Webpack 2.x the loaders should be configured in |
|
FYI: To analyze such behavior you can add |
|
@HyperBrain thanks for the response. How can I tell if the setup triggers the Serverless native packaging? I want to make sure it's optimizing the packaging with webpack properly. Also I am using My handlers are configured as you mentioned. Thanks for the tip on configuring webpack for version 3. Here is my new config export: |
|
@rhlsthrm Apologies for the typo, yes it is |
|
Thanks again for the help! Packaging has been running for quite a while and this is some of the output: I see that some of these functions are taking a long time to zip (293524 ms, 308840 ms). How can I go about finding out what is taking these particular functions so long to pack (there is not a lot of code so it must be the dependencies)? Any ideas on what I can do to make it better? One thing I am noticing is all of these that are taking a long time are using web3 so I wonder if that particular dependency is causing issues. Edit: The packaging step finished with an error. This is the full trace: I am not sure how to debug that final error. Any ideas on where to look? |
|
Ok. What is very interesting are the All other packaging parts are ok - if you do individual packaging it is expected that it needs more time as every function is optimized separately including its referenced modules. So the copy/prune/zip with about 10-15 seconds per function is ok. BUT: There is definitely something wrong with the ZIP function at Can you check, if there is something special with the 2 zips that are finally in |
|
Nothing weird with the zip itself, but the node_modules folder is >300MB: Looking into it, I see that the web3 library is indeed >300MB itself! This must be what is causing the issue. I guess I need to see if I can somehow reduce the size of the library, or only import what I need? Edit: I looked at the web3 package that I have installed with |
|
Yes. That's indeed a problem. You could do one experiment: try to bundle the web3 library with webpack, so that it only fetches code and dependencies used. That might reduce the size dramatically. You can do that with: This will let Webpack bundle the module and the webpack plugin will only package the remaining 2nd level dependencies that are used by the code of web3 that is actually required. Just give it a try - although I cannot give any guarantee that it will work as expected ;-) |
|
That didn't seem to work, I'm getting warnings and errors: If you look at my edit, you can see that the installed package on the disk is much, much smaller. Something is happening when the package is getting processed by webpack. Any idea what I can do? Thanks again for all the help. |
|
Right now I have not really an idea what can be done as I never used the web3 package nor packaged it with any service. I'd have to do some tests with it first. |
HyperBrain
changed the title
|
Thanks for reaching out. Is this problem unique to web3? I'm thinking it could be more generic where some packages bundle into something much bigger than what is desired, and there is some workaround for that. You have never seen an issue where the bundled size is something crazy like this? |
|
Honestly - No 😃 . The biggest module that we used was the unicode7 module, but we bundled that on purpose as the code needed it. And in general, our lambda zips at work are all between 5 and 20 MB. The question is, if web3 is intended to run within a Lambda (as opposed to a server environment) as is or if you might just include/bundle the built js (from the dist directory) which is much smaller (in case it works properly in the node environment). Maybe you should also ask the question on https://gitter.im/ethereum/web3.js |
I'm not sure what you mean by this. There shouldn't be any distinction to the library whether or not it is running on lambda versus a server right? I'm just finding it strange that the size on disk is normal, and only balloons to that insane size after i run the webpack command on it. |
|
I did some tries with web3 and experienced some strange installation issues (errors while |
|
@HyperBrain thank you so much for the help. It appears I'm on the cutting edge here and hopefully all these pains will solve other's problems in the future. I'll wait to see if @emrosenf can help. |
|
@HyperBrain unfortunately the dist/ versions of the library don't work in a node environment and this issue is blocking me now. How can I help to debug this further? If you keep helping me that would be much appreciated! I'll comment on the other issue to see if I can get some help there. Is the code you pushed to fix that issue in the release now? |
|
@HyperBrain <https://github.com/hyperbrain> sadly v3.0.0 did not work with
lerna and I ended up extracting it from the project. I haven't had time to
make you a sample repo yet.
…On Mon, Sep 18, 2017 at 3:48 PM, Rahul Sethuram ***@***.***> wrote:
@HyperBrain <https://github.com/hyperbrain> unfortunately the dist/
versions of the library don't work in a node environment and this issue is
blocking me now. How can I help to debug this further? If you keep helping
me that would be much appreciated! I'll comment on the other issue to see
if I can get some help there. Is the code you pushed to fix that issue in
the release now?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#224 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AAZ-Mz0Lg3s9HivHJFxXZgNI9AmPshMFks5sjvNOgaJpZM4PW9FA>
.
|
|
@rhlsthrm I just found this: https://github.com/uzyn/web3-loader. This loader might be able to remove the web3 dependency from the bundled code and according to the documentation
If that works you should end up with minimal dependencies as you only import the contract and web3 through it. As soon as web3 isn't a dependency of the compiled code anymore, it could work. In general the compilation that is triggered together with If that does not help, you could add some logging to either Additionally you could prepare a minimal project with which I would be able to reproduce the issue. |
|
@HyperBrain this is interesting, but I'm not sure it will work for me. This doesn't look like it will deploy contracts on demand programmatically, but rather it's designed to deploy contracts with a new build or else use previously deployed contracts. I'd love to keep trying to get this to work, since enabling web3 functionality inside a lambda function would be great for the whole Ethereum development community as well! |
|
After some deeper thoughts about the topic it seems to me that the bytecode compilation should be done locally BEFORE serverless triggers the webpack compile. In fact the web3 package is not really needed within the deloy phase in that case. The Serverless deploy step should just upload the contract as well as the JS code in one transaction. Such a scenario would be best implemented as new Serverless plugin, that takes care of the compile and hooks into the deploy event to deploy the contracts. However a "non-deploying" derivate of the webpack plugins (web3 and sol loaders) would be needed, so that the new Serverless plugin can take care of the contract deployment. |
|
@HyperBrain not sure I understand exactly what you are saying here. Web3 is needed as a dependency in order to interact with the Ethereum blockchain. In my app, I need web3 to deploy my Solidity contracts, as well as interact with contracts that are already on the blockchain. It is an in-app dependency that I need when my endpoint code runs. I wonder is there a way to package it with all the dependencies into a sort of "binary" package that can be uploaded to lambda as is? |
|
@HyperBrain major update to share. I updated all my packages in my system and things are behaving VERY different now, seemingly in a good way. When I run Can you help me to figure out what's going on with that error? |
|
That sounds good 😄 . Can you do the package command again with |
|
Sorry, should have done that first, the message explicitly says to 😛. |
|
That seems to be a bug with the function detection in the plugin. There seems to be a mismatch between the fetched functions and the actual entries. I think I can fix that, but I'll need a copy of the |
|
I expand the entry map programmatically. This is what my webpack config looks like: There appears to be other problems that could be caused by things not getting imported correctly but I will deal with those later. |
|
I think the models are already part of the slsw.lib.entries because they are defined as functions, so you should not add them manually. In general function entries must not be set because that will destroy the reverse lookup into Serverless for individual packaging. |
|
BTW: The contract json files are needed as is in the app, i.e. you require them from the code? Then it would be better to bundle them with either the copy file webpack plugin or the file-loader. Then you could use the "raw" slsw.lib.entries without any modification and let webpack bundle the json files. |
|
Yes, the contract json files are needed as is. OK I will look into that, I haven't used webpack much so I'll have to research a little but sounds straightforward. |
|
So try to use just // webpack config
module: {
rules: [
{
test: /\.json$/,
use: [
{
loader: 'file-loader',
options: {}
}
]
}
]
}This will bundle the json files and each require() works as expected. By default the loader renames the jsons (and the requires) to |
|
Thank you so much for all the help! I am very close to having everything working. However now I am running into a couple more issues. I don't see either the models or the contracts appearing in the How does the detection of the required files work? Does the file have to be explicitly In the case of the model, it is imported through Sequelize like so: For the contract, it is actually required by the code: Is there anything else to do to make sure webpack is including the files properly? |
|
Never mind, I fixed that issue by manually adding the files to the entries list and now they show up in the build. Still hitting an error with my endpoint here: I'll keep digging into this more, I'm not sure why that is failing. If you have any ideas let me know. Once this is built successfully then I can close the issue. |
|
I searched a bit about |
Sorry can you elaborate on this? Not sure exactly what you mean. Thank you again, you have truly been the most helpful open source dev I've worked with. I would love to donate to your project if you have a way to do that. |
|
I mean that you should use This will let Webpack replace the As mentioned in the Import section here: http://docs.sequelizejs.com/manual/tutorial/models-definition.html
Sorry, but not yet ;-) |
|
I tried this and unfortunately doesn't work... I'll keep poking on it. EDIT: Also my error is different than what the documentation in that section is showing. My full stack trace is: |
|
I think my issue now is out of the scope of this issue, so I will close it. To summarize: |
|
@rhlsthrm Nevertheless thanks for the good discussion. It might help others as starting point if they run into the same issues, So this thread has not been in vain 😃 |
|
Definitely! It has been a great learning experience for me too. I plan on writing a blog article about my project, which will reference my learnings. |
|
Hi @HyperBrain, now that I am actually trying to use the module I'm running into more issues of course. Here is an error I'm getting: I did some research and found this thread: web3/web3.js#966 There's a snippet of how to get Webpack to work which says to add the following: Problem is, I'm not sure exactly how to refactor my webpack config to incorporate that. I made my config the following: This did not fix the error. Can you tell me if my syntax is correct, and how I can verify that the modules are getting included? |
|
Verification is quite easy: Run For the config ... I'll try to come up with a solution after reading the thread. |
|
Ok. I think the error is, that the snippet above adds all node_modules folders under the separate web3/packages and not the folder itself. So you have to add the whole mechanism in your config like this: const { lstatSync, readdirSync } = require('fs');
// is directory
const isDirectory = source => lstatSync(source).isDirectory();
// get directories
const getDirectories = source => readdirSync(source).map(name => path.join(source, name)).filter(isDirectory);
const web3Modules = getDirectories(`${__dirname}/node_modules/web3/packages`); // web3 node_modules
const nodeModules = [`${__dirname}/node_modules`];
nodeModules.push(...web3Modules.map(m => path.join(m, 'node_modules')));
// Rest of the webpack configThen the Now add them to your resolve section: resolve: {
modules: node_modules
}However I do not know if this will work right away, because I never tested module resolution throughout multiple module folder paths. But it's an interesting experiment. |
|
Same error :(. Here is my new config: I believe you meant to type Thanks again, I really hope we can get this resolved soon, I feel like it's close. |
|
Yes I meant So just comment out the |
|
Interesting, a bunch more errors come up: Interesting, seems like it's now seeing what it needs but unable to find them? |
|
That points to the missing PR I mentioned before (according to the thread it is needed additionally to the fix you already made). But I agree, you're nearly there. ... and: this is somehow a can of worms 😄 |
|
Contents of Looks like the right stuff is in there. I'll try with the latest PR. I might not get to it today, but I'll update when I find out more. |
|
I'm looking back on this whole journey and asking myself why I am going through the trouble of using Webpack to begin with in this project. The reason for me to go down this route initially was so that I could use Babel and transpile ES2017 syntax to work in the Serverless Node environments. If I cut Webpack out of the picture and use raw Node 6 syntax, I will most likely save myself this type of trouble, which isn't productive to me right now. Is there any other reason you see to continue trying to get the transpilation to work? |
|
Not really anything important 🤔 . If you have plain Node 6 compatible code you'd not need a transpiler. Despite of the advanced packagin that webpack/the plugin offer, there is not really a need to integrate it - especially with web3, which has shown that it itself is not yet 100% compatible with webpack (see the PR and the thread). So, just try it with a pure SLS deployment for now. It would be also interesting, if the native SLS packaging can correctly package the modules. |
|
Yes, I think I should give that a try. I will come back to this issue if I run into it again. |
|
Ok. Then I'll close it (again 😄 ) for now. We can reopen it as soon as there are further tries to get it working. |
This is a Bug Report
Description
For bug reports:
Stuck on packaging after running
serverless webpackfor a very long time > 10 minutes.Packaged sooner than that.
What stacktrace or error message from your provider did you see?
This is the message it is stuck on:
Serverless: Packing external modules: babel-runtime@^6.25.0, sequelize@^4.8.3, nodemailer@^4.0.1, hat@0.0.3, promise@^8.0.1, web3@^1.0.0-beta.18, truffle-contract@^3.0.0, basic-auth@^2.0.0Serverless-Webpack Version you're using: 3.0.0
Webpack version you're using: 3.5.6
Serverless Framework Version you're using: 1.22.0
Operating System: macOS 10.12.6
The text was updated successfully, but these errors were encountered: