Skip to content
This repository has been archived by the owner on Oct 11, 2024. It is now read-only.

Mesh Node Dies #377

Closed
pointtoken opened this issue Aug 29, 2019 · 10 comments
Closed

Mesh Node Dies #377

pointtoken opened this issue Aug 29, 2019 · 10 comments
Labels
bug Something isn't working

Comments

@pointtoken
Copy link
Contributor

What is the expected uptime for a mesh node? Ours seems to die after 8 hours. We are running the latest from DockerHub on a VM with 1 Gig ram. Thoughts?

@albrow
Copy link
Contributor

albrow commented Aug 29, 2019

@pointtoken Mesh should be able to run indefinitely and we have some nodes with an uptime of over 2 weeks.

Can you share more information so we can help debug any possible issues? Do you see any errors in the logs?

@albrow albrow added the bug Something isn't working label Aug 29, 2019
@pointtoken
Copy link
Contributor Author

We've been up 25 hours to date, so maybe it was a bug in an earlier build. Where would we find the log files? We are running detached with verbosity = 1

@albrow
Copy link
Contributor

albrow commented Aug 30, 2019

Mesh just outputs logs to stdout, so you can use docker logs. The exact command depends on how you have Docker setup, but it's usually just:

docker logs <container name>

@pointtoken
Copy link
Contributor Author

thanks. bug seems to be fixed in the latest version.

@pointtoken pointtoken reopened this Sep 5, 2019
@pointtoken
Copy link
Contributor Author

I'm reopening this as our node died after about 4 days. @albrow how does one get at the logs from a container that no longer exists?

@albrow
Copy link
Contributor

albrow commented Sep 5, 2019

@pointtoken can you share more information about your setup?

  • What version of Mesh are you running?
  • How are you running it? What Docker command?
  • Where is the container hosted?

If docker logs doesn't show anything then that means the container itself was stopped. Unfortunately there is no way to read from stdout for a container that is no longer running. This is a limitation of Docker.

For our own infrastructure, we pipe all logs to Elasticsearch and Kiabana so we always have a way to search them even if the Docker container is not running. If you want, you can opt-in to send us your logs by following this guide then we could help you debug any potential issues.

Alternatively you could use a simple bash script to redirect the output from Docker to a file. E.g., something like:

# Don't forget to include any needed environment variables.
docker run 0xorg/mesh:3.0.1-beta > /file/on/host/machine

@pointtoken
Copy link
Contributor Author

It is running in an EC2 container. Here's the command used:

docker run -it --rm -d -p 60557:60557 -p 60558:60558 -e ETHEREUM_NETWORK_ID="1" -e ETHEREUM_RPC_URL="https://mainnet.infura.io/v3/XXX" -e  VERBOSITY=2 -v /home/ec2-user/0x_mesh:/usr/mesh/0x_mesh 0xorg/mesh:latest > /home/ec2-user/mesh-logs/log.txt

Running the latest image that's been pushed to DockerHub.

Seems to die after about 48 hours.

@albrow
Copy link
Contributor

albrow commented Sep 12, 2019

@pointtoken what version of Mesh are you running? latest can mean anything depending on when you ran the command and when your local docker images were last updated. Mesh outputs the version on startup.

Also are you able to see any logs and if so can you share them with me? Without seeing the logs there is no way for me to understand what is happening to your node.

One thing I realized is that technically Mesh logs go to stderr instead of stdout. Both stderr and stdout appear in the terminal when running a command but it does make a difference if you are trying to redirect the output. If you are using the > operator to write the logs to a file you will also need to redirect stderr with 2>&1. Something like this:

# Don't forget to include any needed environment variables.
docker run 0xorg/mesh:3.0.1-beta 2>&1 > /file/on/host/machine

Are you able to monitor resource usage? Do you see anything abnormal happening with CPU or memory usage?

@albrow
Copy link
Contributor

albrow commented Oct 16, 2019

Possibly related: #452.

@albrow
Copy link
Contributor

albrow commented Nov 12, 2019

@pointtoken we haven't heard from you in two months, so I'm closing this issue for now. If you are still having problems and are able to share any additional information, I'd be happy to re-open it.

@albrow albrow closed this as completed Nov 12, 2019
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants