Horizon timeout after transaction is added to ledger

[bartekn]

stellar/go#305

[istrau2]

Any updates on this? We've had to regress to depending on stellar.org's horizon instance to submit transactions. I would really like to get off of that dependency (especially since it has other fairly annoying implications for our system).

Thanks for all your awesome work!

[cesarmak]

Seems to me that this issue is not limited only to the Docker version...

Using GO Horizon built on my own VM (not from docker), I repeatedly encountered this issue occasionally, when I tried to pump over 50tps (for short duration, like 1 minute) to it.
(Can't really reproduce it, as it happens occasionally but not guaranteed.)

And this issue seems to happen more rare, when I am using a VM with better RAM & CPU assigned.

So, maybe this issue happens more often on docker, as less resource is available to the docker?

---

@istrau2 could you reveal how the hardware you've been using is?

[andrenarchy]

This is not restricted to this docker image. We observe the same with https://github.com/satoshipay/docker-stellar-horizon and a standalone Stellar Core and Postgres (1x Core, 1x Horizon).

[bartekn]

Not related to Docker: stellar/go#305 (comment)

[thekevshow]

So I believe it is highly possible this is related to docker, I would like to announce my very odd but repeatable findings.

ANNOUNCEMENT:

Debian jessy core i3 machine 8 gig ram, 5400 rpm drive 320 gig
ubuntu 18.04.1 core i7 machine (gaming mobo) 16 gig ram 256 ssd (read write not off hand)
ubuntu 18.04.1 razer blade 15 2018 (see specs for yourself)

load image first time using docker commands, 504 timeout every time.
restart machine (leaving docker running as is for clarification)

run same test for creating account transaction (and other subsequent transactions) success

My speculation would be some interprocess com or networking issue. Might have time to try and look into this at a later date but these are my findings for those who come across this issue. As I am one to not restart a machine unless needed, and went out of my way to verify this was not some resource issue as seen that my first machine is much lower resource than the other two I was using. To which concluded my assumption that resources had nothing to do with this. There is another issue open on the stellar go github that is easy to find if anyone wants to rely that information over there. But I can very much so say this ticket should probably still be open. As docker does seem to possibly play a role here.

[mvaneijk]

I found out that restarting horizon using the supervisorctl fixed the issue.

1. Enter the docker container
2. run supervisorctl
3. restart horizon

[thekevshow]

I found out that restarting horizon using the supervisorctl fixed the issue.

1. Enter the docker container
2. run supervisorctl
3. restart horizon

That makes sense, as restarting computer probably causes this to happen in the container, or amounts to solving the same issue with horizon.

[oryband]

I found out that restarting horizon using the supervisorctl fixed the issue.

restarting horizon does help, never mind if it runs in supervisorctl or docker or not. in my case i just initially launched horizon and it was waiting for cores to initialize as well. after it finished ingesting, all requests timed out. restarting horizon fixed the problem.

[oryband]

I want to emphasize that this doesn't only happen in the quickstart image. It happens in any network whatsoever regardless of it running in Docker or using Supervisord or any other method.

This happens as long as Horizon starts up before Core. Doesn't matter if it's a new Core without any block memory or an existing one. Soon as Horizon starts ingesting it timeouts on submitting transactions.

I'm on 0.12.3 and will verify this with newer version later on.

[bartekn]

The latest version of this image (that includes Horizon 0.15.4) should solve this issue. Please pull the latest version of Docker Hub and try it out!