HOP Sizing

[xProga]

Good day to everyone.

Does Apache HOP (Hop server, Hop web) have some kind of sizing manual? I'd like to use Hop in my project and trying to understand it's requirements. Especially interested in sizing dependencies from Workflow/Pipelines count in system, processes per minute and data load bypass throw Hop Server.

Interested in cases of short living containers (planing use hop in k8s) adn long live (based on static vm).

I have read this article and understand basics but would like to evaluate required resources depending from different criterias.

[hansva]

Hi @xProga,

I'm afraid the answer to all this is "it depends". That's why we don't have these guidelines.

Some of the things that are known:

• Each action/transform (or transform copy) will create a processor thread
  • This means the maximum amount of active transforms equals the maximum amount of threads the CPU supports (-1 for the main process), when the amount of active transforms is higher thread switching will occur
  • So our recommendation is to keep your pipelines as compact as possible (~30 transforms sounds like a sane rule)
• Each transform has a configurable buffer
  • Each transform (copy) has an input buffer, compared to other tools we do not load all data into memory and move them from one transform to the next. We have a buffer system (default 10K rows) and transforms will fill those buffers and get pushback signals to stop processing/fetching data
  • This means the amount of active memory = rows in buffers x (columns x data type)
  • There are a couple of exceptions, eg. Sort Rows needs to have all data so it will load all rows but it has a configurable buffer to spool of data to disk

Hop Web:
Users can run workflows and pipelines inside Hop Web, it is not a client/server application. This does imply that these instances need enough resources to run the processes locally.

Hop Server:
This one is mainly used as a remote extension to local development. It is a stateless server so workloads do not survive restarts. It does not have scheduling.

We recommend using short-lived containers for actual workload scheduling using an orchestration tool of your choice.

Hope this helps.

[bamaer]

in addition to what Hans mentioned, make sure your pipelines are optimized before throwing more (virtual) hardware at them:

• Measure what your actual resource usage is (rows per second, memory and CPU usages, ...) and tweak your scaling accordingly. You may be able to scale down if you only use 30% of the available RAM, Xmx or CPU.
• Identify bottlenecks in your pipelines, fix the worst offenders and readjust your scaling needs accordingly. E.g. adding memory won't help if your pipeline is slow because of missing database indexes
• Tweak for the resources you have available. A larger box that runs your big pipelines a lot faster in a short-lived container or pod may be cheaper than a smaller container or pod that needs more time to run