You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Currently, jobs are run one record at a time and the runner can end up spending a lot of time waiting for a slow transformer or an IO-bound loader (ie POSTing to a slow API endpoint). Chronicle-ETL should offer a multithreaded mode as well (or by default?).
The most impactful change with least effort would be keeping extractors single-threaded but handing off records to a worker pool that can transform and then load records. This would require only minimal changes to:
the job runner UI
the job logging system
making sure loaders are thread-safe (use thread-safe arrays for TableLoader, etc)
Adding concurrency to extractors would be trickier since each extractor would have to specify how to chunk up the work (and wouldn't even be possible for certain extractors like reading from stdin or with an API that uses cursors for pagination)
The text was updated successfully, but these errors were encountered:
Currently, jobs are run one record at a time and the runner can end up spending a lot of time waiting for a slow transformer or an IO-bound loader (ie POSTing to a slow API endpoint). Chronicle-ETL should offer a multithreaded mode as well (or by default?).
The most impactful change with least effort would be keeping extractors single-threaded but handing off records to a worker pool that can transform and then load records. This would require only minimal changes to:
TableLoader
, etc)Adding concurrency to extractors would be trickier since each extractor would have to specify how to chunk up the work (and wouldn't even be possible for certain extractors like reading from stdin or with an API that uses cursors for pagination)
The text was updated successfully, but these errors were encountered: