Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enables running of multiple tasks in batches #1784

Merged
merged 9 commits into from
Aug 12, 2016

Commits on Aug 2, 2016

  1. Enables running of multiple tasks in batches

    Sometimes it's more efficient to run a group of tasks all at once rather than
    one at a time. With luigi, it's difficult to take advantage of this because your
    batch size will also be the minimum granularity you're able to compute. So if
    you have a job that runs hourly, you can't combine their computation when many
    of them get backlogged. When you have a task that runs daily, you can't get
    hourly runs.
    
    For the user, batching is accomplished by setting batch_method in the parameters
    that you wish to batch. This takes a list of parameter values (already parsed)
    and combines them into a single parameter. You can limit the number of tasks
    allowed to run simultaneously in a single batch by setting the class variable
    batch_size. If you want to exempt a specific task from being part of a batch,
    simply set its is_batchable property to False.
    
    Unlike spotify#1538, this does batch combining on the worker side, allowing arbitrary
    batch methods and keeping better track of task ids for things like execution
    summary.
    daveFNbuck committed Aug 2, 2016
    Configuration menu
    Copy the full SHA
    b31018c View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    d7c4c86 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    32fc576 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    9855228 View commit details
    Browse the repository at this point in the history

Commits on Aug 10, 2016

  1. Addresses Arash's comments

    daveFNbuck committed Aug 10, 2016
    Configuration menu
    Copy the full SHA
    096808e View commit details
    Browse the repository at this point in the history

Commits on Aug 11, 2016

  1. Configuration menu
    Copy the full SHA
    a4421ea View commit details
    Browse the repository at this point in the history
  2. Doc changes

    daveFNbuck committed Aug 11, 2016
    Configuration menu
    Copy the full SHA
    81a639e View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    b68bd29 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    6382cc6 View commit details
    Browse the repository at this point in the history