Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Windowing Support for the Dask Runner #23913

Closed
wants to merge 139 commits into from

Commits on Sep 19, 2022

  1. Configuration menu
    Copy the full SHA
    bc6b525 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    665ee61 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    d1095c7 View commit details
    Browse the repository at this point in the history
  4. Fixed a dataclass typo.

    alxmrs committed Sep 19, 2022
    Configuration menu
    Copy the full SHA
    fd13d40 View commit details
    Browse the repository at this point in the history
  5. Expanded translations.

    alxmrs committed Sep 19, 2022
    Configuration menu
    Copy the full SHA
    e3d0c8a View commit details
    Browse the repository at this point in the history
  6. Configuration menu
    Copy the full SHA
    69a660f View commit details
    Browse the repository at this point in the history
  7. Configuration menu
    Copy the full SHA
    ab58334 View commit details
    Browse the repository at this point in the history
  8. Configuration menu
    Copy the full SHA
    5391cd6 View commit details
    Browse the repository at this point in the history
  9. WIP: Alllmost got asserts to work! The current status is:

    - CoGroupByKey is broken due to how tags are used with GroupByKey
    - GroupByKey should output `[('0', None), ('1', 1)]`, however it actually outputs: [(None, ('1', 1)), (None, ('0', None))]
    - Once that is fixed, we may have test pipelines work on Dask.
    alxmrs committed Sep 19, 2022
    Configuration menu
    Copy the full SHA
    5deb598 View commit details
    Browse the repository at this point in the history
  10. With a great 1-liner from @pabloem, groupby is fixed! Now, all three …

    …initial tests pass.
    alxmrs committed Sep 19, 2022
    Configuration menu
    Copy the full SHA
    1768e47 View commit details
    Browse the repository at this point in the history
  11. Configuration menu
    Copy the full SHA
    e6c7106 View commit details
    Browse the repository at this point in the history
  12. Configuration menu
    Copy the full SHA
    1b58d4c View commit details
    Browse the repository at this point in the history
  13. First pass at linting rules.

    alxmrs committed Sep 19, 2022
    Configuration menu
    Copy the full SHA
    5fe9372 View commit details
    Browse the repository at this point in the history
  14. Configuration menu
    Copy the full SHA
    b98330e View commit details
    Browse the repository at this point in the history
  15. WIP: maybe better dask deps?

    alxmrs committed Sep 19, 2022
    Configuration menu
    Copy the full SHA
    e48780a View commit details
    Browse the repository at this point in the history
  16. Configuration menu
    Copy the full SHA
    0dc9e23 View commit details
    Browse the repository at this point in the history
  17. Fixed setup.py (missing ,).

    alxmrs committed Sep 19, 2022
    Configuration menu
    Copy the full SHA
    326d3a3 View commit details
    Browse the repository at this point in the history
  18. Added an additional comma.

    alxmrs committed Sep 19, 2022
    Configuration menu
    Copy the full SHA
    b4cc408 View commit details
    Browse the repository at this point in the history
  19. Configuration menu
    Copy the full SHA
    40a6ebe View commit details
    Browse the repository at this point in the history
  20. Configuration menu
    Copy the full SHA
    3c4204d View commit details
    Browse the repository at this point in the history

Commits on Sep 20, 2022

  1. Configuration menu
    Copy the full SHA
    41623ec View commit details
    Browse the repository at this point in the history

Commits on Sep 21, 2022

  1. Configuration menu
    Copy the full SHA
    676d752 View commit details
    Browse the repository at this point in the history

Commits on Sep 22, 2022

  1. Configuration menu
    Copy the full SHA
    09365f6 View commit details
    Browse the repository at this point in the history

Commits on Sep 28, 2022

  1. Revert "Close client during wait_until_finish; rm async."

    This reverts commit 09365f6.
    pabloem committed Sep 28, 2022
    Configuration menu
    Copy the full SHA
    c6ba4ba View commit details
    Browse the repository at this point in the history
  2. Revert "Changing to async produces a timeout error instead of stuck i…

    …n infinite loop."
    
    This reverts commit 676d752.
    pabloem committed Sep 28, 2022
    Configuration menu
    Copy the full SHA
    a325356 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    ea13125 View commit details
    Browse the repository at this point in the history

Commits on Oct 2, 2022

  1. Configuration menu
    Copy the full SHA
    f855ffc View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    3fd966e View commit details
    Browse the repository at this point in the history
  3. wip - added print stmt.

    alxmrs committed Oct 2, 2022
    Configuration menu
    Copy the full SHA
    173d79b View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    dd2d15c View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    8756618 View commit details
    Browse the repository at this point in the history
  6. wip - rm asserts, add print

    alxmrs committed Oct 2, 2022
    Configuration menu
    Copy the full SHA
    8380d7b View commit details
    Browse the repository at this point in the history
  7. wip - adding named inputs...

    alxmrs committed Oct 2, 2022
    Configuration menu
    Copy the full SHA
    b908dc3 View commit details
    Browse the repository at this point in the history
  8. Configuration menu
    Copy the full SHA
    f444b1e View commit details
    Browse the repository at this point in the history
  9. None --> 'None'

    alxmrs committed Oct 2, 2022
    Configuration menu
    Copy the full SHA
    174d6fd View commit details
    Browse the repository at this point in the history
  10. No default side input.

    alxmrs committed Oct 2, 2022
    Configuration menu
    Copy the full SHA
    60b063a View commit details
    Browse the repository at this point in the history
  11. Pass along args + kwargs.

    alxmrs committed Oct 2, 2022
    Configuration menu
    Copy the full SHA
    90ee474 View commit details
    Browse the repository at this point in the history

Commits on Oct 3, 2022

  1. Configuration menu
    Copy the full SHA
    c62050e View commit details
    Browse the repository at this point in the history

Commits on Oct 10, 2022

  1. Configuration menu
    Copy the full SHA
    79d4603 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    248ec70 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    42452ca View commit details
    Browse the repository at this point in the history
  4. Fixed a dataclass typo.

    alxmrs committed Oct 10, 2022
    Configuration menu
    Copy the full SHA
    1da2ddd View commit details
    Browse the repository at this point in the history
  5. Expanded translations.

    alxmrs committed Oct 10, 2022
    Configuration menu
    Copy the full SHA
    14885a3 View commit details
    Browse the repository at this point in the history
  6. Configuration menu
    Copy the full SHA
    fca2420 View commit details
    Browse the repository at this point in the history
  7. Configuration menu
    Copy the full SHA
    6dd1ada View commit details
    Browse the repository at this point in the history
  8. Configuration menu
    Copy the full SHA
    6675687 View commit details
    Browse the repository at this point in the history
  9. WIP: Alllmost got asserts to work! The current status is:

    - CoGroupByKey is broken due to how tags are used with GroupByKey
    - GroupByKey should output `[('0', None), ('1', 1)]`, however it actually outputs: [(None, ('1', 1)), (None, ('0', None))]
    - Once that is fixed, we may have test pipelines work on Dask.
    alxmrs committed Oct 10, 2022
    Configuration menu
    Copy the full SHA
    88ed36b View commit details
    Browse the repository at this point in the history
  10. With a great 1-liner from @pabloem, groupby is fixed! Now, all three …

    …initial tests pass.
    alxmrs committed Oct 10, 2022
    Configuration menu
    Copy the full SHA
    2e3a126 View commit details
    Browse the repository at this point in the history
  11. Configuration menu
    Copy the full SHA
    6467b0e View commit details
    Browse the repository at this point in the history
  12. Configuration menu
    Copy the full SHA
    793ba86 View commit details
    Browse the repository at this point in the history
  13. First pass at linting rules.

    alxmrs committed Oct 10, 2022
    Configuration menu
    Copy the full SHA
    e535792 View commit details
    Browse the repository at this point in the history
  14. Configuration menu
    Copy the full SHA
    8e32668 View commit details
    Browse the repository at this point in the history
  15. WIP: maybe better dask deps?

    alxmrs committed Oct 10, 2022
    Configuration menu
    Copy the full SHA
    318afc2 View commit details
    Browse the repository at this point in the history
  16. Configuration menu
    Copy the full SHA
    b01855f View commit details
    Browse the repository at this point in the history
  17. Fixed setup.py (missing ,).

    alxmrs committed Oct 10, 2022
    Configuration menu
    Copy the full SHA
    2c2eb8d View commit details
    Browse the repository at this point in the history
  18. Added an additional comma.

    alxmrs committed Oct 10, 2022
    Configuration menu
    Copy the full SHA
    e64e9eb View commit details
    Browse the repository at this point in the history
  19. Configuration menu
    Copy the full SHA
    69b118b View commit details
    Browse the repository at this point in the history
  20. Configuration menu
    Copy the full SHA
    9ffc8d8 View commit details
    Browse the repository at this point in the history
  21. Configuration menu
    Copy the full SHA
    8a2afb7 View commit details
    Browse the repository at this point in the history
  22. Configuration menu
    Copy the full SHA
    93f02f1 View commit details
    Browse the repository at this point in the history
  23. Configuration menu
    Copy the full SHA
    afdcf1b View commit details
    Browse the repository at this point in the history
  24. Configuration menu
    Copy the full SHA
    41b5267 View commit details
    Browse the repository at this point in the history
  25. Revert "Close client during wait_until_finish; rm async."

    This reverts commit 09365f6.
    pabloem authored and alxmrs committed Oct 10, 2022
    Configuration menu
    Copy the full SHA
    e3ac3f8 View commit details
    Browse the repository at this point in the history
  26. Revert "Changing to async produces a timeout error instead of stuck i…

    …n infinite loop."
    
    This reverts commit 676d752.
    pabloem authored and alxmrs committed Oct 10, 2022
    Configuration menu
    Copy the full SHA
    3fddc81 View commit details
    Browse the repository at this point in the history
  27. Configuration menu
    Copy the full SHA
    9eeb9ea View commit details
    Browse the repository at this point in the history
  28. wip - added print stmt.

    alxmrs committed Oct 10, 2022
    Configuration menu
    Copy the full SHA
    b4d0999 View commit details
    Browse the repository at this point in the history
  29. Configuration menu
    Copy the full SHA
    0319ffd View commit details
    Browse the repository at this point in the history
  30. Configuration menu
    Copy the full SHA
    0b13bb0 View commit details
    Browse the repository at this point in the history
  31. wip - rm asserts, add print

    alxmrs committed Oct 10, 2022
    Configuration menu
    Copy the full SHA
    1e7052b View commit details
    Browse the repository at this point in the history
  32. wip - adding named inputs...

    alxmrs committed Oct 10, 2022
    Configuration menu
    Copy the full SHA
    292e023 View commit details
    Browse the repository at this point in the history
  33. Configuration menu
    Copy the full SHA
    31c1e2b View commit details
    Browse the repository at this point in the history
  34. None --> 'None'

    alxmrs committed Oct 10, 2022
    Configuration menu
    Copy the full SHA
    f4ecf2f View commit details
    Browse the repository at this point in the history
  35. No default side input.

    alxmrs committed Oct 10, 2022
    Configuration menu
    Copy the full SHA
    4d24ed9 View commit details
    Browse the repository at this point in the history
  36. Pass along args + kwargs.

    alxmrs committed Oct 10, 2022
    Configuration menu
    Copy the full SHA
    ee62a4a View commit details
    Browse the repository at this point in the history
  37. Applied yapf to dask sources.

    alxmrs committed Oct 10, 2022
    Configuration menu
    Copy the full SHA
    506c719 View commit details
    Browse the repository at this point in the history
  38. Dask sources passing pylint.

    alxmrs committed Oct 10, 2022
    Configuration menu
    Copy the full SHA
    cd0ba8b View commit details
    Browse the repository at this point in the history
  39. Configuration menu
    Copy the full SHA
    d0a7c63 View commit details
    Browse the repository at this point in the history
  40. Applied yapf from tox.

    alxmrs committed Oct 10, 2022
    Configuration menu
    Copy the full SHA
    775bd07 View commit details
    Browse the repository at this point in the history
  41. Include dask in mypy checks.

    alxmrs committed Oct 10, 2022
    Configuration menu
    Copy the full SHA
    efba1c9 View commit details
    Browse the repository at this point in the history
  42. Configuration menu
    Copy the full SHA
    741d961 View commit details
    Browse the repository at this point in the history
  43. Configuration menu
    Copy the full SHA
    f66458a View commit details
    Browse the repository at this point in the history
  44. fix lint: line too long.

    alxmrs committed Oct 10, 2022
    Configuration menu
    Copy the full SHA
    5dcf969 View commit details
    Browse the repository at this point in the history
  45. Configuration menu
    Copy the full SHA
    ec5f613 View commit details
    Browse the repository at this point in the history
  46. Configuration menu
    Copy the full SHA
    04b1f1a View commit details
    Browse the repository at this point in the history
  47. Ran isort.

    alxmrs committed Oct 10, 2022
    Configuration menu
    Copy the full SHA
    712944b View commit details
    Browse the repository at this point in the history
  48. Ran yapf again.

    alxmrs committed Oct 10, 2022
    Configuration menu
    Copy the full SHA
    567b72b View commit details
    Browse the repository at this point in the history
  49. Fix imports (one per line)

    alxmrs committed Oct 10, 2022
    Configuration menu
    Copy the full SHA
    f53c0a4 View commit details
    Browse the repository at this point in the history
  50. isort -- alphabetical.

    alxmrs committed Oct 10, 2022
    Configuration menu
    Copy the full SHA
    fb280ad View commit details
    Browse the repository at this point in the history
  51. Added feature to CHANGES.md.

    alxmrs committed Oct 10, 2022
    Configuration menu
    Copy the full SHA
    80ddfec View commit details
    Browse the repository at this point in the history
  52. Configuration menu
    Copy the full SHA
    40c4e35 View commit details
    Browse the repository at this point in the history

Commits on Oct 13, 2022

  1. Configuration menu
    Copy the full SHA
    a70c5f3 View commit details
    Browse the repository at this point in the history
  2. Change an import to pass CI.

    alxmrs committed Oct 13, 2022
    Configuration menu
    Copy the full SHA
    9fb52e5 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    91115e0 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    26c6016 View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    aec19bf View commit details
    Browse the repository at this point in the history
  6. (Maybe) the last isort fix.

    alxmrs committed Oct 13, 2022
    Configuration menu
    Copy the full SHA
    0673235 View commit details
    Browse the repository at this point in the history

Commits on Oct 14, 2022

  1. Configuration menu
    Copy the full SHA
    de03a32 View commit details
    Browse the repository at this point in the history
  2. Improve formatting of test.

    alxmrs committed Oct 14, 2022
    Configuration menu
    Copy the full SHA
    7e0a2c7 View commit details
    Browse the repository at this point in the history
  3. Self-review: removing side inputs.

    In addition, adding a more helpful property to the base DaskBagOp (tranform).
    alxmrs committed Oct 14, 2022
    Configuration menu
    Copy the full SHA
    39b1e1c View commit details
    Browse the repository at this point in the history

Commits on Oct 17, 2022

  1. Configuration menu
    Copy the full SHA
    6db49fa View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    ed00139 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    2d8f5d6 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    d35e9d6 View commit details
    Browse the repository at this point in the history
  5. Merge pull request #79 from pabloem/dask-runner-windowing

    add mandatory args for _OutputHandler
    alxmrs authored Oct 17, 2022
    Configuration menu
    Copy the full SHA
    973d9f9 View commit details
    Browse the repository at this point in the history
  6. Merge branch 'dask-runner-windowing' of github.com:alxmrs/beam into d…

    …ask-runner-windowing
    alxmrs committed Oct 17, 2022
    Configuration menu
    Copy the full SHA
    0e5d498 View commit details
    Browse the repository at this point in the history
  7. Configuration menu
    Copy the full SHA
    3feeeac View commit details
    Browse the repository at this point in the history

Commits on Oct 18, 2022

  1. Configuration menu
    Copy the full SHA
    036561c View commit details
    Browse the repository at this point in the history
  2. Capture value error in assert.

    alxmrs committed Oct 18, 2022
    Configuration menu
    Copy the full SHA
    191580d View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    365fc87 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    085447e View commit details
    Browse the repository at this point in the history

Commits on Oct 21, 2022

  1. ignoring broken test

    pabloem authored Oct 21, 2022
    Configuration menu
    Copy the full SHA
    1a60a5e View commit details
    Browse the repository at this point in the history
  2. Update CHANGES.md

    pabloem authored Oct 21, 2022
    Configuration menu
    Copy the full SHA
    c1037f8 View commit details
    Browse the repository at this point in the history

Commits on Oct 24, 2022

  1. Configuration menu
    Copy the full SHA
    9e79ffd View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    3e2cc0f View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    4edc970 View commit details
    Browse the repository at this point in the history

Commits on Oct 25, 2022

  1. Configuration menu
    Copy the full SHA
    fd8e361 View commit details
    Browse the repository at this point in the history

Commits on Oct 28, 2022

  1. Configuration menu
    Copy the full SHA
    d88e8a1 View commit details
    Browse the repository at this point in the history

Commits on Oct 31, 2022

  1. Configuration menu
    Copy the full SHA
    36bea9a View commit details
    Browse the repository at this point in the history

Commits on Nov 1, 2022

  1. Ran yapf on Dask sources.

    alxmrs committed Nov 1, 2022
    Configuration menu
    Copy the full SHA
    df315c1 View commit details
    Browse the repository at this point in the history
  2. Ran lint checks.

    alxmrs committed Nov 1, 2022
    Configuration menu
    Copy the full SHA
    ef0d2b6 View commit details
    Browse the repository at this point in the history
  3. (hopefully) final lint check.

    alxmrs committed Nov 1, 2022
    Configuration menu
    Copy the full SHA
    6c2cc4e View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    0fae761 View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    119666c View commit details
    Browse the repository at this point in the history

Commits on Nov 2, 2022

  1. mini self-review.

    alxmrs committed Nov 2, 2022
    Configuration menu
    Copy the full SHA
    2e46d88 View commit details
    Browse the repository at this point in the history

Commits on Nov 5, 2022

  1. Configuration menu
    Copy the full SHA
    2ed8b14 View commit details
    Browse the repository at this point in the history
  2. WIP: Further improvements, more correct windowing impl.

    The error that I'm hitting seems to be in creating tuples during the CoGroupByKey.
    alxmrs committed Nov 5, 2022
    Configuration menu
    Copy the full SHA
    2f193d5 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    577f30a View commit details
    Browse the repository at this point in the history
  4. WIP: All tests are passing :)

    out-of-order issue was fixed by calling dask.optimize.
    alxmrs committed Nov 5, 2022
    Configuration menu
    Copy the full SHA
    b3a70f6 View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    7acd8d5 View commit details
    Browse the repository at this point in the history
  6. Lint + YAPF

    alxmrs committed Nov 5, 2022
    Configuration menu
    Copy the full SHA
    7e90e2b View commit details
    Browse the repository at this point in the history
  7. self-review.

    alxmrs committed Nov 5, 2022
    Configuration menu
    Copy the full SHA
    6e33ce2 View commit details
    Browse the repository at this point in the history

Commits on Nov 18, 2022

  1. fix lint

    pabloem committed Nov 18, 2022
    Configuration menu
    Copy the full SHA
    f54f14c View commit details
    Browse the repository at this point in the history
  2. fixup

    pabloem committed Nov 18, 2022
    Configuration menu
    Copy the full SHA
    8dd2cdb View commit details
    Browse the repository at this point in the history

Commits on Nov 19, 2022

  1. Configuration menu
    Copy the full SHA
    518a8f0 View commit details
    Browse the repository at this point in the history

Commits on Nov 28, 2022

  1. fixing dask version

    pabloem committed Nov 28, 2022
    Configuration menu
    Copy the full SHA
    801b131 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    393c508 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    f89f609 View commit details
    Browse the repository at this point in the history

Commits on Nov 30, 2022

  1. reducing scope of Dask tests

    pabloem committed Nov 30, 2022
    Configuration menu
    Copy the full SHA
    d6486fe View commit details
    Browse the repository at this point in the history
  2. adding datafrems dep

    pabloem committed Nov 30, 2022
    Configuration menu
    Copy the full SHA
    6f05963 View commit details
    Browse the repository at this point in the history