Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Python UDF in Ingestion being used for feature validation #1234

Merged
merged 24 commits into from
Dec 22, 2020
Merged

Python UDF in Ingestion being used for feature validation #1234

merged 24 commits into from
Dec 22, 2020

Commits on Dec 18, 2020

  1. first draft 

    Signed-off-by: Oleksii Moskalenko <moskalenko.alexey@gmail.com>
    @pyalex
    pyalex committed Dec 18, 2020
    Configuration menu
    Copy the full SHA
    0e68ff9 View commit details
    Browse the repository at this point in the history

  2. restore udf in ingestion job 

    Signed-off-by: Oleksii Moskalenko <moskalenko.alexey@gmail.com>
    @pyalex
    pyalex committed Dec 18, 2020
    Configuration menu
    Copy the full SHA
    b9ed047 View commit details
    Browse the repository at this point in the history

  3. it tests for pandas udf 

    Signed-off-by: Oleksii Moskalenko <moskalenko.alexey@gmail.com>
    @pyalex
    pyalex committed Dec 18, 2020
    Configuration menu
    Copy the full SHA
    063d8fc View commit details
    Browse the repository at this point in the history

  4. e2e test 

    Signed-off-by: Oleksii Moskalenko <moskalenko.alexey@gmail.com>
    @pyalex
    pyalex committed Dec 18, 2020
    Configuration menu
    Copy the full SHA
    903e743 View commit details
    Browse the repository at this point in the history

  5. skip udf build 

    Signed-off-by: Oleksii Moskalenko <moskalenko.alexey@gmail.com>
    @pyalex
    pyalex committed Dec 18, 2020
    Configuration menu
    Copy the full SHA
    2b86ea7 View commit details
    Browse the repository at this point in the history

  6. $skipITs 

    Signed-off-by: Oleksii Moskalenko <moskalenko.alexey@gmail.com>
    @pyalex
    pyalex committed Dec 18, 2020
    Configuration menu
    Copy the full SHA
    0ba6a4b View commit details
    Browse the repository at this point in the history

  7. $skipITs 

    Signed-off-by: Oleksii Moskalenko <moskalenko.alexey@gmail.com>
    @pyalex
    pyalex committed Dec 18, 2020
    Configuration menu
    Copy the full SHA
    13ed5ab View commit details
    Browse the repository at this point in the history

  8. add reporting to scalatest 

    Signed-off-by: Oleksii Moskalenko <moskalenko.alexey@gmail.com>
    @pyalex
    pyalex committed Dec 18, 2020
    Configuration menu
    Copy the full SHA
    366605f View commit details
    Browse the repository at this point in the history

  9. platform specific libs path 

    Signed-off-by: Oleksii Moskalenko <moskalenko.alexey@gmail.com>
    @pyalex
    pyalex committed Dec 18, 2020
    Configuration menu
    Copy the full SHA
    bb72032 View commit details
    Browse the repository at this point in the history

  10. continue on error 

    Signed-off-by: Oleksii Moskalenko <moskalenko.alexey@gmail.com>
    @pyalex
    pyalex committed Dec 18, 2020
    Configuration menu
    Copy the full SHA
    afbf8c2 View commit details
    Browse the repository at this point in the history

  11. build pylibs for all platforms 

    Signed-off-by: Oleksii Moskalenko <moskalenko.alexey@gmail.com>
    @pyalex
    pyalex committed Dec 18, 2020
    Configuration menu
    Copy the full SHA
    b5efcd6 View commit details
    Browse the repository at this point in the history

  12. build pylibs for all platforms 

    Signed-off-by: Oleksii Moskalenko <moskalenko.alexey@gmail.com>
    @pyalex
    pyalex committed Dec 18, 2020
    Configuration menu
    Copy the full SHA
    5a908e3 View commit details
    Browse the repository at this point in the history

  13. pull python path from env 

    Signed-off-by: Oleksii Moskalenko <moskalenko.alexey@gmail.com>
    @pyalex
    pyalex committed Dec 18, 2020
    Configuration menu
    Copy the full SHA
    65a2b7b View commit details
    Browse the repository at this point in the history

  14. freeze dataproc python version 

    Signed-off-by: Oleksii Moskalenko <moskalenko.alexey@gmail.com>
    @pyalex
    pyalex committed Dec 18, 2020
    Configuration menu
    Copy the full SHA
    e56dd9e View commit details
    Browse the repository at this point in the history

  15. use python from config 

    Signed-off-by: Oleksii Moskalenko <moskalenko.alexey@gmail.com>
    @pyalex
    pyalex committed Dec 18, 2020
    Configuration menu
    Copy the full SHA
    fb18c92 View commit details
    Browse the repository at this point in the history

  16. add pyarrow to package 

    Signed-off-by: Oleksii Moskalenko <moskalenko.alexey@gmail.com>
    @pyalex
    pyalex committed Dec 18, 2020
    Configuration menu
    Copy the full SHA
    6b67e63 View commit details
    Browse the repository at this point in the history

  17. drop dataclasses to support 3.6 

    Signed-off-by: Oleksii Moskalenko <moskalenko.alexey@gmail.com>
    @pyalex
    pyalex committed Dec 18, 2020
    Configuration menu
    Copy the full SHA
    0fd3c3d View commit details
    Browse the repository at this point in the history

  18. support python 3.6 

    Signed-off-by: Oleksii Moskalenko <moskalenko.alexey@gmail.com>
    @pyalex
    pyalex committed Dec 18, 2020
    Configuration menu
    Copy the full SHA
    2d126c6 View commit details
    Browse the repository at this point in the history

  19. set spark.yarn.isPython=true for emr 

    Signed-off-by: Oleksii Moskalenko <moskalenko.alexey@gmail.com>
    @pyalex
    pyalex committed Dec 18, 2020
    Configuration menu
    Copy the full SHA
    8d4e476 View commit details
    Browse the repository at this point in the history

  20. pass isPython through spark-submit conf 

    Signed-off-by: Oleksii Moskalenko <moskalenko.alexey@gmail.com>
    @pyalex
    pyalex committed Dec 18, 2020
    Configuration menu
    Copy the full SHA
    bf63033 View commit details
    Browse the repository at this point in the history

  21. some cleanup 

    Signed-off-by: Oleksii Moskalenko <moskalenko.alexey@gmail.com>
    @pyalex
    pyalex committed Dec 18, 2020
    Configuration menu
    Copy the full SHA
    54fcb6e View commit details
    Browse the repository at this point in the history

  22. lint-java 

    Signed-off-by: Oleksii Moskalenko <moskalenko.alexey@gmail.com>
    @pyalex
    pyalex committed Dec 18, 2020
    Configuration menu
    Copy the full SHA
    c348481 View commit details
    Browse the repository at this point in the history

Commits on Dec 21, 2020

  1. Update infra/scripts/build-ingestion-py-dependencies.sh 

    Co-authored-by: Oleg Avdeev <oleg.v.avdeev@gmail.com>
Signed-off-by: Oleksii Moskalenko <moskalenko.alexey@gmail.com>
    @pyalex @oavdeev
    pyalex and oavdeev committed Dec 21, 2020
    Configuration menu
    Copy the full SHA
    a6d272f View commit details
    Browse the repository at this point in the history

  2. revert batch sources fix 

    Signed-off-by: Oleksii Moskalenko <moskalenko.alexey@gmail.com>
    @pyalex
    pyalex committed Dec 21, 2020
    Configuration menu
    Copy the full SHA
    4aa0233 View commit details
    Browse the repository at this point in the history