Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support batch ingestion in bulk API (#12457) #13306

Merged
merged 18 commits into from
Apr 30, 2024
Merged

Support batch ingestion in bulk API (#12457) #13306

merged 18 commits into from
Apr 30, 2024

Commits on Apr 26, 2024

  1. [PoC][issues-12457] Support Batch Ingestion 

    Signed-off-by: Liyun Xiu <xiliyun@amazon.com>
    @chishui
    chishui committed Apr 26, 2024
    Configuration menu
    Copy the full SHA
    b7cdcaa View commit details
    Browse the repository at this point in the history

  2. Rewrite batch interface and handle error and metrics 

    Signed-off-by: Liyun Xiu <xiliyun@amazon.com>
    @chishui
    chishui committed Apr 26, 2024
    Configuration menu
    Copy the full SHA
    8d4226d View commit details
    Browse the repository at this point in the history

  3. Remove unnecessary change 

    Signed-off-by: Liyun Xiu <xiliyun@amazon.com>
    @chishui
    chishui committed Apr 26, 2024
    Configuration menu
    Copy the full SHA
    384ba9d View commit details
    Browse the repository at this point in the history

  4. Revert some unnecessary test change 

    Signed-off-by: Liyun Xiu <xiliyun@amazon.com>
    @chishui
    chishui committed Apr 26, 2024
    Configuration menu
    Copy the full SHA
    21f0d71 View commit details
    Browse the repository at this point in the history

  5. Keep executeBulkRequest main logic untouched 

    Signed-off-by: Liyun Xiu <xiliyun@amazon.com>
    @chishui
    chishui committed Apr 26, 2024
    Configuration menu
    Copy the full SHA
    4ba55c2 View commit details
    Browse the repository at this point in the history

  6. Add UT 

    Signed-off-by: Liyun Xiu <xiliyun@amazon.com>
    @chishui
    chishui committed Apr 26, 2024
    Configuration menu
    Copy the full SHA
    d9a832d View commit details
    Browse the repository at this point in the history

  7. Add UT & yamlRest test, fix BulkRequest se/deserialization 

    Signed-off-by: Liyun Xiu <xiliyun@amazon.com>
    @chishui
    chishui committed Apr 26, 2024
    Configuration menu
    Copy the full SHA
    d5a4740 View commit details
    Browse the repository at this point in the history

  8. Add missing java docs 

    Signed-off-by: Liyun Xiu <xiliyun@amazon.com>
    @chishui
    chishui committed Apr 26, 2024
    Configuration menu
    Copy the full SHA
    e6c3b9f View commit details
    Browse the repository at this point in the history

  9. Remove Writable from BatchIngestionOption 

    Signed-off-by: Liyun Xiu <xiliyun@amazon.com>
    @chishui
    chishui committed Apr 26, 2024
    Configuration menu
    Copy the full SHA
    8ef01cd View commit details
    Browse the repository at this point in the history

  10. Add more UTs 

    Signed-off-by: Liyun Xiu <xiliyun@amazon.com>
    @chishui
    chishui committed Apr 26, 2024
    Configuration menu
    Copy the full SHA
    696fe74 View commit details
    Browse the repository at this point in the history

  11. Fix spotlesscheck 

    Signed-off-by: Liyun Xiu <xiliyun@amazon.com>
    @chishui
    chishui committed Apr 26, 2024
    Configuration menu
    Copy the full SHA
    af65dbe View commit details
    Browse the repository at this point in the history

Commits on Apr 27, 2024

  1. Merge branch 'main' into main 

    Signed-off-by: Liyun Xiu <chishui2@gmail.com>
    @chishui
    chishui authored Apr 27, 2024
    Configuration menu
    Copy the full SHA
    54d4919 View commit details
    Browse the repository at this point in the history

Commits on Apr 29, 2024

  1. Rename parameter name to batch_size 

    Signed-off-by: Liyun Xiu <xiliyun@amazon.com>
    @chishui
    chishui committed Apr 29, 2024
    Configuration menu
    Copy the full SHA
    81afff2 View commit details
    Browse the repository at this point in the history

  2. Add more rest yaml tests & update rest spec 

    Signed-off-by: Liyun Xiu <xiliyun@amazon.com>
    @chishui
    chishui committed Apr 29, 2024
    Configuration menu
    Copy the full SHA
    574d2ff View commit details
    Browse the repository at this point in the history

  3. Remove batch_ingestion_option and only use batch_size 

    Signed-off-by: Liyun Xiu <xiliyun@amazon.com>
    @chishui
    chishui committed Apr 29, 2024
    Configuration menu
    Copy the full SHA
    e2fb585 View commit details
    Browse the repository at this point in the history

  4. Throw invalid request exception for invalid batch_size 

    Signed-off-by: Liyun Xiu <xiliyun@amazon.com>
    @chishui
    chishui committed Apr 29, 2024
    Configuration menu
    Copy the full SHA
    334e0c1 View commit details
    Browse the repository at this point in the history

  5. Update server/src/main/java/org/opensearch/action/bulk/BulkRequest.java 

    Co-authored-by: Andriy Redko <drreta@gmail.com>
Signed-off-by: Liyun Xiu <chishui2@gmail.com>
    @chishui @reta
    chishui and reta authored Apr 29, 2024
    Configuration menu
    Copy the full SHA
    68cabe1 View commit details
    Browse the repository at this point in the history

Commits on Apr 30, 2024

  1. Remove version constant 

    Signed-off-by: Liyun Xiu <xiliyun@amazon.com>
    @chishui
    chishui committed Apr 30, 2024
    Configuration menu
    Copy the full SHA
    3cc7f41 View commit details
    Browse the repository at this point in the history