Skip to content

Commit

Permalink
[Test](index compaction) refine index compaction case (#43203)
Browse files Browse the repository at this point in the history
### What problem does this PR solve?
<!--
You need to clearly describe your PR in this part:

1. What problem was fixed (it's best to include specific error reporting
information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what
is it now, why was it modified, and what possible impacts might there
be.
3. What features were added. Why this function was added.
4. Which codes were refactored and why this part of the code was
refactored.
5. Which functions were optimized and what is the difference before and
after the optimization.

The description of the PR needs to enable reviewers to quickly and
clearly understand the logic of the code modification.
-->

<!--
If there are related issues, please fill in the issue number.
- If you want the issue to be closed after the PR is merged, please use
"close #12345". Otherwise, use "ref #12345"
-->

<!--
If this PR is followup a preivous PR, for example, fix the bug that
introduced by a related PR,
link the PR here
-->
Related PR: #41625 

Problem Summary:
Refine case of index compaction, DO NOT need so much data.

### Check List (For Committer)

- Test <!-- At least one of them must be included. -->

    - [x] Regression test
    - [ ] Unit Test
    - [ ] Manual test (add detailed scripts or steps below)
    - [ ] No need to test or manual test. Explain why:
- [ ] This is a refactor/code format and no logic has been changed.
        - [ ] Previous test can cover this change.
        - [ ] No colde files have been changed.
        - [ ] Other reason <!-- Add your reason?  -->

- Behavior changed:

    - [x] No.
    - [ ] Yes. <!-- Explain the behavior change -->

- Does this need documentation?

    - [x] No.
- [ ] Yes. <!-- Add document PR link here. eg:
apache/doris-website#1214 -->

- Release note

    <!-- bugfix, feat, behavior changed need a release note -->
    <!-- Add one line release note for this PR. -->
    None

### Check List (For Reviewer who merge this PR)

- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label <!-- Add branch pick label that this PR
should merge into -->
  • Loading branch information
airborne12 authored Nov 4, 2024
1 parent 10cb92d commit ad7b2bb
Show file tree
Hide file tree
Showing 21 changed files with 20,004 additions and 8 deletions.
1,000 changes: 1,000 additions & 0 deletions regression-test/data/inverted_index_p0/index_compaction/documents-1.json

Large diffs are not rendered by default.

1,000 changes: 1,000 additions & 0 deletions regression-test/data/inverted_index_p0/index_compaction/documents-10.json

Large diffs are not rendered by default.

1,000 changes: 1,000 additions & 0 deletions regression-test/data/inverted_index_p0/index_compaction/documents-11.json

Large diffs are not rendered by default.

1,000 changes: 1,000 additions & 0 deletions regression-test/data/inverted_index_p0/index_compaction/documents-12.json

Large diffs are not rendered by default.

1,000 changes: 1,000 additions & 0 deletions regression-test/data/inverted_index_p0/index_compaction/documents-13.json

Large diffs are not rendered by default.

1,000 changes: 1,000 additions & 0 deletions regression-test/data/inverted_index_p0/index_compaction/documents-14.json

Large diffs are not rendered by default.

1,000 changes: 1,000 additions & 0 deletions regression-test/data/inverted_index_p0/index_compaction/documents-15.json

Large diffs are not rendered by default.

1,000 changes: 1,000 additions & 0 deletions regression-test/data/inverted_index_p0/index_compaction/documents-16.json

Large diffs are not rendered by default.

1,000 changes: 1,000 additions & 0 deletions regression-test/data/inverted_index_p0/index_compaction/documents-17.json

Large diffs are not rendered by default.

1,000 changes: 1,000 additions & 0 deletions regression-test/data/inverted_index_p0/index_compaction/documents-18.json

Large diffs are not rendered by default.

1,000 changes: 1,000 additions & 0 deletions regression-test/data/inverted_index_p0/index_compaction/documents-19.json

Large diffs are not rendered by default.

1,000 changes: 1,000 additions & 0 deletions regression-test/data/inverted_index_p0/index_compaction/documents-2.json

Large diffs are not rendered by default.

1,000 changes: 1,000 additions & 0 deletions regression-test/data/inverted_index_p0/index_compaction/documents-20.json

Large diffs are not rendered by default.

1,000 changes: 1,000 additions & 0 deletions regression-test/data/inverted_index_p0/index_compaction/documents-3.json

Large diffs are not rendered by default.

1,000 changes: 1,000 additions & 0 deletions regression-test/data/inverted_index_p0/index_compaction/documents-4.json

Large diffs are not rendered by default.

1,000 changes: 1,000 additions & 0 deletions regression-test/data/inverted_index_p0/index_compaction/documents-5.json

Large diffs are not rendered by default.

1,000 changes: 1,000 additions & 0 deletions regression-test/data/inverted_index_p0/index_compaction/documents-6.json

Large diffs are not rendered by default.

1,000 changes: 1,000 additions & 0 deletions regression-test/data/inverted_index_p0/index_compaction/documents-7.json

Large diffs are not rendered by default.

1,000 changes: 1,000 additions & 0 deletions regression-test/data/inverted_index_p0/index_compaction/documents-8.json

Large diffs are not rendered by default.

1,000 changes: 1,000 additions & 0 deletions regression-test/data/inverted_index_p0/index_compaction/documents-9.json

Large diffs are not rendered by default.

Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ import org.awaitility.Awaitility

suite("test_index_compaction_p0", "p0, nonConcurrent") {

def compaction_table_name = "httplogs"
def compaction_table_name = "test_index_compaction_p0_httplogs"

def load_json_data = {table_name, file_name ->
// load the json data
Expand Down Expand Up @@ -71,15 +71,11 @@ suite("test_index_compaction_p0", "p0, nonConcurrent") {
"disable_auto_compaction" = "true"
);
"""
def executor = Executors.newFixedThreadPool(20)
(1..20).each { i ->
executor.submit {
def fileName = "documents-" + i + ".json"
load_json_data.call(compaction_table_name, """${getS3Url()}/regression/inverted_index_cases/httplogs/${fileName}""")
}
def fileName = "documents-" + i + ".json"
load_json_data.call(compaction_table_name, """${fileName}""")

}
executor.shutdown()
executor.awaitTermination(1, TimeUnit.MINUTES)

def backendId_to_backendIP = [:]
def backendId_to_backendHttpPort = [:]
Expand Down

0 comments on commit ad7b2bb

Please sign in to comment.