Skip to content

Conversation

@kaijchen
Copy link
Member

@kaijchen kaijchen commented Oct 9, 2022

What changes were proposed in this pull request?

Replace by : .

Does this PR introduce any user-facing change?

No.

How was this patch tested?

No need.

@codecov-commenter
Copy link

codecov-commenter commented Oct 9, 2022

Codecov Report

Merging #255 (1a4dd95) into master (c89f95c) will decrease coverage by 0.02%.
The diff coverage is 0.00%.

@@             Coverage Diff              @@
##             master     #255      +/-   ##
============================================
- Coverage     59.16%   59.13%   -0.03%     
+ Complexity     1340     1339       -1     
============================================
  Files           163      163              
  Lines          8810     8810              
  Branches        833      833              
============================================
- Hits           5212     5210       -2     
- Misses         3332     3333       +1     
- Partials        266      267       +1     
Impacted Files Coverage Δ
.../hadoop/mapreduce/task/reduce/RssBypassWriter.java 61.90% <0.00%> (ø)
...pache/hadoop/mapreduce/task/reduce/RssFetcher.java 89.06% <0.00%> (-1.57%) ⬇️

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

@jerqi jerqi requested a review from zuston October 9, 2022 11:25
@jerqi
Copy link
Contributor

jerqi commented Oct 9, 2022

@zuston Kerberos test seems flaky.

@zuston
Copy link
Member

zuston commented Oct 10, 2022

@zuston Kerberos test seems flaky.

Yes. This test makes me confused, I will take a look

@zuston zuston merged commit f3a6dd8 into apache:master Oct 10, 2022
@zuston
Copy link
Member

zuston commented Oct 10, 2022

Merged. Thanks @kaijchen

@kaijchen
Copy link
Member Author

Thanks @zuston for merge this.

@kaijchen kaijchen deleted the ascii branch October 10, 2022 02:50
@jerryshao
Copy link
Contributor

It would be better to cherry-pick this to 0.6 branch and kick off a new RC release. CC\ @jerqi

@jerqi
Copy link
Contributor

jerqi commented Oct 10, 2022

It would be better to cherry-pick this to 0.6 branch and kick off a new RC release. CC\ @jerqi

Ok, I will cherry-pick this pr.

jerqi pushed a commit that referenced this pull request Oct 10, 2022
What changes were proposed in this pull request?
Replace : by : .

Does this PR introduce any user-facing change?
No.

How was this patch tested?
No need.
smallzhongfeng added a commit that referenced this pull request Mar 28, 2023
<!--
1. Title: [#<issue>] <type>(<scope>): <subject>
   Examples:
     - "[#123] feat(operator): support xxx"
     - "[#233] fix: check null before access result in xxx"
     - "[MINOR] refactor: fix typo in variable name"
     - "[MINOR] docs: fix typo in README"
     - "[#255] test: fix flaky test NameOfTheTest"
   Reference: https://www.conventionalcommits.org/en/v1.0.0/
2. Contributor guidelines:
   https://github.com/apache/incubator-uniffle/blob/master/CONTRIBUTING.md
3. If the PR is unfinished, please mark this PR as draft.
-->

### What changes were proposed in this pull request?
Four util methods for creating thread pools have been added.

### Why are the changes needed?
Fix: #755

### Does this PR introduce _any_ user-facing change?
No.

### How was this patch tested?
Origin uts.
xianjingfeng pushed a commit to xianjingfeng/uniffle that referenced this pull request Apr 5, 2023
…#767)

<!--
1. Title: [#<issue>] <type>(<scope>): <subject>
   Examples:
     - "[apache#123] feat(operator): support xxx"
     - "[apache#233] fix: check null before access result in xxx"
     - "[MINOR] refactor: fix typo in variable name"
     - "[MINOR] docs: fix typo in README"
     - "[apache#255] test: fix flaky test NameOfTheTest"
   Reference: https://www.conventionalcommits.org/en/v1.0.0/
2. Contributor guidelines:
   https://github.com/apache/incubator-uniffle/blob/master/CONTRIBUTING.md
3. If the PR is unfinished, please mark this PR as draft.
-->

### What changes were proposed in this pull request?
Four util methods for creating thread pools have been added.

### Why are the changes needed?
Fix: apache#755

### Does this PR introduce _any_ user-facing change?
No.

### How was this patch tested?
Origin uts.
smallzhongfeng added a commit that referenced this pull request Apr 17, 2023
<!--
1. Title: [#<issue>] <type>(<scope>): <subject>
   Examples:
     - "[#123] feat(operator): support xxx"
     - "[#233] fix: check null before access result in xxx"
     - "[MINOR] refactor: fix typo in variable name"
     - "[MINOR] docs: fix typo in README"
     - "[#255] test: fix flaky test NameOfTheTest"
   Reference: https://www.conventionalcommits.org/en/v1.0.0/
2. Contributor guidelines:
   https://github.com/apache/incubator-uniffle/blob/master/CONTRIBUTING.md
3. If the PR is unfinished, please mark this PR as draft.
-->

### What changes were proposed in this pull request?
Add `tag` as label in the metrics.

So that we can separate the metrics in Grafana.

### Why are the changes needed?
Fix: #778 

### Does this PR introduce _any_ user-facing change?
No.

### How was this patch tested?
fix uts.
maobaolong added a commit that referenced this pull request Jan 7, 2025
… shuffle (#2132)

<!--
1. Title: [#<issue>] <type>(<scope>): <subject>
   Examples:
     - "[#123] feat(operator): support xxx"
     - "[#233] fix: check null before access result in xxx"
     - "[MINOR] refactor: fix typo in variable name"
     - "[MINOR] docs: fix typo in README"
     - "[#255] test: fix flaky test NameOfTheTest"
   Reference: https://www.conventionalcommits.org/en/v1.0.0/
2. Contributor guidelines:
   https://github.com/apache/incubator-uniffle/blob/master/CONTRIBUTING.md
3. If the PR is unfinished, please mark this PR as draft.
-->

### What changes were proposed in this pull request?

Server send heartbeat to coordinator while receive unregister shuffle request.

### Why are the changes needed?

Without this PR, server could not heartbeat the updated app info after unregister this app, so the coordinator and dashboard could display the outdate information.

### Does this PR introduce _any_ user-facing change?

No.

### How was this patch tested?

Tested by a tiny spark job executed by spark-shell

- shell command line

```shell
bin/spark-shell  --master  spark://localhost:7077  --deploy-mode client --conf spark.rss.coordinator.quorum=localhost:19999   --conf spark.shuffle.manager=org.apache.spark.shuffle.RssShuffleManager  --conf spark.rss.storage.type=LOCALFILE --conf spark.serializer=org.apache.spark.serializer.KryoSerializer --conf spark.rss.test.mode.enable=true --conf spark.rss.client.type=GRPC_NETTY --conf spark.default.parallelism=16  -i test.scala
```

- test.scala
```scala
val data = sc.parallelize(Seq(("A", 1), ("B", 2), ("C", 3), ("A", 4), ("B", 5), ("A", 6), ("A", 7),("A", 7), ("A", 7), ("A", 7), ("A", 7), ("A", 7), ("A", 7), ("A", 7), ("A", 7), ("A", 7), ("A", 7), ("A", 7), ("A", 7)));
val result = data.reduceByKey(_ + _);
result.collect().foreach(println);
System.exit(0);
```

<img width="2540" alt="image" src="https://github.com/user-attachments/assets/bbd7c534-9fea-4e86-ba5e-84dff3d9c98c">
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants