Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

merge master into presto-timestamp #2

Merged
merged 17 commits into from
Dec 24, 2021
Merged

merge master into presto-timestamp #2

merged 17 commits into from
Dec 24, 2021

Conversation

tjiuming
Copy link
Owner

(If this PR fixes a github issue, please add Fixes #<xyz>.)

Fixes #

(or if this PR is one task of a github issue, please add Master Issue: #<xyz> to link to the master issue.)

Master Issue: #

Motivation

Explain here the context, and why you're making that change. What is the problem you're trying to solve.

Modifications

Describe the modifications you've done.

Verifying this change

  • Make sure that the change passes the CI checks.

(Please pick either of the following options)

This change is a trivial rework / code cleanup without any test coverage.

(or)

This change is already covered by existing tests, such as (please describe tests).

(or)

This change added tests and can be verified as follows:

(example:)

  • Added integration tests for end-to-end deployment with large payloads (10MB)
  • Extended integration test for recovery after broker failure

Does this pull request potentially affect one of the following parts:

If yes was chosen, please highlight the changes

  • Dependencies (does it add or upgrade a dependency): (yes / no)
  • The public API: (yes / no)
  • The schema: (yes / no / don't know)
  • The default values of configurations: (yes / no)
  • The wire protocol: (yes / no)
  • The rest endpoints: (yes / no)
  • The admin cli options: (yes / no)
  • Anything that affects deployment: (yes / no / don't know)

Documentation

Check the box below and label this PR (if you have committer privilege).

Need to update docs?

  • doc-required

    (If you need help on updating docs, create a doc issue)

  • no-need-doc

    (Please explain why)

  • doc

    (If this PR contains doc changes)

kijanowski and others added 17 commits December 23, 2021 16:59
…pache#13381)

### Motivation

`pulsar-client` runs `conf/pulsar_tools_env.sh` to set `PULSAR_EXTRA_OPTS`. If `PULSAR_EXTRA_OPTS` are given in the command line and contain `-Xmx`, it will be overridden by `PULSAR_MEM`.

### Modifications

Exchange the order of applying env variables. First go the hardcoded PULSAR_MEM settings, then potentially the user settings.
* Deprecate apachepulsar/pulsar-grafana docker image

* Add license to new file
…che#13443)

* [website][upgrade]feat: website upgrade / docs migration - 2.8.1

Signed-off-by: LiLi <urfreespace@gmail.com>

* [website][upgrade]feat: website upgrade / docs migration - 2.8.2

Signed-off-by: LiLi <urfreespace@gmail.com>

Co-authored-by: Anonymitaet <50226895+Anonymitaet@users.noreply.github.com>
---

*Motivation*

The `namespaceEventsSystemTopicFactory` is created when you will
use it. But the `createSystemTopicFactoryIfNeeded()` may failed
which will cause the `namespaceEventsSystemTopicFactory` is null
and throw a NPE error from the method.

*Modifications*

- throw the error and failed the method when there has exceptions in
`createSystemTopicFactoryIfNeeded()`
### Motivation
The time unit in this exception message is ns, which is not very readable. We can change it from ns to ms.
```
org.apache.pulsar.client.api.PulsarClientException$TimeoutException:
The producer xxx can not send message to the topic xxx within given timeout : createdAt 461913074 ns ago, firstSentAt 29545553038276935 ns ago, lastSentAt 29545553038276935 ns ago, retryCount 0 at org.apache.pulsar.client.api.PulsarClientException.unwrap(PulsarClientException.java:916)
at org.apache.pulsar.client.impl.TypedMessageBuilderImpl.send(TypedMessageBuilderImpl.java:93)
at org.apache.pulsar.client.impl.ProducerBase.send(ProducerBase.java:63)
at com.yum.boh.oh.service.impl.StoreOrderPostServiceImpl.generalProcessing(StoreOrderPostServiceImpl.java:272)
at com.yum.boh.oh.service.impl.StoreOrderPostServiceImpl.saveThirdOrder(StoreOrderPostServiceImpl.java:72)
at com.yum.boh.oh.controller.StoreOrderController.postOrderInfo$original$T8425mfx(StoreOrderController.java:39)
at com.yum.boh.oh.controller.StoreOrderController.postOrderInfo$original$T8425mfx$accessor$vJljNzML(StoreOrderController.java)
at com.yum.boh.oh.controller.StoreOrderController$auxiliary$nysalhgy.call(Unknown Source)
at org.apache.skywalking.apm.agent.core.plugin.interceptor.enhance.InstMethodsInter.intercept(InstMethodsInter.java:86)
```

### Modifications
Change the time units from ns to ms for ProducerImpl#OpSendMsg.
…pache#13471)

### Motivation
Currently, the Wireshark dissector doesn't decode send message command metadata correctly, and some TCP packages are marked incorrectly when using pulsar dissector.

### Modifications
- Fix decode send command metadata behavior
- Fix dissect pulsar message marked incorrect TCP packages
- Add DEBUG message for debugging
- Support decode more Send command metadata info
### Motivation
Master Issue: apache#13444

### Modifications

Backlog quota support cross multiple clusters
…pache#13420)

### Motivation

When request the `DELETE: /admin/brokers/configuration/dispatcherMinReadBatchSize`, which return the`org.apache.pulsar.metadata.api.MetadataStoreException$NotFoundException`.  The Pulsar does not create the dynamic configuration resource path on metadata if it does not exist, this behavior is different with Pulsar 2.8.
…13194) (apache#13249)

Fixes apache#13194 

### Motivation
https://github.com/apache/pulsar/blob/38fb839154462fc5c6b0b4293f02762ed4021cd9/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/BacklogQuotaManager.java#L200-L219
BacklogQuotaManager.dropBacklogForTimeLimit may fall into dead loop in some conditions, e.g.
`backlogQuotaDefaultLimitSecond` is enabled
1. producer stop produce after produced some messages, current ledger is A
2. times up, triggered ledger rollover, a new ledger B created which is empty (no entries)
3. now lastConfirmedEntry is `A:last-entry-id`
4. after `backlogQuotaDefaultLimitSecond` times up, it'll reset cursor to position `A:last-entry-id+1` which is only valid, so loop begin until the producer resume produce

### Modifications

Record the previous slowestReaderPosition, if it is same with newer slowestReaderPosition after `resetCursor`, then exit loop.
…3454)

### Motivation

When a large message is sent by chunks, each chunk needs to reserve a spot of the semaphore. However, when it failed, the already reserved memory from limiter and spots from semaphore are not released.

### Modifications

- Release the semaphore and memory when `canEnqueueRequest` returns false for chunks.
- Add `testChunksEnqueueFailed` to cover this case. It sends a large message whose number of chunks is greater than the `maxPendingMessages` so that the first time `canEnqueueRequest` returns true while the following `canEnqueueRequest` calls will return false.
)

* Persistence policies support cross multiple clusters
@tjiuming tjiuming merged commit b6a9e00 into presto-timestamp Dec 24, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.