Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Branch 2.5 All test #14

Open
wants to merge 110 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
110 commits
Select commit Hold shift + click to select a range
f2afad3
Release 2.5.0
sijie Jan 6, 2020
80ad05f
Fix create consumer on partitioned topic while disable topic auto cre…
codelipenghui Jan 9, 2020
2efd47d
Fixed static linking on C++ lib on MacOS (#5581)
merlimat Feb 7, 2020
2074c7b
[pulsar-broker] close managed-ledgers before giving up bundle ownersh…
rdhabalia Jan 6, 2020
f98d4c1
Expose bookkeeper expose explicit lac in broker.conf (#5822)
codelipenghui Feb 10, 2020
0f44118
[build] Skip javadoc task for pulsar-client-kafka-compact modules (#5…
sijie Jan 14, 2020
33b2e24
add_backlogSize_in_topicStat (#5914)
k2la Jan 7, 2020
e3877d8
Allow to enable/disable delayed delivery for messages on namespace (#…
wolfstudy Jan 21, 2020
63a66d0
Fix negative un-ack messages in consumer stats (#5929)
codelipenghui Jan 6, 2020
31fcb38
Upgrade Avro to 1.9.1 (#5938)
Jan 6, 2020
0729333
Avoid using same OpAddEntry between different ledger handles (#5942)
codelipenghui Jan 9, 2020
d662a4a
Prevent creation of regular topic with the same name as existing part…
Jan 6, 2020
6be3149
[pulsar-broker] Clean up closed producer to avoid publish-time for p…
rdhabalia Jan 9, 2020
f3ca73f
Fix unit test (#6006)
codelipenghui Jan 8, 2020
08ef06d
Expose lastConsumedTimestamp and lastAckedTimestamp to consumer stats…
zymap Feb 4, 2020
ffa0b04
Fix issue 5505 (#6060)
ntysdd Jan 23, 2020
4fbdc95
make acker transient (#6064)
yjshen Jan 23, 2020
5680a81
PIP-55: Refresh Authentication Credentials (#6074)
merlimat Feb 10, 2020
46a1b28
Fix zero queue consumer message redelivery (#6076)
codelipenghui Jan 17, 2020
196aa81
Support delete inactive topic when subscriptions caught up (#6077)
codelipenghui Jan 19, 2020
d7aa4d7
Add a message on how to make log refresh immediately when starting a …
murong00 Feb 29, 2020
693b59d
Fix message redelivery for zero queue consumer while using async api …
codelipenghui Jan 20, 2020
e32173e
[Functions] The argument and description for dead letter topic is wro…
sijie Jan 21, 2020
a2c3858
[Websocket] Websocket doesn't set the correct cluster data (#6102)
sijie Jan 24, 2020
13fd6b3
Fix zeroQueueConsumer using listener (#6106)
hrsakai Jan 24, 2020
4c3112a
[Broker]Reset cursor with a non-exists position (#6120)
yjshen Feb 10, 2020
2672574
[pulsar-admin] allow tenant admin to manage subscription permission (…
rdhabalia Mar 18, 2020
ea134ff
Add timeout to search for web service URLs to avoid web threads getti…
Jan 28, 2020
7c13b40
Fix broker client tls settings error (#6128)
jiazhai Feb 24, 2020
077bd71
Output resource usage rate to log on broker (#6152)
Feb 8, 2020
a9df727
[Issue-5994]: Start proxy pods when at least one broker pod is runnin…
roman-popenov Feb 1, 2020
850ede6
add missing check to dashboard-ingress (helm chart) (#6160)
tmemenga Feb 1, 2020
2d9e063
Restore clusterDispatchRate policy for compatibility (#6176)
Feb 10, 2020
1736aa1
Supports evenly distribute topics count when splits bundle (#6241)
codelipenghui Feb 13, 2020
b535ac4
Introduce maxMessagePublishBufferSizeInMB configuration to avoid brok…
codelipenghui Feb 16, 2020
88dda60
Namespace level offloader (#6183)
gaoran10 Feb 10, 2020
e33238f
[Issue 5904]Support `unload` all partitions of a partitioned topic (#…
ltamber Feb 16, 2020
c6d59d9
Create managed ledger path on local zookeeper when create partitions …
codelipenghui Feb 8, 2020
4c19484
Corrected the method name for source implementation (#6190)
abhilashmandaliya Feb 11, 2020
4c86357
[deployement] make kubernetes yamls for aws operational (#6192)
trexinc Feb 7, 2020
4324c5b
Fix get schema version in HttpLookupService. (#6193)
codelipenghui Feb 7, 2020
4f3f5f8
Pin the netty-transport-native-epoll to avoid conflicts (#6194)
merlimat Feb 8, 2020
c40ccf8
[ISSUE-6131]: Ensure JVM memory and GC options are set for bookie (#6…
roman-popenov Feb 8, 2020
a89bd2d
[functions] Default functionAuthProvider when running in k8s (#6203)
addisonj Mar 18, 2020
31e8e59
Fix bug that backlog message that has not yet expired could be delete…
Feb 7, 2020
bb3f0bb
[authentication] Validate tokens for binary connections (#6233)
addisonj Mar 18, 2020
bd439b3
Use fully qualified hostname as default to advertise brokers (#6235)
merlimat Feb 7, 2020
4f1bedc
[Issue 6173][compaction] Fix log compaction for flow control/empty to…
fantapsody Feb 10, 2020
ba60a3f
Fix deploy of WindowFunctions (#6246)
seeday Feb 16, 2020
ef40db5
[C++] Fixed memory corruption on ExecutorService destructor (#6270)
merlimat Feb 10, 2020
8f56884
[C++] Fixed handling of canceled timer events on NegativeAcksTracker …
merlimat Feb 10, 2020
d25cb39
Fix bug that tenants whose allowed clusters include global cannot be …
Feb 12, 2020
d3063da
[Issue 4070][pulsar-client-cpp] Fix for possible deadlock when closin…
heronr Feb 14, 2020
bf0f4a7
Remove problematic semicolon from conf (#6303)
klevy-toast Feb 12, 2020
ad01245
Enable get precise backlog and backlog without delayed messages. (#6310)
codelipenghui Feb 16, 2020
4ba38bb
Fixed casting in ZooKeeperCache.getDataIfPresent() (#6313)
merlimat Feb 12, 2020
b3af045
KeyValue schema support for pulsar sql (#6325)
gaoran10 Feb 17, 2020
c20071b
Should flush the last potential duplicated since can't combine potent…
codelipenghui Feb 14, 2020
eb17206
Upgrade ZooKeeper to 3.5.7 (#6329)
merlimat Feb 15, 2020
aaf87f6
Windows CMake corrections (#6336)
heronr Feb 18, 2020
7112e21
client: make SubscriptionMode a member of ConsumerConfigurationData (…
jiazhai Feb 18, 2020
e44b822
Avoid get partition metadata while the topic name is a partition name…
codelipenghui Feb 17, 2020
e939a51
explicit statement env 'BOOKIE_MEM' and 'BOOKIE_GC' for values-mini.y…
liyuntao Feb 17, 2020
08196be
Fix java doc for key shared policy. (#6341)
codelipenghui Feb 17, 2020
ac59e09
[Java Reader Client] Start reader inside batch result in read first m…
yjshen Feb 24, 2020
37906e3
Fix broker to specify a list of bookie groups. (#6349)
murong00 Feb 24, 2020
64690bf
Independent schema is set for each consumer generated by topic (#6356)
congbobo184 Mar 6, 2020
b4d7dc5
remove future.join() from PulsarSinkEffectivelyOnceProcessor (#6361)
nlu90 Feb 19, 2020
e173f60
[ClientAPI]Fix hasMessageAvailable() (#6362)
yjshen Mar 3, 2020
e7cf371
Creating a topic does not wait for creating cursor of replicators (#6…
codelipenghui Feb 23, 2020
7fe98b8
[Issue 6355][HELM] autorecovery - could not find or load main class (…
jharris- Feb 22, 2020
59f055b
[Pulsar-Client] Stop shade snappy-java in pulsar-client-shaded (#6375)
yjshen Feb 21, 2020
8f1e0d0
fix duplicate key to send propertys (#6390)
liudezhi2098 Mar 20, 2020
8174617
[Issue 6168] Fix Unacked Message Tracker by Using Time Partition on C…
k2la Mar 2, 2020
eaba596
[Reader] Should set either start message id or start message from rol…
yjshen Feb 24, 2020
3313e6c
Seek to the first one >= timestamp (#6393)
yjshen Feb 24, 2020
0cbda7f
[Minor] Fix java code errors reported by lgtm. (#6398)
yjshen Feb 24, 2020
8cebbbb
Close ZK before canceling future with exception (#6228) (#6399)
pawellozinski Feb 29, 2020
d36eee2
Fixed enum package not found (#6401)
tuteng Feb 24, 2020
3ee06e3
Consumer received duplicated deplayed messages upon restart
liudezhi2098 Feb 26, 2020
379bdec
Add verification for SchemaDefinitionBuilderImpl.java (#6405)
codelipenghui Feb 25, 2020
0e9c56a
[Issue 3762][Schema] Fix the problem with parsing of an Avro schema r…
vzhikserg Feb 25, 2020
31e3d19
[Java client] MultiTopics discovery is broken due to discovery task s…
avimas Mar 18, 2020
52d5d7c
Update BatchReceivePolicy.java (#6423)
codelipenghui Feb 26, 2020
f6fb44d
Bump netty version to 4.1.45.Final (#6424)
dzmitryk Feb 27, 2020
6186aef
Fix publish buffer limit does not take effect
ltamber Feb 27, 2020
22fbdc1
[Flink-Connector]Get PulsarClient from cache should always return an …
yjshen Mar 1, 2020
0afcf1b
fix the bug of authenticationData is't initialized. (#6440)
bilahepan Feb 29, 2020
56c7079
Fixed the max backoff configuration for lookups (#6444)
merlimat Mar 3, 2020
eb7aa2a
Use System.nanoTime() instead of System.currentTimeMillis() (#6454)
racorn Mar 3, 2020
4becef9
[Broker] Create namespace failed when TLS is enabled in PulsarStandal…
yjshen Mar 2, 2020
f94eb89
Improve cpp-client-lib: provide another `libpulsarwithdeps.a` in dep/…
jiazhai Mar 18, 2020
85257b5
pulsar-proxy: fix correct name for proxy thread executor name (#6460)
rdhabalia Mar 8, 2020
89e44ec
[pulsar-proxy] fix logging for published messages (#6474)
sijie Mar 5, 2020
b5322bc
Fix create partitioned topic with a substring of an existing topic na…
codelipenghui Mar 6, 2020
c4902d6
Avoid calling ConsumerImpl::redeliverMessages() when message list is …
merlimat Mar 5, 2020
1e1dd06
Fix some async method problems at PersistentTopicsBase. (#6483)
codelipenghui Mar 6, 2020
5911b8b
Fix memory leak when running topic compaction. (#6485)
racorn Mar 6, 2020
4724659
[proxy] Fix proxy routing to functions worker (#6486)
addisonj Mar 6, 2020
e05b786
[pulsar-client] fix deadlock on send failure (#6488)
rdhabalia Mar 6, 2020
53b4073
[broker] Timeout API calls in BrokerService (#6489)
addisonj Mar 18, 2020
1b36a7a
[pulsar-client-cpp] Fix Redelivery of Messages on UnackedMessageTrack…
k2la Mar 9, 2020
77f2c82
Start namespace service and schema registry service before start brok…
codelipenghui Mar 9, 2020
394791d
Instead of always using admin access for topic, use read/write/admin …
srkukarni Mar 8, 2020
b1c9c2f
Fix admin getLastMessageId return batchIndex (#6511)
congbobo184 Mar 18, 2020
b2df780
Disable channel auto read when publish rate or publish buffer exceede…
codelipenghui Mar 19, 2020
bf14a08
Don't increment unacked messages for the consumer with Exclusive/Fail…
codelipenghui Mar 19, 2020
e7459b4
Fix: topic with one partition cannot be updated (#6560)
jerrypeng Mar 19, 2020
58e52e0
Fix NPE while call getLastMessageId. (#6562)
murong00 Mar 19, 2020
68e5b79
Fixed protobuf-shaded version 2.5.0
tuteng Mar 21, 2020
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
1 change: 1 addition & 0 deletions bin/pulsar-daemon
Original file line number Diff line number Diff line change
Expand Up @@ -151,6 +151,7 @@ case $startStop in

rotate_out_log $out
echo starting $command, logging to $logfile
echo Note: Set immediateFlush to true in conf/log4j2.yaml will guarantee the logging event is flushing to disk immediately. The default behavior is switched off due to performance considerations.
pulsar=$PULSAR_HOME/bin/pulsar
nohup $pulsar $command "$@" > "$out" 2>&1 < /dev/null &
echo $! > $pid
Expand Down
2 changes: 1 addition & 1 deletion buildtools/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@

<groupId>org.apache.pulsar</groupId>
<artifactId>buildtools</artifactId>
<version>2.6.0-SNAPSHOT</version>
<version>2.5.0</version>
<packaging>jar</packaging>
<name>Pulsar Build Tools</name>

Expand Down
2 changes: 1 addition & 1 deletion conf/bkenv.sh
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ BOOKIE_CONF=${BOOKIE_CONF:-"$BK_HOME/conf/bookkeeper.conf"}
# BOOKIE_LOG_DIR=

# Memory size options
BOOKIE_MEM=${BOOKIE_MEM:-"-Xms2g -Xmx2g -XX:MaxDirectMemorySize=2g"}
BOOKIE_MEM=${BOOKIE_MEM:-${PULSAR_MEM:-"-Xms2g -Xmx2g -XX:MaxDirectMemorySize=2g"}}

# Garbage collection options
BOOKIE_GC=${BOOKIE_GC:-"-XX:+UseG1GC -XX:MaxGCPauseMillis=10 -XX:+ParallelRefProcEnabled -XX:+UnlockExperimentalVMOptions -XX:+AggressiveOpts -XX:+DoEscapeAnalysis -XX:ParallelGCThreads=32 -XX:ConcGCThreads=32 -XX:G1NewSizePercent=50 -XX:+DisableExplicitGC -XX:-ResizePLAB"}
Expand Down
43 changes: 43 additions & 0 deletions conf/broker.conf
Original file line number Diff line number Diff line change
Expand Up @@ -100,6 +100,17 @@ brokerDeleteInactiveTopicsEnabled=true
# How often to check for inactive topics
brokerDeleteInactiveTopicsFrequencySeconds=60

# Set the inactive topic delete mode. Default is delete_when_no_subscriptions
# 'delete_when_no_subscriptions' mode only delete the topic which has no subscriptions and no active producers
# 'delete_when_subscriptions_caught_up' mode only delete the topic that all subscriptions has no backlogs(caught up)
# and no active producers/consumers
brokerDeleteInactiveTopicsMode=delete_when_no_subscriptions

# Max duration of topic inactivity in seconds, default is not present
# If not present, 'brokerDeleteInactiveTopicsFrequencySeconds' will be used
# Topics that are inactive for longer than this value will be deleted
brokerDeleteInactiveTopicsMaxInactiveDurationSeconds=

# How frequently to proactively check and purge expired messages
messageExpiryCheckIntervalInMinutes=5

Expand Down Expand Up @@ -313,6 +324,18 @@ replicatedSubscriptionsSnapshotTimeoutSeconds=30
# Max number of snapshot to be cached per subscription.
replicatedSubscriptionsSnapshotMaxCachedPerSubscription=10

# Max memory size for broker handling messages sending from producers.
# If the processing message size exceed this value, broker will stop read data
# from the connection. The processing messages means messages are sends to broker
# but broker have not send response to client, usually waiting to write to bookies.
# It's shared across all the topics running in the same broker.
# Use -1 to disable the memory limitation. Default is 1/2 of direct memory.
maxMessagePublishBufferSizeInMB=

# Interval between checks to see if message publish buffer size is exceed the max message publish buffer size
# Use 0 or negative number to disable the max publish buffer limiting.
messagePublishBufferCheckIntervalInMillis=100

### --- Authentication --- ###
# Role names that are treated as "proxy roles". If the broker sees a request with
#role as proxyRoles - it will demand to see a valid original principal.
Expand Down Expand Up @@ -370,6 +393,9 @@ authenticationEnabled=false
# Autentication provider name list, which is comma separated list of class names
authenticationProviders=

# Interval of time for checking for expired authentication credentials
authenticationRefreshCheckSeconds=60

# Enforce authorization
authorizationEnabled=false

Expand Down Expand Up @@ -517,6 +543,10 @@ bookkeeperTLSTrustCertsFilePath=
# Enable/disable disk weight based placement. Default is false
bookkeeperDiskWeightBasedPlacementEnabled=false

# Set the interval to check the need for sending an explicit LAC
# A value of '0' disables sending any explicit LACs. Default is 0.
bookkeeperExplicitLacIntervalInMills=0

### --- Managed Ledger --- ###

# Number of bookies to use when creating a ledger
Expand Down Expand Up @@ -682,6 +712,14 @@ loadBalancerOverrideBrokerNicSpeedGbps=
# Name of load manager to use
loadManagerClassName=org.apache.pulsar.broker.loadbalance.impl.ModularLoadManagerImpl

# Supported algorithms name for namespace bundle split.
# "range_equally_divide" divides the bundle into two parts with the same hash range size.
# "topic_count_equally_divide" divides the bundle into two parts with the same topics count.
supportedNamespaceBundleSplitAlgorithms=[range_equally_divide,topic_count_equally_divide]

# Default algorithm name for namespace bundle split
defaultNamespaceBundleSplitAlgorithm=range_equally_divide

### --- Replication --- ###

# Enable replication metrics
Expand Down Expand Up @@ -747,6 +785,11 @@ exposePublisherStats=true
statsUpdateFrequencyInSecs=60
statsUpdateInitialDelayInSecs=60

# Enable expose the precise backlog stats.
# Set false to use published counter and consumed counter to calculate, this would be more efficient but may be inaccurate.
# Default is false.
exposePreciseBacklogInPrometheus=false

### --- Schema storage --- ###
# The schema storage implementation used by this broker
schemaRegistryStorageClassName=org.apache.pulsar.broker.service.schema.BookkeeperSchemaStorageFactory
Expand Down
9 changes: 9 additions & 0 deletions conf/standalone.conf
Original file line number Diff line number Diff line change
Expand Up @@ -334,6 +334,10 @@ bookkeeperTLSTrustCertsFilePath=
# Enable/disable disk weight based placement. Default is false
bookkeeperDiskWeightBasedPlacementEnabled=false

# Set the interval to check the need for sending an explicit LAC
# A value of '0' disables sending any explicit LACs. Default is 0.
bookkeeperExplicitLacIntervalInMills=0

### --- Managed Ledger --- ###

# Number of bookies to use when creating a ledger
Expand Down Expand Up @@ -527,6 +531,11 @@ exposeTopicLevelMetricsInPrometheus=true
# Enable topic level metrics
exposePublisherStats=true

# Enable expose the precise backlog stats.
# Set false to use published counter and consumed counter to calculate, this would be more efficient but may be inaccurate.
# Default is false.
exposePreciseBacklogInPrometheus=false

### --- Deprecated config variables --- ###

# Deprecated. Use configurationStoreServers
Expand Down
2 changes: 1 addition & 1 deletion dashboard/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@
<parent>
<groupId>org.apache.pulsar</groupId>
<artifactId>docker-images</artifactId>
<version>2.6.0-SNAPSHOT</version>
<version>2.5.0</version>
<relativePath>../docker</relativePath>
</parent>
<modelVersion>4.0.0</modelVersion>
Expand Down
27 changes: 13 additions & 14 deletions deployment/kubernetes/aws/bookkeeper.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -21,13 +21,12 @@

# SSDs for bookie journal storage
kind: StorageClass
apiVersion: storage.k8s.io/v1beta1
apiVersion: storage.k8s.io/v1
metadata:
name: bookie-ssd
provisioner: kubernetes.io/aws-ebs
parameters:
type: gp2
zones: us-east-2a
fsType: xfs
# To create encrypted ebs volume using kms
# encrypted: "true"
Expand All @@ -36,13 +35,12 @@ reclaimPolicy: Retain
---
# HDDs for bookie ledger storage
kind: StorageClass
apiVersion: storage.k8s.io/v1beta1
apiVersion: storage.k8s.io/v1
metadata:
name: bookie-hdd
provisioner: kubernetes.io/aws-ebs
parameters:
type: st1
zones: us-east-2a
fsType: xfs
# To create encrypted ebs volume using kms
# encrypted: "true"
Expand All @@ -54,14 +52,15 @@ kind: ConfigMap
metadata:
name: bookie-config
data:
PULSAR_MEM: "\"-Dio.netty.leakDetectionLevel=disabled -Dio.netty.recycler.linkCapacity=1024 -XX:+UseG1GC -XX:MaxGCPauseMillis=10 -XX:+ParallelRefProcEnabled -XX:+UnlockExperimentalVMOptions -XX:+AggressiveOpts -XX:+DoEscapeAnalysis -XX:ParallelGCThreads=32 -XX:ConcGCThreads=32 -XX:G1NewSizePercent=50 -XX:+DisableExplicitGC -XX:-ResizePLAB -XX:+ExitOnOutOfMemoryError -XX:+PerfDisableSharedMem -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCApplicationStoppedTime -XX:+PrintHeapAtGC -verbosegc -XX:G1LogLevel=finest -Xms28g -Xmx28g -XX:MaxDirectMemorySize=28g\""
PULSAR_PREFIX_dbStorage_writeCacheMaxSizeMb: "2048" # Write cache size (direct memory)
PULSAR_PREFIX_dbStorage_readAheadCacheMaxSizeMb: "2048" # Read cache size (direct memory)
PULSAR_PREFIX_dbStorage_rocksDB_blockCacheSize: "4294967296"
PULSAR_PREFIX_journalMaxSizeMB: "2048"
PULSAR_PREFIX_zkServers: zk-0.zookeeper,zk-1.zookeeper,zk-2.zookeeper
PULSAR_PREFIX_statsProviderClass: org.apache.bookkeeper.stats.prometheus.PrometheusMetricsProvider
PULSAR_PREFIX_useHostNameAsBookieID: "true"
BOOKIE_MEM: "\"-Dio.netty.leakDetectionLevel=disabled -Dio.netty.recycler.linkCapacity=1024 -XX:+ParallelRefProcEnabled -XX:+UnlockExperimentalVMOptions -XX:+AggressiveOpts -XX:+DoEscapeAnalysis -XX:ParallelGCThreads=32 -XX:ConcGCThreads=32 -XX:G1NewSizePercent=50 -XX:+DisableExplicitGC -XX:-ResizePLAB -XX:+ExitOnOutOfMemoryError -XX:+PerfDisableSharedMem -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCApplicationStoppedTime -XX:+PrintHeapAtGC -verbosegc -XX:G1LogLevel=finest -Xms28g -Xmx28g -XX:MaxDirectMemorySize=28g\""
BOOKIE_GC: "\"-XX:+UseG1GC -XX:MaxGCPauseMillis=10\""
dbStorage_writeCacheMaxSizeMb: "2048" # Write cache size (direct memory)
dbStorage_readAheadCacheMaxSizeMb: "2048" # Read cache size (direct memory)
dbStorage_rocksDB_blockCacheSize: "4294967296"
journalMaxSizeMB: "2048"
zkServers: zk-0.zookeeper,zk-1.zookeeper,zk-2.zookeeper
statsProviderClass: org.apache.bookkeeper.stats.prometheus.PrometheusMetricsProvider
useHostNameAsBookieID: "true"
---
apiVersion: apps/v1beta1
kind: StatefulSet
Expand Down Expand Up @@ -203,9 +202,9 @@ spec:
name: bookie-config
env:
## Configure for lower mem usage
- name: PULSAR_MEM
- name: BOOKIE_MEM
value: "\" -Xmx256m \""
- name: PULSAR_GC
- name: BOOKIE_GC
value: "\" \""
# Uncomment the following if bookie should be deployed on a specific instance type
# nodeSelector:
Expand Down
18 changes: 9 additions & 9 deletions deployment/kubernetes/aws/broker.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -27,13 +27,13 @@ data:
# better GC behavior at high throughput
PULSAR_MEM: "\" -Dio.netty.leakDetectionLevel=disabled -Dio.netty.recycler.linkCapacity=1024 -XX:+ParallelRefProcEnabled -XX:+UnlockExperimentalVMOptions -XX:+AggressiveOpts -XX:+DoEscapeAnalysis -XX:ParallelGCThreads=32 -XX:ConcGCThreads=32 -XX:G1NewSizePercent=50 -XX:+DisableExplicitGC -XX:-ResizePLAB -XX:+ExitOnOutOfMemoryError -XX:+PerfDisableSharedMem -Xms12g -Xmx12g -XX:MaxDirectMemorySize=14g -Dpulsar.root.logger=DEBUG,FILE \""
PULSAR_GC: "\" -XX:+UseG1GC -XX:MaxGCPauseMillis=10\""
PULSAR_PREFIX_zookeeperServers: zk-0.zookeeper,zk-1.zookeeper,zk-2.zookeeper
PULSAR_PREFIX_configurationStoreServers: zk-0.zookeeper,zk-1.zookeeper,zk-2.zookeeper
PULSAR_PREFIX_clusterName: us-east
PULSAR_PREFIX_managedLedgerDefaultEnsembleSize: "2"
PULSAR_PREFIX_managedLedgerDefaultWriteQuorum: "2"
PULSAR_PREFIX_managedLedgerDefaultAckQuorum: "2"
PULSAR_PREFIX_deduplicationEnabled: "false"
zookeeperServers: zk-0.zookeeper,zk-1.zookeeper,zk-2.zookeeper
configurationStoreServers: zk-0.zookeeper,zk-1.zookeeper,zk-2.zookeeper
clusterName: us-east
managedLedgerDefaultEnsembleSize: "2"
managedLedgerDefaultWriteQuorum: "2"
managedLedgerDefaultAckQuorum: "2"
deduplicationEnabled: "false"
---
##
## Broker deployment definition
Expand Down Expand Up @@ -63,7 +63,7 @@ spec:
# dnsPolicy: ClusterFirstWithHostNet
containers:
- name: broker
image: apachepulsar/pulsar:1.20.0-incubating
image: apachepulsar/pulsar:latest
command: ["sh", "-c"]
args:
- >
Expand Down Expand Up @@ -127,7 +127,7 @@ metadata:
spec:
containers:
- name: pulsar-admin
image: apachepulsar/pulsar:1.20.0-incubating
image: apachepulsar/pulsar:latest
command: ["sh", "-c"]
args:
- >
Expand Down
42 changes: 42 additions & 0 deletions deployment/kubernetes/aws/cluster-metadata.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,42 @@
#
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
#

apiVersion: batch/v1
kind: Job
metadata:
name: pulsar-cluster-metadata-init
labels:
app: pulsar
component: broker
spec:
template:
spec:
containers:
- name: pulsar-cluster-metadata-init-container
image: apachepulsar/pulsar:latest
command: ["sh", "-c"]
args:
- >
bin/pulsar initialize-cluster-metadata \
--cluster us-east \
--zookeeper zookeeper \
--configuration-store zookeeper \
--web-service-url http://broker:8080/ \
--broker-service-url pulsar://broker:6650/ || true;
restartPolicy: Never
9 changes: 5 additions & 4 deletions deployment/kubernetes/aws/proxy.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -23,10 +23,11 @@ metadata:
name: pulsar-proxy-config
data:
PULSAR_MEM: "\" -Xms4g -Xmx4g -XX:MaxDirectMemorySize=4g\""
PULSAR_PREFIX_brokerServiceURL: pulsar://broker:6650
PULSAR_PREFIX_brokerWebServiceURL: http://broker:8080
PULSAR_PREFIX_clusterName: pulsar-eks

brokerServiceURL: pulsar://broker:6650
brokerWebServiceURL: http://broker:8080
clusterName: us-east
zookeeperServers: zk-0.zookeeper,zk-1.zookeeper,zk-2.zookeeper
configurationStoreServers: zk-0.zookeeper,zk-1.zookeeper,zk-2.zookeeper
---
apiVersion: apps/v1beta1
kind: Deployment
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -115,6 +115,7 @@ spec:
- >
bin/apply-config-from-env.py conf/bookkeeper.conf &&
bin/apply-config-from-env.py conf/pulsar_env.sh &&
bin/apply-config-from-env.py conf/bkenv.sh &&
bin/pulsar bookie
ports:
- name: client
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@
# under the License.
#

{{- if .Values.extra.dashboard }}
{{- if .Values.dashboard.ingress.enabled }}
apiVersion: extensions/v1beta1
kind: Ingress
Expand Down Expand Up @@ -51,3 +52,4 @@ spec:
serviceName: "{{ template "pulsar.fullname" . }}-{{ .Values.dashboard.component }}"
servicePort: {{ .Values.dashboard.ingress.port }}
{{- end }}
{{- end }}
Original file line number Diff line number Diff line change
Expand Up @@ -76,7 +76,7 @@ spec:
terminationGracePeriodSeconds: {{ .Values.proxy.gracePeriod }}
initContainers:
# This init container will wait for zookeeper to be ready before
# deploying the bookies
# deploying the proxies
- name: wait-zookeeper-ready
image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
imagePullPolicy: {{ .Values.image.pullPolicy }}
Expand All @@ -86,6 +86,21 @@ spec:
until bin/pulsar zookeeper-shell -server {{ template "pulsar.fullname" . }}-{{ .Values.zookeeper.component }} ls /admin/clusters/{{ template "pulsar.fullname" . }}; do
sleep 3;
done;
# This init container will wait for at least one broker to be ready before
# deploying the proxy
- name: wait-broker-ready
image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
imagePullPolicy: {{ .Values.image.pullPolicy }}
command: ["bash", "-c"]
args:
- >-
for i in {0..{{ .Values.broker.replicaCount }}}; do
brokerServiceNumber="$(nslookup -timeout=10 {{ template "pulsar.fullname" . }}-{{ .Values.broker.component }} | grep Name | wc -l)"
if [[ ${brokerServiceNumber} -ge 1 ]]; then
break
fi
sleep 30;
done;
containers:
- name: "{{ template "pulsar.fullname" . }}-{{ .Values.proxy.component }}"
image: "{{ .Values.image.repository }}:{{ .Values.image.tag }}"
Expand Down
5 changes: 3 additions & 2 deletions deployment/kubernetes/helm/pulsar/values-mini.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -146,7 +146,8 @@ bookkeeper:
## templates/bookkeeper-configmap.yaml
##
configData:
PULSAR_MEM: "\"-Xms128m -Xmx256m -XX:MaxDirectMemorySize=128m -Dio.netty.leakDetectionLevel=disabled -Dio.netty.recycler.linkCapacity=1024 -XX:+UseG1GC -XX:MaxGCPauseMillis=10 -XX:+ParallelRefProcEnabled -XX:+UnlockExperimentalVMOptions -XX:+AggressiveOpts -XX:+DoEscapeAnalysis -XX:ParallelGCThreads=32 -XX:ConcGCThreads=32 -XX:G1NewSizePercent=50 -XX:+DisableExplicitGC -XX:-ResizePLAB -XX:+ExitOnOutOfMemoryError -XX:+PerfDisableSharedMem -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCApplicationStoppedTime -XX:+PrintHeapAtGC -verbosegc -XX:G1LogLevel=finest\""
BOOKIE_MEM: "\"-Xms128m -Xmx256m -XX:MaxDirectMemorySize=128m -Dio.netty.leakDetectionLevel=disabled -Dio.netty.recycler.linkCapacity=1024 -XX:+ParallelRefProcEnabled -XX:+UnlockExperimentalVMOptions -XX:+AggressiveOpts -XX:+DoEscapeAnalysis -XX:ParallelGCThreads=32 -XX:ConcGCThreads=32 -XX:G1NewSizePercent=50 -XX:+DisableExplicitGC -XX:-ResizePLAB -XX:+ExitOnOutOfMemoryError -XX:+PerfDisableSharedMem\""
BOOKIE_GC: "\"-XX:+UseG1GC -XX:MaxGCPauseMillis=10\""
PULSAR_PREFIX_dbStorage_writeCacheMaxSizeMb: "32"
PULSAR_PREFIX_dbStorage_readAheadCacheMaxSizeMb: "32"
PULSAR_PREFIX_journalMaxSizeMB: "2048"
Expand Down Expand Up @@ -276,7 +277,7 @@ autoRecovery:
## templates/autorecovery-configmap.yaml
##
configData:
PULSAR_MEM: "\" -Xms64m -Xmx128m \""
BOOKIE_MEM: "\" -Xms64m -Xmx128m \""

## Pulsar Extra: Dashboard
## templates/dashboard-deployment.yaml
Expand Down
5 changes: 3 additions & 2 deletions deployment/kubernetes/helm/pulsar/values.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -179,7 +179,8 @@ bookkeeper:
## templates/bookkeeper-configmap.yaml
##
configData:
PULSAR_MEM: "\"-Xms15g -Xmx15g -XX:MaxDirectMemorySize=15g -Dio.netty.leakDetectionLevel=disabled -Dio.netty.recycler.linkCapacity=1024 -XX:+UseG1GC -XX:MaxGCPauseMillis=10 -XX:+ParallelRefProcEnabled -XX:+UnlockExperimentalVMOptions -XX:+AggressiveOpts -XX:+DoEscapeAnalysis -XX:ParallelGCThreads=32 -XX:ConcGCThreads=32 -XX:G1NewSizePercent=50 -XX:+DisableExplicitGC -XX:-ResizePLAB -XX:+ExitOnOutOfMemoryError -XX:+PerfDisableSharedMem -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCApplicationStoppedTime -XX:+PrintHeapAtGC -verbosegc -XX:G1LogLevel=finest\""
BOOKIE_MEM: "\"-Xms15g -Xmx15g -XX:MaxDirectMemorySize=15g -Dio.netty.leakDetectionLevel=disabled -Dio.netty.recycler.linkCapacity=1024 -XX:+ParallelRefProcEnabled -XX:+UnlockExperimentalVMOptions -XX:+AggressiveOpts -XX:+DoEscapeAnalysis -XX:ParallelGCThreads=32 -XX:ConcGCThreads=32 -XX:G1NewSizePercent=50 -XX:+DisableExplicitGC -XX:-ResizePLAB -XX:+ExitOnOutOfMemoryError -XX:+PerfDisableSharedMem -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintGCApplicationStoppedTime -XX:+PrintHeapAtGC -verbosegc -XX:G1LogLevel=finest\""
BOOKIE_GC: "\"-XX:+UseG1GC -XX:MaxGCPauseMillis=10\""
PULSAR_PREFIX_dbStorage_writeCacheMaxSizeMb: "2048"
PULSAR_PREFIX_dbStorage_readAheadCacheMaxSizeMb: "2048"
PULSAR_PREFIX_dbStorage_rocksDB_blockCacheSize: "268435456"
Expand Down Expand Up @@ -310,7 +311,7 @@ autoRecovery:
## templates/autorecovery-configmap.yaml
##
configData:
PULSAR_MEM: "\" -Xms1g -Xmx1g \""
BOOKIE_MEM: "\" -Xms1g -Xmx1g \""

## Pulsar Extra: Dashboard
## templates/dashboard-deployment.yaml
Expand Down
Loading