Skip to content

Commit 9b9ccd0

Browse files
dyunkimdyunkim
authored
Managed DB: Add MongoDB and Kafka Configurations to docs (#864)
* adding kafka and mongo advanced config api docs * fixing formatting issues in kafka docs * add examples to spec
1 parent 96b11dc commit 9b9ccd0

File tree

3 files changed

+382
-0
lines changed

3 files changed

+382
-0
lines changed

‎specification/resources/databases/models/database_config.yml

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,3 +6,5 @@ properties:
66
- $ref: './mysql.yml'
77
- $ref: './postgres.yml'
88
- $ref: './redis.yml'
9+
- $ref: './mongo.yml'
10+
- $ref: './kafka.yml'

‎specification/resources/databases/models/kafka.yml

Lines changed: 328 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,328 @@
1+
type: object
2+
3+
properties:
4+
compression_type:
5+
description: >-
6+
Specify the final compression type for a given topic. This
7+
configuration accepts the standard compression codecs ('gzip', 'snappy',
8+
'lz4', 'zstd'). It additionally accepts 'uncompressed' which is equivalent
9+
to no compression; and 'producer' which means retain the original
10+
compression codec set by the producer.
11+
type: string
12+
enum:
13+
- gzip
14+
- snappy
15+
- lz4
16+
- zstd
17+
- uncompressed
18+
- producer
19+
example: gzip
20+
group_initial_rebalance_delay_ms:
21+
description: >-
22+
The amount of time, in milliseconds, the group coordinator
23+
will wait for more consumers to join a new group before performing
24+
the first rebalance. A longer delay means potentially fewer rebalances,
25+
but increases the time until processing begins. The default value
26+
for this is 3 seconds. During development and testing it might be
27+
desirable to set this to 0 in order to not delay test execution time.
28+
type: integer
29+
example: 3000
30+
minimum: 0
31+
maximum: 300000
32+
group_min_session_timeout_ms:
33+
description: >-
34+
The minimum allowed session timeout for registered consumers.
35+
Longer timeouts give consumers more time to process messages in between
36+
heartbeats at the cost of a longer time to detect failures.
37+
type: integer
38+
example: 6000
39+
minimum: 0
40+
maximum: 60000
41+
group_max_session_timeout_ms:
42+
description: >-
43+
The maximum allowed session timeout for registered consumers.
44+
Longer timeouts give consumers more time to process messages in between
45+
heartbeats at the cost of a longer time to detect failures.
46+
type: integer
47+
example: 1800000
48+
minimum: 0
49+
maximum: 1800000
50+
connections_max_idle_ms:
51+
description: >-
52+
Idle connections timeout: the server socket processor
53+
threads close the connections that idle for longer than this.
54+
type: integer
55+
minimum: 1000
56+
example: 540000
57+
maximum: 3600000
58+
max_incremental_fetch_session_cache_slots:
59+
description: >-
60+
The maximum number of incremental fetch sessions that the
61+
broker will maintain.
62+
type: integer
63+
example: 1000
64+
minimum: 1000
65+
maximum: 10000
66+
message_max_bytes:
67+
description: >-
68+
The maximum size of message that the server can receive.
69+
type: integer
70+
example: 1048588
71+
minimum: 0
72+
maximum: 100001200
73+
offsets_retention_minutes:
74+
description: >-
75+
Log retention window in minutes for offsets topic
76+
type: integer
77+
example: 10080
78+
minimum: 1
79+
maximum: 2147483647
80+
log_cleaner_delete_retention_ms:
81+
description: >-
82+
How long are delete records retained?
83+
type: integer
84+
minimum: 0
85+
maximum: 315569260000
86+
example: 86400000
87+
log_cleaner_min_cleanable_ratio:
88+
description: >-
89+
Controls log compactor frequency. Larger value means more
90+
frequent compactions but also more space wasted for logs. Consider
91+
setting log_cleaner_max_compaction_lag_ms to enforce compactions sooner,
92+
instead of setting a very high value for this option.
93+
type: number
94+
minimum: 0.2
95+
maximum: 0.9
96+
example: 0.5
97+
log_cleaner_max_compaction_lag_ms:
98+
description: >-
99+
The maximum amount of time message will remain uncompacted.
100+
Only applicable for logs that are being compacted
101+
type: integer
102+
minimum: 30000
103+
maximum: 9223372036854776000
104+
example: 60000
105+
log_cleaner_min_compaction_lag_ms:
106+
description: >-
107+
The minimum time a message will remain uncompacted in the
108+
log. Only applicable for logs that are being compacted.
109+
type: integer
110+
minimum: 0
111+
maximum: 9223372036854776000
112+
example: 100000
113+
log_cleanup_policy:
114+
description: >-
115+
The default cleanup policy for segments beyond the retention
116+
window
117+
type: string
118+
enum:
119+
- delete
120+
- compact
121+
- compact,delete
122+
example: delete
123+
log_flush_interval_messages:
124+
description: >-
125+
The number of messages accumulated on a log partition before
126+
messages are flushed to disk
127+
type: integer
128+
minimum: 1
129+
maximum: 9223372036854776000
130+
example: 9223372036854776000
131+
log_flush_interval_ms:
132+
description: >-
133+
The maximum time in ms that a message in any topic is kept
134+
in memory before flushed to disk. If not set, the value in log.flush.scheduler.interval.ms
135+
is used
136+
type: integer
137+
minimum: 0
138+
maximum: 9223372036854776000
139+
example: 1000000
140+
log_index_interval_bytes:
141+
description: >-
142+
The interval with which Kafka adds an entry to the offset
143+
index
144+
type: integer
145+
minimum: 0
146+
maximum: 104857600
147+
example: 4096
148+
log_index_size_max_bytes:
149+
description: >-
150+
The maximum size in bytes of the offset index
151+
type: integer
152+
minimum: 1048576
153+
maximum: 104857600
154+
example: 10485760
155+
log_message_downconversion_enable:
156+
description: >-
157+
This configuration controls whether down-conversion of
158+
message formats is enabled to satisfy consume requests.
159+
type: boolean
160+
example: true
161+
log_message_timestamp_type:
162+
description: >-
163+
Define whether the timestamp in the message is message
164+
create time or log append time.
165+
type: string
166+
enum:
167+
- CreateTime
168+
- LogAppendTime
169+
example: CreateTime
170+
log_message_timestamp_difference_max_ms:
171+
description: >-
172+
The maximum difference allowed between the timestamp when
173+
a broker receives a message and the timestamp specified in the message
174+
type: integer
175+
minimum: 0
176+
maximum: 9223372036854776000
177+
example: 1000000
178+
log_preallocate:
179+
description: >-
180+
Controls whether to preallocate a file when creating a new segment
181+
type: boolean
182+
example: false
183+
log_retention_bytes:
184+
description: >-
185+
The maximum size of the log before deleting messages
186+
type: integer
187+
minimum: -1
188+
maximum: 9223372036854776000
189+
example: 1000000
190+
log_retention_hours:
191+
description: >-
192+
The number of hours to keep a log file before deleting it
193+
type: integer
194+
minimum: -1
195+
maximum: 2147483647
196+
example: 1000000
197+
log_retention_ms:
198+
description: >-
199+
The number of milliseconds to keep a log file before deleting
200+
it (in milliseconds), If not set, the value in log.retention.minutes
201+
is used. If set to -1, no time limit is applied.
202+
type: integer
203+
minimum: -1
204+
maximum: 9223372036854776000
205+
example: 100000000
206+
log_roll_jitter_ms:
207+
description: >-
208+
The maximum jitter to subtract from logRollTimeMillis (in
209+
milliseconds). If not set, the value in log.roll.jitter.hours is used
210+
type: integer
211+
minimum: 0
212+
maximum: 9223372036854776000
213+
example: 10000000
214+
log_roll_ms:
215+
description: >-
216+
The maximum time before a new log segment is rolled out
217+
(in milliseconds).
218+
type: integer
219+
minimum: 1
220+
maximum: 9223372036854776000
221+
example: 1000000
222+
log_segment_bytes:
223+
description: >-
224+
The maximum size of a single log file
225+
type: integer
226+
minimum: 10485760
227+
maximum: 1073741824
228+
example: 100000000
229+
log_segment_delete_delay_ms:
230+
description: >-
231+
The amount of time to wait before deleting a file from
232+
the filesystem
233+
type: integer
234+
minimum: 0
235+
maximum: 3600000
236+
example: 60000
237+
auto_create_topics_enable:
238+
description: >-
239+
Enable auto creation of topics
240+
type: boolean
241+
example: true
242+
min_insync_replicas:
243+
description: >-
244+
When a producer sets acks to 'all' (or '-1'), min_insync_replicas
245+
specifies the minimum number of replicas that must acknowledge a write
246+
for the write to be considered successful.
247+
type: integer
248+
minimum: 1
249+
maximum: 7
250+
example: 1
251+
num_partitions:
252+
description: >-
253+
Number of partitions for autocreated topics
254+
type: integer
255+
minimum: 1
256+
maximum: 1000
257+
example: 10
258+
default_replication_factor:
259+
description: >-
260+
Replication factor for autocreated topics
261+
type: integer
262+
minimum: 1
263+
maximum: 10
264+
example: 2
265+
replica_fetch_max_bytes:
266+
description: >-
267+
The number of bytes of messages to attempt to fetch for
268+
each partition (defaults to 1048576). This is not an absolute maximum,
269+
if the first record batch in the first non-empty partition of the
270+
fetch is larger than this value, the record batch will still be returned
271+
to ensure that progress can be made.
272+
type: integer
273+
minimum: 1048576
274+
maximum: 104857600
275+
example: 2097152
276+
replica_fetch_response_max_bytes:
277+
description: >-
278+
Maximum bytes expected for the entire fetch response (defaults
279+
to 10485760). Records are fetched in batches, and if the first record
280+
batch in the first non-empty partition of the fetch is larger than
281+
this value, the record batch will still be returned to ensure that
282+
progress can be made. As such, this is not an absolute maximum.
283+
type: integer
284+
minimum: 10485760
285+
maximum: 1048576000
286+
example: 20971520
287+
max_connections_per_ip:
288+
description: >-
289+
The maximum number of connections allowed from each ip
290+
address (defaults to 2147483647).
291+
type: integer
292+
minimum: 256
293+
maximum: 2147483647
294+
example: 512
295+
producer_purgatory_purge_interval_requests:
296+
description: >-
297+
The purge interval (in number of requests) of the producer
298+
request purgatory (defaults to 1000).
299+
type: integer
300+
minimum: 10
301+
maximum: 10000
302+
example: 100
303+
socket_request_max_bytes:
304+
description: >-
305+
The maximum number of bytes in a socket request (defaults
306+
to 104857600).
307+
type: integer
308+
minimum: 10485760
309+
maximum: 209715200
310+
example: 20971520
311+
transaction_state_log_segment_bytes:
312+
description: >-
313+
The transaction topic segment bytes should be kept relatively
314+
small in order to facilitate faster log compaction and cache loads
315+
(defaults to 104857600 (100 mebibytes)).
316+
type: integer
317+
minimum: 1048576
318+
maximum: 2147483647
319+
example: 104857600
320+
transaction_remove_expired_transaction_cleanup_interval_ms:
321+
description: >-
322+
The interval at which to remove transactions that have
323+
expired due to transactional.id.expiration.ms passing (defaults to
324+
3600000 (1 hour)).
325+
type: integer
326+
minimum: 600000
327+
maximum: 3600000
328+
example: 3600000

‎specification/resources/databases/models/mongo.yml

Lines changed: 52 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,52 @@
1+
type: object
2+
3+
properties:
4+
default_read_concern:
5+
description: >-
6+
Specifies the default consistency behavior of reads from the database. Data that is returned from the query with may
7+
or may not have been acknowledged by all nodes in the replicaset depending on this value.
8+
Learn more [here](https://www.mongodb.com/docs/manual/reference/read-concern/).
9+
type: string
10+
enum: [local, available, majority]
11+
default: local
12+
example: local
13+
default_write_concern:
14+
description: >-
15+
Describes the level of acknowledgment requested from MongoDB for write operations clusters. This field can set to either `majority`
16+
or a number `0...n` which will describe the number of nodes that must acknowledge the write operation before it is fully accepted.
17+
Setting to `0` will request no acknowledgement of the write operation.
18+
Learn more [here](https://www.mongodb.com/docs/manual/reference/write-concern/).
19+
type: string
20+
default: majority
21+
example: majority
22+
transaction_lifetime_limit_seconds:
23+
description: >-
24+
Specifies the lifetime of multi-document transactions. Transactions that exceed this limit are considered expired and will be
25+
aborted by a periodic cleanup process. The cleanup process runs every `transactionLifetimeLimitSeconds/2 seconds` or at least
26+
once every 60 seconds. *Changing this parameter will lead to a restart of the MongoDB service.*
27+
Learn more [here](https://www.mongodb.com/docs/manual/reference/parameters/#mongodb-parameter-param.transactionLifetimeLimitSeconds).
28+
type: integer
29+
minimum: 1
30+
default: 60
31+
example: 100
32+
slow_op_threshold_ms:
33+
description: >-
34+
Operations that run for longer than this threshold are considered slow which are then recorded to the diagnostic logs.
35+
Higher log levels (verbosity) will record all operations regardless of this threshold on the primary node.
36+
*Changing this parameter will lead to a restart of the MongoDB service.*
37+
Learn more [here](https://www.mongodb.com/docs/manual/reference/configuration-options/#mongodb-setting-operationProfiling.slowOpThresholdMs).
38+
type: integer
39+
minimum: 0
40+
default: 100
41+
example: 200
42+
verbosity:
43+
description: >-
44+
The log message verbosity level. The verbosity level determines the amount of Informational and Debug messages MongoDB outputs.
45+
0 includes informational messages while 1...5 increases the level to include debug messages.
46+
*Changing this parameter will lead to a restart of the MongoDB service.*
47+
Learn more [here](https://www.mongodb.com/docs/manual/reference/configuration-options/#mongodb-setting-systemLog.verbosity).
48+
type: integer
49+
minimum: 0
50+
maximum: 5
51+
default: 0
52+
example: 3

0 commit comments

Comments
 (0)