Skip to content

Commit

Permalink
[KYUUBI #4352] Support System.gc() with periodic GC interval
Browse files Browse the repository at this point in the history
### _Why are the changes needed?_

### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible

- [ ] Add screenshots for manual tests if appropriate

- [ ] [Run test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests) locally before make a pull request

Closes #4382 from lightning-L/kyuubi-4352.

Closes #4352

2beb1ef [Tianlin Liao] update config name
a9eb126 [Tianlin Liao] update config setting
75cdfbb [Tianlin Liao] [KYUUBI #4352] Support System.gc() with periodic GC interval

Authored-by: Tianlin Liao <tiliao@ebay.com>
Signed-off-by: fwang12 <fwang12@ebay.com>
  • Loading branch information
lightning-L authored and turboFei committed Feb 21, 2023
1 parent ae374c1 commit e5ade90
Show file tree
Hide file tree
Showing 4 changed files with 69 additions and 12 deletions.
25 changes: 13 additions & 12 deletions docs/deployment/settings.md
Original file line number Diff line number Diff line change
Expand Up @@ -449,18 +449,19 @@ You can configure the Kyuubi properties in `$KYUUBI_HOME/conf/kyuubi-defaults.co

### Server

| Key | Default | Meaning | Type | Since |
|----------------------------------------------------------|-------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|--------|-------|
| kyuubi.server.batch.limit.connections.per.ipaddress | &lt;undefined&gt; | Maximum kyuubi server batch connections per ipaddress. Any user exceeding this limit will not be allowed to connect. | int | 1.7.0 |
| kyuubi.server.batch.limit.connections.per.user | &lt;undefined&gt; | Maximum kyuubi server batch connections per user. Any user exceeding this limit will not be allowed to connect. | int | 1.7.0 |
| kyuubi.server.batch.limit.connections.per.user.ipaddress | &lt;undefined&gt; | Maximum kyuubi server batch connections per user:ipaddress combination. Any user-ipaddress exceeding this limit will not be allowed to connect. | int | 1.7.0 |
| kyuubi.server.info.provider | ENGINE | The server information provider name, some clients may rely on this information to check the server compatibilities and functionalities. <li>SERVER: Return Kyuubi server information.</li> <li>ENGINE: Return Kyuubi engine information.</li> | string | 1.6.1 |
| kyuubi.server.limit.connections.per.ipaddress | &lt;undefined&gt; | Maximum kyuubi server connections per ipaddress. Any user exceeding this limit will not be allowed to connect. | int | 1.6.0 |
| kyuubi.server.limit.connections.per.user | &lt;undefined&gt; | Maximum kyuubi server connections per user. Any user exceeding this limit will not be allowed to connect. | int | 1.6.0 |
| kyuubi.server.limit.connections.per.user.ipaddress | &lt;undefined&gt; | Maximum kyuubi server connections per user:ipaddress combination. Any user-ipaddress exceeding this limit will not be allowed to connect. | int | 1.6.0 |
| kyuubi.server.limit.connections.user.unlimited.list || The maximum connections of the user in the white list will not be limited. | seq | 1.7.0 |
| kyuubi.server.name | &lt;undefined&gt; | The name of Kyuubi Server. | string | 1.5.0 |
| kyuubi.server.redaction.regex | &lt;undefined&gt; | Regex to decide which Kyuubi contain sensitive information. When this regex matches a property key or value, the value is redacted from the various logs. || 1.6.0 |
| Key | Default | Meaning | Type | Since |
|----------------------------------------------------------|-------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------|-------|
| kyuubi.server.batch.limit.connections.per.ipaddress | &lt;undefined&gt; | Maximum kyuubi server batch connections per ipaddress. Any user exceeding this limit will not be allowed to connect. | int | 1.7.0 |
| kyuubi.server.batch.limit.connections.per.user | &lt;undefined&gt; | Maximum kyuubi server batch connections per user. Any user exceeding this limit will not be allowed to connect. | int | 1.7.0 |
| kyuubi.server.batch.limit.connections.per.user.ipaddress | &lt;undefined&gt; | Maximum kyuubi server batch connections per user:ipaddress combination. Any user-ipaddress exceeding this limit will not be allowed to connect. | int | 1.7.0 |
| kyuubi.server.info.provider | ENGINE | The server information provider name, some clients may rely on this information to check the server compatibilities and functionalities. <li>SERVER: Return Kyuubi server information.</li> <li>ENGINE: Return Kyuubi engine information.</li> | string | 1.6.1 |
| kyuubi.server.limit.connections.per.ipaddress | &lt;undefined&gt; | Maximum kyuubi server connections per ipaddress. Any user exceeding this limit will not be allowed to connect. | int | 1.6.0 |
| kyuubi.server.limit.connections.per.user | &lt;undefined&gt; | Maximum kyuubi server connections per user. Any user exceeding this limit will not be allowed to connect. | int | 1.6.0 |
| kyuubi.server.limit.connections.per.user.ipaddress | &lt;undefined&gt; | Maximum kyuubi server connections per user:ipaddress combination. Any user-ipaddress exceeding this limit will not be allowed to connect. | int | 1.6.0 |
| kyuubi.server.limit.connections.user.unlimited.list || The maximum connections of the user in the white list will not be limited. | seq | 1.7.0 |
| kyuubi.server.name | &lt;undefined&gt; | The name of Kyuubi Server. | string | 1.5.0 |
| kyuubi.server.periodicGC.interval | PT30M | How often to trigger a garbage collection. | duration | 1.7.0 |
| kyuubi.server.redaction.regex | &lt;undefined&gt; | Regex to decide which Kyuubi contain sensitive information. When this regex matches a property key or value, the value is redacted from the various logs. || 1.6.0 |

### Session

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2412,6 +2412,14 @@ object KyuubiConf {
.regexConf
.createOptional

val SERVER_PERIODIC_GC_INTERVAL: ConfigEntry[Long] =
buildConf("kyuubi.server.periodicGC.interval")
.doc("How often to trigger a garbage collection.")
.version("1.7.0")
.serverOnly
.timeConf
.createWithDefaultString("PT30M")

val OPERATION_SPARK_LISTENER_ENABLED: ConfigEntry[Boolean] =
buildConf("kyuubi.operation.spark.listener.enabled")
.doc("When set to true, Spark engine registers an SQLOperationListener before executing " +
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -169,6 +169,9 @@ class KyuubiServer(name: String) extends Serverable(name) {
val kinit = new KinitAuxiliaryService()
addService(kinit)

val periodicGCService = new PeriodicGCService
addService(periodicGCService)

if (conf.get(MetricsConf.METRICS_ENABLED)) {
addService(new MetricsSystem)
}
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the "License"); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/

package org.apache.kyuubi.server

import java.util.concurrent.TimeUnit

import org.apache.kyuubi.config.KyuubiConf
import org.apache.kyuubi.service.AbstractService
import org.apache.kyuubi.util.ThreadUtils

class PeriodicGCService(name: String) extends AbstractService(name) {
def this() = this(classOf[PeriodicGCService].getSimpleName)

private val gcTrigger = ThreadUtils.newDaemonSingleThreadScheduledExecutor("periodic-gc-trigger")

override def start(): Unit = {
startGcTrigger()
super.start()
}

override def stop(): Unit = {
super.stop()
ThreadUtils.shutdown(gcTrigger)
}

private def startGcTrigger(): Unit = {
val interval = conf.get(KyuubiConf.SERVER_PERIODIC_GC_INTERVAL)
gcTrigger.scheduleWithFixedDelay(() => System.gc(), interval, interval, TimeUnit.MILLISECONDS)
}
}

0 comments on commit e5ade90

Please sign in to comment.