Enable Multi DB #20305
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,5 +1,6 @@ | ||
| {% if INSTANCES %} | ||
| {% for redis_inst, redis_items in INSTANCES.items() %} | ||
|
|
||
|
There was a problem hiding this comment. Remove this unnecessary change? There was a problem hiding this comment. If there is no blank line here, all program:{{ redis_inst }} of critical_processes will be concentrated on one line, which will cause subsequent regular expression matching to fail. There was a problem hiding this comment. Does changing There was a problem hiding this comment. Changing {%- endfor %} to {% endfor %} works. I apply this change instead of adding blank line. |
||
| program:{{ redis_inst }} | ||
| {%- endfor %} | ||
| {%- endif %} | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -50,7 +50,11 @@ mkdir -p /etc/supervisor/conf.d/ | |
| if [ -f /etc/sonic/database_config$NAMESPACE_ID.json ]; then | ||
| cp /etc/sonic/database_config$NAMESPACE_ID.json $REDIS_DIR/sonic-db/database_config.json | ||
| else | ||
| if [ -f /etc/sonic/enable_multidb ]; then | ||
| HOST_IP=$host_ip REDIS_PORT=$redis_port DATABASE_TYPE=$DATABASE_TYPE BMP_DB_PORT=$BMP_DB_PORT j2 /usr/share/sonic/templates/multi_database_config.json.j2 > $REDIS_DIR/sonic-db/database_config.json | ||
| else | ||
| HOST_IP=$host_ip REDIS_PORT=$redis_port DATABASE_TYPE=$DATABASE_TYPE BMP_DB_PORT=$BMP_DB_PORT j2 /usr/share/sonic/templates/database_config.json.j2 > $REDIS_DIR/sonic-db/database_config.json | ||
| fi | ||
|
There was a problem hiding this comment. @Pan-XT any sonic-mgmt tests for multi DB? |
||
| fi | ||
|
|
||
| # on VoQ system, we only publish redis_chassis instance and CHASSIS_APP_DB when | ||
|
|
@@ -125,6 +129,12 @@ do | |
| else | ||
| echo -n > /var/lib/$inst/dump.rdb | ||
| fi | ||
|
|
||
| chown -R redis:redis /var/lib/$inst | ||
|
There was a problem hiding this comment. Add comments to this file, help other understand. There was a problem hiding this comment. Here are the necessary permissions for enable_multidb, see /etc/supervisor/conf.d/supervisord.conf.j2 There was a problem hiding this comment. Hi @Pan-XT , can you add your comments to this file? that will help other understand the propose of this code. There was a problem hiding this comment. ok，I add commnets “the Redis process is operating under the 'redis' user in supervisord and make redis user own /var/lib/$inst inside db container.” |
||
| sock_file="$REDIS_DIR/$inst.sock" | ||
| if [[ -f $sock_file ]]; then | ||
| chgrp -f redis $sock_file && chmod -f 0760 $sock_file | ||
| fi | ||
| done | ||
|
|
||
| TZ=$(cat /etc/timezone) | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,192 @@ | ||
| {% set include_remote_db = (REMOTE_DB_IP is defined and REMOTE_DB_PORT is defined) %} | ||
| { | ||
| "INSTANCES": { | ||
| "redis":{ | ||
| "hostname" : "{{HOST_IP}}", | ||
| "port" : {{REDIS_PORT}}, | ||
| "unix_socket_path" : "/var/run/redis{{DEV}}/redis.sock", | ||
| "persistence_for_warm_boot" : "yes" | ||
| {% if DATABASE_TYPE is defined and DATABASE_TYPE != "" %} | ||
| ,"database_type": "{{DATABASE_TYPE}}" | ||
|
There was a problem hiding this comment. I didn't find code in sonic-swss-common to parse and handle 'database_type', is there a PR? There was a problem hiding this comment. I search that PR, not find any code read "database_type" Otherwise, sonic-swss-common lib can't handle multi-DB, so please make sure sonic-swss-common also change to handle it. There was a problem hiding this comment. There was a problem hiding this comment. This field was added via It is current SONiC behavior Can you help to check with him to see what the need for this field? There was a problem hiding this comment. #Closed, I will offline sync with Ze |
||
| {% endif %} | ||
| }, | ||
| "redis1": { | ||
| "hostname" : "{{HOST_IP}}", | ||
| "port" : 6380, | ||
|
There was a problem hiding this comment. The port selection conflicts with both the chassis DB and Smart Switch DB service and must be changed. Port 6380 is used by the chassis Redis server, while ports 6381-6388 are assigned to the DPU databases. There was a problem hiding this comment. @oleksandrivantsiv where does chassis DB's port get defined? This multi-db approach is enabled only on the platform which needs performance and controlled via a compile flag. ENABLE_MULTIDB=y The detail information could be found at https://lists.sonicfoundation.dev/g/sonic-wg-routing/wiki/37961 During the discussion in routing WG, we feel there is NO use case to enable it for chassis based platform for the following reasons.
There was a problem hiding this comment. Discussed with Guahan on this one. 6380 will be kept for Chassis DB. It is not easy to change chassis's port number. We use 6379, 6378, 6377, ... a.k.a going down for new redis instances. @Pan-XT can you update your diff? There was a problem hiding this comment. Update redis1,reids2,redis3,redis4 port number to 6378,6377,6376,6375 |
||
| "unix_socket_path" : "/var/run/redis{{DEV}}/redis1.sock", | ||
| "persistence_for_warm_boot" : "yes" | ||
| {% if DATABASE_TYPE is defined and DATABASE_TYPE != "" %} | ||
| ,"database_type": "{{DATABASE_TYPE}}" | ||
| {% endif %} | ||
| }, | ||
| "redis2": { | ||
| "hostname" : "{{HOST_IP}}", | ||
| "port" : 6381, | ||
| "unix_socket_path" : "/var/run/redis{{DEV}}/redis2.sock", | ||
| "persistence_for_warm_boot" : "yes" | ||
| {% if DATABASE_TYPE is defined and DATABASE_TYPE != "" %} | ||
| ,"database_type": "{{DATABASE_TYPE}}" | ||
| {% endif %} | ||
| }, | ||
| "redis3": { | ||
| "hostname" : "{{HOST_IP}}", | ||
| "port" : 6382, | ||
| "unix_socket_path" : "/var/run/redis{{DEV}}/redis3.sock", | ||
| "persistence_for_warm_boot" : "yes" | ||
| {% if DATABASE_TYPE is defined and DATABASE_TYPE != "" %} | ||
| ,"database_type": "{{DATABASE_TYPE}}" | ||
| {% endif %} | ||
| }, | ||
| "redis4": { | ||
| "hostname" : "{{HOST_IP}}", | ||
| "port" : 6383, | ||
| "unix_socket_path" : "/var/run/redis{{DEV}}/redis4.sock", | ||
| "persistence_for_warm_boot" : "yes" | ||
| {% if DATABASE_TYPE is defined and DATABASE_TYPE != "" %} | ||
| ,"database_type": "{{DATABASE_TYPE}}" | ||
| {% endif %} | ||
| }, | ||
| "redis_chassis":{ | ||
| "hostname" : "redis_chassis.server", | ||
| "port": 6380, | ||
| "unix_socket_path": "/var/run/redis-chassis/redis_chassis.sock", | ||
| "persistence_for_warm_boot" : "yes" | ||
| } | ||
| {% if include_remote_db %} | ||
| ,"remote_redis":{ | ||
| "hostname" : "{{REMOTE_DB_IP}}", | ||
| "port" : {{REMOTE_DB_PORT}}, | ||
| "unix_socket_path": "", | ||
| "persistence_for_warm_boot" : "yes" | ||
| } | ||
| {% endif %}, | ||
| "redis_bmp":{ | ||
| "hostname" : "{{HOST_IP}}", | ||
| "port" : {{BMP_DB_PORT}}, | ||
| "unix_socket_path" : "/var/run/redis{{DEV}}/redis_bmp.sock", | ||
| "persistence_for_warm_boot" : "yes" | ||
| } | ||
| }, | ||
| "DATABASES" : { | ||
| "APPL_DB": { | ||
| "id": 0, | ||
| "separator": ":", | ||
| "instance": "redis1" | ||
| }, | ||
| "ASIC_DB": { | ||
| "id": 1, | ||
| "separator": ":", | ||
| "instance": "redis2" | ||
| }, | ||
| "COUNTERS_DB": { | ||
| "id": 2, | ||
| "separator": ":", | ||
| "instance": "redis3" | ||
| }, | ||
| "LOGLEVEL_DB": { | ||
| "id": 3, | ||
| "separator": ":", | ||
| "instance": "redis" | ||
| }, | ||
| "CONFIG_DB": { | ||
| "id": 4, | ||
| "separator": "|", | ||
| "instance": "redis" | ||
| }, | ||
| "PFC_WD_DB": { | ||
| "id": 5, | ||
| "separator": ":", | ||
| "instance": "redis3" | ||
| }, | ||
| "FLEX_COUNTER_DB": { | ||
| "id": 5, | ||
| "separator": ":", | ||
| "instance": "redis3" | ||
| }, | ||
| "STATE_DB": { | ||
| "id": 6, | ||
| "separator": "|", | ||
| "instance": "redis" | ||
| }, | ||
| "SNMP_OVERLAY_DB": { | ||
| "id": 7, | ||
| "separator": "|", | ||
| "instance": "redis" | ||
| }, | ||
| "SYSMON_DB": { | ||
| "id": 10, | ||
| "separator": "|", | ||
| "instance": "redis" | ||
| }, | ||
| "BMC_DB": { | ||
| "id": 12, | ||
| "separator": ":", | ||
| "instance": "redis4" | ||
| }, | ||
| "RESTAPI_DB": { | ||
| "id": 8, | ||
| "separator": "|", | ||
| "instance": "redis" | ||
| }, | ||
| "GB_ASIC_DB": { | ||
| "id": 9, | ||
| "separator": ":", | ||
| "instance": "redis" | ||
| }, | ||
| "GB_COUNTERS_DB": { | ||
| "id": 10, | ||
| "separator": ":", | ||
| "instance": "redis" | ||
| }, | ||
| "GB_FLEX_COUNTER_DB": { | ||
| "id": 11, | ||
| "separator": ":", | ||
| "instance": "redis" | ||
| }, | ||
| "APPL_STATE_DB": { | ||
| "id": 14, | ||
| "separator": ":", | ||
| "instance": "redis" | ||
| }, | ||
| "CHASSIS_APP_DB" : { | ||
| "id" : 12, | ||
| "separator": "|", | ||
| "instance" : "redis_chassis" | ||
| }, | ||
| "CHASSIS_STATE_DB" : { | ||
| "id" : 13, | ||
| "separator": "|", | ||
| "instance" : "redis_chassis" | ||
| } | ||
| {% if DATABASE_TYPE is defined and DATABASE_TYPE == "dpudb" %} | ||
| , | ||
| "DPU_APPL_DB" : { | ||
| "id" : 15, | ||
| "separator": ":", | ||
| "instance" : {% if include_remote_db %} "remote_redis" {% else %} "redis" {% endif %}, | ||
|
There was a problem hiding this comment. Suggest define a variable for all these dupe template There was a problem hiding this comment. Currently, the multi_database_config.json.j2 is consistent with the database_config.json.j2, except for the added content related to redis1,redis2,redis3,redis4. Defining a variable for all these dupe templates will be implemented in future modifications. |
||
| "format": "proto" | ||
| }, | ||
| "DPU_APPL_STATE_DB" : { | ||
| "id" : 16, | ||
| "separator": "|", | ||
| "instance" : {% if include_remote_db %} "remote_redis" {% else %} "redis" {% endif %} | ||
| }, | ||
| "DPU_STATE_DB" : { | ||
| "id" : 17, | ||
| "separator": "|", | ||
| "instance" : {% if include_remote_db %} "remote_redis" {% else %} "redis" {% endif %} | ||
| }, | ||
| "DPU_COUNTERS_DB" : { | ||
| "id" : 18, | ||
| "separator": ":", | ||
| "instance" : {% if include_remote_db %} "remote_redis" {% else %} "redis" {% endif %} | ||
| } | ||
| {% endif %}, | ||
| "BMP_STATE_DB" : { | ||
| "id" : 20, | ||
| "separator": "|", | ||
| "instance" : "redis_bmp" | ||
| } | ||
| }, | ||
| "VERSION" : "1.0" | ||
| } | ||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -31,6 +31,8 @@ dependent_startup=true | |
| {% if INSTANCES %} | ||
| {% for redis_inst, redis_items in INSTANCES.items() %} | ||
| {%- if redis_inst != 'remote_redis' %} | ||
|
|
||
|
There was a problem hiding this comment. Suggest remove unnecessary change in this file There was a problem hiding this comment. This change is the same as #20545 , so I remove it. |
||
|
|
||
| [program:{{ redis_inst }}] | ||
| {% if redis_items['hostname'] != '127.0.0.1' %} | ||
| {%- set ADDITIONAL_OPTS = '--protected-mode no' %} | ||
|
|
@@ -53,6 +55,7 @@ stderr_logfile=syslog | |
| {% endfor %} | ||
| {% endif %} | ||
|
|
||
|
|
||
| [program:flushdb] | ||
| command=/bin/bash -c "sleep 300 && /usr/local/bin/flush_unused_database" | ||
| priority=3 | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@Pan-XT Could you upload a write-up for this multi-DB including the current issues and usage.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
https://github.com/dzhangalibaba/SONiC/blob/master/doc/database/multi_database_instances.md
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
#Closed