Issues with RabbitMQ after upgrading from 3.12 to v4

[CTIBurn0ut]

After upgrading OpenCTI from 6.2.1 to 6.4.5, rabbitMQ was also upgraded from 3.12 to 4.0.

As a result, the platform does not start and getting this error:

=INFO REPORT==== 19-Dec-2024::21:16:36.429796 === alarm_handler: {set,{system_memory_high_watermark,[]}} 2024-12-19 21:16:38.936506+00:00 [error] <0.216.0> Feature flags: stream_filtering: required feature flag not enabled! It must be enabled before upgrading RabbitMQ. 2024-12-19 21:16:38.945036+00:00 [error] <0.216.0> Failed to initialize feature flags registry: {disabled_required_feature_flag, 2024-12-19 21:16:38.945036+00:00 [error] <0.216.0> stream_filtering} 2024-12-19 21:16:38.950377+00:00 [error] <0.216.0> 2024-12-19 21:16:38.950377+00:00 [error] <0.216.0> BOOT FAILED 2024-12-19 21:16:38.950377+00:00 [error] <0.216.0> =========== 2024-12-19 21:16:38.950377+00:00 [error] <0.216.0> Error during startup: {error,failed_to_initialize_feature_flags_registry} 2024-12-19 21:16:38.950377+00:00 [error] <0.216.0> BOOT FAILED =========== Error during startup: {error,failed_to_initialize_feature_flags_registry} 2024-12-19 21:16:39.953143+00:00 [notice] <0.45.0> Application rabbit exited with reason: {failed_to_initialize_feature_flags_registry,{rabbit,start,[normal,[]]}} {exit,terminating,[{application_controller,call,2,[{file,"application_controller.erl"},{line,511}]},{application,'-ensure_all_started/3-lc$^0/1-0-',1,[{file,"application.erl"},{line,367}]},{application,ensure_all_started,3,[{file,"application.erl"},{line,367}]},{rabbit,'-start_it/1-fun-0-',1,[{file,"rabbit.erl"},{line,430}]},{timer,tc,2,[{file,"timer.erl"},{line,595}]},{rabbit,start_it,1,[{file,"rabbit.erl"},{line,426}]},{init,start_it,1,[]},{init,start_em,1,[]}]} Runtime terminating during boot (terminating) Crash dump is being written to: erl_crash.dump...[os_mon] cpu supervisor port (cpu_sup): Erlang has closed [os_mon] memory supervisor port (memsup): Erlang has closed

The error indicates that RabbitMQ requires the stream_filtering feature flag to be enabled before upgrading. Additionally, the system memory watermark alarm suggests that RabbitMQ might have run into resource constraints.

We need guidance/recommendations for customers that will be upgrading on how to solve this.

Steps to Fix the Issue

1 - Enable the Required Feature Flag:

• RabbitMQ uses feature flags to manage backward-compatible changes. To enable stream_filtering, you'll need to access the RabbitMQ management UI or CLI.

Via RabbitMQ CLI:

rabbitmqctl enable_feature_flag stream_filtering

Via RabbitMQ Management UI:

• Log in to the management UI (http://:15672).
• Navigate to the Feature Flags section.
• Enable the stream_filtering flag.
  Check RabbitMQ Version: Ensure your RabbitMQ version supports the required feature flag. Use:
• rabbitmqctl status
• Look for the RabbitMQ and Erlang/OTP versions.
• Verify the stream_filtering feature is available in the version you're running.

System Resource Configuration:

• The system memory watermark alarm indicates high memory usage.
• Modify the memory watermark settings if you have sufficient system resources:
  rabbitmqctl set_vm_memory_high_watermark 0.8
  Replace 0.8 (80%) with a suitable value.
• Consider tuning RabbitMQ for the system:
• Check for the rabbitmq.conf file (usually in /etc/rabbitmq/ or /usr/local/etc/rabbitmq/).

Set resource limits:

vm_memory_high_watermark.relative = 0.8

• Restart RabbitMQ: After enabling the feature flag and updating the configuration, restart RabbitMQ:
  sudo systemctl restart rabbitmq-server
  Check Logs for Errors: Review RabbitMQ logs after restarting:
  tail -f /var/log/rabbitmq/rabbit@<hostname>.log

Upgrade Process: Once the required feature flag is enabled and the system is stable:

• Re-run the RabbitMQ upgrade process.
• Verify the upgrade logs for any additional warnings or errors.

[CTIBurn0ut]

Docker:

`version: '3.8'

services:
rabbitmq:
image: rabbitmq:3.12-management # Use a version that supports stream_filtering
container_name: rabbitmq
ports:
- "5672:5672" # AMQP port
- "15672:15672" # Management UI
environment:
- RABBITMQ_DEFAULT_USER=admin
- RABBITMQ_DEFAULT_PASS=admin
- RABBITMQ_VM_MEMORY_HIGH_WATERMARK=0.8 # Set memory watermark
volumes:
- ./rabbitmq.conf:/etc/rabbitmq/rabbitmq.conf # Mount custom config
command: >
sh -c "
rabbitmq-server &
sleep 5 &&
rabbitmqctl wait --timeout 60 /var/lib/rabbitmq/mnesia/rabbit@${HOSTNAME}.pid &&
rabbitmqctl enable_feature_flag stream_filtering &&
rabbitmqctl stop_app &&
rabbitmqctl start_app
"
`

Explanation of Key Sections

1 - Image Selection:
Replace rabbitmq:3.12-management with a specific version that supports stream_filtering.

2 - Feature Flag Enablement:
The command section runs a sequence of commands to enable the stream_filtering feature flag after the RabbitMQ server starts.

3 - Memory High Watermark:
The RABBITMQ_VM_MEMORY_HIGH_WATERMARK=0.8 environment variable sets the memory limit to 80%.

4 - Custom Configuration:
Mount a rabbitmq.conf file into the container to manage advanced settings.