Spring Cloud GCP Pub/Sub Binder: default consumer configuration property effect is nullified when binding level config defined

[mayur-solace]

I'm using v3.6.3 of spring-cloud-gcp-pubsub-stream-binder. I have multiple consumer and producer bindings in my sample application.

I'm trying to disable auto resource (topic/subscription) creation globally for all consumers, however the resources are still being created in case of below mentioned scenario.

I'm using this sample sink for reproduction with below configuration changes.

My configuration:

1. add disable auto resource creation: spring.cloud.stream.gcp.pubsub.default.consumer.auto-create-resources=false
2. comment group name: #spring.cloud.stream.bindings.input.group=my-group
3. add consumer subscription name: spring.cloud.stream.gcp.pubsub.bindings.input.consumer.subscription-name=gcp-pubsub-topic-sub

As soon as spring.cloud.stream.gcp.pubsub.bindings.input.consumer.subscription-name property is added, effect of
spring.cloud.stream.gcp.pubsub.default.consumer.auto-create-resources is nullified.

i.e. in other words if spring.cloud.stream.gcp.pubsub.bindings.input.consumer.subscription-name is removed, spring.cloud.stream.gcp.pubsub.default.consumer.auto-create-resources=false would not create any resources as expected.

Here is complete configuration:

# Binding name is mapped to a simpler shortcut name "input". If you omit the mapping, make sure to
# use the full name (spring.cloud.stream.bindings.logUserMessage-in-0.) instead of the shortcut
# (spring.cloud.stream.bindings.input.) as configuration prefix for all properties related to this
# binding.
spring.cloud.stream.function.bindings.logUserMessage-in-0=input
spring.cloud.stream.bindings.input.destination=my-topic

# Optional, as Spring Cloud Stream will autodiscover the correct functional bean.
spring.cloud.function.definition=logUserMessage

# If group is specified, the Pub/Sub subscription name will be [PUBSUB_TOPIC_NAME].[PUBSUB_GROUP_NAME]
#spring.cloud.stream.bindings.input.group=my-group

# Custom port to avoid conflict with the Source app.
server.port=8081

#spring.cloud.gcp.project-id=[YOUR_GCP_PROJECT_ID]
#spring.cloud.gcp.credentials.location=file:[LOCAL_PATH_TO_CREDENTIALS]

spring.cloud.stream.gcp.pubsub.default.consumer.auto-create-resources=false
spring.cloud.stream.gcp.pubsub.bindings.input.consumer.subscription-name=gcp-pubsub-topic-sub

Background
I'm developing an application that has multiple consumer functions and multiple publisher functions.
I want an ability to globally enable/disable resource creation for all consumers, publisher.

[meltsufin]

It looks like the way Cloud Stream works is that defaults are only used when no properties are specified for the specific binding.

I'm not sure why there isn't some kind of merging of the properties. @artembilan can you please advise here?

[artembilan]

The default means for all not specified.
I would suggest to try like this:

spring.cloud.stream.gcp.pubsub.bindings.input.consumer.auto-create-resources=false
spring.cloud.stream.gcp.pubsub.bindings.input.consumer.subscription-name=gcp-pubsub-topic-sub

Apparently you are right: there is no merge if we have binding-specific properties and that default private boolean autoCreateResources = true; has an effect in this case.

[mayur-solace]

@artembilan, I know the config you mentioned would work. I was hoping merging to work, so if I have many bindings I don't have to repeat auto-create-resources=false.

I know Solace Binder does automatic merging. I'm will have to verify if Kafka binder does merging or not.

[artembilan]

So, here is an answer.
There is a merge with defaults in Spring Cloud Stream core, but only for common binder properties.
If your have binder-specific, like this stream.gcp.pubsub, then you need to implement something similar to this:
https://github.com/spring-cloud/spring-cloud-stream/blob/main/binders/kafka-binder/spring-cloud-stream-binder-kafka/src/main/java/org/springframework/cloud/stream/binder/kafka/config/ExtendedBindingHandlerMappingsProviderConfiguration.java
or this:
https://github.com/spring-cloud/spring-cloud-stream/blob/main/binders/rabbit-binder/spring-cloud-stream-binder-rabbit/src/main/java/org/springframework/cloud/stream/binder/rabbit/config/ExtendedBindingHandlerMappingsProviderConfiguration.java

and add it to org.springframework.boot.autoconfigure.AutoConfiguration.imports.

And that is exactly what is done in the Solace Binder as well: https://github.com/SolaceProducts/solace-spring-cloud/blob/master/solace-spring-cloud-stream-binder/solace-spring-cloud-stream-binder/src/main/java/com/solace/spring/cloud/stream/binder/config/ExtendedBindingHandlerMappingsProviderConfiguration.java

[meltsufin]

@artembilan Interesting, something like this was done back in https://github.com/spring-attic/spring-cloud-gcp/pull/1107/files#diff-740698cdd78cb559d5ba86af224be4e027dfafa169ec34544052662c4313fb0aR59. However, it looks like adding it to org.springframework.boot.autoconfigure.AutoConfiguration.imports was missed??

[artembilan]

Yes, that correct.
That is indeed a mistake according to the logic where that MappingsProvider is used.
See BindingServiceConfiguration:

@Bean
public BindingHandlerAdvise BindingHandlerAdvise(
			@Nullable MappingsProvider[] providers) {

But every binder and its configuration is processed only later on by the next bean in this config: DefaultBinderFactory binderFactory.
Therefore all the beans we are hiding into binder configuration (essentially, an entry of the spring.binders), are not visible in the root (parent) application context.

Please, consider to fix this problem same way as it is done in other binders.

Sorry for inconvenience. I also feel like I would add simple beans to the binder config. However this is not how things work.