Changing tenant configuration writes only changed values to disk, continues with different config

[koivunej]

Modifying the added test in #3446 to instead of changing gc_horizon twice to in different http PUT requests:

1. set gc_horizon
2. set pitr_interval

Will write to disk as last version:

# This file contains a specific per-tenant's config.
#  It is read in case of pageserver restart.

[tenant_config]
pitr_interval = "0s"

But adding logging to Tenant::update_tenant_config shows that the tenant config is first updated to:

2023-01-27T15:27:39.514582Z  INFO tenant_config{tenant=db6238f481f8459ce03705f1fa96ab42}: updated config TenantConfOpt {
    checkpoint_distance: None,
    checkpoint_timeout: None,
    compaction_target_size: None,
    compaction_period: None,
    compaction_threshold: None,
    gc_horizon: Some(
        1000000,
    ),
    gc_period: None,
    image_creation_threshold: None,
    pitr_interval: None,
    walreceiver_connect_timeout: None,
    lagging_wal_timeout: None,
    max_lsn_wal_lag: None,
    trace_read_requests: None,
}

Then by the second request:

2023-01-27T15:27:39.521233Z  INFO tenant_config{tenant=db6238f481f8459ce03705f1fa96ab42}: updated config TenantConfOpt {
    checkpoint_distance: None,
    checkpoint_timeout: None,
    compaction_target_size: None,
    compaction_period: None,
    compaction_threshold: None,
    gc_horizon: Some(
        1000000,
    ),
    gc_period: None,
    image_creation_threshold: None,
    pitr_interval: Some(
        0ns,
    ),
    walreceiver_connect_timeout: None,
    lagging_wal_timeout: None,
    max_lsn_wal_lag: None,
    trace_read_requests: None,
}

What should happen, the written to disk version should be the union of settings (with default tenant config values removed):

# This file contains a specific per-tenant's config.
#  It is read in case of pageserver restart.

[tenant_config]
gc_horizon = 1000000
pitr_interval = "0s"

Perhaps we should also add a GET /v1/tenant/:tenant_id/config to get the current tenant config out, so that we can easily compare it always matches the one on disk after single-threaded updates over HTTP.

Origin:

• Allow creating config for attached tenant #3446 (comment)