Skip to content

Commit 80b142d

Browse files
dadoonetdadoonet
committed
Azure repository: Move to named configurations as we do for S3 repository
We should have the same behavior for Azure repositories as we have for S3 (see #22762). Instead of: ```yml cloud: azure: storage: my_account1: account: your_azure_storage_account1 key: your_azure_storage_key1 default: true my_account2: account: your_azure_storage_account2 key: your_azure_storage_key2 ``` Support something like: ``` azure.client: default: account: your_azure_storage_account1 key: your_azure_storage_key1 my_account2: account: your_azure_storage_account2 key: your_azure_storage_key2 ``` Then instead of: ``` PUT _snapshot/my_backup3 { "type": "azure", "settings": { "account": "my_account2" } } ``` Use: ``` PUT _snapshot/my_backup3 { "type": "azure", "settings": { "config": "my_account2" } } ``` If someone uses: ``` PUT _snapshot/my_backup3 { "type": "azure" } ``` It will use the `default` azure repository settings. And mark as deprecated old settings. Closes #22763.
1 parent f0c1e30 commit 80b142d

File tree

17 files changed

+482
-208
lines changed

17 files changed

+482
-208
lines changed

‎docs/plugins/repository-azure.asciidoc‎

Lines changed: 26 additions & 44 deletions
Original file line numberDiff line numberDiff line change
@@ -10,35 +10,28 @@ include::install_remove.asciidoc[]
1010
[[repository-azure-usage]]
1111
==== Azure Repository
1212

13-
To enable Azure repositories, you have first to set your azure storage settings in `elasticsearch.yml` file:
13+
To enable Azure repositories, you have first to define your azure storage settings as
14+
{ref}/secure-settings.html[secured settings]:
1415

15-
[source,yaml]
16-
----
17-
cloud:
18-
azure:
19-
storage:
20-
my_account:
21-
account: your_azure_storage_account
22-
key: your_azure_storage_key
23-
----
16+
[source,sh]
17+
----------------------------------------------------------------
18+
bin/elasticsearch-keystore add azure.client.default.account
19+
bin/elasticsearch-keystore add azure.client.default.key
20+
----------------------------------------------------------------
21+
22+
Where `account` is the azure account name and `key` the azure secret key.
2423

2524
Note that you can also define more than one account:
2625

27-
[source,yaml]
28-
----
29-
cloud:
30-
azure:
31-
storage:
32-
my_account1:
33-
account: your_azure_storage_account1
34-
key: your_azure_storage_key1
35-
default: true
36-
my_account2:
37-
account: your_azure_storage_account2
38-
key: your_azure_storage_key2
39-
----
26+
[source,sh]
27+
----------------------------------------------------------------
28+
bin/elasticsearch-keystore add azure.client.default.account
29+
bin/elasticsearch-keystore add azure.client.default.key
30+
bin/elasticsearch-keystore add azure.client.secondary.account
31+
bin/elasticsearch-keystore add azure.client.secondary.key
32+
----------------------------------------------------------------
4033

41-
`my_account1` is the default account which will be used by a repository unless you set an explicit one.
34+
`default` is the default account name which will be used by a repository unless you set an explicit one.
4235

4336
You can set the client side timeout to use when making any single request. It can be defined globally, per account or both.
4437
It's not set by default which means that elasticsearch is using the
@@ -53,23 +46,13 @@ before retrying after a first timeout or failure. The maximum backoff period is
5346

5447
[source,yaml]
5548
----
56-
cloud:
57-
azure:
58-
storage:
59-
timeout: 10s
60-
my_account1:
61-
account: your_azure_storage_account1
62-
key: your_azure_storage_key1
63-
default: true
64-
max_retries: 7
65-
my_account2:
66-
account: your_azure_storage_account2
67-
key: your_azure_storage_key2
68-
timeout: 30s
49+
cloud.azure.storage.timeout: 10s
50+
azure.client.default.max_retries: 7
51+
azure.client.secondary.timeout: 30s
6952
----
7053

71-
In this example, timeout will be `10s` per try for `my_account1` with `7` retries before failing
72-
and `30s` per try for `my_account2` with `3` retries.
54+
In this example, timeout will be `10s` per try for `default` with `7` retries before failing
55+
and `30s` per try for `secondary` with `3` retries.
7356

7457
[IMPORTANT]
7558
.Supported Azure Storage Account types
@@ -89,10 +72,9 @@ https://azure.microsoft.com/en-gb/documentation/articles/storage-premium-storage
8972

9073
The Azure repository supports following settings:
9174

92-
`account`::
75+
`client`::
9376

94-
Azure account settings to use. Defaults to the only one if you set a single
95-
account or to the one marked as `default` if you have more than one.
77+
Azure named client to use. Defaults to `default`.
9678

9779
`container`::
9880

@@ -153,14 +135,14 @@ PUT _snapshot/my_backup3
153135
{
154136
"type": "azure",
155137
"settings": {
156-
"account": "my_account1"
138+
"client": "secondary"
157139
}
158140
}
159141
PUT _snapshot/my_backup4
160142
{
161143
"type": "azure",
162144
"settings": {
163-
"account": "my_account2",
145+
"client": "secondary",
164146
"location_mode": "primary_only"
165147
}
166148
}

‎plugins/repository-azure/build.gradle‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -45,4 +45,8 @@ thirdPartyAudit.excludes = [
4545
integTestCluster {
4646
setting 'cloud.azure.storage.my_account_test.account', 'cloudazureresource'
4747
setting 'cloud.azure.storage.my_account_test.key', 'abcdefgh'
48+
keystoreSetting 'azure.client.default.account', 'cloudazureresource'
49+
keystoreSetting 'azure.client.default.key', 'abcdefgh'
50+
keystoreSetting 'azure.client.secondary.account', 'cloudazureresource'
51+
keystoreSetting 'azure.client.secondary.key', 'abcdefgh'
4852
}

‎plugins/repository-azure/src/main/java/org/elasticsearch/cloud/azure/blobstore/AzureBlobStore.java‎

Lines changed: 11 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -22,7 +22,6 @@
2222
import com.microsoft.azure.storage.LocationMode;
2323
import com.microsoft.azure.storage.StorageException;
2424
import org.elasticsearch.cloud.azure.storage.AzureStorageService;
25-
import org.elasticsearch.cloud.azure.storage.AzureStorageService.Storage;
2625
import org.elasticsearch.cluster.metadata.RepositoryMetaData;
2726
import org.elasticsearch.common.Strings;
2827
import org.elasticsearch.common.blobstore.BlobContainer;
@@ -45,7 +44,7 @@ public class AzureBlobStore extends AbstractComponent implements BlobStore {
4544

4645
private final AzureStorageService client;
4746

48-
private final String accountName;
47+
private final String clientName;
4948
private final LocationMode locMode;
5049
private final String container;
5150
private final String repositoryName;
@@ -56,7 +55,7 @@ public AzureBlobStore(RepositoryMetaData metadata, Settings settings,
5655
this.client = client;
5756
this.container = Repository.CONTAINER_SETTING.get(metadata.settings());
5857
this.repositoryName = metadata.name();
59-
this.accountName = Repository.ACCOUNT_SETTING.get(metadata.settings());
58+
this.clientName = Repository.CLIENT_NAME.get(metadata.settings());
6059

6160
String modeStr = Repository.LOCATION_MODE_SETTING.get(metadata.settings());
6261
if (Strings.hasLength(modeStr)) {
@@ -91,7 +90,7 @@ public BlobContainer blobContainer(BlobPath path) {
9190
public void delete(BlobPath path) {
9291
String keyPath = path.buildAsString();
9392
try {
94-
this.client.deleteFiles(this.accountName, this.locMode, container, keyPath);
93+
this.client.deleteFiles(this.clientName, this.locMode, container, keyPath);
9594
} catch (URISyntaxException | StorageException e) {
9695
logger.warn("can not remove [{}] in container {{}}: {}", keyPath, container, e.getMessage());
9796
}
@@ -103,41 +102,41 @@ public void close() {
103102

104103
public boolean doesContainerExist(String container)
105104
{
106-
return this.client.doesContainerExist(this.accountName, this.locMode, container);
105+
return this.client.doesContainerExist(this.clientName, this.locMode, container);
107106
}
108107

109108
public void deleteFiles(String container, String path) throws URISyntaxException, StorageException
110109
{
111-
this.client.deleteFiles(this.accountName, this.locMode, container, path);
110+
this.client.deleteFiles(this.clientName, this.locMode, container, path);
112111
}
113112

114113
public boolean blobExists(String container, String blob) throws URISyntaxException, StorageException
115114
{
116-
return this.client.blobExists(this.accountName, this.locMode, container, blob);
115+
return this.client.blobExists(this.clientName, this.locMode, container, blob);
117116
}
118117

119118
public void deleteBlob(String container, String blob) throws URISyntaxException, StorageException
120119
{
121-
this.client.deleteBlob(this.accountName, this.locMode, container, blob);
120+
this.client.deleteBlob(this.clientName, this.locMode, container, blob);
122121
}
123122

124123
public InputStream getInputStream(String container, String blob) throws URISyntaxException, StorageException, IOException
125124
{
126-
return this.client.getInputStream(this.accountName, this.locMode, container, blob);
125+
return this.client.getInputStream(this.clientName, this.locMode, container, blob);
127126
}
128127

129128
public OutputStream getOutputStream(String container, String blob) throws URISyntaxException, StorageException
130129
{
131-
return this.client.getOutputStream(this.accountName, this.locMode, container, blob);
130+
return this.client.getOutputStream(this.clientName, this.locMode, container, blob);
132131
}
133132

134133
public Map<String,BlobMetaData> listBlobsByPrefix(String container, String keyPath, String prefix) throws URISyntaxException, StorageException
135134
{
136-
return this.client.listBlobsByPrefix(this.accountName, this.locMode, container, keyPath, prefix);
135+
return this.client.listBlobsByPrefix(this.clientName, this.locMode, container, keyPath, prefix);
137136
}
138137

139138
public void moveBlob(String container, String sourceBlob, String targetBlob) throws URISyntaxException, StorageException
140139
{
141-
this.client.moveBlob(this.accountName, this.locMode, container, sourceBlob, targetBlob);
140+
this.client.moveBlob(this.clientName, this.locMode, container, sourceBlob, targetBlob);
142141
}
143142
}

‎plugins/repository-azure/src/main/java/org/elasticsearch/cloud/azure/storage/AzureStorageService.java‎

Lines changed: 9 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -45,12 +45,19 @@ public interface AzureStorageService {
4545
ByteSizeValue MAX_CHUNK_SIZE = new ByteSizeValue(64, ByteSizeUnit.MB);
4646

4747
final class Storage {
48+
@Deprecated
4849
public static final String PREFIX = "cloud.azure.storage.";
4950

51+
@Deprecated
5052
public static final Setting<Settings> STORAGE_ACCOUNTS = Setting.groupSetting(Storage.PREFIX, Setting.Property.NodeScope);
5153

52-
public static final Setting<TimeValue> TIMEOUT_SETTING =
53-
Setting.timeSetting("cloud.azure.storage.timeout", TimeValue.timeValueMinutes(-1), Property.NodeScope);
54+
/**
55+
* Azure timeout (defaults to -1 minute)
56+
* @deprecated We don't want to support global timeout settings anymore
57+
*/
58+
@Deprecated
59+
static final Setting<TimeValue> TIMEOUT_SETTING =
60+
Setting.timeSetting("cloud.azure.storage.timeout", TimeValue.timeValueMinutes(-1), Property.NodeScope, Property.Deprecated);
5461
}
5562

5663
boolean doesContainerExist(String account, LocationMode mode, String container);

‎plugins/repository-azure/src/main/java/org/elasticsearch/cloud/azure/storage/AzureStorageServiceImpl.java‎

Lines changed: 46 additions & 35 deletions
Original file line numberDiff line numberDiff line change
@@ -45,39 +45,57 @@
4545
import java.io.OutputStream;
4646
import java.net.URI;
4747
import java.net.URISyntaxException;
48-
import java.security.AccessController;
49-
import java.security.PrivilegedActionException;
50-
import java.security.PrivilegedExceptionAction;
5148
import java.util.HashMap;
5249
import java.util.Map;
5350

5451
public class AzureStorageServiceImpl extends AbstractComponent implements AzureStorageService {
5552

56-
final AzureStorageSettings primaryStorageSettings;
57-
final Map<String, AzureStorageSettings> secondariesStorageSettings;
53+
final Map<String, AzureStorageSettings> storageSettings;
54+
final Map<String, AzureStorageSettings> deprecatedStorageSettings;
5855

5956
final Map<String, CloudBlobClient> clients;
6057

61-
public AzureStorageServiceImpl(Settings settings) {
58+
public AzureStorageServiceImpl(Settings settings, Map<String, AzureStorageSettings> regularStorageSettings) {
6259
super(settings);
6360

64-
Tuple<AzureStorageSettings, Map<String, AzureStorageSettings>> storageSettings = AzureStorageSettings.parse(settings);
65-
this.primaryStorageSettings = storageSettings.v1();
66-
this.secondariesStorageSettings = storageSettings.v2();
61+
if (regularStorageSettings.isEmpty()) {
62+
this.storageSettings = new HashMap<>();
63+
// We have deprecated settings so we need to migrate them to the new implementation
64+
Tuple<AzureStorageSettings, Map<String, AzureStorageSettings>> storageSettingsMapTuple = AzureStorageSettings.loadLegacy(settings);
65+
deprecatedStorageSettings = storageSettingsMapTuple.v2();
66+
if (storageSettingsMapTuple.v1() != null) {
67+
if (storageSettingsMapTuple.v1().getName().equals("default") == false) {
68+
// We add the primary configuration to the list of all settings with its deprecated name in case someone is
69+
// forcing a specific configuration name when creating the repository instance
70+
deprecatedStorageSettings.put(storageSettingsMapTuple.v1().getName(), storageSettingsMapTuple.v1());
71+
}
72+
// We add the primary configuration to the list of all settings as the "default" one
73+
deprecatedStorageSettings.put("default", storageSettingsMapTuple.v1());
74+
} else {
75+
// If someone did not register any settings or deprecated settings, they
76+
// basically can't use the plugin
77+
throw new IllegalArgumentException("If you want to use an azure repository, you need to define a client configuration.");
78+
}
79+
80+
81+
} else {
82+
this.storageSettings = regularStorageSettings;
83+
this.deprecatedStorageSettings = new HashMap<>();
84+
}
6785

6886
this.clients = new HashMap<>();
6987

7088
logger.debug("starting azure storage client instance");
7189

72-
// We register the primary client if any
73-
if (primaryStorageSettings != null) {
74-
logger.debug("registering primary client for account [{}]", primaryStorageSettings.getAccount());
75-
createClient(primaryStorageSettings);
90+
// We register all regular azure clients
91+
for (Map.Entry<String, AzureStorageSettings> azureStorageSettingsEntry : this.storageSettings.entrySet()) {
92+
logger.debug("registering regular client for account [{}]", azureStorageSettingsEntry.getKey());
93+
createClient(azureStorageSettingsEntry.getValue());
7694
}
7795

78-
// We register all secondary clients
79-
for (Map.Entry<String, AzureStorageSettings> azureStorageSettingsEntry : secondariesStorageSettings.entrySet()) {
80-
logger.debug("registering secondary client for account [{}]", azureStorageSettingsEntry.getKey());
96+
// We register all deprecated azure clients
97+
for (Map.Entry<String, AzureStorageSettings> azureStorageSettingsEntry : this.deprecatedStorageSettings.entrySet()) {
98+
logger.debug("registering deprecated client for account [{}]", azureStorageSettingsEntry.getKey());
8199
createClient(azureStorageSettingsEntry.getValue());
82100
}
83101
}
@@ -107,32 +125,25 @@ void createClient(AzureStorageSettings azureStorageSettings) {
107125

108126
CloudBlobClient getSelectedClient(String account, LocationMode mode) {
109127
logger.trace("selecting a client for account [{}], mode [{}]", account, mode.name());
110-
AzureStorageSettings azureStorageSettings = null;
111-
112-
if (this.primaryStorageSettings == null) {
113-
throw new IllegalArgumentException("No primary azure storage can be found. Check your elasticsearch.yml.");
114-
}
115-
116-
if (Strings.hasLength(account)) {
117-
azureStorageSettings = this.secondariesStorageSettings.get(account);
118-
}
119-
120-
// if account is not secondary, it's the primary
128+
AzureStorageSettings azureStorageSettings = this.storageSettings.get(account);
121129
if (azureStorageSettings == null) {
122-
if (Strings.hasLength(account) == false || primaryStorageSettings.getName() == null || account.equals(primaryStorageSettings.getName())) {
123-
azureStorageSettings = primaryStorageSettings;
130+
// We can't find a client that has been registered using regular settings so we try deprecated client
131+
azureStorageSettings = this.deprecatedStorageSettings.get(account);
132+
if (azureStorageSettings == null) {
133+
// We did not get an account. That's bad.
134+
if (Strings.hasLength(account)) {
135+
throw new IllegalArgumentException("Can not find named azure client [" + account +
136+
"]. Check your elasticsearch.yml.");
137+
}
138+
throw new IllegalArgumentException("Can not find primary/secondary client using deprecated settings. " +
139+
"Check your elasticsearch.yml.");
124140
}
125141
}
126142

127-
if (azureStorageSettings == null) {
128-
// We did not get an account. That's bad.
129-
throw new IllegalArgumentException("Can not find azure account [" + account + "]. Check your elasticsearch.yml.");
130-
}
131-
132143
CloudBlobClient client = this.clients.get(azureStorageSettings.getAccount());
133144

134145
if (client == null) {
135-
throw new IllegalArgumentException("Can not find an azure client for account [" + account + "]");
146+
throw new IllegalArgumentException("Can not find an azure client for account [" + azureStorageSettings.getAccount() + "]");
136147
}
137148

138149
// NOTE: for now, just set the location mode in case it is different;

0 commit comments

Comments
 (0)