Skip to content

Commit

Permalink
v23.3.0
Browse files Browse the repository at this point in the history
  • Loading branch information
@ryan-lacy-veeva
ryan-lacy-veeva committed Dec 14, 2023
1 parent fc10d78 commit 759cc80
Show file tree
Hide file tree
Showing 5 changed files with 115 additions and 25 deletions.
33 changes: 20 additions & 13 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -8,8 +8,8 @@ This tool is distributed as a single JAR file, and does not require installation
click on the "Download" button. From there, the jar file can be run from a command line console.

Download the
latest <a href="https://github.com/veeva/Vault-Data-Tools/blob/main/vault-data-tools-23.2.0.jar">
vault-data-tools-23.2.0.jar</a>.
latest <a href="https://github.com/veeva/Vault-Data-Tools/blob/main/vault-data-tools-23.3.0.jar">
vault-data-tools-23.3.0.jar</a>.

## Quick Start

Expand Down Expand Up @@ -43,31 +43,31 @@ java -jar {jarFile} -datatype {datatype} -action {actionName} -input {filepath}
1. Count All Data

```
java -jar vault-data-tools-23.2.0.jar -datatype ALL -action COUNT -vaultDNS cholecap.veevavault.com -username my-username@cholecap.veevavault.com -password "my-password"
java -jar vault-data-tools-23.3.0.jar -datatype ALL -action COUNT -vaultDNS cholecap.veevavault.com -username my-username@cholecap.veevavault.com -password "my-password"
```

2. Count All Objects

```
java -jar vault-data-tools-23.2.0.jar -datatype OBJECTS -action COUNT -vaultDNS cholecap.veevavault.com -username my-username@cholecap.veevavault.com -password "my-password"
java -jar vault-data-tools-23.3.0.jar -datatype OBJECTS -action COUNT -vaultDNS cholecap.veevavault.com -username my-username@cholecap.veevavault.com -password "my-password"
```

3. Count All Documents

```
java -jar vault-data-tools-23.2.0.jar -datatype DOCUMENTS -action COUNT -vaultDNS cholecap.veevavault.com -username my-username@cholecap.veevavault.com -password "my-password"
java -jar vault-data-tools-23.3.0.jar -datatype DOCUMENTS -action COUNT -vaultDNS cholecap.veevavault.com -username my-username@cholecap.veevavault.com -password "my-password"
```

4. Count Specific Objects

```
java -jar vault-data-tools-23.2.0.jar -datatype DOCUMENTS -action COUNT -input ./objects-to-count.csv -vaultDNS cholecap.veevavault.com -username my-username@cholecap.veevavault.com -password "my-password"
java -jar vault-data-tools-23.3.0.jar -datatype DOCUMENTS -action COUNT -input ./objects-to-count.csv -vaultDNS cholecap.veevavault.com -username my-username@cholecap.veevavault.com -password "my-password"
```

5. Count All Custom Objects (Exclude System/Standard)

```
java -jar vault-data-tools-23.2.0.jar -datatype OBJECTS -action COUNT -exclude SYSTEM,STANDARD -vaultDNS cholecap.veevavault.com -username my-username@cholecap.veevavault.com -password "my-password"
java -jar vault-data-tools-23.3.0.jar -datatype OBJECTS -action COUNT -exclude SYSTEM,STANDARD -vaultDNS cholecap.veevavault.com -username my-username@cholecap.veevavault.com -password "my-password"
```

<br />
Expand All @@ -89,6 +89,7 @@ java -jar {jarFile} -datatype {datatype} -action {actionName} -input {filepath}
| -datatype | OBJECTS | ```-datatype OBJECTS``` | Used to delete all object record data in the specified Vault |
| | DOCUMENTS | ```-datatype DOCUMENTS``` | Used to delete all documents in the specified Vault |
| | ALL | ```-datatype ALL``` | Used to delete all data (object records and documents) from a specified Vault |
| -readonly | TRUE | ```-readonly TRUE``` | Read-only mode writes the data that would be deleted given the current inputs to CSV. No data is deleted. |
| -input | {.csv file} | ```-input ./objects-to-delete.csv``` | Optional path to location of input file containing a list of specific objects to delete when using the ```-datatype OBJECTS``` command. Find an example input file here: <a href="https://gitlab.veevadev.com/veevavaultdevsupport/vault-data-tools/-/blob/main/objects-to-delete.csv">objects-to-delete.csv</a>. To delete all records for a specific object, provide the object name in the first column. To optionally delete only specific records within that object, provide a unique idParam and idParamValue to identify those records. |
| | {.csv file} | ```-input ./documents-to-delete.csv``` | Optional path to location of input file containing a list of specific document types to delete when using the ```-datatype DOCUMENTS``` command. Find an example input file here: <a href="https://gitlab.veevadev.com/veevavaultdevsupport/vault-data-tools/-/blob/main/document-types-to-delete.csv">document-types-to-delete.csv</a> |
| -vaultDNS | {vault DNS} | ```-vaultDNS cholecap.veevavault.com``` | Vault DNS to delete data from (must be a Sandbox) |
Expand All @@ -104,35 +105,41 @@ java -jar {jarFile} -datatype {datatype} -action {actionName} -input {filepath}
1. Delete All Data

```
java -jar vault-data-tools-23.2.0.jar -datatype ALL -action DELETE -vaultDNS cholecap.veevavault.com -username my-username@cholecap.veevavault.com -password "my-password"
java -jar vault-data-tools-23.3.0.jar -datatype ALL -action DELETE -vaultDNS cholecap.veevavault.com -username my-username@cholecap.veevavault.com -password "my-password"
```

2. Delete All Objects

```
java -jar vault-data-tools-23.2.0.jar -datatype OBJECTS -action DELETE -vaultDNS cholecap.veevavault.com -username my-username@cholecap.veevavault.com -password "my-password"
java -jar vault-data-tools-23.3.0.jar -datatype OBJECTS -action DELETE -vaultDNS cholecap.veevavault.com -username my-username@cholecap.veevavault.com -password "my-password"
```

3. Delete All Documents

```
java -jar vault-data-tools-23.2.0.jar -datatype DOCUMENTS -action DELETE -vaultDNS cholecap.veevavault.com -username my-username@cholecap.veevavault.com -password "my-password"
java -jar vault-data-tools-23.3.0.jar -datatype DOCUMENTS -action DELETE -vaultDNS cholecap.veevavault.com -username my-username@cholecap.veevavault.com -password "my-password"
```

4. Delete Specific Objects

```
java -jar vault-data-tools-23.2.0.jar -datatype OBJECTS -action DELETE -input ./objects-to-delete.csv -vaultDNS cholecap.veevavault.com -username my-username@cholecap.veevavault.com -password "my-password"
java -jar vault-data-tools-23.3.0.jar -datatype OBJECTS -action DELETE -input ./objects-to-delete.csv -vaultDNS cholecap.veevavault.com -username my-username@cholecap.veevavault.com -password "my-password"
```

5. Delete Specific Document Types

```
java -jar vault-data-tools-23.2.0.jar -datatype DOCUMENTS -action DELETE -input ./documents-to-delete.csv -vaultDNS cholecap.veevavault.com -username my-username@cholecap.veevavault.com -password "my-password"
java -jar vault-data-tools-23.3.0.jar -datatype DOCUMENTS -action DELETE -input ./documents-to-delete.csv -vaultDNS cholecap.veevavault.com -username my-username@cholecap.veevavault.com -password "my-password"
```

6. Delete All Custom Objects (Exclude System/Standard)

```
java -jar vault-data-tools-23.2.0.jar -datatype OBJECTS -action DELETE -exclude SYSTEM,STANDARD -vaultDNS cholecap.veevavault.com -username my-username@cholecap.veevavault.com -password "my-password"
java -jar vault-data-tools-23.3.0.jar -datatype OBJECTS -action DELETE -exclude SYSTEM,STANDARD -vaultDNS cholecap.veevavault.com -username my-username@cholecap.veevavault.com -password "my-password"
```

7. Read Only Mode

```
java -jar vault-data-tools-23.3.0.jar -datatype ALL -action DELETE -readonly TRUE -vaultDNS cholecap.veevavault.com -username my-username@cholecap.veevavault.com -password "my-password"
```
4 changes: 2 additions & 2 deletions pom.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@

<groupId>com.veeva.vault</groupId>
<artifactId>vault-data-tools</artifactId>
<version>23.2.0</version>
<version>23.3.0</version>

<packaging>jar</packaging>

Expand All @@ -32,7 +32,7 @@
<dependency>
<groupId>com.veeva.vault</groupId>
<artifactId>vapil</artifactId>
<version>23.2.0</version>
<version>23.3.0</version>
</dependency>
<dependency>
<groupId>com.opencsv</groupId>
Expand Down
13 changes: 13 additions & 0 deletions src/main/java/com/veeva/vault/tools/cli/DataToolOptions.java
View file
Original file line number Diff line number Diff line change
Expand Up @@ -94,6 +94,19 @@ public void setExclude(String exclude) {
}
}

//------------------------------------------------------------------------------------------------
// Is Read Only: Expected Values [TRUE, FALSE]
//------------------------------------------------------------------------------------------------
@JsonProperty("readOnly")
@JsonAlias({"readonly"})
public String getReadyOnly() {
return this.getString("readOnly");
}

public void setReadOnly(String readOnly) {
this.set("readOnly", readOnly);
}

//------------------------------------------------------------------------------------------------
// Vault Authentication Details
//------------------------------------------------------------------------------------------------
Expand Down
90 changes: 80 additions & 10 deletions src/main/java/com/veeva/vault/tools/delete/DeleteVaultData.java
View file
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,7 @@ public class DeleteVaultData {
private HashMap<String, List<String>> inputData;
private List<DataToolOptions.Exclude> excludeSources;
private CSVWriter csvWriter;
private Boolean isReadOnly;

/**
* Main driver method for DeleteVaultData. Processes the given DataToolOptions and performs data deletion based on
Expand Down Expand Up @@ -71,15 +72,32 @@ public void process(DataToolOptions dataToolOptions) {
excludeSources = dataToolOptions.getExcludeList();
}

if (dataToolOptions.getReadyOnly() != null) {
if (dataToolOptions.getReadyOnly().equalsIgnoreCase("true") || dataToolOptions.getReadyOnly().equalsIgnoreCase("false")) {
isReadOnly = Boolean.valueOf(dataToolOptions.getReadyOnly());
} else {
logger.error("Unknown value provided for readOnly; Expected values = [TRUE, FALSE]");
return;
}
}

// Confirm user wants to proceed with deleting data
if (dataType != null && !confirmDataDeletion()) {
return;
}

String outputFileName = FileUtil.formatFileName("delete-data-output.csv");
String readOnlyFileNameModifier = "";
if (isReadOnly != null && isReadOnly) {
readOnlyFileNameModifier = "read-only-";
}
String outputFileName = FileUtil.formatFileName(readOnlyFileNameModifier + "delete-data-output.csv");
csvWriter = FileUtil.getCsvWriter(outputFileName);

String[] outputHeaders = {"action", "data_type", "name", "id", "status", "error_message"};
if (isReadOnly != null && isReadOnly) {
outputHeaders = new String[] {"data_type", "name", "id"};
}

List<String[]> headerData = Collections.singletonList(outputHeaders);
FileUtil.writeDataToCsv(headerData, csvWriter);

Expand Down Expand Up @@ -135,6 +153,14 @@ private boolean confirmDataDeletion() {
}
}

String warningMessage = "";
if (isReadOnly != null && isReadOnly) {
warningMessage = "READ-ONLY mode enabled. No data will be deleted.\n" +
"The data that would be deleted based on the current inputs will be written to file.";
} else {
warningMessage = "You are about to permanently delete this data from your Vault. THIS CANNOT BE UNDONE.";
}

/*
Intentionally using System.out instead of Logger to draw attention to this section and make it obvious
the user needs to confirm before proceeding with bulk deletion
Expand All @@ -145,8 +171,7 @@ private boolean confirmDataDeletion() {
System.out.print(selectedDataToDelete);
System.out.print(excludedSourcesString);
System.out.println();
System.out.println("You are about to permanently delete this data from your Vault.");
System.out.println("THIS CANNOT BE UNDONE.");
System.out.println(warningMessage);
System.out.println();
System.out.println("-------------------------------------------------------------------------------------");
System.out.print("Do you wish to proceed? (Y/N) ");
Expand Down Expand Up @@ -209,7 +234,11 @@ private void deleteObjectsHandler() {
// Delete the data in sorted order
for (String object : sorted) {
if (allDataToDelete.containsKey(object)) {
deleteData(object, "", allDataToDelete.get(object));
if (isReadOnly != null && isReadOnly) {
writeReadOnlyResultsToCSV("OBJECTS", object, allDataToDelete.get(object));
} else {
deleteData(object, "", allDataToDelete.get(object));
}
}
}
}
Expand Down Expand Up @@ -443,9 +472,23 @@ private void deleteDocumentsHandler() {
continue;
}

String query = String.format("SELECT id FROM documents WHERE type__v = '%s'", docType.getLabel());

deleteDataHandler("documents", docType.getName(), query);
StringBuilder query = new StringBuilder();
query.append("SELECT id FROM documents WHERE type__v = '").append(docType.getLabel()).append("'");

// If the input contains this document type and has idParams, add those idParamValues to query
if (inputData != null && inputData.containsKey(docType.getName())) {
// Only add idParamValues to the query if provided in the input
if (inputData.get(docType.getName()).size() > 1) {
String idParam = inputData.get(docType.getName()).get(0);
int paramValuesSize = inputData.get(docType.getName()).size();
List<String> idParamValues = inputData.get(docType.getName()).subList(1, paramValuesSize);
if (!idParam.isEmpty() && idParamValues.size() > 0) {
query.append(" AND ");
appendListToQuery(query, idParam, idParamValues);
}
}
}
deleteDataHandler("documents", docType.getName(), query.toString());
}
}
}
Expand All @@ -464,8 +507,12 @@ private void deleteDataHandler(String target, String type, String query) {

if (queryResponse != null && !queryResponse.getResponseStatus().equalsIgnoreCase("FAILURE") && queryResponse.getData().size() > 0) {

// Delete the first page
deleteData(target, type, queryResponse.getData());
if (isReadOnly != null && isReadOnly) {
writeReadOnlyResultsToCSV(target, type, queryResponse.getData());
} else {
// Delete the first page
deleteData(target, type, queryResponse.getData());
}

if (queryResponse.isPaginated()) {

Expand All @@ -474,7 +521,12 @@ private void deleteDataHandler(String target, String type, String query) {
queryResponse = Client.getVaultClient().newRequest(QueryRequest.class).queryByPage(nextPage);

if (queryResponse != null && !queryResponse.getResponseStatus().equalsIgnoreCase("FAILURE")) {
deleteData(target, type, queryResponse.getData());

if (isReadOnly != null && isReadOnly) {
writeReadOnlyResultsToCSV(target, type, queryResponse.getData());
} else {
deleteData(target, type, queryResponse.getData());
}
}
}
}
Expand Down Expand Up @@ -632,4 +684,22 @@ private String handleErrors(VaultResponse resp, boolean isIndividualResponse) {
return "";
}
}

/**
* In read-only mode, writes the data to be deleted to CSV
*
* @param dataType - OBJECTS or DOCUMENTS
* @param name - the object name or document type to be deleted
* @param dataToDelete - List of QueryResults to be deleted
*/
private void writeReadOnlyResultsToCSV(String dataType, String name, List<QueryResponse.QueryResult> dataToDelete) {
List<String[]> outputData = new ArrayList<>();

for (QueryResponse.QueryResult row : dataToDelete) {
String[] currentOutput = { dataType.toUpperCase(), name, row.get("id").toString()};
outputData.add(currentOutput);
}

FileUtil.writeDataToCsv(outputData, csvWriter);
}
}
Binary file renamed vault-data-tools-23.2.0.jar → vault-data-tools-23.3.0.jar
View file
Binary file not shown.

0 comments on commit 759cc80

Please sign in to comment.