Note: this service only works with Java 11
Every migration starts with creating a ChangeSet (annotated with @ChangeSet). It contains the following attributes:
| Attribute | Description | Mandatory? |
|---|---|---|
| id | Returns the ChangeSet's id that will be stored in the ChangeSet history table/collection and will the way to identify a ChangeSet. The combination of this field and the author must be unique among the changesets. | Yes |
| order | Returns the ChangeSet's execution order. | Yes |
| author | Returns the ChangeSet's author. The combination of this and the author must be unique among the changesets. | No |
The Loader has been tested with MongoDB and Azure CosmosDB API for MongoDB.
There is an additional flag for supporting CosmosDB -Dflow.mongo.cosmosdbttl=true as TTL indexes are a known limitation. Additionally, support for CosmosDB means we can't use renameCollection() in our changesets.
The changelog and lock collections are defined in /loader/src/main/java/net/boomerangplatform/migration/config/BoomerangFlowConfig.java
https://www.mongodb.com/docs/manual/reference/method/js-database/
To create a collection, use the db.createCollection() method.
db.createCollection(collectionPrefix + "settings");
https://www.mongodb.com/docs/manual/reference/method/js-collection/
In /loader/src/main/resources/flow, there are several numbered folders. The numbering for the folder matches the changeset in which it is applied.
For example,
@ChangeSet(order = "026", id = "026", author = "Adrienne Hudson")
public void createFlowSettings(MongoDatabase db) throws IOException {
db.createCollection(collectionPrefix + "settings");
final List<String> files = fileloadingService.loadFiles("flow/026/flow_settings/*.json");
for (final String fileContents : files) {
final Document doc = Document.parse(fileContents);
final MongoCollection<Document> collection = db.getCollection(collectionPrefix + "settings");
collection.insertOne(doc);
}
}
where the documents added with changeset 026 are contained within the flow/026/flow_settings/ folder in /loader/src/main/resources/
https://www.mongodb.com/docs/manual/reference/method/js-collection/
Modifying existing documents requires identification of the document(s).
@ChangeSet(order = "064", id = "064", author = "Adrienne Hudson")
public void updateWorker(MongoDatabase db) throws IOException {
MongoCollection<Document> collection = db.getCollection(collectionPrefix + "settings");
Document workers = collection.find(eq("name", "Task Configuration")).first();
List<Document> configs = (List<Document>) workers.get("config");
for (Document config : configs) {
if (config.get("key").equals("worker.image")) {
config.put("value", "boomerangio/worker-flow:2.8.11");
}
}
workers.put("config", configs);
collection.replaceOne(eq("name", "Task Configuration"), workers);
}
In this example, we query to find the first document in the collectionPrefix + "settings collection with the "name": "Task Configuration" with the statement Document workers = collection.find(eq("name", "Task Configuration")).first();. Once the document has been identified, the config with "key":"worker.image" is updated with "value":"boomerangio/worker-flow:2.8.11"
and the update is written back onto the document.
https://mongodb.github.io/mongo-java-driver/3.4/javadoc/org/bson/Document
@ChangeSet(order = "073", id = "073", author = "Adrienne Hudson")
public void migrateEnablePersistentStorage(MongoDatabase db) throws IOException {
final MongoCollection<Document> flowWorkflowsCollection =
db.getCollection(collectionPrefix + "workflows");
final FindIterable<Document> flowWorkflows = flowWorkflowsCollection.find();
for (final Document flowWorkflow : flowWorkflows) {
boolean enablePersistentStorage = (boolean) flowWorkflow.get("enablePersistentStorage");
Document storage =
(Document) flowWorkflow.get("storage") != null ? (Document) flowWorkflow.get("storage")
: new Document();
Document workflowStorage =
(Document) storage.get("workflow") != null ? (Document) storage.get("workflow")
: new Document();
workflowStorage.put("enabled", enablePersistentStorage);
storage.put("workflow", workflowStorage);
flowWorkflow.put("storage", storage);
flowWorkflow.remove("enablePersistentStorage");
flowWorkflowsCollection.replaceOne(eq("_id", flowWorkflow.getObjectId("_id")), flowWorkflow);
}
}
In this example, the update applies to every document in the collectionPrefix + "workflows" collection. By using the org.bson.Document.put(String key, Object value) and org.bson.Document.remove(Object key), object can be created, updated, or removed from a document.
https://www.mongodb.com/docs/manual/reference/method/js-database/
To remove a document in a collection, use the db.collection.findOneAndDelete() method.
@ChangeSet(order = "108", id = "108", author = "Adrienne Hudson")
public void removeTemplates(MongoDatabase db) throws IOException {
MongoCollection<Document> collection = db.getCollection(collectionPrefix + "workflows");
collection.findOneAndDelete(eq("name", "Looking through planets with HTTP Call "));
collection.findOneAndDelete(eq("name", "MongoDB email query results"));
}
docker run --name local-mongo -d mongo:latest
mvn clean package
java -Dspring.profiles.active={profile} -jar target/loader.jar
If you are running a higher version of Java, you will need to locate the Java 11 version by running
/usr/libexec/java_home -V
Using the output you can then run the particular version, such as
/Library/Java/JavaVirtualMachines/ibm-semeru-open-11.jdk/Contents/Home/bin/java -Dspring.profiles.active=flow -jar target/loader.jar
docker run -e JAVA_OPTS="-Dspring.data.mongodb.uri=mongodb://localhost:27017/boomerang -Dflow.mongo.collection.prefix=flow -Dspring.profiles.active=flow" --network host --platform linux/amd64 boomerangio/flow-loader:latest