Merge pull request #315 from jeffngo/merge_3.0.0_master

Merge 'release/3.0.0' into master

.bumpversion.cfg

@@ -1,5 +1,5 @@
 [bumpversion]
-current_version = 2.1.0
+current_version = 3.0.0
 commit = False
 tag = False
 parse = (?P<major>\d+)\.(?P<minor>\d+)\.(?P<patch>\d+)(\.(?P<release>[a-z]+)(?P<dev>\d+))?

.dependabot/config.yaml

@@ -3,21 +3,21 @@ update_configs:
  # Keep requirements.txt files up-to-date in each package.
  - package_manager: "python"
  directory: "/dvp"
- update_schedule: "daily"
+ update_schedule: "monthly"
  target_branch: "develop"
  - package_manager: "python"
  directory: "/common"
- update_schedule: "daily"
+ update_schedule: "monthly"
  target_branch: "develop"
  - package_manager: "python"
  directory: "/platform"
- update_schedule: "daily"
+ update_schedule: "monthly"
  target_branch: "develop"
  - package_manager: "python"
  directory: "/libs"
- update_schedule: "daily"
+ update_schedule: "monthly"
  target_branch: "develop"
  - package_manager: "python"
  directory: "/tools"
- update_schedule: "daily"
+ update_schedule: "monthly"
  target_branch: "develop"

.github/CODEOWNERS

@@ -1 +1 @@
-* @ankursarin @fdrozdowski @nhlien93 @crystalplumage
+* @ankursarin @nhlien93 @mothslaw

.github/workflows/publish-docs.yml

@@ -33,8 +33,9 @@ jobs:
  run: |
  CURRENT_BRANCH_VAR=${GITHUB_REF#refs/heads/}
  echo "Current branch: $CURRENT_BRANCH_VAR"
- # Set CURRENT_BRANCH environment variable to the current branch name.
- echo "::set-env name=CURRENT_BRANCH::$(echo $CURRENT_BRANCH_VAR)"
+ # Set CURRENT_BRANCH environment variable to the current branch name. Refrain from using 'set-env' as
+ # GitHub has identified the command as a moderate security vulnerability.
+ echo "CURRENT_BRANCH=$(echo $CURRENT_BRANCH_VAR)" >> $GITHUB_ENV
  - name: Display all remote branches
  working-directory: ${{ matrix.package }}
  run: |
@@ -46,8 +47,9 @@ jobs:
  # Get only docs branches, extract the "docs/x.y.z" part, sort them in descending order, and get the first one.
  LATEST_DOCS_BRANCH_VAR=$(git branch -r | grep -e ".*\/*docs\/[0-9].[0-9].[0-9]" | sed -n "s/.*\/*\(docs\/[0-9].[0-9].[0-9]\).*/\1/p" | sort -r | head -n 1)
  echo "Latest docs branch: $LATEST_DOCS_BRANCH_VAR"
- # Set the LATEST_DOCS_BRANCH environment variable.
- echo "::set-env name=LATEST_DOCS_BRANCH::$(echo $LATEST_DOCS_BRANCH_VAR)"
+ # Set the LATEST_DOCS_BRANCH environment variable. Refrain from using 'set-env' as GitHub has identified the
+ # command as a moderate security vulnerability.
+ echo "LATEST_DOCS_BRANCH=$(echo $LATEST_DOCS_BRANCH_VAR)" >> $GITHUB_ENV
  - name: Check that the current branch is the latest docs branch, fail otherwise
  working-directory: ${{ matrix.package }}
  run: |

README-dev.md

@@ -125,17 +125,17 @@ To run blackbox tests, follow these steps:
 2. Navigate to the app-gate directory and start tests using `git blackbox`. For the guide on which test suite to use,
 see the next sections.
 
-At a minimum, each pull request should pass `appdata_python_samples` and `appdata_sanity` tests with a direct or staged plugin.
+At a minimum, each pull request should pass `appdata_python_samples` and `appdata_basic` tests with a direct or staged plugin.
 See the section below for the description of each test suite.
 
 #### Blackbox tests targeting wrappers (mostly Delphix Engine workflows)
 * appdata_python_samples (sample plugins from the app-gate):
 `git blackbox -s appdata_python_samples --extra-params="-p virt-sdk-repo=https://github.com/<username>/virtualization-sdk.git -p virt-sdk-branch=my-feature"`,
-* appdata_sanity with a direct Python plugin on CentOS 7.3: `git blackbox -s appdata_sanity -c APPDATA_PYTHON_DIRECT_CENTOS73 -a --extra-params="-p virt-sdk-repo=https://github.com/<username>/virtualization-sdk.git -p virt-sdk-branch=my-feature"`,
-* appdata_sanity with a staged Python plugin on CentOS 7.3: `git blackbox -s appdata_sanity -c APPDATA_PYTHON_STAGED_CENTOS73 -a --extra-params="-p virt-sdk-repo=https://github.com/<username>/virtualization-sdk.git -p virt-sdk-branch=my-feature"`.
+* appdata_basic with a direct Python plugin on CentOS 7.3: `git blackbox -s appdata_basic -c APPDATA_PYTHON_DIRECT_CENTOS73 -a --extra-params="-p virt-sdk-repo=https://github.com/<username>/virtualization-sdk.git -p virt-sdk-branch=my-feature"`,
+* appdata_basic with a staged Python plugin on CentOS 7.3: `git blackbox -s appdata_basic -c APPDATA_PYTHON_STAGED_CENTOS73 -a --extra-params="-p virt-sdk-repo=https://github.com/<username>/virtualization-sdk.git -p virt-sdk-branch=my-feature"`.
 
 #### Blackbox tests targeting the CLI (~80% CLI tests)
 * virtualization_sdk (installs and tests a direct Python plugin on Ubuntu 18): 
 `git blackbox -s virtualization_sdk -c APPDATA_SDK_UBUNTU18_DIRECT_CENTOS73 --extra-params="-p virt-sdk-repo=https://github.com/<username>/virtualization-sdk.git -p virt-sdk-branch=my-feature"`,
 * virtualization_sdk (installs and tests a staged Python plugin on Ubuntu 18): 
-`git blackbox -s virtualization_sdk -c APPDATA_SDK_UBUNTU18_STAGED_CENTOS73 --extra-params="-p virt-sdk-repo=https://github.com/<username>/virtualization-sdk.git -p virt-sdk-branch=my-feature"`.
+`git blackbox -s virtualization_sdk -c APPDATA_SDK_UBUNTU18_STAGED_CENTOS73 --extra-params="-p virt-sdk-repo=https://github.com/<username>/virtualization-sdk.git -p virt-sdk-branch=my-feature"`.

common/setup.py

@@ -4,7 +4,7 @@
 PYTHON_SRC = 'src/main/python'
 
 install_requires = [
- "dvp-api == 1.3.0",
+ "dvp-api == 1.4.0",
 ]
 
 with open(os.path.join(PYTHON_SRC, 'dlpx/virtualization/common/VERSION')) as version_file:

common/src/main/python/dlpx/virtualization/common/VERSION

@@ -1 +1 @@
-2.1.0
+3.0.0

docs/docs/Best_Practices/.pages

@@ -6,3 +6,4 @@ arrange:
  - Sensitive_Data.md
  - Unicode_Data.md
  - Working_with_Powershell.md
+ - Scratch_Paths.md

docs/docs/Best_Practices/Scratch_Paths.md

@@ -0,0 +1,19 @@
+#Scratch Paths
+
+A scratch path is a directory reserved for plugin use on each remote host. This is intended for uses such as:
+
+- Storage of small amounts of persistent data
+- A place to mount VDB data
+- Temporary logs for debugging (Be careful that you don't use too much space though!)
+
+The location of this scratch area is given by the `scratch_path` property on the [RemoteHost](/References/Classes/#remotehost) object.
+
+
+Things to note about the scratch path:
+
+- No guarantees are made about where the path is located on the system.
+- No guarantees are made about how much space might be available in this directory. It is strongly advised that you use only a small amount of disk space here.
+- The directory will be owned by the "primary user" associated with the remote host. This might be a completely different user from the one that is associated with a particular dsource or VDB.
+- If you need to store dSource- or VDB-specific data, it is highly recommended that you create a separate subdirectory for each dSource/VDB inside this scratch area. It's also recommended to name this subdirectory using the GUID of the dSource/VDB, so that you avoid accidental name collisions.
+- The Delphix Engine will not do any cleanup for you, so be sure to delete anything you're no longer using. For example, any VDB-specific information must be deleted in your [unconfigure](/References/Plugin_Operations/#virtual-source-unconfigure) operation (and dSource data gets deleted in your [stopStaging](/References/Plugin_Operations/#staged-linked-source-stop-staging) operation.)
+- Do not store any [sensitive information](Sensitive_Data.md) here!

docs/docs/Building_Your_First_Plugin/Data_Ingestion.md

@@ -21,7 +21,7 @@ quite limiting.
 
 For our first plugin, we will be using the more flexible [staging](/References/Glossary.md#staged-linkingsyncing) strategy. With this strategy, the Delphix Engine uses NFS for Unix environments (or iSCSI on Windows environments) to mount storage onto a [staging environment](/References/Glossary.md#staging-environment). Our plugin will then be in full control of how to get data from the source environment onto this storage mount.
 
-With the staging strategy, there are two types of syncs: sync and resync. A `sync` is used to ingestion incremental changes while a `resync` is used to re-ingest all the data for the dSource. For databases, this could mean re-ingesting from a full database backup to reset the dSource. A `sync` and a `resync` execute the same plugin operations and are differentiated by a boolean flag in the [snapshot_parameters](/References/Classes.md#snapshotparametersdefinition) argument passed into [linked.pre_snapshot](/References/Plugin_Operations.md#staged-linked-source-pre-snapshot) and [linked.post_snapshot](/References/Plugin_Operations.md#staged-linked-source-post-snapshot).
+With the staging strategy, there are two types of syncs: sync and resync. A `sync` is used to ingest incremental changes while a `resync` is used to re-ingest all the data for the dSource. For databases, this could mean re-ingesting from a full database backup to reset the dSource. A `sync` and a `resync` will execute the same plugin operations. To differentiate a `sync` from a `resync`, simply add a boolean property (i.e. `resync`) in the plugin's [snapshot parameters definition](References/Schemas_and_Autogenerated_Classes.md#snapshotparametersdefinition-schema). Once `sync` or `resync` is selected, the property will be passed into [linked.pre_snapshot](/References/Plugin_Operations.md#staged-linked-source-pre-snapshot) and [linked.post_snapshot](/References/Plugin_Operations.md#staged-linked-source-post-snapshot) as a [snapshot parameter](/References/Glossary.md#snapshot-parameters).
 
 A regular `sync` is the default and is executed as part of policy driven syncs. A `resync` is only executed during initial ingestion or if the Delphix user manually starts one. The customer can manually trigger a `resync` via the UI by selecting the dSource, going to more options and selecting **Resynchronize dSource**. ![Screenshot](images/Resync.png)
 
@@ -168,7 +168,7 @@ Next, we'll add a new function:
 
 ```python
 @plugin.linked.pre_snapshot()
-def copy_data_from_source(staged_source, repository, source_config, snapshot_parameters):
+def copy_data_from_source(staged_source, repository, source_config, optional_snapshot_parameters):
  stage_mount_path = staged_source.mount.mount_path
  data_location = "{}@{}:{}".format(staged_source.parameters.username,
  staged_source.parameters.source_address,
@@ -249,7 +249,7 @@ In fact, the default implementation that was generated by `dvp init` will work j
 def linked_post_snapshot(staged_source,
  repository,
  source_config,
- snapshot_parameters):
+ optional_snapshot_parameters):
  return SnapshotDefinition()
 ```
 

docs/docs/References/Classes.md

@@ -144,29 +144,6 @@ Field | Type | Description
 mounts | list[[Mount](#mount)] | The list of mounts to export the data sets to.
 ownership_specification | [OwnershipSpecification](#ownershipspecification) | **Optional.** Control the ownership attributes for the data set. It defaults to the environment user of the remote environment if it is not specified.
 
-## SnapshotParametersDefinition
-
-User provided parameters for the snapshot operation. It includes a boolean property named `resync` that can be used to indicate to the plugin whether or not to initiate a full ingestion of the dSource. The parameters are only set during a manual snapshot. When using a sync policy, `resync` defaults to `false`.
-
-```python
-from dlpx.virtualization.platform import Plugin
-
-plugin = Plugin()
-
-@plugin.linked.pre_snapshot()
-def linked_pre_snapshot(staged_source, repository, source_config, snapshot_parameters):
- if snapshot_parameter.resync:
- print(snapshot_parameter.resync)
-```
-
-> This class will be generated during build and is located with the autogenerated classes. As it is passed into the operation, importing it is not neccessary.
-
-### Fields
-
-Field | Type | Description
------ | ---- | -----------
-resync | Boolean | Determines if this snapshot should ingest the dSource from scratch.
-
 ## RemoteEnvironment
 
 Represents a remote environment.
@@ -203,7 +180,8 @@ Field | Type | Description
 name | String | Host address.
 reference | String | Unique identifier for the host.
 binary_path | String | Path to Delphix provided binaries on the host, which are present in the toolkit pushed to the remote host like `dlpx_db_exec`, `dlpx_pfexec`, etc. This property is only available for Unix hosts.
-scratch_path | String | Path to scratch path on the host.
+scratch_path | String | Path to scratch area on the host. See details [here](/Best_Practices/Scratch_Paths.md).
+
 
 ## RemoteUser
 
@@ -220,4 +198,4 @@ user = RemoteUser(name, reference)
 Field | Type | Description
 ----- | ---- | -----------
 name | String | User name.
-reference | String | Unique identifier for the user.
+reference | String | Unique identifier for the user.

docs/docs/References/Decorators.md

@@ -13,13 +13,13 @@ plugin = Plugin()
 # Use the decorator to annotate the function that corresponds to the "Virtual Source Start" Plugin Operation
 @plugin.virtual_source.start()
 def my_start(virtual_source, repository, source_config):
- print "running start" 
+ print "running start"
 ```
 
 !!! info
  Decorators exposed by the Virtualization SDK are inherently python function calls and needs parentheses `()` appended at the end.
 
-Assuming the name of the object, is `plugin` as above, the table below lists the corresponding decorators for each plugin operation.
+Assuming the name of the object is `plugin` as above, the table below lists the corresponding decorators for each plugin operation.
 
 Plugin Operation | Decorator
 ---------------- | --------
@@ -35,6 +35,7 @@ Plugin Operation | Decorator
 [Staged Linked Source Worker](Plugin_Operations.md#staged-linked-source-worker) | `@plugin.linked.worker()`
 [Staged Linked Source Mount Specification](Plugin_Operations.md#staged-linked-source-mount-specification) | `@plugin.linked.mount_specification()`
 [Virtual Source Configure](Plugin_Operations.md#virtual-source-configure) | `@plugin.virtual.configure()`
+[Virtual Source Initialize](Plugin_Operations.md#virtual-source-initialize) | `@plugin.virtual.initialize()`
 [Virtual Source Unconfigure](Plugin_Operations.md#virtual-source-unconfigure) | `@plugin.virtual.unconfigure()`
 [Virtual Source Reconfigure](Plugin_Operations.md#virtual-source-reconfigure) | `@plugin.virtual.reconfigure()`
 [Virtual Source Start](Plugin_Operations.md#virtual-source-start) | `@plugin.virtual.start()`

docs/docs/References/Glossary.md

@@ -28,6 +28,13 @@ The process by which the Delphix Engine learns about how a particular environmen
 ## dSource
 See [Linked Dataset](#linked-dataset)
 
+## Empty VDB
+A VDB that is created from scratch, without provisioning from another dataset. Users can create empty VDBs when they want to construct a brand-new dataset from within Delphix, instead of creating it externally and then ingesting it.
+
+This "empty" VDB, of course, will typically not stay empty for long. Data will be added as users work with the new dataset.
+
+A plugin can support this functionality by implementing the [initialize](Plugin_Operations.md#virtual-source-initialize) operation.
+
 ## Environment
 A remote system that the Delphix Engine can interact with. An environment can be used as a [source](#source-environment), [staging](#staging-environment) or [target](#target-environment) environment (or any combination of those). For example, a Linux machine that the Delphix Engine can connect to is an environment.
 
@@ -81,10 +88,10 @@ Information that represents a set of dependencies that a dataset requires in ord
 A formal description of a data type. Plugins use JSON format for their [schemas](Schemas_and_Autogenerated_Classes.md#schemas-and-autogenerated-classes).
 
 ## Snapshot
-A point-in-time read-only copy of a dataset. A snapshot includes associated metadata represented by the [SnapshotDefinition Schema](Schemas_and_Autogenerated_Classes.md#snapshotdefinition)
+A point-in-time read-only copy of a dataset. A snapshot includes associated metadata represented by the [SnapshotDefinition Schema](Schemas_and_Autogenerated_Classes.md#snapshotdefinition).
 
-## Snapshot Parameter
-User provided parameters for the snapshot operation. Currently the only properties the parameter has is resync.
+## Snapshot Parameters
+User provided parameters for the snapshot operation which can be defined in a [Snapshot Parameters Definition](Schemas_and_Autogenerated_Classes.md#snapshotparametersdefinition).
 
 ## Source Config
 A collection of information that the Delphix Engine needs to interact with a dataset (whether [linked](#linked-dataset) or [virtual](#virtual-dataset) on an [environment](#environment).
@@ -105,7 +112,7 @@ The process by which the Delphix Engine ingests data from a dataset on a [source
 An [environment](#environment) on which Delphix-provided virtualized datasets can be used.
 
 ## Lua Toolkit
-Legacy model for writing "plugins" in Lua, with limited documentation and support for writing, building and uploading toolkits. This was the predecessor to the Virtualization SDK. 
+Legacy model for writing "plugins" in Lua, with limited documentation and limited support for writing, building and uploading toolkits. This was the predecessor to the Virtualization SDK.
 
 ## Upgrade Operation
 A special plugin operation that takes data produced by an older version of a plugin, and transforms it into the format expected by the new version of the plugin.

docs/docs/References/Plugin_Config.md

@@ -20,6 +20,7 @@ The name of the file can be specified during the build. By default, the build lo
 |language|Y|enum|Must be `PYTHON27`.|
 |defaultLocale|N|enum|The locale to be used by the plugin if the Delphix user does not specify one. Plugin messages will be displayed in this locale by default. The default value is `en-us`.|
 |rootSquashEnabled|N|boolean|This dictates whether "root squash" is enabled on NFS mounts for the plugin (i.e. whether the `root` user on remote hosts has access to the NFS mounts). Setting this to `false` allows processes usually run as `root`, like Docker daemons, access to the NFS mounts. The default value is `true`. This field only applies to Unix hosts.|
+|extendedStartStopHooks|N|boolean|This controls whether the user's pre-start and post-start hooks will run during enable operations (and, likewise, whether pre-stop and post-stop hooks will run during disable operations). The default value is `false`.|
 
 ## Example
 Assume the following basic plugin structure: