Docs update (#2919)

* Doc 'Creation of SampleSet metadata item via GUI' was added
* Doc 'Environments synchronization' was added
* 'Enironments sync' doc was added to RN0.17
* (Issues #3020, #3028) Docs added
* (Issue #3080) Billing docs and RN update
* (Issue #3015) Push notifications docs were added

docs/md/installation/change_kube_certificates.md

@@ -25,10 +25,11 @@ systemctl start kubelet
 
 There is a script to backup old one and generate new one certificates for kube cluster
 In order to change certificates:
+
  - Copy this script to some file on kube master node
- - Replace placeholders for <KUBE_IP_HOST> <KUBE_API_HOSTNAME>
+ - Replace placeholders for `<KUBE_IP_HOST>`, `<KUBE_API_HOSTNAME>`
  - Increase DAYS_CERT if needed (365 by default)
- - chmod +x this file
+ - `chmod +x` this file
  - Run this script, it will save old certificates in `./kubernetes-bk` and generates and print to the console new one
  - You will need manually replace `client-certificate-data` in several files in according to script's output
  - Reboot kube master node

docs/md/installation/management/environments_sync.md

@@ -0,0 +1,100 @@
+# Environments synchronization
+
+In some cases, you may need to synchronize two different environments of the Cloud Pipeline.  
+Special routine in the [`pipectl`](https://github.com/epam/cloud-pipeline/tree/develop/deploy/README.md) utility shall be used for that - `pipectl sync`.
+
+It allows to synchronize from the source environment to the destination one the following objects:
+
+- users / user groups / user roles
+- docker registry / tool groups / tools
+
+Synchronization can be performed with or without synchronization of attributes (metadata) for the specified Platform objects.
+
+During the synchronization, changes are being performed only in the **_destination_** environment, the **_source_** environment remains the same.
+
+## Prerequisites
+
+For the correct work, launch `pipectl sync` utility only at Linux machine, in the separate terminal, not from the run.
+
+If you are going to synchronize tools between environments, Docker shall be installed at the machine as well.
+
+## Users synchronization process
+
+The format of the command to synchronize users between environments:
+
+`./pipectl sync --users --source-url <source_url> --source-token <source_token> --target-url <target_url> --target-token <target_token>`
+
+**_Note_**: the command above shall be performed with admin rights in the the same directory where `pipectl` utility was downloaded.
+
+Details:
+
+- `<source_url>` is the _source_ environment address. Shall be specified in the following format: `<protocol>`://`<IP/hostname>`:`<port>` (e.g. `https://cloud-pipeline.epam.com:443`)
+- `<source_token>` is the _source_ environment's access token. It can be obtained via the GUI (at the "Pipe CLI" settings [form](../../manual/12_Manage_Settings/12._Manage_Settings.md#cli-tab)) or via [`pipe token`](../../manual/14_CLI/14.1._Install_and_setup_CLI.md#using-pipe-token-command) CLI command
+- `<target_url>` is the _destination_ environment address. Shall be specified in the following format: `<protocol>`://`<IP/hostname>`:`<port>` (e.g. `https://cloud-pipeline-dev.epam.com:443`)
+- `<target_token>` is the _destination_ environment's access token. It can be obtained via the GUI (at the "Pipe CLI" settings [form](../../manual/12_Manage_Settings/12._Manage_Settings.md#cli-tab)) or via [`pipe token`](../../manual/14_CLI/14.1._Install_and_setup_CLI.md#using-pipe-token-command) CLI command
+
+During the `--users` synchronization, the following processes are being performed:
+
+- **Roles/groups synchronization**
+    - roles/groups from the _source_ environment, that are missing in the _destination_ environment, are being created in the _destination_ environment ("blocked" statuses are being set according to the _source_ environment as well)
+    - for roles/groups from the _destination_ environment matching roles/groups from the _source_ environment, their blocked status are being overridden:
+        - if a group/role is blocked in the _source_ environment - that status will be set for the group/role in the _destination_ environment too
+        - if a group/role is not blocked in the _source_ environment but blocked in the _destination_ environment - that status will be removed in the _destination_ environment
+- **Users synchronization**
+    - users from the _source_ environment, that are missing in the _destination_ environment, are being created in the _destination_ environment ("blocked" statuses, groups/roles lists are being set according to the _source_ environment)
+    - for users from the _destination_ environment matching users from the _source_ environment, their blocked statuses are being overridden:
+        - if a user is blocked in the _source_ environment - that status is being set for the user in the _destination_ environment too
+        - if a user is not blocked in the _source_ environment but blocked in the _destination_ environment - that status will be removed in the _destination_ environment
+    - for users from the _destination_ environment matching users from the _source_ environment, their groups/roles lists are being merged with the corresponding lists of the _source_ environment
+- **Metadata synchronization**
+    - by default, metadata (attributes) for users/groups/roles are being created/merged too. For users/groups/roles from the _destination_ environment matching users/groups/roles from the _source_ environment, if metadata keys for the _source_ and _destination_ are the same - the values of such metadata will be overridden by the _source_ environment values
+    - to exclude overriding of the matching metadata, the specific environment variable shall be set **before** the running of the `pipectl` utility - **`CP_SYNC_USERS_METADATA_SKIP_KEYS`**:
+        - for that variable, the comma-separated list of users' metadata keys to skip during the synchronization shall be specified
+        - example of usage: `export CP_SYNC_USERS_METADATA_SKIP_KEYS="ssh_pub,ssh_prv"` - to exclude public and private keys merging during the synchronization
+
+Command example:  
+    ![CP_EnvironmentsSynchronization](attachments/EnvironmentsSynchronization_01.png)
+
+Example of the output (part):  
+    ![CP_EnvironmentsSynchronization](attachments/EnvironmentsSynchronization_02.png)
+
+## Tools synchronization process
+
+The format of the command to synchronize tools between environments:
+
+`./pipectl sync --tools --source-url <source_url> --source-token <source_token> --target-url <target_url> --target-token <target_token>`
+
+**_Note_**: the command above shall be performed with admin rights in the the same directory where `pipectl` utility was downloaded.
+
+Details:
+
+- `<source_url>` is the _source_ environment address. Shall be specified in the following format: `<protocol>`://`<IP/hostname>`:`<port>` (e.g. `https://cloud-pipeline.epam.com:443`)
+- `<source_token>` is the _source_ environment's access token. It could be gotten via the GUI (at the "Pipe CLI" settings [form](../../manual/12_Manage_Settings/12._Manage_Settings.md#cli-tab)) or via [`pipe token`](../../manual/14_CLI/14.1._Install_and_setup_CLI.md#using-pipe-token-command) CLI command
+- `<target_url>` is the _destination_ environment address. Shall be specified in the following format: `<protocol>`://`<IP/hostname>`:`<port>` (e.g. `https://cloud-pipeline-dev.epam.com:443`)
+- `<target_token>` is the _destination_ environment's access token. It could be gotten via the GUI (at the "Pipe CLI" settings [form](../../manual/12_Manage_Settings/12._Manage_Settings.md#cli-tab)) or via [`pipe token`](../../manual/14_CLI/14.1._Install_and_setup_CLI.md#using-pipe-token-command) CLI command
+
+During the `--tools` synchronization, the following processes are being performed:
+
+- If the default registry is not attached to the _destination_ environment - this will be performed
+- **Tools groups' synchronization**
+    - if a group doesn't exist in the _destination_ environment registry - a new group will be created with all the properties of a _source_ one
+    - for tool groups from the _destination_ environment matching tool groups from the _source_ environment, their permissions/OWNER property will be overridden according to corresponding _source_ tool groups' settings (in case, when these settings are different)
+- **Tools'synchronization**
+    - tools from the _source_ environment, that are missing in the _destination_ environment, will be transferred to the _destination_ environment registry (with their versions, settings, descriptions according to the _source_ environment)
+    - for tools from the _destination_ environment matching tools from the _source_ environment:
+        - the same identical versions (that exist in both environments) remain in the _destination_
+        - versions from the _destination_ not matching the _source_ ones will be removed
+        - versions from the _source_ not matching the _destination_ will be transferred to the _destination_ environment registry
+        - settings and descriptions will be set according to the _source_ environment
+- **"Symlinked" tools' synchronization**
+- **Metadata synchronization**
+    - by default, metadata (attributes) for tools/tool groups/registry are being created/merged too. For tools/tool groups/registry from the _destination_ environment matching tools/tool groups/registry from the _source_ environment, if metadata keys for the _source_ and _destination_ are the same - the values of such metadata will be overridden according to the _source_ environment values
+    - to exclude overriding of the matching metadata, the specific environment variable shall be set **before** the running of the `pipectl` utility - **`CP_SYNC_TOOLS_METADATA_SKIP_KEYS`**:
+        - for that variable, the comma-separated list of tools' metadata keys to skip during the synchronization shall be specified
+        - example of usage: `export CP_SYNC_TOOLS_METADATA_SKIP_KEYS="ke1,key2"` - to exclude merging of "**_key1_**" and "**_key2_**" metadata keys during the synchronization
+
+Command example:  
+    ![CP_EnvironmentsSynchronization](attachments/EnvironmentsSynchronization_03.png)
+
+Example of the output (part):  
+    ![CP_EnvironmentsSynchronization](attachments/EnvironmentsSynchronization_04.png)

docs/md/installation/prerequisites/aws.md

@@ -438,4 +438,4 @@ The following policy shall be attached to the key:
         }
     ]
 }
-```
+```

docs/md/installation/prerequisites/common.md

@@ -15,21 +15,21 @@ Please refer to the corresponding section for the details:
 The following DNS entries are considered to access the Cloud Pipeline GUI and route the requests to the other services.
 
 * Cloud Pipeline GUI/API
-  * Record type: `A`
-  * Record value: `<Cloud Pipeline Core Instance IP>`
-  * Example: cloud-pipeline.epam.com
+    * Record type: `A`
+    * Record value: `<Cloud Pipeline Core Instance IP>`
+    * Example: cloud-pipeline.epam.com
 * Embedded git access for the "well-established" pipelines
-  * Record type: `CNAME`
-  * Record value: `<Cloud Pipeline GUI/API>`
-  * Example: git.cloud-pipeline.epam.com
+    * Record type: `CNAME`
+    * Record value: `<Cloud Pipeline GUI/API>`
+    * Example: git.cloud-pipeline.epam.com
 * Access to the docker registry for pull/push operation
-  * Record type: `CNAME`
-  * Record value: `<Cloud Pipeline GUI/API>`
-  * Example: docker.cloud-pipeline.epam.com
+    * Record type: `CNAME`
+    * Record value: `<Cloud Pipeline GUI/API>`
+    * Example: docker.cloud-pipeline.epam.com
 * Access to the "interactive" services, controlled by the Cloud Pipeline
-  * Record type: `CNAME`
-  * Record value: `<Cloud Pipeline GUI/API>`
-  * Example: edge.cloud-pipeline.epam.com
+    * Record type: `CNAME`
+    * Record value: `<Cloud Pipeline GUI/API>`
+    * Example: edge.cloud-pipeline.epam.com
 
 ## SSL/TLS certificates
 
@@ -40,20 +40,20 @@ SSL/TLS certificate shall be issued using an available CA (shall be trusted by t
 The following IdP configuration is required:
 
 * Cloud Pipeline GUI
-  * Service Provider URL: https://`<Cloud Pipeline GUI/API>`/pipeline
-  * Service Provider ACS URL: https://`<Cloud Pipeline GUI/API>`/pipeline/saml
-  * SAML Binding: `HTTP Redirect`
-  * Assertion information:
-    * NameID
-    * Email
-    * FirstName
-    * LastName
+    * Service Provider URL: https://`<Cloud Pipeline GUI/API>`/pipeline
+    * Service Provider ACS URL: https://`<Cloud Pipeline GUI/API>`/pipeline/saml
+    * SAML Binding: `HTTP Redirect`
+    * Assertion information:
+        * NameID
+        * Email
+        * FirstName
+        * LastName
 * Cloud Pipeline Git
-  * Service Provider URL: https://`<Embedded git>`
-  * Service Provider ACS URL: https://`<Embedded git>`/users/auth/saml/callback
-  * SAML Binding: `HTTP POST`
-  * Assertion information:
-    * NameID
-    * Email
-    * FirstName
-    * LastName
+    * Service Provider URL: https://`<Embedded git>`
+    * Service Provider ACS URL: https://`<Embedded git>`/users/auth/saml/callback
+    * SAML Binding: `HTTP POST`
+    * Assertion information:
+        * NameID
+        * Email
+        * FirstName
+        * LastName

docs/md/installation/prerequisites/gcp.md

@@ -21,38 +21,37 @@ Note: for small deployments or POCs - a single small routable subnet is enough
 ## Firewall rules
 
 * CP-Cluster-Internal:
-  * Targets: ALL
-  * Ports:
-    * TCP: 0-65535
-    * UDP: 0-65535
-    * ICMP
-  * IP Ranges: `VPC Subnets CIDR`
-  * Type: Ingress
-
+    * Targets: ALL
+    * Ports:
+        * TCP: 0-65535
+        * UDP: 0-65535
+        * ICMP
+    * IP Ranges: `VPC Subnets CIDR`
+    * Type: Ingress
 * CP-HTTPS-Access:
-  * Targets: ALL
-  * Ports:
-    * TCP: 443
-  * IP Ranges: `Internal (on-prem) networks or 0.0.0.0 (for the Public IPs usage)`
-  * Type: Ingress
-
+    * Targets: ALL
+    * Ports:
+        * TCP: 443
+    * IP Ranges: `Internal (on-prem) networks or 0.0.0.0 (for the Public IPs usage)`
+    * Type: Ingress
 * CP-Internet-Access:
-  * Targets: ALL
-  * Ports:
-    * TCP: 3128
-  * IP Ranges: `Egress HTTP proxy, if applicable`
-  * Type: Egress
+    * Targets: ALL
+    * Ports:
+        * TCP: 3128
+    * IP Ranges: `Egress HTTP proxy, if applicable`
+    * Type: Egress
 
 ## IAM
 
 The following service accounts shall be created:
+
 * `cp-service`
-  * Description: This account is used by the Cloud Pipeline to communicate to the GCP API (create VMs, manage data, etc.)
-  * Roles:
-    * `Compute Admin`
-    * `Service Account Token Creator`
-    * `Storage Admin`
+    * Description: This account is used by the Cloud Pipeline to communicate to the GCP API (create VMs, manage data, etc.)
+    * Roles:
+        * `Compute Admin`
+        * `Service Account Token Creator`
+        * `Storage Admin`
 * `cp-storage`
-  * Description: This account is used by the end-users to communicate to the GCS. Users are not granted access to the account directly, instead - temporary tokens are generated to perform CLI/GUI operations
-  * Roles:
-    * `Storage Object Admin`
+    * Description: This account is used by the end-users to communicate to the GCS. Users are not granted access to the account directly, instead - temporary tokens are generated to perform CLI/GUI operations
+    * Roles:
+        * `Storage Object Admin`

docs/md/installation/support_windows_runs.md

@@ -17,25 +17,23 @@ Base AMI which has to be used for new AMIs building is _ami-041bf3c49db945dbb_.
 Similar AMI in different regions can be found by this [name](https://aws.amazon.com/marketplace/pp/Amazon-Web-Services-Microsoft-Windows-Server-2019-/B07R3BJL99). 
 The following steps help to build new version of Cloud Pipeline Windows AMI.
 
-1. Launch an instance using _ami-041bf3c49db945dbb_ instance image, _m5.large_ instance type, proper _subnet_, _100 gb_ disk and proper _security groups_.
-Subnet and security groups should be the same as in Cloud Pipeline deployment itself.
-
-2. Once instance is ready generate password using default private key and connect using RDP to this machine with Administrator login and generated password.
-
-3. Once connected open powershell console and manually execute contents of _deploy/infra/aws/install-common-win-node.ps1_ script excluding a single `New-EC2Tag` call. 
+1) Launch an instance using _ami-041bf3c49db945dbb_ instance image, _m5.large_ instance type, proper _subnet_, _100 gb_ disk and proper _security groups_.
+Subnet and security groups should be the same as in Cloud Pipeline deployment itself.  
+2) Once instance is ready generate password using default private key and connect using RDP to this machine with Administrator login and generated password.  
+3) Once connected open powershell console and manually execute contents of _deploy/infra/aws/install-common-win-node.ps1_ script excluding a single `New-EC2Tag` call. 
 At this point any changes can be performed on the machine. Notice that before reboot the following command has to be executed otherwise created AMI won't work.
 
 ```powershell
 C:\ProgramData\Amazon\EC2-Windows\Launch\Scripts\InitializeInstance.ps1 -Schedule
 ```
 
-4. Create a new ami from the running instance.
+4) Create a new ami from the running instance.
 
 ## Build Windows tool
 
-The following steps help to build and push a new version of Cloud Pipeline Windows tool.
+The following steps help to build and push a new version of Cloud Pipeline Windows tool:
 
-1. Launch an instance using ami-041bf3c49db945dbb instance image or launch Windows run in Cloud Pipeline, connect to the instance via RDP or NoMachine, build Window docker image using _deploy/docker/cp-tools/base/windows/Dockerfile_ and push it to Cloud Pipeline private docker registry using the commands below.
+Launch an instance using ami-041bf3c49db945dbb instance image or launch Windows run in Cloud Pipeline, connect to the instance via RDP or NoMachine, build Window docker image using _deploy/docker/cp-tools/base/windows/Dockerfile_ and push it to Cloud Pipeline private docker registry using the commands below.  
 Please use the actual Cloud Pipeline deployment ip.
 
 ```powershell
@@ -68,7 +66,7 @@ docker push "$registry/library/windows:latest"
 
 To configure Windows runs support for brand-new Cloud Pipeline deployments the following actions have to be performed.
 
-1. Update cluster.networks.config system preference by adding platform field to all amis and adding a new Windows ami.
+1) Update cluster.networks.config system preference by adding platform field to all amis and adding a new Windows ami.
 
 _Before_
 
@@ -113,13 +111,13 @@ _After_
 }
 ```
 
-2. Configure CP_REPO_ENABLED=false run parameter for Windows tool.
+2) Configure CP_REPO_ENABLED=false run parameter for Windows tool.
 
 ### Configure existing deployment
 
 To configure Windows runs support for existing Cloud Pipeline deployments the following actions have to be performed.
 
-1. Two additional parameters have to be added to Kubernetes global config map.
+1) Two additional parameters have to be added to Kubernetes global config map.
 
 ```bash
 CP_KUBE_KUBEADM_CERT_HASH="$(openssl x509 -in /etc/kubernetes/pki/ca.crt -noout -pubkey | openssl rsa -pubin -outform DER 2>/dev/null | sha256sum | cut -d' ' -f1)"
@@ -131,7 +129,7 @@ CP_KUBE_NODE_TOKEN="$(kubectl --namespace=kube-system describe sa canal \
   | base64 --decode)"
 ```
 
-2. Update cluster.networks.config system preference by adding platform field to all amis and adding a new Windows ami.
+2) Update cluster.networks.config system preference by adding platform field to all amis and adding a new Windows ami.
 
 _Before_
 
@@ -176,22 +174,21 @@ _After_
 }
 ```
 
-3. Several existing application properties have to be updated.
+3) Several existing application properties have to be updated.
+
 ```
 launch.script.url --renamed-to--> launch.script.url.linux
 api.security.public.urls <--added-another-public-url-- launch.py
 ```
 
-4. Several new application properties have to be added.
+4) Several new application properties have to be added.
 
 ```properties
 launch.script.url.windows=https://${CP_API_SRV_INTERNAL_HOST}:${CP_API_SRV_INTERNAL_PORT}/pipeline/launch.py
 kube.kubeadm.cert.hash=${CP_KUBE_KUBEADM_CERT_HASH}
 kube.node.token=${CP_KUBE_NODE_TOKEN}
 ```
 
-5. Kubernetes canal config map has to be updated and all canal pods have to be restarted.
-
-6. All edge pods have to be restarted.
-
-7. Configure _CP_REPO_ENABLED=false_ run parameter for Windows tool.
+5) Kubernetes canal config map has to be updated and all canal pods have to be restarted.  
+6) All edge pods have to be restarted.  
+7) Configure _CP_REPO_ENABLED=false_ run parameter for Windows tool.