diff --git a/docs/assets/images/hopsworksai/aws/connect-aws-profile.png b/docs/assets/images/hopsworksai/aws/connect-aws-profile.png new file mode 100644 index 0000000000..78adce2dd6 Binary files /dev/null and b/docs/assets/images/hopsworksai/aws/connect-aws-profile.png differ diff --git a/docs/assets/images/hopsworksai/aws/connect-aws-review.png b/docs/assets/images/hopsworksai/aws/connect-aws-review.png new file mode 100644 index 0000000000..9ab270ee4a Binary files /dev/null and b/docs/assets/images/hopsworksai/aws/connect-aws-review.png differ diff --git a/docs/assets/images/hopsworksai/aws/connect-aws-ssh.png b/docs/assets/images/hopsworksai/aws/connect-aws-ssh.png new file mode 100644 index 0000000000..b3bb4c29af Binary files /dev/null and b/docs/assets/images/hopsworksai/aws/connect-aws-ssh.png differ diff --git a/docs/assets/images/hopsworksai/aws/create-instance-general.png b/docs/assets/images/hopsworksai/aws/create-instance-general.png new file mode 100644 index 0000000000..650352e0a7 Binary files /dev/null and b/docs/assets/images/hopsworksai/aws/create-instance-general.png differ diff --git a/docs/assets/images/hopsworksai/aws/create-key-pair-2.png b/docs/assets/images/hopsworksai/aws/create-key-pair-2.png new file mode 100644 index 0000000000..7b9f16e706 Binary files /dev/null and b/docs/assets/images/hopsworksai/aws/create-key-pair-2.png differ diff --git a/docs/assets/images/hopsworksai/aws/create-key-pair.png b/docs/assets/images/hopsworksai/aws/create-key-pair.png new file mode 100644 index 0000000000..6b7861a08b Binary files /dev/null and b/docs/assets/images/hopsworksai/aws/create-key-pair.png differ diff --git a/docs/assets/images/hopsworksai/aws/create-s3-bucket-1.png b/docs/assets/images/hopsworksai/aws/create-s3-bucket-1.png new file mode 100644 index 0000000000..8e331f66d0 Binary files /dev/null and b/docs/assets/images/hopsworksai/aws/create-s3-bucket-1.png differ diff --git a/docs/assets/images/hopsworksai/aws/create-s3-bucket-2.png b/docs/assets/images/hopsworksai/aws/create-s3-bucket-2.png new file mode 100644 index 0000000000..16383c9afb Binary files /dev/null and b/docs/assets/images/hopsworksai/aws/create-s3-bucket-2.png differ diff --git a/docs/assets/images/hopsworksai/aws/import-key-pair-2.png b/docs/assets/images/hopsworksai/aws/import-key-pair-2.png new file mode 100644 index 0000000000..4e3a387851 Binary files /dev/null and b/docs/assets/images/hopsworksai/aws/import-key-pair-2.png differ diff --git a/docs/assets/images/hopsworksai/aws/import-key-pair.png b/docs/assets/images/hopsworksai/aws/import-key-pair.png new file mode 100644 index 0000000000..a9bbfb8bcf Binary files /dev/null and b/docs/assets/images/hopsworksai/aws/import-key-pair.png differ diff --git a/docs/hopsworksai/aws/databricks_quickstart.md b/docs/hopsworksai/aws/databricks_quickstart.md index 9ec7133192..76d97fac8a 100644 --- a/docs/hopsworksai/aws/databricks_quickstart.md +++ b/docs/hopsworksai/aws/databricks_quickstart.md @@ -30,7 +30,6 @@ As region, choose the region you use for Databricks: :align: center :figclass: align-center -{!hopsworksai/aws/s3_permissions.md!} Select the VPC with a name starting with *databricks-*: diff --git a/docs/hopsworksai/aws/getting_started.md b/docs/hopsworksai/aws/getting_started.md index 1e4153843b..c77850a9b0 100644 --- a/docs/hopsworksai/aws/getting_started.md +++ b/docs/hopsworksai/aws/getting_started.md @@ -1,13 +1,13 @@ # Getting started with Hopsworks.ai (AWS) Hopsworks.ai is our managed platform for running Hopsworks and the Feature Store -in the cloud. It integrates seamlessly with third party platforms such as Databricks, +in the cloud. It integrates seamlessly with third-party platforms such as Databricks, SageMaker and KubeFlow. This guide shows how to set up Hopsworks.ai with your organization's AWS account. ## Step 1: Connecting your AWS account Hopsworks.ai deploys Hopsworks clusters to your AWS account. To enable this you have to -give us permission to do so. This can be either achieved by using AWS cross-account roles or +permit us to do so. This can be either achieved by using AWS cross-account roles or AWS access keys. We strongly recommend the usage of cross-account roles whenever possible due to security reasons. @@ -110,7 +110,7 @@ Copy the *Role ARN* from the summary of your cross-account role:

-Paste the *Role ARN* into Hopsworks.ai and select *Configure*: +Paste the *Role ARN* into Hopsworks.ai and click on *Finish*:

@@ -201,7 +201,7 @@ Copy the *Access Key ID* and the *Secret Access Key*:

-Paste the *Access Key ID* and the *Secret Access Key* into Hopsworks.ai and select *Configure*: +Paste the *Access Key ID* and the *Secret Access Key* into Hopsworks.ai and click on *Finish*:

@@ -212,7 +212,87 @@ Paste the *Access Key ID* and the *Secret Access Key* into Hopsworks.ai and sele

-## Step 2: Deploying a Hopsworks cluster +## Step 2: Creating and configuring a storage + +The Hopsworks clusters deployed by hopsworks.ai store their data in an S3 bucket in your AWS account. +To enable this you need to create an S3 bucket and an instance profile to give cluster nodes access to the bucket. + +### Step 2.1: Creating an S3 bucket +Proceed to the [S3 Management Console](https://s3.console.aws.amazon.com/s3/home) and click on *Create bucket*: +

+

+ + Create an S3 bucket + +
Create an S3 bucket
+
+

+ +Name your bucket and select the region where your Hopsworks cluster will run. Click on *Create bucket* at the bottom of the page. + +

+

+ + Create an S3 bucket + +
Create an S3 bucket
+
+

+ +### Step 2.2: Creating an instance profile and giving it access to the bucket + +Follow the instructions in this guide to create an IAM instance profile with access to your S3 bucket: [Guide](https://docs.aws.amazon.com/codedeploy/latest/userguide/getting-started-create-iam-instance-profile.html) + +When creating the policy, paste the following in the JSON tab. +{!hopsworksai/aws/s3_permissions.md!} + +## Step 3: Create an SSH key +When deploying clusters, Hopsworks.ai installs an ssh key on the cluster's instances so that you can access them if necessary. For this purpose, you need to add an ssh key to your AWS EC2 environment. This can be done in two ways: [creating a new key pair](#step-31-create-a-new-key-pair) or [importing an existing key pair](#step-32-import-a-key-pair). + +### Step 3.1: Create a new key pair + +Proceed to [Key pairs in the EC2 console](https://us-east-2.console.aws.amazon.com/ec2/v2/home?#KeyPairs) and click on *Create key pair* +

+

+ + Create a key pair + +
Create a key pair
+
+

+ +Name your key, select the file format you prefer and click on *Create key pair*. +

+

+ + Create a key pair + +
Create a key pair
+
+

+ +### Step 3.2: Import a key pair +Proceed to [Key pairs in the EC2 console](https://us-east-2.console.aws.amazon.com/ec2/v2/home?#KeyPairs), click on *Action* and click on *Import key pair* +

+

+ + Import a key pair + +
Import a key pair
+
+

+ +Name your key pair, upload your public key and click on *Import key pair*. +

+

+ + Import a key pair + +
Import a key pair
+
+

+ +## Step 4: Deploying a Hopsworks cluster In Hopsworks.ai, select *Create cluster*: @@ -225,15 +305,66 @@ In Hopsworks.ai, select *Create cluster*:

-Configure the cluster by selecting the location, instance type and optionally the VPC, -subnet and security group. Select *Deploy*. +Select the *Region* in which you want your cluster to run (1), name your cluster (2). -!!! note "SSH key configuration" - We recommend that you always configure a SSH key under advanced options to ensure you can troubleshoot the cluster if necessary. +Select the *Instance type* (3) and *Local storage* (4) size for the cluster *Head node*. -{!hopsworksai/aws/s3_permissions.md!} +Select the number of workers you want to start the cluster with (5). +Select the *Instance type* (6) and *Local storage* size (7) for the *worker nodes*. + +!!! note + It is possible to add or remove workers once the cluster is running. + +Enter the name of the *S3 bucket* (8) you created above in *S3 bucket*. + +!!! note + The S3 bucket you are using must be empty. -The cluster will start. This might take a couple of minutes: +Press *Next* (9): + +

+

+ + Create a Hopsworks cluster, general Information + +
Create a Hopsworks cluster, general information
+
+

+ +Select the *SSH key* that you want to use to access cluster instances: + +

+

+ + Choose SSH key + +
Choose SSH key
+
+

+ +Select the *Instance Profile* that you created above and click on *Review and Submit*: + +

+

+ + Choose the instance profile + +
Choose the instance profile
+
+

+ +Review all information and select *Create*: + +

+

+ + Review cluster information + +
Review cluster information
+
+

+ +The cluster will start. This will take a few minutes:

@@ -244,8 +375,8 @@ The cluster will start. This might take a couple of minutes:

-As soon as the cluster has started, you will be able to log in to your new Hopsworks cluster with the username -and password provided. You will also able to stop or terminate the cluster. +As soon as the cluster has started, you will be able to log in to your new Hopsworks cluster with the username and password provided. You will also be able to stop, restart, or terminate the cluster. +

@@ -256,12 +387,12 @@ and password provided. You will also able to stop or terminate the cluster.

-## Step 3: Outside Access to the Feature Store +## Step 5: Outside Access to the Feature Store By default, only the Hopsworks UI is made available to clients on external networks, like the Internet. To integrate with external platforms and access APIs for services such as the Feature Store, you have to open the service's ports. -Open ports by going to *Services* tab, selecting a service and pressing *Update*. This will update the *Security Group* attached to the Hopsworks cluster to allow incoming traffic on the relevant ports. +Open ports by going to the *Services* tab, selecting a service, and pressing *Update*. This will update the *Security Group* attached to the Hopsworks cluster to allow incoming traffic on the relevant ports.

@@ -272,7 +403,7 @@ Open ports by going to *Services* tab, selecting a service and pressing *Update*

-## Step 4: Next steps +## Step 6: Next steps Check out our other guides for how to get started with Hopsworks and the Feature Store: diff --git a/docs/hopsworksai/aws/s3_permissions.md b/docs/hopsworksai/aws/s3_permissions.md index 0ce0e17705..bdd6a3e3f0 100644 --- a/docs/hopsworksai/aws/s3_permissions.md +++ b/docs/hopsworksai/aws/s3_permissions.md @@ -1,41 +1,26 @@ -If you select a S3 bucket then HopsFS will store all the files in the S3 bucket. Do not forget to set -appropriate instance profile so that the cluster instances can access the selected bucket. - -

-

- - Configuring Instance Profile for Hopsworks cluster - -
Configuring Instance Profile for Hopsworks cluster
-
-

- -Following is an example of an instance profile needed by HopsFS to store the file system blocks in a S3 bucket. - +Replace *BUCKET_NAME* with appropriate S3 bucket name. ```json { - "Version": "2012-10-17", - "Statement": [ - { - "Sid": "HopsFSS3Permissions", - "Effect": "Allow", - "Action": [ - "S3:PutObject", - "S3:ListBucket", - "S3:GetBucketLocation", - "S3:GetObject", - "S3:DeleteObject", - "S3:AbortMultipartUpload", - "S3:ListBucketMultipartUploads" - ], - "Resource": [ - "arn:aws:s3:::bucket.name/*", - "arn:aws:s3:::bucket.name" - ] - } + "Version": "2012-10-17", + "Statement": [ + { + "Sid": "HopsFSS3Permissions", + "Effect": "Allow", + "Action": [ + "S3:PutObject", + "S3:ListBucket", + "S3:GetBucketLocation", + "S3:GetObject", + "S3:DeleteObject", + "S3:AbortMultipartUpload", + "S3:ListBucketMultipartUploads" + ], + "Resource": [ + "arn:aws:s3:::BUCKET_NAME/*", + "arn:aws:s3:::BUCKET_NAME" ] - } + } + ] +} ``` - -Replace *bucket.name* with appropriate S3 bucket name. diff --git a/docs/hopsworksai/aws/sagemaker_quickstart.md b/docs/hopsworksai/aws/sagemaker_quickstart.md index 79fc136f69..3d364f9ec2 100644 --- a/docs/hopsworksai/aws/sagemaker_quickstart.md +++ b/docs/hopsworksai/aws/sagemaker_quickstart.md @@ -28,7 +28,6 @@ Select the region of your SageMaker instance as region and create the cluster. :align: center :figclass: align-center -.. include:: ../../../getting_started/hopsworksai/guides/s3_permissions.rst By default, only the Hopsworks UI is exposed (made available to clients on external networks, like the Internet) by your Hopsworks cluster. To reach the Feature Store from SageMaker, you have diff --git a/docs/hopsworksai/azure/getting_started.md b/docs/hopsworksai/azure/getting_started.md index d7e22c3457..9c3c120e20 100644 --- a/docs/hopsworksai/azure/getting_started.md +++ b/docs/hopsworksai/azure/getting_started.md @@ -440,7 +440,7 @@ The cluster will start. This will take a few minutes:

-As soon as the cluster has started, you will be able to log in to your new Hopsworks cluster with the username and password provided. You will also able to stop, restart or terminate the cluster. +As soon as the cluster has started, you will be able to log in to your new Hopsworks cluster with the username and password provided. You will also be able to stop, restart or terminate the cluster.