Skip to content
This repository has been archived by the owner on Oct 3, 2023. It is now read-only.

docs(samples): the bash scripts for environment setup are added #392

Merged
merged 10 commits into from
Apr 7, 2022
140 changes: 91 additions & 49 deletions samples/interactive-tutorials/README.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
#Retail Search Interactive Tutorials
# Retail Search Interactive Tutorials

##Run tutorials in Cloud Shell
## Run tutorials in Cloud Shell

To advance with the interactive tutorials, use Retail Search step-by-step manuals on the right side of the Cloud Shell IDE:
![Interactive tutorials](images/tutorial1.png)
@@ -36,55 +36,61 @@ The Retail API provides you with the following possibilities to:

You can find the information about the Retail services in the [documentation](https://cloud.google.com/retail/docs)

<!--TODO(tkarasova) update the link to the tutorials when will be published-->
If you would like to have a closer look at the Retail API features and try them yourself,
the best option is to use the [Interactive Tutorials](https://cloud.google.com/retail/docs/overview). The tutorials will be launched in the CloudShell environment, and you will be able to request the Retail services and check the response with minimum time and effort.
the best option is to use the [Interactive Tutorials](https://cloud.google.com/retail/docs/overview). In the documentation chapters find the "Guide me" button, the tutorials will be launched in the CloudShell environment, and you will be able to request the Retail services and check the response with minimum time and effort.

The code samples in the directory **python-retail/samples/interactive-tutorials** are explicitly created for use with the Retail Interactive Tutorials.
The code samples in the directory **java-retail/samples/interactive-tutorials** are explicitly created for use with the Retail Interactive Tutorials.

If, for some reason, you have decided to proceed with these code samples without the tutorial, please go through the following steps and set up the required preconditions.

### Select your project and enable the Retail API
## Prepare your work environment

Google Cloud organizes resources into projects. This lets you
collect all the related resources for a single application in one place.
To prepare the work environment you should perform the following steps:
- Create a service account.
- Create a service account key and set it to authorize your calls to the Retail API.
- Install Google Cloud Retail library.

If you don't have a Google Cloud project yet or you're not the owner of an existing one, you can
[create a new project](https://console.cloud.google.com/projectcreate).
### There are two ways to set up your work environment:

- If you want to **speed up the process** of setting up the working environment, run the script java-retail/samples/interactive-tutorials/samples/interactive-tutorials/src/main/java/user_environment_setup.sh and skip the next **set up the work environment step-by-step** tutorial step:

After the project is created, set your PROJECT_ID to a ```project``` variable.
1. Run the following command in Terminal:
```bash
gcloud config set project <YOUR_PROJECT_ID>
bash java-retail/samples/interactive-tutorials/user_environment_setup.sh
```

1. To check that the Retail API is enabled for your Project, go to the [Admin Console](https://console.cloud.google.com/ai/retail/).
- If you want to perform the environment set up step by step along with getting the explanation you should proceed with the next tutorial step.

## Set up the work environment step-by-step

### Create service account

To access the Retail API, you must create a service account.
To access the Retail API, you must create a service account. Check that you are an owner of your Google Cloud project on the [IAM page](https://console.cloud.google.com/iam-admin/iam).

1. To create a service account, follow this [instruction](https://cloud.google.com/retail/docs/setting-up#service-account)
1. To create a service account, perform the following command:

1. Find your service account on the [IAM page](https://console.cloud.google.com/iam-admin/iam),
click `Edit` icon, add the 'Storage Admin' and 'BigQuery Admin' roles. It may take some time for changes to apply.
```bash
gcloud iam service-accounts create <YOUR_SERVICE_ACCOUNT_ID>
```

1. Copy the service account email in the Principal field.
1. Assign the needed roles to your service account:

### Set up authentication
```bash
for role in {retail.admin,editor,bigquery.admin}
do gcloud projects add-iam-policy-binding <YOUR_PROJECT_ID> --member="serviceAccount:<YOUR_SERVICE_ACCOUNT_ID>@<YOUR_PROJECT_ID>.iam.gserviceaccount.com" --role="roles/${role}"
done
```

To run a code sample from the Cloud Shell, you need to be authenticated using the service account credentials.
1. Use the following command to show the service account email:

1. Login with your user credentials.
```bash
gcloud auth login
gcloud iam service-accounts list|grep <YOUR_SERVICE_ACCOUNT_ID>
```

1. Type `Y` and press **Enter**. Click the link in a Terminal. A browser window should appear asking you to log in using your Gmail account.
Copy the service account email.

1. Provide the Google Auth Library with access to your credentials and paste the code from the browser to the Terminal.

1. Upload your service account key JSON file and use it to activate the service account:
1. Upload your service account key JSON file and use it to activate the service
account:

```bash
gcloud iam service-accounts keys create ~/key.json --iam-account <YOUR_SERVICE_ACCOUNT_EMAIL>
@@ -94,57 +100,93 @@ To run a code sample from the Cloud Shell, you need to be authenticated using th
gcloud auth activate-service-account --key-file ~/key.json
```

1. To request the Retail API, set your service account key JSON file as the GOOGLE_APPLICATION_CREDENTIALS environment variable :
1. Set the key as the GOOGLE_APPLICATION_CREDENTIALS environment variable to
use it for sending requests to the Retail API.

```bash
export GOOGLE_APPLICATION_CREDENTIALS=~/key.json
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Aren't we trying to de-emphasize GOOGLE_APPLICATION_CREDENTIALS in favor of gcloud auth application-default login ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

using GOOGLE_APPLICATION_CREDENTIALS to request the Retail servicei is a recommented way to do that.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Really? Most of the time we are providing you with credentials.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is discribed in the Retail documentation we are relying creating this tutorials https://cloud.google.com/retail/docs/setting-up#local-environment

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That isn't the way things are supposed to be these days.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you reach out to a TechWriter?

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

https://cloud.google.com/docs/authentication/production#automatically
From our previous experiences, gcloud CLI often results in using end user credentials instead of the service account credentials in some languages, and causing errors like "Your application has authenticated using end user credentials from the Google Cloud SDK or Google Cloud Shell which are not supported."

In this case, we prefer to have users explicitly set GOOGLE_APPLICATION_CREDENTIALS so that end user credentials won't be used.

```

### Set the GOOGLE_CLOUD_PROJECT environment variable
### Google Cloud Retail libraries

Learn more about the [Java Google Cloud Retail library](https://googleapis.dev/java/google-cloud-retail/latest/index.html).

## Congrats! You have configured your work environment

You will run the code samples in your own Google Cloud project. To use the **project_id** in every request to the Retail API, you should first specify them as environment variables.
1. Check that you are in the directory with code samples.

1. Find the project ID in the Project Info card displayed on **Home/Dashboard**.
The code samples for each of the Retail services are stored in different directories.

Go to the code samples directory, your starting point to run more commands.

1. Set the **project_id** with the following command:
```bash
export GOOGLE_CLOUD_PROJECT=<YOUR_PROJECT_ID>
cd java-retail/samples/interactive-tutorials/
```

## Import Catalog Data
## Import catalog data

This step is required if this is the first Retail API Tutorial you run.
Otherwise, you can skip it.
<i>This step is required if this is the first Retail API tutorial that you run.
Otherwise, you can skip it.</i>

### Upload catalog data to Cloud Storage
There is a java-retail/samples/interactive-tutorials/src/main/resources/products.json file with valid products prepared in the `resources` directory.

There is a JSON file with valid products prepared in the `product` directory:
`product/resources/products.json`.
The other file, java-retail/samples/interactive-tutorials/src/main/resources/products_some_invalid.json, contains both valid and invalid products. You will use it to check the error handling.

Another file, `product/resources/products_some_invalid.json`, contains both valid and invalid products, and you will use it to check the error handling.
- If you want to **speed up the process**, run the following script in the Terminal directory to import all products to catalog and skip the next **Prepare the catalog data step-by-step** tutorial step:

In your own project, create a Cloud Storage bucket and put the JSON file there.
```bash
bash java-retail/samples/interactive-tutorials/user_import_data_to_catalog.sh
```

- If you want to upload products to the catalog step by step along with getting the explanation, you should proceed with the next tutorial step.

## Prepare the catalog data step-by-step

### Upload catalog data to Cloud Storage

In your own project you need to create a Cloud Storage bucket and put the JSON file there.
The bucket name must be unique. For convenience, you can name it `<YOUR_PROJECT_ID>_<TIMESTAMP>`.

1. To create the bucket and upload the JSON file, run the following command in the Terminal:
<!--TODO update with the correct file path when will be merged-->
1. The code samples for each of the Retail services are stored in different directories.

Go to the code samples directory, your starting point to run more commands.

```bash
pmvn compile exec:java -Dexec.mainClass=CreateGcsBucket
cd java-retail/samples/interactive-tutorials
```

1. To create the bucket and upload the JSON file, open java-retail/samples/interactive-tutorials/src/main/java/product/setup/ProductsCreateGcsBucket.java file

1. Go to the **product** directory and run the following command in the Terminal:

```bash
mvn compile exec:java
-Dexec.mainClass=product.setup.ProductsCreateGcsBucket
```

Now you can see the bucket is created in the [Cloud Storage](https://console.cloud.google.com/storage/browser), and the files are uploaded.

1. The name of the created Retail Search bucket is printed in the Terminal. Copy the name and set it as the environment variable `BUCKET_NAME`:
1. The name of the created Cloud Storage bucket is shown in the Terminal.

```
The gcs bucket <YOUR_PROJECT_ID>_<TIMESTAMP> was created
```

Copy the name and set it as the environment variable `BUCKET_NAME`:

```bash
export BUCKET_NAME=<YOUR_BUCKET_NAME>
```

### Import products to the Retail Catalog

To import the prepared products to a catalog, run the following command in the Terminal:
<!--TODO update with the correct file path when will be merged-->
```bash
pmvn compile exec:java -Dexec.mainClass=ImportProductsGcs
```
To import the prepared products to a catalog, open java-retail/samples/interactive-tutorials/src/main/java/product/ImportProductsGcs.java file and run the following command in the Terminal:

```bash
mvn compile exec:java -Dexec.mainClass=product.ImportProductsGcs
```

## Your Retail catalog is ready to use!

### Running code samples

53 changes: 53 additions & 0 deletions samples/interactive-tutorials/user_environment_setup.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,53 @@
#!/bin/bash

# Copyright 2022 Google Inc. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

# set the Google Cloud Project ID
project_id=$1
echo "Project ID: $project_id"
gcloud config set project "$project_id"

timestamp=$(date +%s)

service_account_id="service-acc-$timestamp"
echo "Service Account: $service_account_id"

# create service account (your service-acc-$timestamp)
gcloud iam service-accounts create "$service_account_id"

# assign necessary roles to your new service account
for role in {retail.admin,editor,bigquery.admin}
do
gcloud projects add-iam-policy-binding "$project_id" --member="serviceAccount:$service_account_id@$project_id.iam.gserviceaccount.com" --role=roles/"${role}"
done

echo "Wait ~60 seconds to be sure the appropriate roles have been assigned to your service account"
sleep 60

# upload your service account key file
service_acc_email="$service_account_id@$project_id.iam.gserviceaccount.com"
gcloud iam service-accounts keys create ~/key.json --iam-account "$service_acc_email"

# activate the service account using the key
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, if your doing this, then why tell the user to provide a service account in the README?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the tutorials we give the user 2 options - eanther users want to setup the work env by themselfs, then we provide steps similar to those discribed in the README, or they want to speedup the process and start exploring the Retail features - for tha purpose we give them this scripts.

But you are right, it is good to tell users that they may use the scripts in the Readme as well

gcloud auth activate-service-account --key-file ~/key.json

# install needed Google client libraries
cd ~/cloudshell_open/java-retail/samples/interactive-tutorials || exit
mvn clean install -DskipTests

echo "========================================"
echo "The Google Cloud setup is completed."
echo "Please proceed with the Tutorial steps"
echo "========================================"
36 changes: 36 additions & 0 deletions samples/interactive-tutorials/user_import_data_to_catalog.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
#!/bin/bash

# Copyright 2022 Google Inc. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

# set the key as GOOGLE_APPLICATION_CREDENTIALS
export GOOGLE_APPLICATION_CREDENTIALS=~/key.json
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You do know that this will not be set once the script ends?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I know that, but we still need this variable to run the java code samples within the script


# Change the working directory
cd ~/cloudshell_open/java-retail/samples/interactive-tutorials/ || exit

# Run the sample for creating the GCS bucket and extract the output of that execution
output=$(mvn compile exec:java -Dexec.mainClass="product.setup.ProductsCreateGcsBucket")

# Get the bucket name and store it in the env variable BUCKET_NAME
temp="${output#*gcs bucket }"
bucket_name="${temp% was created*}"
export BUCKET_NAME=$bucket_name

# Import products to the Retail catalog
mvn compile exec:java -Dexec.mainClass="product.ImportProductsGcs"

echo "====================================="
echo "Your Retail catalog is ready to use!"
echo "====================================="