AWS DynamoDB setup

UXF 2.0 uses the data handler system, which means data can be handled in any number of ways, not just written to a file on your PC. One of these data handlers allows uploading data to an AWS DynamoDB database. This is only supported on the WebGL build target because it uses the AWS Javascript SDK.

You should read about DynamoDB here before you start. There will be some costs associated with collecting data in a database but for most studies it should be only a few dollars. See the costs section below.

Make sure you have the LATEST version of UXF added to your project before continuing. Earlier versions had some issues which have now been fixed.

Outline

To use this, you need to change the build target to WebGL. Then open UXF Wizard (UXF menu at top, Show setup Wizard) and ensure the provided WebGL template is used.

The Web AWS DynamoDB data handler can be enabled in the DataHandler tab of the Session component.

It works like the other data handlers - it automatically handles your data at points when it needs to be written. For example, at the end of the session, UXF takes your trial results, and in this case it will automatically upload them to the DynamoDB. No coding change is needed. UXF automatically creates the tables you need, and constructs the requests on your behalf. You can still manually access the database, too.

WebGL

This only works with WebGL builds (i.e. your experiment running in a web browser). That means any other packages you use must also be compatible with WebGL. You cannot use built-in VR support for example, but you can use WebXR.

There is a provided HTML template for WebGL to work properly. You can optionally create your own, but make sure it carries a reference to the AWS DynamoDB SDK:

<!-- AWS SDK required for DynamoDB -->
<script src="https://sdk.amazonaws.com/js/aws-sdk-2.776.0.min.js"></script>

Getting your credentials

First we need to set up our credentials, so that your application can securely write (and perhaps read) from the database. You'll need to use 3 AWS services.

DynamoDB, the database where data will be stored.
IAM (Identity and Access Management), which allows you to create permissions to allow reading/writing to the database.
Cognito, which allows you to create identites (groups of users or applications) with specific IAM permissions.

Using this method means you do not grant your application unchecked access to all of your AWS services.

Lets set up the services. We need to set them up all under one region. Use the one closest to you. You can select the region with the dropdown in the top right (I chose eu-west-2).

Steps:

In the AWS console, go to the Cognito service.
1. Click manage Identity Pools.
2. This may take you to automatically create an identity pool. If not, click Create new identity pool.
3. Enter a memorable name for the pool, e.g. UXF Participants
4. Enable access to unauthenticated identities - this allows uploading of data without users having to sign up.
5. Press create pool.
6. Expand the View details foldout, and note the Role Name of the unathenticated identity. Is is probably similar to: Cognito_UXFParticipantsUnauth_Role.
7. Press Allow.
8. The sample code in the "Get AWS Credentials" box here shows the credentials we will need later.
9. Copy the Identity Pool ID, without quotes. It will look like this: eu-west-2:2b0947a6-762c-4484-924f-e7cb7a7bc285.
10. You can get back to this page by selecting the identity you created in Cognito.
In the AWS console, go to Identity and Access Management (IAM).
1. Go to Policies and Create Policy.
2. In the visual editor, select DynamoDB as the service.
3. We now need to select the Actions for the policy.
  - Under Write, select BatchWriteItem, CreateTable, & PutItem.
  - Optionally, under Read, select GetItem. Leave this blank if you do not need to read from the database. You can also edit this policy later.
4. Under Resources, select All resources. (You could also set up conditional access to tables that start with the string UXF for extra security).
5. Press Review Policy, enter a name e.g. UXF_DynamoDB_Access, press create policy.
6. Now, go to Roles, and Create Role.
7. Select Web identity as the trusted entity and Amazon Cognito as the identity provider.
8. Enter the Identity Pool ID of the Cognito Identity you created earlier, e.g. eu-west-2:2b0947a6-762c-4484-924f-e7cb7a7bc285.
9. Here we search for and select the Policy (UXF_DynamoDB_Access) we created earlier.
10. Press Next, and Next again (no tags needed). Enter a name e.g. UXF_DynamoDB_Role.
11. Press Create Role.
Go back to the Cognito service.
1. Find the identity pool you created (UXF Participants), and press Edit identity pool in the top right.
2. In the Unauthenticated role, select the role we created previously UXF_DynamoDB_Role.
3. Save the changes.

Putting the credentials into your project

We need to now enter the credentials into our UXF project.

Go to your UXF prefab, under the session component, click Data Handlers. Untick the File Saver component. Enable (tick) the Web AWS DynamoDB data hander. Press configure.
Here you need to create a new credentials asset. Press the small create button, and save it in your Assets folder with some name e.g. MyCredentials.
It should now allow you to enter the two items needed: The region code, and the Identity Pool. The identity pool is the one you set up in Cognito - and you can find it again by clicking the identity in AWS Cognito. The region code is just the part before the :, but this should be entered separately.

Costs

AWS & DynamoDB have a free tier which should cover smaller experiments. Make sure to regularly visit your DynamoDB console and delete any unused tables. Tables created by UXF are automatically set to On-Demand mode which AWS recommends for unpredictable workloads. Unless you are running a very large testing operation the On-Demand mode looks far more cost efficent. You can use this tool to estimate costs. But if you collect data for <100 participants, with a normal amount of data per session (trial results, perhaps some movement data) you should only see a cost of a few dollars. It is up to you to manage the costs of your database.

When you have finished your data collection, you can download the tables as CSVs, delete the tables, and also delete the Cognito Identity Pool you created, which will stop any further data coming in.

Testing it out

Make sure your platform is WebGL in Build Settings. It will ask you to download the WebGL module if you haven't already.
You need to make sure the UXF WebGL template is selected under Player Settings, or it will not work.
Test your experiment with Build and run! Unfortunately the Web AWS DynamoDB Data Handler cannot communicate with the database whilst running in the Unity Editor, because it relies on the AWS JavaScript SDK. You can still test in the Unity Editor and run it normally, but data will not be uploaded. You could enable File Saver for testing in the editor.
Before building, you may need to change some UI settings, since some UI features are not supported in WebGL (e.g. settings json files)

Downloading data

Head to DynamoDB on the AWS console, click tables, and find the table you want to download data from. Press actions, and export to CSV. You will need to further process the CSVs to get data in a "raw" form, as it by default spits out key-value pairs. See this section on the Wiki.

Possible issues

DynamoDB has a limit of 400KB for a single entity. Therefore, if you are using Tracker components that capture (e.g.) movement data, the files entities can exceed 400KB if your trials are particularly long. Trials should be less than a few minutes- if they are longer, try to split them into multiple trials.

Manual DynamoDB access

UXF automatically handles the database calls. But sometimes you may want to manually access the database. You can reference the WebAWSDynamoDB component and use the following methods:

Put data into the database

The PutCustomDataInDB takes an item in the form of Dictionary<string, object>. Example:

var myItem = new Dictionary<string, object>() {
    { "col_name", 12345 },
    { "other_col_name", "hello" }
};

Put the item in the database.

webAWSDynamoDB.PutCustomDataInDB(tableName, myItem);

Getting items from the database

Get an item in the database that has been stored by UXF. The method happens asynchronously and so calls callback when it has finished. This method will automatically construct the table name and request for you.

webAWSDynamoDB.GetUXFDataFromDB(
    experimentName,
    dataType, 
    ppid, 
    sessionNum, 
    dataName, 
    callback, 
    optionalTrialNumber = 0
);

Get an item in the database that has not been stored by UXF. The method happens asynchronously and so calls callback when it has finished. This method will automatically construct the request for you.

webAWSDynamoDB.GetCustomDataFromDB(
    tableName, 
    primaryKeyName, 
    primaryKeyValue, 
    callback, 
    optionalSortKeyName = "", 
    optionalSortKeyValue = null
);

The two methods GetUXFDataFromDB and GetCustomDataFromDB call a callback function when retrieving the data. So you need to create a method that takes Dictionary<string, object> as argument, e.g.:

void HandleMyData(Dictionary<string, object> item)
{
    // do something with your item
    Debug.Log(item["my_column"]);
    Debug.Log(item["my_other_column"]);
}

Example usage, getting row 10 from a TrialResults data output, using GetUXFDataFromDB:

webAWSDynamoDB.GetUXFDataFromDB(
    "my_experiment",
    UXF.UXFDataType.TrialResults, 
    "participant_id_12345", 
    1, 
    "trial_results", 
    HandleMyData, 
    optionalTrialNumber = 10
);