Skip to content

Latest commit

 

History

History
132 lines (93 loc) · 16.6 KB

README.md

File metadata and controls

132 lines (93 loc) · 16.6 KB
Raw

MATLAB Parallel Server on Amazon Web Services (Windows VM)

Step 1. Launch the Template

Click the Launch Stack button for your desired region below to deploy the cloud resources on AWS. This will open the AWS console in your web browser.

Region Launch Link
us-east-1 alt text
us-east-2 alt text
us-west-1 alt text
us-west-2 alt text
ca-central-1 alt text
eu-central-1 alt text
eu-west-1 alt text
eu-west-2 alt text
eu-west-3 alt text
eu-north-1 alt text
sa-east-1 alt text
me-south-1 alt text
ap-east-1 alt text
ap-south-1 alt text
ap-northeast-1 alt text
ap-northeast-2 alt text
ap-southeast-1 alt text
ap-southeast-2 alt text

Step 2. Configure the Cloud Resources

After you click the Launch Stack button above, the “Create stack” page will open in your browser where you can configure the parameters. It is easier to complete the steps if you position these instructions and the AWS console window side by side.

  1. Specify a stack name. This will be shown in the AWS CloudFormation console and must be unique within the AWS account.

  2. Specify and check the defaults for these resource parameters:

Parameter label Description
VPC to deploy this stack to ID of an existing VPC in which to deploy this stack
Subnets for the head node and worker nodes List of existing subnets IDs for the head node and workers
CIDR IP address range of client IP address range that will be allowed to connect to this cluster from outside of the VPC. This field should be formatted as <ip_address>/<mask>. E.g. 10.0.0.1/32. This is the public IP address which can be found by searching for 'what is my ip address' on the web. The mask determines the number of IP addresses to include. A mask of 32 is a single IP address. This calculator can be used to build a specific range: https://www.ipaddressguide.com/cidr. You may need to contact your IT administrator to determine which address is appropriate.
RDP Key Pair Name of an existing EC2 KeyPair to allow RDP access to all the instances. See https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-key-pairs.html for details on creating these.
Cluster name Name to use for this cluster. This name will be shown in MATLAB as the cluster profile name.
Instance type for the head node AWS instance type to use for the head node, which will run the job manager. No workers will be started on this node, so this can be a smaller instance type than the worker nodes. See https://aws.amazon.com/ec2/instance-types for a list of instance types. Must be available in the Availability Zone of the first subnet in the configured list
InstanceAmiCustom Custom Amazon Machine Image (AMI) in the target region
Storage size for the MJS database Size in GiB of the EBS volume storing the MJS database (all job and task information, including input and output data). Ensure that the volume has enough capacity to store the data.
Instance type for the worker nodes AWS instance type to use for the workers. See https://aws.amazon.com/ec2/instance-types for a list of instance types.
Number of worker nodes Number of AWS instances to start for the workers to run on.
Minimum number of worker nodes Minimum number of AWS instances that can run at any time.
Maximum number of worker nodes Maximum number of AWS instances that can run at any time.
Number of workers to start on each node Number of MATLAB workers to start on each instance. Specify 1 worker per physical core (1 worker for every 2 vCPU). For example an m4.16xlarge instance has 64 vCPUs, so can support 32 MATLAB workers. See https://aws.amazon.com/ec2/instance-types for details on vCPUs for each instance type.
License Manager for MATLAB connection string Optional License Manager for MATLAB, specified as a string in the form <port>@<hostname>. If not specified, use online licensing. If specified, the network license manager (NLM) must be accessible from the specified VPC and subnets. To use the private hostname of the NLM host instead of the public hostname, specify the security group of the NLM deployment in the AdditionalSecurityGroup parameter. For more information, see https://github.com/mathworks-ref-arch/license-manager-for-matlab-on-aws.
Additional security group to place instances in ID of an additional (optional) Security Group for the instances to be placed in. Often the License Manager for MATLAB's Security Group.
Enable instance autoscaling Flag indicating whether instance autoscaling is enabled. For more information about autoscaling, refer to the Use Autoscaling section in the deployment README.
  1. Tick the box to accept that the template uses IAM roles. These roles allow:
  • the instances to transfer the shared secret information between the nodes, via the S3 bucket, to establish SSL encrypted communications
  • the instances to write the cluster profile to the S3 bucket for secure access to the cluster from the client MATLAB
  • a custom lambda function to delete the contents of this S3 bucket when the stack is deleted
  1. Click the Create button.

When you click Create, the cluster is created using AWS CloudFormation templates.

Step 3: Connect to Your Cluster From MATLAB

  1. After clicking Create you will be taken to the Stack Detail page for your Stack. Wait for the Status to reach CREATE_COMPLETE. This may take up to 10 minutes; manually refreshing the page might be required to update the displayed Status.

  2. Select Outputs.

    Stack Outputs On Completion

  3. Click the link next to BucketURL under Outputs.

  4. Select the profile (ClusterName.mlsettings) and click Download. Manually refreshing the page might be required for the settings file to be shown; generating the file may take up to 10 minutes from when access to the bucket is available.

  5. Open MATLAB.

  6. In the Parallel drop-down menu in the MATLAB toolstrip select Create and Manage Clusters....

  7. Click Import.

  8. Select the downloaded profile and click open.

  9. Click Set as Default.

  10. (Optional) Validate your cluster by clicking the Validate button.

After setting the cloud cluster as default, the next time you run a parallel language command (such as parfor, spmd, parfeval or batch) MATLAB connects to the cluster. The first time you connect, you will be prompted for your MathWorks account login. The first time you run a task on a worker it will take several minutes for the worker MATLAB to start. This delay is due to initial loading of data from the EBS volumes. This is a one-time operation, and subsequent tasks begin much faster.

Your cluster is now ready to use. It will remain running after you close MATLAB. Delete your cluster by following the instructions below.

NOTE: Use the profile and client IP address range to control access to your cloud resources. Anyone with this file can connect to your resources from a machine within the specified IP address range and run jobs on it.

Additional Information

Port requirements for accessing MATLAB Parallel Server

To access a MATLAB Parallel Server cluster from your client MATLAB, your client machine must be able to communicate on specific ports. Make sure that the network firewall allows the following outgoing connections:

Required ports Description
TCP 27350 to 27358 + 4*N Ports 27350 to 27358 + 4*N, where N is the maximum number of workers on a single node
TCP 443 HTTPS access to (at least) *.mathworks and *.amazonaws.com
TCP 3389 RDP access to the cluster nodes for troubleshooting

Table 1: Outgoing port requirements

Delete Your Cloud Resources

You can remove the CloudFormation stack and all associated resources when you are done with them. Note that there is no undo. After you delete the cloud resources you cannot use the downloaded profile again.

  1. Select the Stack in the CloudFormation Stacks screen. Select Delete.

    CloudFormation Stacks Delete

  2. Confirm the delete when prompted. CloudFormation will now delete your resources which can take a few minutes.

Use Autoscaling

To optimize the number of Amazon EC2 instances running MATLAB workers, enable autoscaling by setting Enable instance autoscaling to Yes when you create the stack. Autoscaling is optional and is disabled by default.

When autoscaling is disabled, the AWS Auto Scaling group launches Number of worker nodes instances. To change the number of worker nodes, use the AWS Management Console.

When autoscaling is enabled, the desired capacity of the AWS Auto Scaling group is regulated by the number of workers needed by the cluster. The number of Amazon EC2 instances fluctuates between the Minimum and Maximum number of worker nodes. To change these limits after you create the stack, use the AWS Management Console. To change the amount of time for which idle nodes are preserved, adjust the value of the tag mwWorkerIdleTimeoutMinutes.

To disable autoscaling in a deployed stack, redeploy the stack with autoscaling disabled.

Troubleshooting

If your stack fails to create, check the events section of the CloudFormation console. It will indicate which resources caused the failure and why.

If the stack created successfully but you are unable to validate it, check the logs on the instances to diagnose the error. This can be achieved by logging into the instance nodes using remote desktop.

  1. Expand the Resources section in the Stack Detail page.
  2. Look for the Logical ID named Headnode and click the Physical ID, then select the Instance ID.
  3. Click the Connect button in the AWS console and select the RDP Client tab, and download the remote desktop file as prompted.
  4. Open the downloaded .rdp file, and In the login screen that's displayed, use the username Administrator. The associated password can be obtained from the Get password prompt on the same AWS console page that was used to download the remote desktop file.

The files of interest are those under C:\ProgramData\MJS\Log.