Why does terraform give show "Error: Unauthorized" when deploying on AWS?

[emiliom]

I got very close to my first QHub cluster deployment (on AWS), but ran into a terraform error. I followed the steps in https://docs.qhub.dev/en/latest/source/03_tutorials_and_samples/1_project_setup_tutorial.html and https://docs.qhub.dev/en/latest/source/02_get_started/02_setup.html (with some necessary tweaks [1]). The qhub deploy process ended in error after about 15 minutes:

qhub deploy -c qhub-config.yaml --dns-provider cloudflare --dns-auto-provision

Here are the last several lines of the run-time messages:

[terraform]: Warning: Applied changes may be incomplete
[terraform]: 
[terraform]: The plan was created with the -target option in effect, so some changes
[terraform]: requested in the configuration may have been ignored and the output values may
[terraform]: not be fully updated. Run the following command to verify that no other
[terraform]: changes are pending:
[terraform]:     terraform plan
[terraform]: 	
[terraform]: Note that the -target option is not suitable for routine use, and is provided
[terraform]: only for exceptional situations such as recovering from errors or mistakes, or
[terraform]: when Terraform specifically suggests to use it as part of an error message.
[terraform]: 
[terraform]: 
[terraform]: Error: error waiting for EKS Node Group (qhubcluster-dev:user) creation: AccessDenied: Amazon EKS Nodegroups was unable to assume the service-linked role in your account. Resource IDs: []
[terraform]: 
[terraform]:   on .terraform/modules/kubernetes/modules/aws/kubernetes/main.tf line 18, in resource "aws_eks_node_group" "main":
[terraform]:   18: resource "aws_eks_node_group" "main" {
[terraform]: 
[terraform]: 

Afterwards I ran qhub destroy --config qhub-config.yaml to be safe. It looks like it really was destroying AWS resources that were in set up. The job ran for about 5 minutes, but also ended in error. Here are the last several lines:

...
[terraform]: module.kubernetes.aws_iam_policy.worker_autoscaling: Destruction complete after 32s
[terraform]: module.kubernetes.aws_iam_role.node-group: Destruction complete after 32s
[terraform]: 
[terraform]: Error: Unauthorized
[terraform]: 
[terraform]: 

Problem encountered: Terraform error

Thanks in advance. I can post complete logs if it helps. BTW, I wasn't sure where to post this. Traffic on both gitter and here stopped the last week of April, but I couldn't find concrete pointers to a more current communication platform other than a passing reference to Slack and an incomplete reference to gitter possibly being deprecated.

Update: I'm using qhub 0.3.11

[1] For the Auth0 domain, I had to append .us.auth0.com rather than .auth0.com as found in the QHub guide. I also needed to set up the AWS_DEFAULT_REGION environment variable.

[danlester]

Thank you very much for your report. GitHub Issues is the best place to post these for now.

One point of confusion is that the qhub.dev docs redirected to 'latest' until yesterday, and now redirects to 'stable' (so the docs that would be in line with the 0.3.11 release). So that would have told you to set AWS_DEFAULT_REGION, whereas in the 'latest' (in development) branch that is actually not needed.

I'm not too sure why Auth0 sometimes gives different subdomains. I have seen .eu.auth0.com whereas others do just get .auth0.com... I'll see if I can find out why, or at least update the docs to note this discrepancy.

None of this really addresses your issue... I would guess (as I'm sure you would too!) that somehow the AWS keys you've provided don't have permissions to do everything needed on EKS, or it could just be a timing issue where the permissions aren't ready.

I haven't come across this myself, but I'll ask around to see if anyone else has.

[emiliom]

I tried deploying it again (w/o double checking the AWS key), and this time it worked!! So, it must have been the permissions lag issue @danlester mentioned. Thanks!

One unexpected behavior is that the hub was deployed directly at the domain name specified in the qhub init command (--domain amazon-riverbgc.org), rather than at jupyter.amazon-riverbgc.org. I followed the docs to the t, so I'm puzzled. Any ideas? FYI, that domain name is brand new and prior to deploying qhub it wasn't pointing to any site.

[tylerpotts]

@emiliom Glad it deployed!

Also thanks for bringing this up. We recently removed the requirement for DNS names to start with jupyter.<domain>. I just created this PR to update the documention #631.

With the latest version of QHub, the domain name passed with the --domain no longer prepends the DNS with jupyter

[emiliom]

Thanks!

I've run into a 500 error when starting the server, but if I follow up I'll do it separately via a github issue.

[dharhas]

@emiliom 👋 good to see you :)

[emiliom]

Likewise, @dharhas. I've been tracking QHub since it was released and have been itching for the opportunity to try it.

[Adam-D-Lewis]

I'm closing old discussions in a repo cleanup effort. Feel free to re-open if needed.