error: runtime error: invalid memory address or nil pointer dereference

[wise-east]

Version

0.15.0

Description

After installing cortex with $ bash -c "$(curl -sS https://raw.githubusercontent.com/cortexlabs/cortex/0.15/get-cli.sh)" and running cortex cluster up, I get the following error message after providing the aws region I would like to use:

error: runtime error: invalid memory address or nil pointer dereference

I could not figure out why, so I removed this installation and went to the docs and used version 0.14 and the same exact commands seem to work fine.

Configuration

I'll get the message after providing the region if I don't provide a config file but if I do, I'll get the error message right away.

Steps to reproduce

1. $ bash -c "$(curl -sS https://raw.githubusercontent.com/cortexlabs/cortex/0.15/get-cli.sh
2. $ cortex cluster up
3. Provide aws region and return

Expected behavior

Move on to ask configuration for s3 bucket, etc.

Actual behavior

Screenshots

Stack traces

N/A

[deliahu]

@wise-east Thank you for reporting this!

I believe this is an issue with a new IAM check we added for 0.15. Do you mind trying two things:

1. What does aws iam get-user show for you (using the same credentials you are using for the cluster)? There shouldn't be sensitive information in there, but feel free to obscure anything if you'd like.

2. I have built a slightly modified version of the CLI, which will print stack traces for panics (this change #921 has been merged into master, but not yet released). I also added extra debug statements to try to see what is going on. Do you mind trying this CLI and sending the output?

# make a temporary directory for the test
mkdir cli-test
cd cli-test

# download the modified CLI

# on Mac:
curl -o cortex-test https://cortex-cli-david.s3-us-west-2.amazonaws.com/0.15.0/cli/darwin/cortex
# on Linux:
curl -o cortex-test https://cortex-cli-david.s3-us-west-2.amazonaws.com/0.15.0/cli/linux/cortex

chmod +x cortex-test

# run the command that was breaking for you
./cortex-test cluster up -c <path to cluster.yaml>

# ctrl+c if it asks "do you want to continue", but it will probably crash before then

# clean up
cd ..
rm -rf cli-test

Feel free to email me at david@cortex.dev if you prefer, I'd also be happy to jump on a call if you'd like to go through it together, since this is on the critical path and I'd like to get it fixed as soon as possible.

Thanks again for reporting this!

[wise-east]

My pleasure.

1. Here's my output for aws iam get-user

{
    "User": {
        "UserId": "493676065566",
        "Arn": "arn:aws:iam::493676065566:root",
        "CreateDate": "2018-06-07T08:09:37Z",
        "PasswordLastUsed": "2020-04-01T17:59:10Z"
    }
}

2. Here's the output I get for the commands you've asked me to run.

(interface {}) <nil>
---------------------
(*iam.GetUserOutput)(({
  User: {
    Arn: "arn:aws:iam::493676065566:root",
    CreateDate: 2018-06-07 08:09:37 +0000 UTC,
    PasswordLastUsed: 2020-04-01 17:59:10 +0000 UTC,
    UserId: "493676065566"
  }
}) {
  _: (struct {}) {
  },
  User: (*iam.User)(({
  Arn: "arn:aws:iam::493676065566:root",
  CreateDate: 2018-06-07 08:09:37 +0000 UTC,
  PasswordLastUsed: 2020-04-01 17:59:10 +0000 UTC,
  UserId: "493676065566"
}) {
    _: (struct {}) {
    },
    Arn: (*string)((len=30) "arn:aws:iam::493676065566:root"),
    CreateDate: (*time.Time)((2018-06-07 08:09:37 +0000 UTC) {
      wall: (uint64) 0,
      ext: (int64) 63663955777,
      loc: (*time.Location)(<nil>)
    }),
    PasswordLastUsed: (*time.Time)((2020-04-01 17:59:10 +0000 UTC) {
      wall: (uint64) 0,
      ext: (int64) 63721360750,
      loc: (*time.Location)(<nil>)
    }),
    Path: (*string)(<nil>),
    PermissionsBoundary: (*iam.AttachedPermissionsBoundary)(<nil>),
    Tags: ([]*iam.Tag) <nil>,
    UserId: (*string)((len=12) "493676065566"),
    UserName: (*string)(<nil>)
  })
})
=====================
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
	panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x1b5bf0b]

goroutine 1 [running]:
github.com/cortexlabs/cortex/pkg/lib/exit.Panic(0x2e88a80, 0xc00007f680, 0x0, 0x0, 0x0)
	/home/ubuntu/src/github.com/cortexlabs/cortex/pkg/lib/exit/exit.go:60 +0x1c0
github.com/cortexlabs/cortex/pkg/lib/exit.RecoverAndExit(0x0, 0x0, 0x0)
	/home/ubuntu/src/github.com/cortexlabs/cortex/pkg/lib/exit/exit.go:66 +0x9d
panic(0x27f3be0, 0x3e06950)
	/usr/local/go/src/runtime/panic.go:967 +0x166
github.com/cortexlabs/cortex/pkg/lib/aws.(*Client).IsAdmin(0xc00030a070, 0x15)
	/home/ubuntu/src/github.com/cortexlabs/cortex/pkg/lib/aws/iam.go:104 +0x7b
github.com/cortexlabs/cortex/cli/cmd.promptIfNotAdmin(0xc00030a070)
	/home/ubuntu/src/github.com/cortexlabs/cortex/cli/cmd/lib_aws_creds.go:60 +0x59
github.com/cortexlabs/cortex/cli/cmd.getInstallClusterConfig(0xc0000b6f80, 0x14, 0xc0000ac420, 0x28, 0xc0000b6f80, 0x14, 0xc0000ac420, 0x28, 0xc0000ac420, 0x28, ...)
	/home/ubuntu/src/github.com/cortexlabs/cortex/cli/cmd/lib_cluster_config.go:143 +0x10c
github.com/cortexlabs/cortex/cli/cmd.glob..func1(0x3e18540, 0x3e55938, 0x0, 0x0)
	/home/ubuntu/src/github.com/cortexlabs/cortex/cli/cmd/cluster.go:89 +0x10c
github.com/spf13/cobra.(*Command).execute(0x3e18540, 0x3e55938, 0x0, 0x0, 0x3e18540, 0x3e55938)
	/home/ubuntu/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:830 +0x29d
github.com/spf13/cobra.(*Command).ExecuteC(0x3e1a0c0, 0xc00065ff40, 0x1, 0x1)
	/home/ubuntu/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:914 +0x2fb
github.com/spf13/cobra.(*Command).Execute(...)
	/home/ubuntu/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:864
github.com/cortexlabs/cortex/cli/cmd.Execute()
	/home/ubuntu/src/github.com/cortexlabs/cortex/cli/cmd/root.go:132 +0x300
main.main()
	/home/ubuntu/src/github.com/cortexlabs/cortex/cli/main.go:24 +0x20

Hope this helps!

[deliahu]

Yes this is super helpful, thank you! It explains why I was unable to reproduce it: I am using an IAM user, and this issue only happens with the root user.

I will work on a fix tonight, and we'll try to have it released tomorrow. In the mean time you can continue using 0.14, or you can create an IAM user (give it the "AdministratorAccess" policy) and use that instead of your root user.

[deliahu]

I just pushed the fix; v0.15.1 is building now, and we plan on testing and releasing it tomorrow.

Thanks again for the heads up!