Add Oracle Cloud support

[lixuna]

Thank you for your time today, Jon and TJ.

Here are some CNCF CI Cross Cloud resources available to share with the Oracle Cloud team:

• README: https://github.com/cncf/cross-cloud/blob/master/README.md

• FAQ: https://github.com/cncf/cross-cloud/blob/master/FAQ.md

• Dashboard Design Mocks (v1.0 in progress):
  https://github.com/cncf/cross-cloud/blob/master/DASHBOARD.md

• Source Code: https://gitlab.cncf.ci/cncf/cross-cloud

• CNCF CI Public Mailing List:
  https://lists.cncf.io/mailman/listinfo/cncf-ci-public

Next steps:

• To gain access to CI jobs on the GitLab repo, please create an account
  on GitLab, visit https://gitlab.cncf.ci/cncf/cross-cloud and press
  "Request Access."

• To share credentials and example docs, let's coordinate on our email thread.

Thanks again,
Lucina

Lucina Stricko
Partner / Project Manager
vulk.coop

cc @denverwilliams

[lixuna]

Question for @denverwilliams:

The terraform provider for OCI is hosted out of the oracle github org: https://github.com/oracle/terraform-provider-oci

None of the other providers are in a separate repo.

Is it ok to hard code the version for this provider in the docker file?

[doberloh]

Another Question. What is the preferred public/private availability of the infrastructure? Do I need to have both public and private options?

[taylor]

Howdy, @doberloh.

Is it ok to hard code the version for this provider in the docker file?

Yes

What is the preferred public/private availability of the infrastructure? Do I need to have both public and private options?

No preference for us. It's really up to you for this.

For what is tested and shown on https://cncf.ci it can be either.

[doberloh]

Wonderful. Thanks for the response. Just back from 4th vacation so ill be on it this week.

[doberloh]

Another question @taylor i see that vsphere is using straight ignition configs, but all the other providers are not. What is the difference here. I have attempted both workflows (b64gzip ignition config from vspheres implementation and b64gzip straight config from the templates module similar to aws and azure) and it seems to not be working either way. I see the system pulling user data but then nothing happens and the server reboots. Metadata polls return what i expect based on what terraform sends in as well. Any pointers there would be awesome! :)

Second, coreOS has dropped support for oracle cloud in their recent builds. I have pulled the most recent version that works with OCI (currently the stable version of 1745.7.0) but cannot guarantee that anything will work past that as there is no official build with oracle support for the most recent versions.

[denverwilliams]

@doberloh
They both end up using cloud-init in the end https://github.com/crosscloudci/cross-cloud/blob/master/vsphere/modules/master/data_sources.tf#L31
On VMware there are some restrictions on using the de-facto cloud-init data-source, the solution was to use ignition to write the meta-data (as a file) to disk and then manually execute cloud-init using the file as the data-source.

Are you able to ssh into to the vm, if so it might be worth checking out the journalctl logs for the cloud-init service, you should be able to see if/why cloud-init is crashing.

If it's not getting that far, it could be that the Oracle cloud doesn't support being given gzip encoded meta-data (the only reason we do this is for AWS, who have a 16K meta data limit), so you can try removing it and see if you get any further.

e.g https://github.com/crosscloudci/cross-cloud/blob/master/aws/modules/master/ec2.tf#L24
Using aws as an example you would update L24 to look like this.

user_data = "${ element(split("`", var.master_cloud_init), count.index) }"

[doberloh]

Thanks for the response. Looks like b64 gzip data is a no go. I ran with just b64 encoded data and looks like it found it. That being said, I still don't see things running when I ssh in to the box.

When I run journalctl for ignition I see it get the user data and then is spits this out:
Jul 17 16:36:40 localhost ignition[752]: failed to fetch config: not a config (found coreos-cloudconfig) Jul 17 16:36:40 localhost ignition[752]: not a config (found coreos-cloudconfig): ignoring user-provided config
I think this is because ignition has replaced cloudconfig in newer versions of the OS and is not meant to run cloudconfig files. I tried VM wares way I run into metadata size limits and cant provision. My next question is, what would the recommendation be to get this running?

[doberloh]

more notes. Running cloudinit directly produces this output:

coreos-cloudinit --from-url http://169.254.169.254/opc/v1/instance/metadata/user_data 2018/07/17 20:26:39 Checking availability of "url" 2018/07/17 20:26:39 Fetching user-data from datasource of type "url" 2018/07/17 20:26:39 Fetching data from http://169.254.169.254/opc/v1/instance/metadata/user_data. Attempt #1 2018/07/17 20:26:39 line 1: error: must be "#cloud-config" or begin with "#!" 2018/07/17 20:26:39 Fetching meta-data from datasource of type "url" Failed to parse user-data: Unrecognized user-data format Continuing... 2018/07/17 20:26:39 Merging cloud-config from meta-data and user-data 2018/07/17 20:26:39 Ensuring runtime unit file "etcd.service" is unmasked 2018/07/17 20:26:39 Ensuring runtime unit file "etcd2.service" is unmasked 2018/07/17 20:26:39 Ensuring runtime unit file "fleet.service" is unmasked 2018/07/17 20:26:39 Ensuring runtime unit file "locksmithd.service" is unmasked

I have downloaded using wget to the instance directly and decoded, and verified the #cloud-config is there.

[taylor]

@doberloh,

It should be possible to use cloud-init directly (and do not use ignition at all). It looks like you should use the raw version of the metadata user_data instead of b64 encoded.

Using aws as an example you would update L17

    user_data = "${ base64encode(element(split("`", var.master_cloud_init), count.index)) }"

to look like this:

    user_data = "${ element(split("`", var.master_cloud_init), count.index) }"

(AWS example e.g https://github.com/crosscloudci/cross-cloud/blob/master/aws/modules/master/ec2.tf#L24)

Look at the jounalctl for cloud-init (vs ignition) for debugging the issues with cloud-init.

[doberloh]

Ill try this and report my results.

[doberloh]

I am able to grab the userdata and execute cloudinit directly through a remote-exec terraform provisioner, so I will go that route for now.

Two more questions:

1. after the cloud init runs i should see a kube master and worker running on their respective nodes, correct?

and

2. I put this in the slack as well, but what is the plan for the linux distro used for the cross-cloud project moving forward. I know that container linux as it exists today will be absorbed by RedHat and moved into RHEL. Are there plans to change the distro in the future, or will we be using whatever the RedHat incarnation is?

[doberloh]

answer to question 2 is there is going to be a topic in the CI WG call next tues.

[doberloh]

@denverwilliams or @taylor Based on what i have read i think that oracle needs to be added to the DNS server. Who can do that for me?
#147 (comment)

[denverwilliams]

@doberloh I have added oracle.local to our dns config, so you should now be able to create and resolve records.

[doberloh]

Hey there,

I was able to get through a full run of the provision script on friday. What are the next steps?

[lixuna]

@doberloh That's great news!

Please submit a New Pull Request on https://github.com/crosscloudci/cross-cloud/pulls and compare the code on your branch to the master branch.

Once the PR is available, we will review and merge for testing.

If we encounter any issues on master, we will open a new ticket for your attention.

[doberloh]

For reference: #166