[deps] Update HCL2 to 2.17.0+tweaks

[angrycub]

In hashicorp/hcl#620, I created an updated version of the Nomad-tweaked HCL2 code for HCL v2.17.0. We leverage this version's Extra attribute of an hcl.Diagnostic in nomad-pack.

[ghthor]

@angrycub

Our internal tooling has started using this 2.17.0 hcl branch because the older hcl2 branch has some bugs that have been fixed since then.

Once we did that, jobspec2.Parse() started returning the follow error for jobs that define constraint { distinct_hosts = true } }

Unsupported argument; An argument named "distinct_hosts" is not expected here.

The following repo contains a fully reproducible example

https://github.com/ghthor/nomad-bug-reports/tree/main/nomad-hcl-2.17.0-branch-pr-18122

[angrycub]

Okay. That is another case where we hacked in support for old HCL shapes. Will research it and see what is missing. If you switch to the operator value form it should work as expected in the interim.

[ghthor]

It looks like there are some failing test cases between this branch and main, interested why CI is not running go test here.

nomad/jobspec2 on  dep-hcl2+tweaks [!] via 🐹 v1.21.3 on ☁️  (eu-west-1) 
✦ ❯ go test | grep '^--- FAIL'
--- FAIL: TestParse_TaskEnvs_Multiple (0.00s)
--- FAIL: TestParse_Meta_Alternatives (0.00s)
--- FAIL: Test_TaskEnvs_Invalid (0.00s)
--- FAIL: TestParse_InvalidScalingSyntax (0.01s)
--- FAIL: TestParse_TaskEnvs (0.01s)
--- FAIL: TestEquivalentToHCL1 (0.05s)

Thu Nov 16 04:57:11 2023 exit 0 🟢[ exit 1 ❌ ERROR| exit 0 ] => 

I'm wondering if it would be worthwhile to keep the changes required in the hcl2 fork to a minimum and perform some preprocessing of the HCL using the hclwrite package before we pass it into hclsyntax.ParseConfig.

diff --git a/jobspec2/parse.go b/jobspec2/parse.go
index 4b91ba76f7..bd2b44664b 100644
--- a/jobspec2/parse.go
+++ b/jobspec2/parse.go
@@ -14,6 +14,7 @@ import (
 
 	"github.com/hashicorp/hcl/v2"
 	"github.com/hashicorp/hcl/v2/hclsyntax"
+	"github.com/hashicorp/hcl/v2/hclwrite"
 	hcljson "github.com/hashicorp/hcl/v2/json"
 	"github.com/hashicorp/nomad/api"
 )
@@ -161,9 +162,37 @@ func parseHCLOrJSON(src []byte, filename string) (*hcl.File, hcl.Diagnostics) {
 		return hcljson.Parse(src, filename)
 	}
 
+	src, diags := convertOldHCLFormats(src, filename)
+	if diags != nil {
+		return nil, diags
+	}
+
 	return hclsyntax.ParseConfig(src, filename, hcl.Pos{Line: 1, Column: 1})
 }
 
+func convertOldHCLFormats(src []byte, filename string) ([]byte, hcl.Diagnostics) {
+	var diags hcl.Diagnostics
+
+	f, diags := hclwrite.ParseConfig(src, filename, hcl.InitialPos)
+	if diags != nil {
+		return src, diags
+	}
+
+	// TODO: preprocess old HCL shapes
+
+	var b bytes.Buffer
+	_, err := f.WriteTo(&b)
+	if err != nil {
+		return src, append(diags, &hcl.Diagnostic{
+			Severity: hcl.DiagError,
+			Summary:  "failed write to buffer",
+			Detail:   "failed to write hclwrite.File into a bytes.Buffer",
+		})
+	}
+
+	return b.Bytes(), nil
+}
+
 func isJSON(src []byte) bool {
 	for _, c := range src {
 		if c == ' ' {

We should add some test cases for these alternative constraint {} block formats.

[hashicorp-cla]

All committers have signed the CLA.

[tgross]

LGTM!

I've left suggestions to update from testify to shoenig/test assertions (we should be gradually migrating over because testify has correctness issues that can't be fixed backwards compatibly).

[tgross]

@angrycub I think we can close this now that #22439 has been merged?

[angrycub]

Superseded #22439