Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Reconciler crashes with Missing field 'SetIdentifier' #446

Closed
humphd opened this issue Mar 25, 2023 · 3 comments
Closed

Reconciler crashes with Missing field 'SetIdentifier' #446

humphd opened this issue Mar 25, 2023 · 3 comments
Labels
bug Something isn't working
Milestone
Milestone 0.8

Comments

@humphd
Copy link
Contributor

humphd commented Mar 25, 2023

I tried running the reconciler on staging, and it crashes with this in the server logs (cc @dadolhay):

starchart_mycustomdomain.1.szxwrejre3cw@cudm-mgmt01dv.dcm.senecacollege.ca    | {"changeSet":[{"Action":"DELETE","ResourceRecordSet":{"MultiValueAnswer":false,"Name":"stage.mystudentproject.ca.","ResourceRecords":[{"Value":"\"v=spf1 -all\""}],"TTL":300,"Type":"TXT"}},{"Action":"DELETE","ResourceRecordSet":{"MultiValueAnswer":false,"Name":"_dmarc.stage.mystudentproject.ca.","ResourceRecords":[{"Value":"\"v=DMARC1; p=reject; sp=reject\""}],"TTL":300,"Type":"TXT"}},{"Action":"DELETE","ResourceRecordSet":{"MultiValueAnswer":false,"Name":"foo.davidhumphrey.stage.mystudentproject.ca.","ResourceRecords":[{"Value":"github.com"}],"TTL":300,"Type":"CNAME"}},{"Action":"DELETE","ResourceRecordSet":{"MultiValueAnswer":false,"Name":"test1.wno.stage.mystudentproject.ca.","ResourceRecords":[{"Value":"192.168.0.1"}],"TTL":300,"Type":"A"}}],"level":"debug","message":"Reconciler intends to push the following 4 changes","timestamp":"2023-03-25T23:43:50.576Z"}
starchart_mycustomdomain.1.szxwrejre3cw@cudm-mgmt01dv.dcm.senecacollege.ca    | {"error":{"$fault":"client","$metadata":{"attempts":1,"httpStatusCode":400,"requestId":"07d2e314-a5a1-4c94-9a2d-9a12c9ff0350","totalRetryDelay":0},"Code":"InvalidInput","Type":"Sender","name":"InvalidInput"},"level":"error","message":"DNS Error - Failed to execute changeSet","timestamp":"2023-03-25T23:43:50.742Z"}
starchart_mycustomdomain.1.szxwrejre3cw@cudm-mgmt01dv.dcm.senecacollege.ca    | InvalidInput: Invalid request: Missing field 'SetIdentifier' in Change with [Action=DELETE, Name=stage.mystudentproject.ca., Type=TXT, SetIdentifier=null]
starchart_mycustomdomain.1.szxwrejre3cw@cudm-mgmt01dv.dcm.senecacollege.ca    |     at deserializeAws_restXmlInvalidInputResponse (/app/node_modules/@aws-sdk/client-route-53/dist-cjs/protocols/Aws_restXml.js:5262:23)
starchart_mycustomdomain.1.szxwrejre3cw@cudm-mgmt01dv.dcm.senecacollege.ca    |     at deserializeAws_restXmlChangeResourceRecordSetsCommandError (/app/node_modules/@aws-sdk/client-route-53/dist-cjs/protocols/Aws_restXml.js:1928:25)
starchart_mycustomdomain.1.szxwrejre3cw@cudm-mgmt01dv.dcm.senecacollege.ca    |     at processTicksAndRejections (node:internal/process/task_queues:95:5)
starchart_mycustomdomain.1.szxwrejre3cw@cudm-mgmt01dv.dcm.senecacollege.ca    |     at /app/node_modules/@aws-sdk/middleware-serde/dist-cjs/deserializerMiddleware.js:7:24
starchart_mycustomdomain.1.szxwrejre3cw@cudm-mgmt01dv.dcm.senecacollege.ca    |     at /app/node_modules/@aws-sdk/middleware-signing/dist-cjs/middleware.js:14:20
starchart_mycustomdomain.1.szxwrejre3cw@cudm-mgmt01dv.dcm.senecacollege.ca    |     at /app/node_modules/@aws-sdk/middleware-retry/dist-cjs/retryMiddleware.js:27:46
starchart_mycustomdomain.1.szxwrejre3cw@cudm-mgmt01dv.dcm.senecacollege.ca    |     at /app/node_modules/@aws-sdk/middleware-logger/dist-cjs/loggerMiddleware.js:7:26
starchart_mycustomdomain.1.szxwrejre3cw@cudm-mgmt01dv.dcm.senecacollege.ca    |     at executeChangeSet (/app/build/index.js:769:20)
starchart_mycustomdomain.1.szxwrejre3cw@cudm-mgmt01dv.dcm.senecacollege.ca    |     at reconcile (/app/build/index.js:2696:7)
starchart_mycustomdomain.1.szxwrejre3cw@cudm-mgmt01dv.dcm.senecacollege.ca    |     at action7 (/app/build/index.js:2712:14) {
starchart_mycustomdomain.1.szxwrejre3cw@cudm-mgmt01dv.dcm.senecacollege.ca    |   '$fault': 'client',
starchart_mycustomdomain.1.szxwrejre3cw@cudm-mgmt01dv.dcm.senecacollege.ca    |   '$metadata': {
starchart_mycustomdomain.1.szxwrejre3cw@cudm-mgmt01dv.dcm.senecacollege.ca    |     httpStatusCode: 400,
starchart_mycustomdomain.1.szxwrejre3cw@cudm-mgmt01dv.dcm.senecacollege.ca    |     requestId: '07d2e314-a5a1-4c94-9a2d-9a12c9ff0350',
starchart_mycustomdomain.1.szxwrejre3cw@cudm-mgmt01dv.dcm.senecacollege.ca    |     extendedRequestId: undefined,
starchart_mycustomdomain.1.szxwrejre3cw@cudm-mgmt01dv.dcm.senecacollege.ca    |     cfId: undefined,
starchart_mycustomdomain.1.szxwrejre3cw@cudm-mgmt01dv.dcm.senecacollege.ca    |     attempts: 1,
starchart_mycustomdomain.1.szxwrejre3cw@cudm-mgmt01dv.dcm.senecacollege.ca    |     totalRetryDelay: 0
starchart_mycustomdomain.1.szxwrejre3cw@cudm-mgmt01dv.dcm.senecacollege.ca    |   },
starchart_mycustomdomain.1.szxwrejre3cw@cudm-mgmt01dv.dcm.senecacollege.ca    |   Type: 'Sender',
starchart_mycustomdomain.1.szxwrejre3cw@cudm-mgmt01dv.dcm.senecacollege.ca    |   Code: 'InvalidInput'
starchart_mycustomdomain.1.szxwrejre3cw@cudm-mgmt01dv.dcm.senecacollege.ca    | }

Here is is cleaned up:

{
  "changeSet": [
    {
      "Action": "DELETE",
      "ResourceRecordSet": {
        "MultiValueAnswer": false,
        "Name": "stage.mystudentproject.ca.",
        "ResourceRecords": [{ "Value": "\"v=spf1 -all\"" }],
        "TTL": 300,
        "Type": "TXT"
      }
    },
    {
      "Action": "DELETE",
      "ResourceRecordSet": {
        "MultiValueAnswer": false,
        "Name": "_dmarc.stage.mystudentproject.ca.",
        "ResourceRecords": [{ "Value": "\"v=DMARC1; p=reject; sp=reject\"" }],
        "TTL": 300,
        "Type": "TXT"
      }
    },
    {
      "Action": "DELETE",
      "ResourceRecordSet": {
        "MultiValueAnswer": false,
        "Name": "foo.davidhumphrey.stage.mystudentproject.ca.",
        "ResourceRecords": [{ "Value": "github.com" }],
        "TTL": 300,
        "Type": "CNAME"
      }
    },
    {
      "Action": "DELETE",
      "ResourceRecordSet": {
        "MultiValueAnswer": false,
        "Name": "test1.wno.stage.mystudentproject.ca.",
        "ResourceRecords": [{ "Value": "192.168.0.1" }],
        "TTL": 300,
        "Type": "A"
      }
    }
  ],
  "level": "debug",
  "message": "Reconciler intends to push the following 4 changes",
  "timestamp": "2023-03-25T23:43:50.576Z"
}

{
  "error": {
    "$fault": "client",
    "$metadata": {
      "attempts": 1,
      "httpStatusCode": 400,
      "requestId": "07d2e314-a5a1-4c94-9a2d-9a12c9ff0350",
      "totalRetryDelay": 0
    },
    "Code": "InvalidInput",
    "Type": "Sender",
    "name": "InvalidInput"
  },
  "level": "error",
  "message": "DNS Error - Failed to execute changeSet",
  "timestamp": "2023-03-25T23:43:50.742Z"
}

And:

InvalidInput: Invalid request: Missing field 'SetIdentifier' in Change with [Action=DELETE, Name=stage.mystudentproject.ca., Type=TXT, SetIdentifier=null]

I'm not finding a specific answer to why this is happening in my searches so far.

It looks like there are records in here that aren't related to our app, so we're going to have to rethink the idea of owning the entire zone.
@humphd humphd added the bug Something isn't working label Mar 25, 2023
@humphd
Copy link
Contributor Author

humphd commented Mar 25, 2023

My guess is that SetIdentifier is set on one of those records we don't own, and it's refusing to delete them because we haven't included it where it expects to find it.

I wonder if we should limit our scope to only records that follow the form {something}.{someusername}.root.domain., which would eliminate both stage.mystudentproject.ca. and _dmarc.stage.mystudentproject.ca. in the case above.

@humphd
Copy link
Contributor Author

humphd commented Mar 26, 2023

Despite the work to add limp mode in #447, I'd like to avoid stepping on records we don't actually own in the hosted zone.

@Myrfion Myrfion assigned ghost Mar 27, 2023
@Myrfion Myrfion added this to the Milestone 0.8 milestone Mar 27, 2023
@humphd
Copy link
Contributor Author

humphd commented Mar 28, 2023

This was fixed in our debugging tonight.

@humphd humphd closed this as completed Mar 28, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
Milestone 0.8
Development

No branches or pull requests
2 participants
@humphd @Myrfion