Build and send pre-processing reports

[antoineludeau]

Context

Our back-end system is composed of several components :

Pre-processing component (id-fix)
Processing component (ban-plateforme)
Our logs are separated and also are only written on server files.
Our system need to have a better visibility on the pre-processing and processing logs of the BAL sent to the BAN.

Enhancement

This PR aims to :

• build a pre-processing report for each BAL processed
• send the pre-processing report to Ban-plateforme

How to test

Need to start ban-plateforme and id-fix, connected locally.
Here is the corresponding ban-plateforme PR : BaseAdresseNationale/ban-plateforme#433

I. Legacy pre-processing report :

• Send a BAL through id-fix that is not part of the whitelist or that is not using ban-IDs
• The BAL is sent to ban-plateforme legacy API
• A pre-processing report is generated and sent to ban-plateforme
• The report that is generated in the processing-reports collection is the following one :

{
  "districtID": "a950efd3-69e7-41df-b5d8-a47dc660b66e",
  "preProcessingStatusKey": 0,
  "preProcessingStatus": "success",
  "preProcessingMessage": "Redirected to legacy : District id a950efd3-69e7-41df-b5d8-a47dc660b66e (cog: 59183) is not part of the whitelist.",
  "preProcessingResponse": {},
  "meta": {
    "targetedPlateform": "legacy",
    "revision": {...revisionData*},
    "cog": "59183"
  },
  "preProcessingDate": {
    "$date": "2024-06-21T09:39:19.027Z"
  }
}

*revisionData being the data from the last revision of api-de-dépôt.

II. Pre-processing reports - Success:

• Send a BAL throgh id-fix that is part of the whitelist
• The BAL is processed by id-fix and all the different components (addresses and common toponyms) are sent to BAN BAN-IDs APIs (/address /common-toponym)
• A pre-processing report is generated and sent to ban-plateforme
• The report that is generated in the processing-reports collection is the following one :

{
  "districtID": "a950efd3-69e7-41df-b5d8-a47dc660b66e",
  "preProcessingStatusKey": 0,
  "preProcessingStatus": "success",
  "preProcessingMessage": "Pre-processed : District id a950efd3-69e7-41df-b5d8-a47dc660b66e (cog: 59183) pre-processed and sent to BAN APIs.",
  "preProcessingResponse": {
    "addresses": {
      "update": [
        {
          "date": "2024-06-21T09:46:32.527Z",
          "status": "success",
          "message": "Check the status of your request : http://localhost:5000/api/job-status/H5BY3S7AF",
          "response": {
            "statusID": "H5BY3S7AF"
          }
        }
      ],
      "delete": [
        {
          "date": "2024-06-21T09:46:32.531Z",
          "status": "success",
          "message": "Check the status of your request : http://localhost:5000/api/job-status/94DX1NB9Q",
          "response": {
            "statusID": "94DX1NB9Q"
          }
        }
      ]
    },
    "commonToponyms": {
      "update": [
        {
          "date": "2024-06-21T09:46:32.524Z",
          "status": "success",
          "message": "Check the status of your request : http://localhost:5000/api/job-status/26DRMPNQ1",
          "response": {
            "statusID": "26DRMPNQ1"
          }
        }
      ],
      "delete": [
        {
          "date": "2024-06-21T09:46:32.534Z",
          "status": "success",
          "message": "Check the status of your request : http://localhost:5000/api/job-status/BDTT4AGXC",
          "response": {
            "statusID": "BDTT4AGXC"
          }
        }
      ]
    }
  },
  "meta": {
    "targetedPlateform": "ban",
    "revision": {...revisionData*},
    "cog": "59183"
  },
  "preProcessingDate": {
    "$date": "2024-06-21T09:46:32.535Z"
  }
}

*revisionData being the data from the last revision of api-de-dépôt.

This pre-processing report contains a pre-processing response with the different status ID from our asynchronous ban apis. This pre-processing report can be "rebuilt" in a final state by two mechanisms :

• make an API call to GET api/report/district/{districtID} (or /api/report/district/cog/{cog})
• make the daily batch run by changing the time between two jobs :
  In worker.js file, change 1d to 10s for example :
  queue('build-reports').add({}, {jobId: 'buildReportsJobId', repeat: {every: ms('10s')}, removeOnComplete: true})

III. Pre-processing reports - Error:

The pre-processing report can also sent pre-processing errors such as :

• a missing ID on a BAL line :

To test this behaviour :

• Send a BAL that is part of the whitelist, that have ban ids but that has an error on a line (delete an address ID for example).
• The BAL is processed by id-fix and the error is detected
• A pre-processing report is generated and sent to ban-plateforme
• The report that is generated in the processing-reports collection is the following one :

{
  "districtID": "a950efd3-69e7-41df-b5d8-a47dc660b66e",
  "preProcessingStatusKey": 1,
  "preProcessingStatus": "error",
  "preProcessingMessage": "Not Processed : Missing mainTopoID for bal address {\"objectid\":\"61419\",\"cle_interop\":\"59183_3577_00005\",\"commune_insee\":\"59183\",\"commune_nom\":\"Dunkerque\",\"commune_deleguee_insee\":\"59540\",\"commune_deleguee_nom\":\"Saint-Pol-sur-Mer\",\"voie_nom\":\"Square Delvallez\",\"lieudit_complement_nom\":\"\",\"numero\":5,\"suffixe\":\"\",\"position\":\"entrée\",\"x\":653301.9598168375,\"y\":7103868.546534107,\"long\":2.3358292732844284,\"lat\":51.02876869564352,\"cad_parcelles\":[\"591183540AV0192\",\"591183540AV0191\"],\"source\":\"BAN\",\"date_der_maj\":\"2024-01-04T00:00:00.000Z\",\"certification_commune\":false,\"id_ban_adresse\":\"b611de43-6578-4a07-9ac2-0f58d5b8f0ba\",\"id_ban_toponyme\":\"\",\"id_ban_commune\":\"a950efd3-69e7-41df-b5d8-a47dc660b66e\"}",
  "preProcessingResponse": {},
  "meta": {
    "revision": {...revisionData*},
    "cog": "59183"
  },
  "preProcessingDate": {
    "$date": "2024-06-21T10:12:49.077Z"
  }
}

*revisionData being the data from the last revision of api-de-dépôt.