From 2a170a34927a7ac0c8f8e4379e0f9b8e900f78a1 Mon Sep 17 00:00:00 2001 From: t83714 Date: Sun, 8 Oct 2023 09:41:15 +0000 Subject: [PATCH] v2.1.0-alpha.0 --- README.md | 80 +++++++++++++------------- deploy/magda-ckan-connector/Chart.yaml | 2 +- package.json | 2 +- 3 files changed, 42 insertions(+), 42 deletions(-) diff --git a/README.md b/README.md index 711d2de..fa2c9ee 100644 --- a/README.md +++ b/README.md @@ -14,52 +14,52 @@ It's recommended to deploy connectors with as [dependencies](https://helm.sh/doc ```yaml dependencies: - - name: magda-ckan-connector - version: "2.0.0" - alias: connector-xxx - repository: "oci://ghcr.io/magda-io/charts" - tags: - - connectors - - connector-xxx + - name: magda-ckan-connector + version: "2.0.0" + alias: connector-xxx + repository: "oci://ghcr.io/magda-io/charts" + tags: + - connectors + - connector-xxx ``` ## Requirements Kubernetes: `>= 1.21.0` -| Repository | Name | Version | -| ----------------------------- | ------------ | ------- | -| oci://ghcr.io/magda-io/charts | magda-common | 2.0.1 | +| Repository | Name | Version | +|------------|------|---------| +| oci://ghcr.io/magda-io/charts | magda-common | 2.0.1 | ## Values -| Key | Type | Default | Description | -| ------------------------------- | ------ | ----------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -| config.allowedOrganisationNames | string | `nil` | allowOrganisationNames: a list of organisation names that will be allowed to be harvested. should an array of string; | -| config.customJsFilterCode | string | `""` | Custome JS filter code to filter out records the code will be executed with the following variables available in the context: - jsonData: the rawJson record data retrieved by connector directly from remote API. Also the record to be filtered; - recordType: a string represent the type of the record. e.g. "Organization" | "Dataset" | "Distribution" the code should return a boolean value to indicate whether the record should be filtered out (true) or not (false) Any non-false return value (or no return value) will be treated as `false` | -| config.extras | string | `nil` | extras: an object whose data will be copied to records' source aspect as additional data automatically should be an object with key-value pairs | -| config.id | string | `"dga"` | Unique id to identify this connector and records that are harvested from it | -| config.ignoreHarvestSources | list | `["*"]` | CKAN-specific config: what harvest sources to ignore. \* will ignore everything that's been harvested from another portal | -| config.ignoreOrganisationNames | string | `nil` | ignoreOrganisationNames: a list of organisation names that will be ignored to be harvested. should an array of string; | -| config.name | string | `"data.gov.au"` | Friendly readable name | -| config.pageSize | int | `100` | When crawling through from beginning to end, how big should the individual requests be in records? | -| config.presetRecordAspects | string | `nil` | Any aspects that will be `preset` on any records created by the connector This field should be an array of objects with the following fields: - `id`: aspect id - `opType`: operation type; Describe how to add the aspect to the record (Default value (If not specified) will be `MERGE_LEFT`) - MERGE_LEFT: merge `presetAspect` with records aspects. - i.e. `presetAspect` will be overwritten by records aspects data if any - MEREG_RIGHT: merge records aspects with `presetAspect`. - i.e. records aspects data (if any) will be overwritten by `presetAspect` - REPLACE: `presetAspect` will replace any existing records aspect - FILL: `presetAspect` will be added if no existing aspect - ` recordType``: Describes which type of records this aspect will be added to; If this field is omitted, the aspect will be added to every records. - `data`: Object; aspect data | -| config.schedule | string | `""` | Crontab schedule for how often this should happen. default = "0 14 \* \* 6" i.e. 12am Sydney time on Sunday | -| config.sourceUrl | string | `"https://data.gov.au/data/"` | The base URL of the place to source data from | -| createConfigMap | bool | `true` | | -| createFunction | bool | `false` | | -| defaultImage.imagePullSecret | bool | `false` | | -| defaultImage.pullPolicy | string | `"IfNotPresent"` | | -| defaultImage.repository | string | `"ghcr.io/magda-io"` | | -| defaultSettings | object | `{"includeCronJobs":true,"includeInitialJobs":false}` | defaultSettings Chart level .Values.includeInitialJobs is higher priority than .Values.global.includeInitialJobs The value of includeInitialJobs or includeCronJobs is determined by: - .Values.includeInitialJobs if its value is set - Then .Values.global.includeInitialJobs if its value is set - Then .Values.defaultSettings.includeInitialJobs By default, .Values.includeInitialJobs is not set. Therefore, if .Values.global.includeInitialJobs is not set, .Values.defaultSettings.includeInitialJobs will be used. | -| defaultTenantId | int | `0` | | -| global.connectors.image | object | `{}` | | -| global.image | object | `{}` | | -| global.openfaas | object | `{}` | | -| image.name | string | `"magda-ckan-connector"` | | -| includeCronJobs | bool | `true` | see `defaultSettings` | -| includeInitialJobs | string | `nil` | see `defaultSettings` | -| resources.limits.cpu | string | `"100m"` | | -| resources.requests.cpu | string | `"50m"` | | -| resources.requests.memory | string | `"30Mi"` | | -| secrets | string | `nil` | Set a list of secrets that pass to function (this config field is only for deploying connector code as openfaas functions) Secrets are accessible from `/var/openfaas/secrets/` as file | +| Key | Type | Default | Description | +|-----|------|---------|-------------| +| config.allowedOrganisationNames | string | `nil` | allowOrganisationNames: a list of organisation names that will be allowed to be harvested. should an array of string; | +| config.customJsFilterCode | string | `""` | Custome JS filter code to filter out records the code will be executed with the following variables available in the context: - jsonData: the rawJson record data retrieved by connector directly from remote API. Also the record to be filtered; - recordType: a string represent the type of the record. e.g. "Organization" | "Dataset" | "Distribution" the code should return a boolean value to indicate whether the record should be filtered out (true) or not (false) Any non-false return value (or no return value) will be treated as `false` | +| config.extras | string | `nil` | extras: an object whose data will be copied to records' source aspect as additional data automatically should be an object with key-value pairs | +| config.id | string | `"dga"` | Unique id to identify this connector and records that are harvested from it | +| config.ignoreHarvestSources | list | `["*"]` | CKAN-specific config: what harvest sources to ignore. * will ignore everything that's been harvested from another portal | +| config.ignoreOrganisationNames | string | `nil` | ignoreOrganisationNames: a list of organisation names that will be ignored to be harvested. should an array of string; | +| config.name | string | `"data.gov.au"` | Friendly readable name | +| config.pageSize | int | `100` | When crawling through from beginning to end, how big should the individual requests be in records? | +| config.presetRecordAspects | string | `nil` | Any aspects that will be `preset` on any records created by the connector This field should be an array of objects with the following fields: - `id`: aspect id - `opType`: operation type; Describe how to add the aspect to the record (Default value (If not specified) will be `MERGE_LEFT`) - MERGE_LEFT: merge `presetAspect` with records aspects. - i.e. `presetAspect` will be overwritten by records aspects data if any - MEREG_RIGHT: merge records aspects with `presetAspect`. - i.e. records aspects data (if any) will be overwritten by `presetAspect` - REPLACE: `presetAspect` will replace any existing records aspect - FILL: `presetAspect` will be added if no existing aspect - `recordType``: Describes which type of records this aspect will be added to; If this field is omitted, the aspect will be added to every records. - `data`: Object; aspect data | +| config.schedule | string | `""` | Crontab schedule for how often this should happen. default = "0 14 * * 6" i.e. 12am Sydney time on Sunday | +| config.sourceUrl | string | `"https://data.gov.au/data/"` | The base URL of the place to source data from | +| createConfigMap | bool | `true` | | +| createFunction | bool | `false` | | +| defaultImage.imagePullSecret | bool | `false` | | +| defaultImage.pullPolicy | string | `"IfNotPresent"` | | +| defaultImage.repository | string | `"ghcr.io/magda-io"` | | +| defaultSettings | object | `{"includeCronJobs":true,"includeInitialJobs":false}` | defaultSettings Chart level .Values.includeInitialJobs is higher priority than .Values.global.includeInitialJobs The value of includeInitialJobs or includeCronJobs is determined by: - .Values.includeInitialJobs if its value is set - Then .Values.global.includeInitialJobs if its value is set - Then .Values.defaultSettings.includeInitialJobs By default, .Values.includeInitialJobs is not set. Therefore, if .Values.global.includeInitialJobs is not set, .Values.defaultSettings.includeInitialJobs will be used. | +| defaultTenantId | int | `0` | | +| global.connectors.image | object | `{}` | | +| global.image | object | `{}` | | +| global.openfaas | object | `{}` | | +| image.name | string | `"magda-ckan-connector"` | | +| includeCronJobs | bool | `true` | see `defaultSettings` | +| includeInitialJobs | string | `nil` | see `defaultSettings` | +| resources.limits.cpu | string | `"100m"` | | +| resources.requests.cpu | string | `"50m"` | | +| resources.requests.memory | string | `"30Mi"` | | +| secrets | string | `nil` | Set a list of secrets that pass to function (this config field is only for deploying connector code as openfaas functions) Secrets are accessible from `/var/openfaas/secrets/` as file | \ No newline at end of file diff --git a/deploy/magda-ckan-connector/Chart.yaml b/deploy/magda-ckan-connector/Chart.yaml index e6261de..d33ca32 100644 --- a/deploy/magda-ckan-connector/Chart.yaml +++ b/deploy/magda-ckan-connector/Chart.yaml @@ -1,7 +1,7 @@ apiVersion: v2 name: magda-ckan-connector description: A Helm chart for Magda Ckan Connector -version: "2.0.0" +version: "2.1.0-alpha.0" kubeVersion: ">= 1.21.0" home: "https://github.com/magda-io/magda-ckan-connector" sources: [ "https://github.com/magda-io/magda-ckan-connector" ] diff --git a/package.json b/package.json index 45247f2..46874ba 100644 --- a/package.json +++ b/package.json @@ -1,7 +1,7 @@ { "name": "@magda/ckan-connector", "description": "MAGDA CKAN Connector", - "version": "2.0.0", + "version": "2.1.0-alpha.0", "scripts": { "prebuild": "rimraf dist tsconfig.tsbuildinfo", "build": "yarn run compile",