Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Script to delete all entitities of a Kind in the datastore #3349

Closed
iperdomo opened this issue Dec 11, 2019 · 1 comment
Closed

Script to delete all entitities of a Kind in the datastore #3349

iperdomo opened this issue Dec 11, 2019 · 1 comment
Assignees
@iperdomo
Labels
Backend Deployment & infrastructure

Comments

@iperdomo
Copy link
Contributor

Documentation & resources

Improvements on the base script:

  • Use SELECT __key__ FORM {kind} since we have unlimited small operations in the Datastore
    • keys only queries are considered small Datastore operations
  • Use json and jq output when querying the gcloud

Related: https://github.com/akvo/akvo-platform/issues/204
@iperdomo iperdomo self-assigned this Dec 11, 2019
iperdomo added a commit that referenced this issue Dec 11, 2019
@iperdomo
[#3349] Use Dataflow for deleting all entities of a Kind
a9e403c 
* See https://cloud.google.com/datastore/docs/bulk-delete
@iperdomo
Copy link
Contributor Author

Example of the output

./scripts/data/delete-kind.sh akvoflow-dev3 SurveyalValue
Updated property [core/project].
BucketNotFoundException: 404 gs://tmpdeleteakvoflowdev3 bucket does not exist.
Creating gs://tmpdeleteakvoflowdev3/...
Enabling dataflow api
Operation "operations/acf.e841b065-7086-4e8f-aa26-a425d3441fda" finished successfully.
createTime: '2019-12-11T09:17:15.574482Z'
currentStateTime: '1970-01-01T00:00:00Z'
id: 2019-12-11_01_17_14-6062327673874293807
location: europe-west1
name: deletejob-1576055658
projectId: akvoflow-dev3
startTime: '2019-12-11T09:17:15.574482Z'
type: JOB_TYPE_BATCH
JOB_ID                                   NAME                  TYPE   CREATION_TIME        STATE    REGION
2019-12-11_01_17_14-6062327673874293807  deletejob-1576055658  Batch  2019-12-11 09:17:15  Pending  europe-west1
Done
Removing gs://tmpdeleteakvoflowdev3/...
Disabling Dataflow API
Operation "operations/acf.8c40fd1b-4e1e-437b-9a71-864489008571" finished successfully.

iperdomo added a commit that referenced this issue Dec 11, 2019
@iperdomo
Merge pull request #3350 from akvo/issue/3349-delete-kind
890f8ad 
[#3349] Use Dataflow for deleting all entities of a Kind
iperdomo added a commit that referenced this issue Dec 13, 2019
@iperdomo
[#3349] Use project as parameter to be able to execute in parallel
204cc63 
Instead of `gcloud config set core/project` to define the project
we want, use `--project` in all calls to APIs.

This has the benefit of running the same script in parallel with
multiple projects at the same time.
dlebrero added a commit that referenced this issue Dec 13, 2019
@dlebrero
Merge pull request #3357 from akvo/issue/3349-script-improvements
7c7ddae 
[#3349] Use project as parameter to be able to execute in parallel
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@iperdomo iperdomo

Labels
Backend Deployment & infrastructure
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@iperdomo