Anonymization & Archive functionality

doc/compodoc_sources/concepts/entity-schema-system.md

@@ -1,8 +1,14 @@
-# Entity Schema 
+# Entities & Entity Schema 
 -----
-The Entity Schema defines details of the properties of an entity type.
-(An "entity" is an object of a certain type that users work with and save to the database, like "Child" or "School")
+For us, an "Entity" is an object in the database (and a representation of something in the user's real world, e.g. a "Child" or "School").
+Entities are at the core of the Aam Digital platform and the primary way to customize the system is to adapt and add new entity types.
+
+The Entity Schema defines the data structure as well as how it is displayed in the UI.
+Entity instances also have some generic functionality inherited from the `Entity` base class.
 
+
+## Entity Schema
+The Entity Schema defines details of the properties of an entity type.
 We define an entity type and its schema in code through a plain TypeScript class and some custom annotations.
 Read more on the background and practical considerations in [How to create a new Entity Type](../how-to-guides/create-a-new-entity-type.html).
 
@@ -45,9 +51,45 @@ providing these through Angular dependency injection using `multi: true`.
 
 Also see: [How to create a new Datatype](../how-to-guides/create-a-new-datatype.html).
 
-## Schema options
+### Schema options
 
 The schema definitions contains information regarding the schema transformation as well as how a property can be displayed.
 The [EntitySchemaField](../../interfaces/EntitySchemaField.html) interface shows all configuration options.
 If the `editComponent` and the `viewComponent` are not set, the default components of this property's datatype will be used.
 The `description` field allows adding further explanation which will be displayed as a tooltip.
+
+
+## Generic Entity functionalities
+
+### Metadata (created, updated)
+Each record automatically holds basic data of timestamp and user who created and last updated the record.
+(see `Entity` class)
+
+### Archive / Anonymize
+Any entity can be archived (i.e. marked as inactive and hidden from UI by default) or anonymized (i.e. discarding most data and keeping a few selected properties for statistical reports).
+This is often preferable to deleting a record completely. Deleting data also affects statistical reports, even for previous time periods.
+By anonymizing records, all personal identifiable data can be removed and the remaining stub record can be stored indefinitely, as it is not subject to data protection regulations like GDPR anymore.
+
+Anonymization is configured as part of the entity schema.
+Data of fields that are not explicitly marked to be retained during anonymization is always deleted (anonymization by default).
+
+To keep some data even after the user "anonymized" a record, configure the `anonymize` property of the `@DatabaseField` decorator:
+- `anonymize: "retain"` will keep this field unchanged and prevent it from being deleted
+- `anonymize: "retain-anonymized"` will trigger a special "partial" deletion that depends on the dataType (e.g. date types will be changed to 1st July of the given year, thereby largely removing details but keeping data to calculate a rough age)
+
+#### Data Protection & GDPR regarding anonymization / pseudonomyzation
+The "anonymize" function is implemented specifically for data protection rules requiring to delete personal data.
+According to the EU's "General Data Protection Regulation" (GDPR) "anonymous" data does not fall under its regulations:
+
+- GDPR is not applicable to anonymous data: "The principles of data protection should therefore not apply to [...] personal data rendered anonymous in such a manner that the data subject is not or no longer identifiable." [<sup>GDPR Recital 26</sup>](https://gdpr-info.eu/recitals/no-26/)
+  - "To determine whether a natural person is identifiable, account should be taken of all the means reasonably likely to be used, such as singling out, either by the controller or by another person to identify the natural person directly or indirectly."
+  - "To ascertain whether means are reasonably likely to be used to identify the natural person, account should be taken of all objective factors, such as the costs of and the amount of time required for identification, taking into consideration the available technology at the time of the processing and technological developments."
+- "Pseudonymisation enables the personal data to become unidentifiable unless more information is available whereas anonymization allows the processing of personal data to irreversibly prevent re-identification." [<sup>source</sup>](https://www.privacycompany.eu/blogpost-en/what-are-the-differences-between-anonymisation-and-pseudonymisation)
+- _also see this [good overview of anonymization misunderstandings and considerations](https://edps.europa.eu/system/files/2021-04/21-04-27_aepd-edps_anonymisation_en_5.pdf)_
+
+In the case of records being retained "anonymized" in Aam Digital, we provide a context that makes re-identification even harder:
+- only authorized users of the system can access even the anonymized record (where only a few properties have been retained). Unless the organisation actively shares the data, it remains as securely protected as the personal data managed in Aam Digital.
+- those authorized users with access to the anonymized records (and therefor a theoretical chance to attempt re-identification) are team members of an organization. They have been screened to be responsible persons and are usually legally bound to keep information confidential.
+- by default only a few, explicitly selected properties in anonymized records are retained (data minimization by default). As such, both re-identification likelihood and the impact in case of re-identification are reduced as far as possible.
+
+--> If our anonymization process is configured thoughfully on a case by case basis to only retain a few data fields that are not easy indirect identifiers, it seems reasonably unlikely that the person can be identified after the anonymization process. Therefore, GDPR should not apply to these records and it is legitimate to retain these for statistical reporting.

src/app/child-dev-project/children/model/child.spec.ts

@@ -57,5 +57,13 @@ describe("Child", () => {
     const testEntity3 = new Child();
     testEntity3["status"] = "Dropout";
     expect(testEntity3.isActive).withContext("Dropout").toBeFalse();
+
+    // always give "inactive" precedence over other logic (as it is trigger in UI)
+    const testEntityPrec = new Child();
+    testEntityPrec["inactive"] = false;
+    testEntityPrec["status"] = "Dropout";
+    expect(testEntityPrec.isActive)
+      .withContext("inactive taking precedence")
+      .toBeTrue();
   });
 });

src/app/child-dev-project/children/model/child.ts

@@ -54,19 +54,22 @@ export class Child extends Entity {
     label: $localize`:Label for the project number of a child:Project Number`,
     labelShort: $localize`:Short label for the project number:PN`,
     searchable: true,
+    anonymize: "retain",
   })
   projectNumber: string;
 
   @DatabaseField({
     label: $localize`:Label for the date of birth of a child:Date of birth`,
     labelShort: $localize`:Short label for the date of birth:DoB`,
+    anonymize: "retain-anonymized",
   })
   dateOfBirth: DateWithAge;
 
   @DatabaseField({
     dataType: "configurable-enum",
     label: $localize`:Label for the gender of a child:Gender`,
     innerDataType: "genders",
+    anonymize: "retain",
   })
   gender: ConfigurableEnumValue;
 
@@ -120,6 +123,11 @@ export class Child extends Entity {
   phone: string;
 
   get isActive(): boolean {
+    if (this.inactive !== undefined) {
+      // explicit property set through UI has to take precedence
+      return super.isActive;
+    }
+
     return (
       this.status !== "Dropout" &&
       !this["dropoutDate"] &&

src/app/child-dev-project/notes/note-details/note-details.component.html

@@ -18,51 +18,55 @@
 <h1 mat-dialog-title>{{ tmpEntity.date | date }}: {{ tmpEntity.subject }}</h1>
 <app-dialog-close mat-dialog-close=""></app-dialog-close>
 
-<mat-dialog-content class="flex-column gap-regular">
-  <app-entity-form [columns]="topForm" [entity]="entity" [form]="form">
-  </app-entity-form>
+<mat-dialog-content>
+  <div class="flex-column gap-regular">
+    <app-entity-archived-info [entity]="entity"></app-entity-archived-info>
 
-  <!-- Primary information of Note -->
-  <div
-    class="mat-elevation-z2 flex-column gap-small padding-small margin-bottom-regular"
-  >
-    <div class="middle-form-field">
-      <ng-container
-        [appDynamicComponent]="{
-          component: middleForm[0].edit,
-          config: {
-            formFieldConfig: middleForm[0],
-            propertySchema: entity.getSchema().get(middleForm[0].id),
-            formControl: form.get(middleForm[0].id),
-            entity: entity
-          }
-        }"
-      >
-      </ng-container>
-    </div>
+    <app-entity-form [columns]="topForm" [entity]="entity" [form]="form">
+    </app-entity-form>
+
+    <!-- Primary information of Note -->
+    <div
+      class="mat-elevation-z2 flex-column gap-small padding-small margin-bottom-regular"
+    >
+      <div class="middle-form-field">
+        <ng-container
+          [appDynamicComponent]="{
+            component: middleForm[0].edit,
+            config: {
+              formFieldConfig: middleForm[0],
+              propertySchema: entity.getSchema().get(middleForm[0].id),
+              formControl: form.get(middleForm[0].id),
+              entity: entity
+            }
+          }"
+        >
+        </ng-container>
+      </div>
 
-    <div class="textarea middle-form-field">
-      <ng-container
-        [appDynamicComponent]="{
-          component: middleForm[1].edit,
-          config: {
-            formFieldConfig: middleForm[1],
-            propertySchema: entity.getSchema().get(middleForm[1].id),
-            formControl: form.get(middleForm[1].id),
-            entity: entity
-          }
-        }"
-      >
-      </ng-container>
+      <div class="textarea middle-form-field">
+        <ng-container
+          [appDynamicComponent]="{
+            component: middleForm[1].edit,
+            config: {
+              formFieldConfig: middleForm[1],
+              propertySchema: entity.getSchema().get(middleForm[1].id),
+              formControl: form.get(middleForm[1].id),
+              entity: entity
+            }
+          }"
+        >
+        </ng-container>
+      </div>
     </div>
-  </div>
 
-  <app-entity-form
-    [columns]="[bottomForm]"
-    [entity]="entity"
-    [form]="form"
-    style="margin-top: 10px"
-  ></app-entity-form>
+    <app-entity-form
+      [columns]="[bottomForm]"
+      [entity]="entity"
+      [form]="form"
+      style="margin-top: 10px"
+    ></app-entity-form>
+  </div>
 </mat-dialog-content>
 
 <mat-dialog-actions>

src/app/child-dev-project/notes/note-details/note-details.component.ts

@@ -26,6 +26,7 @@ import { DynamicComponentDirective } from "../../../core/config/dynamic-componen
 import { MAT_DIALOG_DATA, MatDialogModule } from "@angular/material/dialog";
 import { DialogButtonsComponent } from "../../../core/form-dialog/dialog-buttons/dialog-buttons.component";
 import { DialogCloseComponent } from "../../../core/common-components/dialog-close/dialog-close.component";
+import { EntityArchivedInfoComponent } from "../../../core/entity-details/entity-archived-info/entity-archived-info.component";
 
 /**
  * Component responsible for displaying the Note creation/view window
@@ -46,6 +47,7 @@ import { DialogCloseComponent } from "../../../core/common-components/dialog-clo
     DialogButtonsComponent,
     MatMenuModule,
     DialogCloseComponent,
+    EntityArchivedInfoComponent,
   ],
   standalone: true,
   encapsulation: ViewEncapsulation.None,

src/app/child-dev-project/schools/child-school-overview/child-school-overview.component.ts

@@ -1,4 +1,4 @@
-import { Component, OnInit } from "@angular/core";
+import { Component, Input, OnInit } from "@angular/core";
 import { DynamicComponent } from "../../../core/config/dynamic-components/dynamic-component.decorator";
 import { Child } from "../../children/model/child";
 import { School } from "../model/school";
@@ -41,6 +41,7 @@ export class ChildSchoolOverviewComponent
   implements OnInit
 {
   mode: "child" | "school" = "child";
+  @Input() showInactive = false;
 
   constructor(private childrenService: ChildrenService) {
     super(null, null);
@@ -60,7 +61,7 @@ export class ChildSchoolOverviewComponent
     this.switchRelatedEntityColumnForMode();
 
     await this.loadData();
-    super.filterActiveInactive();
+    super.onIsActiveFilterChange(this.showInactive);
   }
 
   private inferMode(entity: Entity): "child" | "school" {

src/app/core/basic-datatypes/array/array.datatype.ts

@@ -97,6 +97,25 @@ export class ArrayDatatype<
       ),
     );
   }
+
+  async anonymize(
+    value: EntityType[],
+    schemaField: EntitySchemaField,
+    parent,
+  ): Promise<any[]> {
+    const arrayElementDatatype: DefaultDatatype =
+      this.schemaService.getDatatypeOrDefault(schemaField.innerDataType);
+
+    const mappedPromises = value.map(async (el) =>
+      arrayElementDatatype.anonymize(
+        el,
+        generateSubSchemaField(schemaField),
+        parent,
+      ),
+    );
+
+    return Promise.all(mappedPromises);
+  }
 }
 
 /**

src/app/core/basic-datatypes/date-with-age/edit-age/edit-age.component.html

@@ -1,5 +1,10 @@
 <div [formGroup]="parent">
-  <mat-form-field style="width: 70%">
+  <mat-form-field
+    style="width: 70%"
+    *ngIf="
+      !(entity.anonymized && propertySchema.anonymize === 'retain-anonymized')
+    "
+  >
     <mat-label>{{ label }}</mat-label>
     <input
       matInput
@@ -16,6 +21,7 @@
       <app-error-hint [form]="formControl"></app-error-hint>
     </mat-error>
   </mat-form-field>
+
   <mat-form-field style="width: 30%">
     <mat-label i18n="Placeholder for the input that displays the age"
       >Age</mat-label

src/app/core/basic-datatypes/date/date.datatype.spec.ts

@@ -7,4 +7,12 @@ describe("Schema data type: date", () => {
     new Date(2023, 10, 25),
     new Date(2023, 10, 25),
   );
+
+  it("should anonymize dates and only retain year", async () => {
+    const datatype = new DateDatatype();
+    const testDate = new Date(2023, 10, 25);
+
+    const actualAnonymized = await datatype.anonymize(testDate);
+    expect(actualAnonymized).toEqual(new Date(2023, 6, 1));
+  });
 });

src/app/core/basic-datatypes/date/date.datatype.ts

@@ -71,4 +71,10 @@ export class DateDatatype<DBFormat = any> extends DefaultDatatype<
       return undefined;
     }
   }
+
+  async anonymize(value: Date): Promise<Date> {
+    // normalize to 01.06. of the year, which has less statistical distortion than 01.01.
+    // (roughly half the dates before anonymization will be earlier and half will be later)
+    return new Date(value.getFullYear(), 6, 1);
+  }
 }