feat(*): cascading delete and anonymize of related entities (#2041)

closes #220

---------
This functionality has been developed for the project “codo”.
codo is developed under the projects “Landungsbrücken – Patenschaften in Hamburg stärken” and “openTransfer Patenschaften”. It is funded through the program “Menschen stärken Menschen” by the German Federal Ministry of Family Affairs, Senior Citizens, Women and Youth.
More information at https://github.com/codo-mentoring

“Landungsbrücken – Patenschaften in Hamburg stärken” is a project of BürgerStiftung Hamburg in cooperation with the Mentor.Ring Hamburg. With a mix of networking opportunities, capacity building and financial support the project strengthens Hamburg’s scene of mentoring projects since its founding in 2016.

The “Stiftung Bürgermut” foundation since 2007 supports the digital and real exchange of experiences and connections of active citizens. Within the federal program “Menschen stärken Menschen” the foundation as part of its program “openTransfer Patenschaften” offers support services for connecting, spreading and upskilling mentoring organisations across Germany.

Diese Funktion wurde entwickelt für das Projekt codo.
codo wird entwickelt im Rahmen der Projekte Landungsbrücken – Patenschaften in Hamburg stärken und openTransfer Patenschaften. Er ist gefördert durch das Bundesprogramm Menschen stärken Menschen des Bundesministeriums für Familie, Senioren, Frauen und Jugend.
Mehr Informationen unter https://github.com/codo-mentoring

“Landungsbrücken – Patenschaften in Hamburg stärken” ist ein Projekt der BürgerStiftung Hamburg in Kooperation mit dem Mentor.Ring Hamburg. Mit einer Mischung aus Vernetzungsangeboten, Qualifizierungsmaßnahmen und finanzieller Förderung stärkt das Projekt die Hamburger Szene der Patenschaftsprojekte seit der Gründung im Jahr 2016.

Die Stiftung Bürgermut fördert seit 2007 den digitalen und realen Erfahrungsaustausch und die Vernetzung von engagierten Bürger:innen. Innerhalb des Bundesprogramms „Menschen stärken Menschen” bietet die Stiftung im Rahmen ihres Programms openTransfer Patenschaften Unterstützungsleistungen zur Vernetzung, Verbreitung und Qualifizierung von Patenschafts- und Mentoringorganisationen bundesweit.

Co-authored-by: codo-mentoring <117934638+codo-mentoring@users.noreply.github.com>
Co-authored-by: Simon <simon@aam-digital.com>

doc/compodoc_sources/concepts/entities.md

@@ -0,0 +1,10 @@
+# Entities & Entity Schema
+-----
+For us, an "Entity" is an object in the database (and a representation of something in the user's real world, e.g. a "Child" or "School").
+Entities are at the core of the Aam Digital platform and the primary way to customize the system is to adapt and add new entity types.
+
+The Entity Schema defines the data structure as well as how it is displayed in the UI.
+Entity instances also have some generic functionality inherited from the `Entity` base class.
+
+------
+_see the sub-pages here for details of the various concepts related to the Entity system_

doc/compodoc_sources/concepts/entity-anonymization.md

@@ -0,0 +1,39 @@
+# Archive / Anonymize Entities
+-----
+Any entity can be archived (i.e. marked as inactive and hidden from UI by default) or anonymized (i.e. discarding most data and keeping a few selected properties for statistical reports).
+This is often preferable to deleting a record completely. Deleting data also affects statistical reports, even for previous time periods.
+By anonymizing records, all personal identifiable data can be removed and the remaining stub record can be stored indefinitely, as it is not subject to data protection regulations like GDPR anymore.
+
+Anonymization is configured as part of the entity schema.
+Data of fields that are not explicitly marked to be retained during anonymization is always deleted (anonymization by default).
+
+To keep some data even after the user "anonymized" a record, configure the `anonymize` property of the `@DatabaseField` decorator:
+- `anonymize: "retain"` will keep this field unchanged and prevent it from being deleted
+- `anonymize: "retain-anonymized"` will trigger a special "partial" deletion that depends on the dataType (e.g. date types will be changed to 1st July of the given year, thereby largely removing details but keeping data to calculate a rough age)
+
+
+## Cascading anonymization / deletion
+Relationships between entities are automatically handled when the user anonymizes or deletes an entity.
+Any related entities that reference the anonymized/deleted entity are checked 
+and - depending on their configured role - may be updated or anonymized as well.
+
+The logic follows the scenarios shown below:
+![](../../images/cascading-delete.png)
+
+
+## Data Protection & GDPR regarding anonymization / pseudonomyzation
+The "anonymize" function is implemented specifically for data protection rules requiring to delete personal data.
+According to the EU's "General Data Protection Regulation" (GDPR) "anonymous" data does not fall under its regulations:
+
+- GDPR is not applicable to anonymous data: "The principles of data protection should therefore not apply to [...] personal data rendered anonymous in such a manner that the data subject is not or no longer identifiable." [<sup>GDPR Recital 26</sup>](https://gdpr-info.eu/recitals/no-26/)
+  - "To determine whether a natural person is identifiable, account should be taken of all the means reasonably likely to be used, such as singling out, either by the controller or by another person to identify the natural person directly or indirectly."
+  - "To ascertain whether means are reasonably likely to be used to identify the natural person, account should be taken of all objective factors, such as the costs of and the amount of time required for identification, taking into consideration the available technology at the time of the processing and technological developments."
+- "Pseudonymisation enables the personal data to become unidentifiable unless more information is available whereas anonymization allows the processing of personal data to irreversibly prevent re-identification." [<sup>source</sup>](https://www.privacycompany.eu/blogpost-en/what-are-the-differences-between-anonymisation-and-pseudonymisation)
+- _also see this [good overview of anonymization misunderstandings and considerations](https://edps.europa.eu/system/files/2021-04/21-04-27_aepd-edps_anonymisation_en_5.pdf)_
+
+In the case of records being retained "anonymized" in Aam Digital, we provide a context that makes re-identification even harder:
+- only authorized users of the system can access even the anonymized record (where only a few properties have been retained). Unless the organisation actively shares the data, it remains as securely protected as the personal data managed in Aam Digital.
+- those authorized users with access to the anonymized records (and therefor a theoretical chance to attempt re-identification) are team members of an organization. They have been screened to be responsible persons and are usually legally bound to keep information confidential.
+- by default only a few, explicitly selected properties in anonymized records are retained (data minimization by default). As such, both re-identification likelihood and the impact in case of re-identification are reduced as far as possible.
+
+--> If our anonymization process is configured thoughfully on a case by case basis to only retain a few data fields that are not easy indirect identifiers, it seems reasonably unlikely that the person can be identified after the anonymization process. Therefore, GDPR should not apply to these records and it is legitimate to retain these for statistical reporting.

doc/compodoc_sources/concepts/entity-schema-system.md

@@ -1,13 +1,5 @@
-# Entities & Entity Schema 
+# Entity Schema 
 -----
-For us, an "Entity" is an object in the database (and a representation of something in the user's real world, e.g. a "Child" or "School").
-Entities are at the core of the Aam Digital platform and the primary way to customize the system is to adapt and add new entity types.
-
-The Entity Schema defines the data structure as well as how it is displayed in the UI.
-Entity instances also have some generic functionality inherited from the `Entity` base class.
-
-
-## Entity Schema
 The Entity Schema defines details of the properties of an entity type.
 We define an entity type and its schema in code through a plain TypeScript class and some custom annotations.
 Read more on the background and practical considerations in [How to create a new Entity Type](../how-to-guides/create-a-new-entity-type.html).
@@ -64,32 +56,3 @@ The `description` field allows adding further explanation which will be displaye
 ### Metadata (created, updated)
 Each record automatically holds basic data of timestamp and user who created and last updated the record.
 (see `Entity` class)
-
-### Archive / Anonymize
-Any entity can be archived (i.e. marked as inactive and hidden from UI by default) or anonymized (i.e. discarding most data and keeping a few selected properties for statistical reports).
-This is often preferable to deleting a record completely. Deleting data also affects statistical reports, even for previous time periods.
-By anonymizing records, all personal identifiable data can be removed and the remaining stub record can be stored indefinitely, as it is not subject to data protection regulations like GDPR anymore.
-
-Anonymization is configured as part of the entity schema.
-Data of fields that are not explicitly marked to be retained during anonymization is always deleted (anonymization by default).
-
-To keep some data even after the user "anonymized" a record, configure the `anonymize` property of the `@DatabaseField` decorator:
-- `anonymize: "retain"` will keep this field unchanged and prevent it from being deleted
-- `anonymize: "retain-anonymized"` will trigger a special "partial" deletion that depends on the dataType (e.g. date types will be changed to 1st July of the given year, thereby largely removing details but keeping data to calculate a rough age)
-
-#### Data Protection & GDPR regarding anonymization / pseudonomyzation
-The "anonymize" function is implemented specifically for data protection rules requiring to delete personal data.
-According to the EU's "General Data Protection Regulation" (GDPR) "anonymous" data does not fall under its regulations:
-
-- GDPR is not applicable to anonymous data: "The principles of data protection should therefore not apply to [...] personal data rendered anonymous in such a manner that the data subject is not or no longer identifiable." [<sup>GDPR Recital 26</sup>](https://gdpr-info.eu/recitals/no-26/)
-  - "To determine whether a natural person is identifiable, account should be taken of all the means reasonably likely to be used, such as singling out, either by the controller or by another person to identify the natural person directly or indirectly."
-  - "To ascertain whether means are reasonably likely to be used to identify the natural person, account should be taken of all objective factors, such as the costs of and the amount of time required for identification, taking into consideration the available technology at the time of the processing and technological developments."
-- "Pseudonymisation enables the personal data to become unidentifiable unless more information is available whereas anonymization allows the processing of personal data to irreversibly prevent re-identification." [<sup>source</sup>](https://www.privacycompany.eu/blogpost-en/what-are-the-differences-between-anonymisation-and-pseudonymisation)
-- _also see this [good overview of anonymization misunderstandings and considerations](https://edps.europa.eu/system/files/2021-04/21-04-27_aepd-edps_anonymisation_en_5.pdf)_
-
-In the case of records being retained "anonymized" in Aam Digital, we provide a context that makes re-identification even harder:
-- only authorized users of the system can access even the anonymized record (where only a few properties have been retained). Unless the organisation actively shares the data, it remains as securely protected as the personal data managed in Aam Digital.
-- those authorized users with access to the anonymized records (and therefor a theoretical chance to attempt re-identification) are team members of an organization. They have been screened to be responsible persons and are usually legally bound to keep information confidential.
-- by default only a few, explicitly selected properties in anonymized records are retained (data minimization by default). As such, both re-identification likelihood and the impact in case of re-identification are reduced as far as possible.
-
---> If our anonymization process is configured thoughfully on a case by case basis to only retain a few data fields that are not easy indirect identifiers, it seems reasonably unlikely that the person can be identified after the anonymization process. Therefore, GDPR should not apply to these records and it is legitimate to retain these for statistical reporting.

doc/compodoc_sources/summary.json

@@ -38,8 +38,18 @@
         "file": "concepts/extendability.md"
       },
       {
-        "title": "Entity Schema",
-        "file": "concepts/entity-schema-system.md"
+        "title": "Entity System",
+        "file": "concepts/entities.md",
+        "children": [
+          {
+            "title": "Entity Schema",
+            "file": "concepts/entity-schema-system.md"
+          },
+          {
+            "title": "Archiving, Anonymizing and Deleting Entities",
+            "file": "concepts/entity-anonymization.md"
+          }
+        ]
       },
       {
         "title": "Configuration",

src/app/child-dev-project/attendance/add-day-attendance/roll-call/roll-call.component.html

@@ -5,7 +5,7 @@
     [value]="(currentIndex / children.length) * 100"
   ></mat-progress-bar>
 
-  <div class="progress-nav">
+  <div class="progress-nav flex-row">
     <div style="margin-left: -10px">
       <button
         mat-icon-button
@@ -25,12 +25,12 @@
       </button>
     </div>
     <div
-      class="progress-label"
+      class="progress-label flex-grow"
       [style.visibility]="currentIndex < children.length ? 'visible' : 'hidden'"
     >
       {{ currentIndex + 1 }} / {{ children.length }}
     </div>
-    <div style="margin-right: -10px">
+    <div>
       <button
         mat-icon-button
         (click)="goToNext()"
@@ -48,6 +48,19 @@
         <fa-icon icon="angle-double-right"></fa-icon>
       </button>
     </div>
+
+    <div *ngIf="inactiveParticipants?.length > 0" style="margin-right: -10px">
+      <button
+        mat-icon-button
+        (click)="includeInactive()"
+        [disabled]="isFinished"
+        color="warn"
+        matTooltip="Excluded some archived participants. Click to include."
+        i18n-matTooltip
+      >
+        <fa-icon icon="warning"></fa-icon>
+      </button>
+    </div>
   </div>
 
   <app-child-block

src/app/child-dev-project/attendance/add-day-attendance/roll-call/roll-call.component.scss

@@ -39,8 +39,6 @@
 
 .progress-nav {
   margin-top: -4px;
-  display: flex;
-  flex-direction: row;
   justify-content: space-between;
   align-items: center;
 }

src/app/child-dev-project/attendance/add-day-attendance/roll-call/roll-call.component.ts

@@ -33,6 +33,8 @@ import {
 } from "@angular/platform-browser";
 import Hammer from "hammerjs";
 import { ConfigurableEnumService } from "../../../../core/basic-datatypes/configurable-enum/configurable-enum.service";
+import { MatTooltipModule } from "@angular/material/tooltip";
+import { ConfirmationDialogService } from "../../../../core/common-components/confirmation-dialog/confirmation-dialog.service";
 
 // Only allow horizontal swiping
 @Injectable()
@@ -69,6 +71,7 @@ class HorizontalHammerConfig extends HammerGestureConfig {
     NgClass,
     RollCallTabComponent,
     HammerModule,
+    MatTooltipModule,
   ],
   providers: [
     {
@@ -114,12 +117,14 @@ export class RollCallComponent implements OnChanges {
   availableStatus: AttendanceStatusType[];
 
   children: Child[] = [];
+  inactiveParticipants: Child[];
 
   constructor(
     private enumService: ConfigurableEnumService,
     private entityMapper: EntityMapperService,
     private formDialog: FormDialogService,
     private loggingService: LoggingService,
+    private confirmationDialog: ConfirmationDialogService,
   ) {}
 
   async ngOnChanges(changes: SimpleChanges) {
@@ -165,8 +170,9 @@ export class RollCallComponent implements OnChanges {
 
   private async loadParticipants() {
     this.children = [];
+    this.inactiveParticipants = [];
     for (const childId of this.eventEntity.children) {
-      let child;
+      let child: Child;
       try {
         child = await this.entityMapper.load(Child, childId);
       } catch (e) {
@@ -179,7 +185,12 @@ export class RollCallComponent implements OnChanges {
         this.eventEntity.removeChild(childId);
         continue;
       }
-      this.children.push(child);
+
+      if (child.isActive) {
+        this.children.push(child);
+      } else {
+        this.inactiveParticipants.push(child);
+      }
     }
     this.sortParticipants();
   }
@@ -254,4 +265,15 @@ export class RollCallComponent implements OnChanges {
   showDetails() {
     this.formDialog.openFormPopup(this.eventEntity, [], NoteDetailsComponent);
   }
+
+  async includeInactive() {
+    const confirmation = await this.confirmationDialog.getConfirmation(
+      $localize`Also include archived participants?`,
+      $localize`This event has some participants who are "archived". We automatically remove them from the attendance list for you. Do you want to also include archived participants for this event?`,
+    );
+    if (confirmation) {
+      this.children = [...this.children, ...this.inactiveParticipants];
+      this.inactiveParticipants = [];
+    }
+  }
 }

src/app/child-dev-project/attendance/model/recurring-activity.ts

@@ -55,7 +55,6 @@ export class RecurringActivity extends Entity {
     validators: {
       required: true,
     },
-    anonymize: "retain",
   })
   title: string = "";
 
@@ -68,7 +67,6 @@ export class RecurringActivity extends Entity {
     label: $localize`:Label for the interaction type of a recurring activity:Type`,
     dataType: "configurable-enum",
     innerDataType: INTERACTION_TYPE_CONFIG_ID,
-    anonymize: "retain",
   })
   type: InteractionType;
 
@@ -77,7 +75,6 @@ export class RecurringActivity extends Entity {
     label: $localize`:Label for the participants of a recurring activity:Participants`,
     dataType: "entity-array",
     additional: Child.ENTITY_TYPE,
-    anonymize: "retain",
   })
   participants: string[] = [];
 
@@ -86,7 +83,6 @@ export class RecurringActivity extends Entity {
     label: $localize`:Label for the linked schools of a recurring activity:Groups`,
     dataType: "entity-array",
     additional: School.ENTITY_TYPE,
-    anonymize: "retain",
   })
   linkedGroups: string[] = [];
 
@@ -95,7 +91,6 @@ export class RecurringActivity extends Entity {
     label: $localize`:Label for excluded participants of a recurring activity:Excluded Participants`,
     dataType: "entity-array",
     additional: Child.ENTITY_TYPE,
-    anonymize: "retain",
   })
   excludedParticipants: string[] = [];
 
@@ -104,7 +99,6 @@ export class RecurringActivity extends Entity {
     label: $localize`:Label for the assigned user(s) of a recurring activity:Assigned user(s)`,
     dataType: "entity-array",
     additional: User.ENTITY_TYPE,
-    anonymize: "retain",
   })
   assignedTo: string[] = [];
 

src/app/child-dev-project/children/aser/model/aser.ts

@@ -22,15 +22,26 @@ import { SkillLevel } from "./skill-levels";
 import { WarningLevel } from "../../../warning-level";
 import { ConfigurableEnumDatatype } from "../../../../core/basic-datatypes/configurable-enum/configurable-enum-datatype/configurable-enum.datatype";
 import { PLACEHOLDERS } from "../../../../core/entity/schema/entity-schema-field";
+import { Child } from "../../model/child";
 
 @DatabaseEntity("Aser")
 export class Aser extends Entity {
-  @DatabaseField() child: string; // id of Child entity
+  static override hasPII = true;
+
+  @DatabaseField({
+    dataType: "entity",
+    additional: Child.ENTITY_TYPE,
+    entityReferenceRole: "composite",
+  })
+  child: string;
+
   @DatabaseField({
     label: $localize`:Label for date of the ASER results:Date`,
     defaultValue: PLACEHOLDERS.NOW,
+    anonymize: "retain-anonymized",
   })
   date: Date;
+
   @DatabaseField({
     label: $localize`:Label of the Hindi ASER result:Hindi`,
     dataType: "configurable-enum",
@@ -55,6 +66,7 @@ export class Aser extends Entity {
     innerDataType: "math-levels",
   })
   math: SkillLevel;
+
   @DatabaseField({
     label: $localize`:Label for the remarks of a ASER result:Remarks`,
   })

src/app/child-dev-project/children/health-checkup/model/health-check.ts

@@ -19,18 +19,24 @@ import { Entity } from "../../../../core/entity/model/entity";
 import { DatabaseEntity } from "../../../../core/entity/database-entity.decorator";
 import { DatabaseField } from "../../../../core/entity/database-field.decorator";
 import { WarningLevel } from "../../../warning-level";
+import { Child } from "../../model/child";
 
 /**
  * Model Class for the Health Checks that are taken for a Child.
  * It stores the Child's ID in a String and both, the height and weight in cm as a number, and the Date
  */
 @DatabaseEntity("HealthCheck")
 export class HealthCheck extends Entity {
+  static override hasPII = true;
+
   static create(contents: Partial<HealthCheck>) {
     return Object.assign(new HealthCheck(), contents);
   }
 
   @DatabaseField({
+    dataType: "entity",
+    additional: Child.ENTITY_TYPE,
+    entityReferenceRole: "composite",
     anonymize: "retain",
   })
   child: string;

src/app/child-dev-project/children/model/child.ts

@@ -31,6 +31,7 @@ export class Child extends Entity {
   static label = $localize`:label for entity:Participant`;
   static labelPlural = $localize`:label (plural) for entity:Participants`;
   static color = "#1565C0";
+  static override hasPII = true;
 
   static create(name: string): Child {
     const instance = new Child();