Adding sentry data synchronization

[rfuelsh]

See more details below

[rfuelsh]

@Voxelot @tirkesi -

Here are the results from the deployment of sentry-10 in beta3 from a clone snapshot of sentry-3 in beta3

Ref PRs:

• Infrastructure Repo: Create AWS EBS Snapshot script infrastructure#109
• Fuel-Deployment Repo: https://github.com/FuelLabs/fuel-deployment/pull/145
• Fuel-Deployment CI/CD Run: https://github.com/FuelLabs/fuel-deployment/actions/runs/4711498568/jobs/8355904294

Steps to re-create:

• Ran script: https://github.com/FuelLabs/infrastructure/blob/add-ebs-snapshot-script/scripts/create-ebs-snapshot.sh

NAMESPACE= beta3

SERVICE = sentry-3

this created a EBS snapshot of sentry-3 - you get the "fuel_core_pvc_snapshot_ref" from the running the script which is an output along the lines of "snap-*"

• This is then being used in https://github.com/FuelLabs/fuel-deployment/blob/feature/beta3-sentry-4-6/config/eks/fuel-prod-1/beta3-sentry-10.env

these envs are updated

fuel_core_pvc_clone_enabled=true
fuel_core_pvc_snapshot_id="beta3-sentry-3-clone"
fuel_core_pvc_snapshot_ref="snap-030f4ceb6089d754d"

fuel_core_pvc_snapshot_id env can be anything but keeping a syntax like "namespace-service_name-clone" is best

• Deploy the regular fuel-core helm chart using this fuel-core PR and the fuel-deployment PR- https://github.com/FuelLabs/fuel-deployment/pull/145

Resources created:

 % kubectl get VolumeSnapshot
NAME                   READYTOUSE   SOURCEPVC   SOURCESNAPSHOTCONTENT   RESTORESIZE   SNAPSHOTCLASS   SNAPSHOTCONTENT        CREATIONTIME   AGE
beta3-sentry-3-clone   true                     beta3-sentry-3-clone    100Gi         ebs-csi-aws     beta3-sentry-3-clone   22m            10m

Which further creates a pvc for sentry-10 based on the ebs snapshot from sentry-3

% kubectl get pvc sentry-10-db-pv-claim -o yaml        
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  annotations:
    meta.helm.sh/release-name: sentry-10
    meta.helm.sh/release-namespace: beta3
    pv.kubernetes.io/bind-completed: "yes"
    pv.kubernetes.io/bound-by-controller: "yes"
    volume.beta.kubernetes.io/storage-provisioner: ebs.csi.aws.com
    volume.kubernetes.io/selected-node: ip-172-34-28-209.ec2.internal
    volume.kubernetes.io/storage-provisioner: ebs.csi.aws.com
  creationTimestamp: "2023-04-16T04:39:29Z"
  finalizers:
  - kubernetes.io/pvc-protection
  labels:
    app.kubernetes.io/managed-by: Helm
    heritage: Helm
    release: sentry-10
  name: sentry-10-db-pv-claim
  namespace: beta3
  resourceVersion: "99011086"
  uid: 6ee4d9ab-84a9-4e86-9e8a-cebea742e07d
spec:
  accessModes:
  - ReadWriteOnce
  dataSource:
    apiGroup: snapshot.storage.k8s.io
    kind: VolumeSnapshot
    name: beta3-sentry-3-clone
  dataSourceRef:
    apiGroup: snapshot.storage.k8s.io
    kind: VolumeSnapshot
    name: beta3-sentry-3-clone
  resources:
    requests:
      storage: 100Gi
  storageClassName: gp3
  volumeMode: Filesystem
  volumeName: pvc-6ee4d9ab-84a9-4e86-9e8a-cebea742e07d
status:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 100Gi
  phase: Bound

notice the dataSourceRef

Further more the sentry-10 within 3 minutes starts to sync with the other sentries very quickly

% kubectl get pods | grep sentry-10 
sentry-10-k8s-6688cf7f9b-b72kv     1/1     Running   2 (18m ago)   20m

first initial minute

{"timestamp":"2023-04-16T04:42:27.969866Z","level":"INFO","fields":{"message":"Status change from: Processing(263954..=264062), to: Processing(263955..=264062)"},"target":"fuel_core_sync::state","line_number":194,"span":{"height":263954,"self":"State { status: Processing(263954..=264062) }","name":"commit"},"spans":[{"name":"sentry-10","name":"new"},{"service":"fuel-core-sync","name":"initialize_loop"},{"name":"sync"},{"height":263954,"self":"State { status: Processing(263954..=264062) }","name":"commit"}]}

after 4 minutes

{"timestamp":"2023-04-16T04:47:48.848581Z","level":"INFO","fields":{"message":"Status change from: Processing(264104..=264104), to: Committed(264104)"},"target":"fuel_core_sync::state","line_number":194,"span":{"height":264104,"self":"State { status: Processing(264104..=264104) }","name":"commit"},"spans":[{"name":"sentry-10","name":"new"},{"service":"fuel-core-sync","name":"initialize_loop"},{"name":"sync"},{"height":264104,"self":"State { status: Processing(264104..=264104) }","name":"commit"}]}

Main point:

This process is not perfect, as the ebs snapshot takes 5 minutes or so to be fully 100% and we are only getting a snapshot of the data at the time we run the script

so there is still a few minutes : about 5 minutes I experienced for the sentry to full catch up but this is better than the old process

[rfuelsh]

The case of not deploying volume snapshots and PVC that use volumesnapshots which is what our regular fuel-core passed in fuel-dev: https://github.com/FuelLabs/fuel-deployment/actions/runs/4711671457/jobs/8356182509 for authority node

So helm chart changes are valid