From 77dc52789d5937089f84da881bb1086e865d919e Mon Sep 17 00:00:00 2001 From: Erica Sadun Date: Thu, 30 Jan 2025 15:42:20 -0700 Subject: [PATCH 1/3] EDU-3861: HAN prototyping --- .../cloud/high-availability/enable.mdx | 296 ++++++++++++++++++ .../cloud/high-availability/faq.mdx | 219 +++++++++++++ .../cloud/high-availability/guarantees.mdx | 61 ++++ .../cloud/high-availability/how-it-works.mdx | 197 ++++++++++++ .../cloud/high-availability/index.mdx | 113 +++++++ .../cloud/high-availability/monitor.mdx | 81 +++++ .../cloud/high-availability/work-file.txt | 67 ++++ sidebars.js | 16 + 8 files changed, 1050 insertions(+) create mode 100644 docs/production-deployment/cloud/high-availability/enable.mdx create mode 100644 docs/production-deployment/cloud/high-availability/faq.mdx create mode 100644 docs/production-deployment/cloud/high-availability/guarantees.mdx create mode 100644 docs/production-deployment/cloud/high-availability/how-it-works.mdx create mode 100644 docs/production-deployment/cloud/high-availability/index.mdx create mode 100644 docs/production-deployment/cloud/high-availability/monitor.mdx create mode 100644 docs/production-deployment/cloud/high-availability/work-file.txt diff --git a/docs/production-deployment/cloud/high-availability/enable.mdx b/docs/production-deployment/cloud/high-availability/enable.mdx new file mode 100644 index 0000000000..839b0fba79 --- /dev/null +++ b/docs/production-deployment/cloud/high-availability/enable.mdx @@ -0,0 +1,296 @@ +--- +id: enable +title: Enable high availability +sidebar_label: Enable high availability +slug: /cloud/high-availability/choosing-high-availability +description: Temporal Cloud's High-Availability Namespaces offer automated failover, synchronized data replication, and high availability for workloads requiring disaster-tolerant deployment and 99.99% uptime. Use Global Namespace for self-hosted. +tags: + - Temporal Cloud + - Production + - High availability +keywords: + - availability + - explanation + - failover + - high-availability + - multi-region + - multi-region namespace + - namespaces + - temporal-cloud + - term +--- +import { RelatedReadContainer, RelatedReadItem } from '@site/src/components/related-read/RelatedRead'; + +:::tip Support, stability, and dependency info + +High-availability Namespaces are in [Public Preview](/evaluate/development-production-features/release-stages#public-preview) for Temporal Cloud. + +::: + +
**Some audits, updates. Needs intros, re-org. Suggest breaking down into "opting in", "setting up (worker and privatelink)", and "testing" because the content feels really mixed up right now and too long and the metrics section is now a little too short**
+ +You can enable the high-availability Namespace feature for your existing Namespace by [adding a second zone](#add-zones) to your Namespace. +After adding the second zone, Temporal Cloud begins data replication for your new standby replica. +Temporal Cloud notifies you once the replication has caught up and both Namespace zones are in sync. + +**Advantages of using a high-availability Namespace:** + +- No manual deployment or configuration needed, just simple push-button operation. +- Open Workflows continue in the standby region with minimal interruption and data loss. +- No changes needed for Worker and Workflow code during setup or failover. +- 99.99% Contractual SLA. + +### Create a multi-region Namespace {#create} + +The following sections explain how to create a new multi-region Namespace (MRN). +MRNs provide multi-region deployment backed by Temporal's data replication and active-standby features. + +:::tip + +While reading through this coverage, remember that pairing is currently limited to regions within the same continent. + +::: + +#### Temporal Cloud Web UI + +During Namespace creation, specify the first region for the Namespace. +Then, select the “Add a region” option. +Adding a second region enables multi-region Namespace capabilities. + +#### Temporal 'tcld' CLI + +Start with the following command to create the new multi-region Namespace: + +``` +tcld namespace create \ + --namespace . \ + --region +``` + +Include both regions by specifying the [region codes](/cloud/service-availability) as arguments to the `--region` flags. +Before pressing return, add your authentication credentials. For example, `--ca-certificate-file `. + + + + +## Upgrade an existing single-zone Namespace for high-availability functionality {#add-zones} + +You can upgrade existing ssingle-zone Namespace for high-availability by adding a standby zone. +The following sections show you how. + +
**The following material has not been audited for MRN/HAN**
+ +#### Temporal Cloud Web UI + +To upgrade an existing Namespace to a multi-region Namespace: + +1. Visit Temporal Cloud [Namespaces](https://cloud.temporal.io/namespaces) in your Web browser +1. Navigate to the Namespace details page +1. Select the “Add a region” button. +1. Select the standby region you want to add to this Namespace + +You will see an estimated time for replication. +This time is based on your selection and the size and scale of Workflows in your Namespace, +An email alert is sent once your multi-region Namespace is ready for use. + +#### Temporal 'tcld' CLI + +At the command line, enter: + +``` +tcld namespace add-region \ + --namespace . \ + --region +``` + +Specify the region code for the new region to add. +Before pressing return, add your authentication credentials. For example, `--ca-certificate-file `. +An email alert is sent once your multi-region Namespace is ready for use. + +### Discontinuing multi-region availability {#discontinuing} + +Disabling multi-region removes the high availability and automatic failover features that provide Temporal's highest service level agreement. +To disable the feature and end charges, users must contact [Temporal Support](https://support.temporal.io) directly. +MRN-specific charges for replication will stop once this decommissioning procedure completes. + +- When making your request you must let us know which region you want the Namespace to land in after removing the standby region. +- If you cease services in the middle of the month, your Namespace will be converted to a single region Namespace within 1 business day. +- Temporal won't retain replicated data in the standby region once multi-region has been disabled. +- After disabling multi-region, Temporal Cloud cannot re-enable the feature for a given Namespace for seven days. + +## Triggering failovers {#triggering-failovers} + +Failovers happen automatically in Temporal when a regional outage or disaster affects a multi-region Namespace. +You can also trigger a failover based on custom alerts or for testing purposes. +This section explains how to manually trigger a failover and what to expect afterward. + +Regular failover testing ensures your app can handle disruptions and continue running smoothly in production. +Whether responding to incident warnings or conducting tests, follow the steps in the next sections to move your active Namespace to its standby region and learn how to handle failovers effectively. + +For details on how Temporal detects conditions and triggers failovers automatically, see [Failovers](/cloud/multi-region/#failovers). + +:::warning Check Your Replication Lag + +Always check the [metric replication lag](/production-deployment/cloud/metrics/reference#temporal_cloud_v0_replication_lag_bucket) before initiating a failover. +A forced failover when there is a large replication lag has a higher likelihood of rolling back Workflow progress. + +::: + +**Performing manual failovers** + +You can trigger a failover manually using the Temporal Cloud Web UI or the `tcld` CLI, depending on your preference and setup. +The following table outlines the steps for each method: + +| Method | Instructions | +| ------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| **Temporal Cloud Web UI** | 1. Visit the [Namespace page](https://cloud.temporal.io/namespaces) on the Temporal Cloud Web UI.
2. Navigate to your Namespace details page and select the **Trigger a failover** option from the menu.
3. After confirming, the failover will be initiated. | +| **Temporal `tcld` CLI** | To manually trigger a failover, run the following command in your terminal:
tcld namespace failover \
    --namespace \.\ \
    --region \ | + +**Post-failover event information** + +After any failover, whether triggered by you or by Temporal, event information appears in both the [Temporal Cloud Web UI](https://cloud.temporal.io/namespaces) (on the Namespace detail page) and in your audit logs. +The audit log entry for Failover uses the `"operation": "FailoverNamespace"` event. +After failover, the Namespace is active in the new region. + +You don't need to monitor Temporal Cloud's failover response in real-time. +Whenever there is a failover event, users with the Account Owner and Global Admin roles automatically receive an alert email. + +**Failbacks** + +After Temporal-initiated failovers, Temporal Cloud shifts Workflow Execution processing back to the original region that was active before the incident (a "failback") once the incident is resolved. + +**Reasons to test failing over** + +Microservices and external dependencies will fail at some point. +Testing failovers ensures your app can handle these failures effectively. +Temporal recommends regular and periodic failover testing for mission-critical applications in production. +By testing in non-emergency conditions, you verify that your app continues to function even when parts of the infrastructure fail. + +:::tip Safety First + +If this is your first time performing a failover test, run it with a test-specific namespace and application. +This helps you gain operational experience before applying it to your production environment. +Practice runs help ensure the process runs smoothly during real incidents in production. + +::: + +Trigger testing can: + +- **Validate multi-region deployments**: + In multi-region setups, failover testing ensures your app can run from another region when the primary region experiences outages. + This maintains high availability in mission-critical deployments. + Manual testing confirms the failover mechanism works as expected, so your system handles regional outages or disasters effectively. + +- **Assess replication lag**: + Monitoring [replication lag](/cloud/multi-region#metrics-operations) between regions is crucial in multi-region setups. + Check the lag before initiating a failover to avoid rolling back Workflow progress. + Manual testing helps you practice this critical step and understand its impact. + When there's no real incident, the switch over (recovery) should happen almost instantly. + +- **Assess recovery time**: + Manual testing helps you measure actual recovery time. + You can check if it meets your expected Recovery Time Objective (RTO) of 20 minutes or less, as stated in the [Multi-region Namespace SLA](/cloud/multi-region#sla). + +- **Identify potential issues**: + Failover testing uncovers problems not visible during normal operation. + This includes issues like [backlogs and capacity planning](https://temporal.io/blog/workers-in-production#testing-failure-paths-2438) and how external dependencies behave during a failover event. + +- **Validate fault-oblivious programming**: + Temporal uses a "fault-oblivious programming" model, where your app doesn’t need to explicitly handle many types of failures. + Testing failovers ensures that this model works as expected in your app. + +- **Operational readiness**: + Regular testing familiarizes your team with the failover process, improving their ability to handle real incidents when they arise. + +Testing failovers regularly ensures your Temporal-based applications remain resilient and reliable, even when infrastructure fails. + + +## Worker Deployment {#worker-deployment} + +Enabling the multi-region Namespace does not require specific Worker configuration. +The process is invisible to the Workers. +When a Namespace fails over to the standby region, the DNS redirection orchestrated by Temporal ensures that your existing Workers continue to poll the Namespace without interruption. +More details are available in the [Routing](/cloud/multi-region#routing) section below. + +:::info + +- When a Namespace fails over to a standby region, Workers will be communicating cross-region. + +- In case of a complete regional outage, Workers in the original region may fail alongside the original Namespace. + To keep Workflows moving during this level of outage, deploy a second set of Workers to your standby region. + +::: + +## Routing {#routing} + +When using multi-region for a Namespace, the Namespace's DNS record `..` targets a regional DNS record in the format `.region.`. +In this format, `` is the currently active region for your Namespace. +Clients resolving the Namespace’s DNS record are directed to connect to the active region for that Namespace, thanks to the regional DNS record. + +During failover, Temporal Cloud changes the target of the Namespace DNS record from one region to another. +Namespace DNS records are configured with a 15 seconds TTL. +Any DNS cache should re-resolve the record within this delay. As a rule of thumb, DNS reconciliation takes no longer than twice (2x) the TTL. +Clients should converge to the newly targeted region within, at, most a 30-second delay. + + +## PrivateLink routing {#privatelink-routing} + +:::important + +Some networking configuration is required for failover to be transparent to clients and workers when using PrivateLink. +This section describes how to configure routing for multi-region Namespaces for PrivateLink customers only. + +::: + +PrivateLink customers may need to change certain configurations for multi-region Namespace use. +Routing configuration depends on networking setup and use of PrivateLink. +You may need to: + +- override a DNS zone; and +- ensure the network connectivity between the two regions. + +![Customer side solution example](/img/multi-region/private-link.png) + +When using PrivateLink, you connect to Temporal Cloud using IP addresses local to your network. +The `region.` zone is configured in the Temporal systems as an independent zone. +This allows you to override it to make sure traffic is routed internally for the regions in use. +You can check the Namespace's active region using the Namespace record CNAME, which is public. + +To set up the DNS override, you override specific regions to target the relevant IP addresses (e.g. aws-us-west-1.region.tmprl.cloud to target 192.168.1.2). +Using AWS, this can be done using a private hosted zone in Route53 for `region.`. +Link that private zone to the VPCs you use for Workers. +Private Link is not yet offered for GCP multi-region Namespaces. + +When your Workers connect to the Namespace, they first resolve the `..` record. +This targets `.region.` using a CNAME. Your private zone overrides that second DNS resolution, leading traffic to reach the internal IP you're using. + +Consider how you'll configure Workers to run in this scenario. +You might set Workers to run in both regions at all times. +Alternately, you could establish connectivity between the regions to redirect Workers once failover occurs. + +The following table lists Temporal's available regions, PrivateLink endpoints, and DNS record overrides. +The `sa-east-1` region listed here is not yet available for use with multi-region Namespaces. + +| Region | PrivateLink Service Name | DNS Record Override | +| ---------------- | -------------------------------------------------------------- | --------------------------------------- | +| `ap-northeast-1` | `com.amazonaws.vpce.ap-northeast-1.vpce-svc-08f34c33f9fb8a48a` | `aws-ap-northeast-1.region.tmprl.cloud` | +| `ap-northeast-2` | `com.amazonaws.vpce.ap-northeast-2.vpce-svc-08c4d5445a5aad308` | `aws-ap-northeast-2.region.tmprl.cloud` | +| `ap-south-1` | `com.amazonaws.vpce.ap-south-1.vpce-svc-0ad4f8ed56db15662` | `aws-ap-south-1.region.tmprl.cloud` | +| `ap-south-2` | `com.amazonaws.vpce.ap-south-2.vpce-svc-08bcf602b646c69c1` | `aws-ap-south-2.region.tmprl.cloud` | +| `ap-southeast-1` | `com.amazonaws.vpce.ap-southeast-1.vpce-svc-05c24096fa89b0ccd` | `aws-ap-southeast-1.region.tmprl.cloud` | +| `ap-southeast-2` | `com.amazonaws.vpce.ap-southeast-2.vpce-svc-0634f9628e3c15b08` | `aws-ap-southeast-2.region.tmprl.cloud` | +| `ca-central-1` | `com.amazonaws.vpce.ca-central-1.vpce-svc-080a781925d0b1d9d` | `aws-ca-central-1.region.tmprl.cloud` | +| `eu-central-1` | `com.amazonaws.vpce.eu-central-1.vpce-svc-073a419b36663a0f3` | `aws-eu-central-1.region.tmprl.cloud` | +| `eu-west-1` | `com.amazonaws.vpce.eu-west-1.vpce-svc-04388e89f3479b739` | `aws-eu-west-1.region.tmprl.cloud` | +| `eu-west-2` | `com.amazonaws.vpce.eu-west-2.vpce-svc-0ac7f9f07e7fb5695` | `aws-eu-west-2.region.tmprl.cloud` | +| `sa-east-1` | `com.amazonaws.vpce.sa-east-1.vpce-svc-0ca67a102f3ce525a` | `aws-sa-east-1.region.tmprl.cloud` | +| `us-east-1` | `com.amazonaws.vpce.us-east-1.vpce-svc-0822256b6575ea37f` | `aws-us-east-1.region.tmprl.cloud` | +| `us-east-2` | `com.amazonaws.vpce.us-east-2.vpce-svc-01b8dccfc6660d9d4` | `aws-us-east-2.region.tmprl.cloud` | +| `us-west-2` | `com.amazonaws.vpce.us-west-2.vpce-svc-0f44b3d7302816b94` | `aws-us-west-2.region.tmprl.cloud` | + +:::tip Learn more about multi-region Namespaces + +If you have more questions or feedback about this feature, reach out to the product team. + +::: + diff --git a/docs/production-deployment/cloud/high-availability/faq.mdx b/docs/production-deployment/cloud/high-availability/faq.mdx new file mode 100644 index 0000000000..85b75a94b4 --- /dev/null +++ b/docs/production-deployment/cloud/high-availability/faq.mdx @@ -0,0 +1,219 @@ +--- +id: faq +title: Frequently Asked Questions +sidebar_label: Frequently Asked Questions +slug: /cloud/high-availability/faq +description: Temporal Cloud's High-Availability Namespaces offer automated failover, synchronized data replication, and high availability for workloads requiring disaster-tolerant deployment and 99.99% uptime. Use Global Namespace for self-hosted. +tags: + - Temporal Cloud + - Production + - High availability +keywords: + - availability + - explanation + - failover + - high-availability + - multi-region + - multi-region namespace + - namespaces + - temporal-cloud + - term +--- +import { RelatedReadContainer, RelatedReadItem } from '@site/src/components/related-read/RelatedRead'; + +:::tip Support, stability, and dependency info + +High-availability Namespaces are in [Public Preview](/evaluate/development-production-features/release-stages#public-preview) for Temporal Cloud. + +::: + +
**Repurposed material. No audits, updates, intros, re-org. Converted to markdown automatically from Google Doc rich text, so there are many errors**
+ +Failovers + +**Q: What is a failover****** + +A failover shifts Workflow Execution processing from an active Temporal Namespace to a standby Temporal Namespace during outages or other incidents. Standby Namespaces use replication to duplicate data and prevent data loss during failover. + +**Q: What failover modes does Temporal use internally?****** + +Users cannot configure failover modes. The following descriptions explain Temporal Cloud’s internal failover system: + + * ******[Graceful failover**](https://docs.temporal.io/cloud/multi-region#graceful-failover): Replication tasks are fully processed and drained before transferring control to the standby region. Temporal Cloud pauses traffic to the active Namespace before the failover, minimizing the rewind of progress and avoiding data conflicts. The Namespace experiences a short period of unavailability, defaulting to 10 seconds at most. Under most circumstances, the actual time the Namespace is unavailable is much, much shorter than that. + +During this period, existing Workflows stop progress. Temporal Cloud returns a "Service unavailable error". State transitions will not happen and tasks are not dispatched. User requests like start/signal Workflow will be rejected while operations are paused during handover. This mode favors _consistency_ over availability. + + * ******[Forced failover**](https://docs.temporal.io/cloud/multi-region#forced-failover): In this mode, a Namespace immediately activates in the standby region. Events not replicated due to replication lag will undergo conflict resolution upon reaching the new active region. This mode prioritizes _availability_ over consistency. + * ******[Hybrid failover**](https://docs.temporal.io/cloud/multi-region#hybrid-failover) (Default mode): While graceful failovers are consistent, they aren’t always practical in certain circumstances such as when cells experience outages and/or a critical database is unavailable. Temporal Cloud’s hybrid failover mode limits an initial Graceful failover attempt to 10 seconds or less. If the graceful approach doesn’t resolve the issue, Temporal Cloud automatically switches to a forced failover. This strategy balances consistency and availability requirements. This strategy balances _consistency_ and _availability_ requirements. + +**Q: What is the difference between a handover and a failover?****** + +They are essentially the same thing. It is the process of transferring control from the active to the standby region during outages or other incidents. + + + + + + + + + + +**Q: What situation triggers a graceful failover vs a forced one? Who or what triggers it? What are the differences in results?****** + +Users can initiate a failover, but they can’t control or configure the failover mode. The three failover modes are internal operations on the Temporal side. They are explained for user education. + +**Q: Under what circumstances would I want to initiate a failover?****** + +Normally, we don't expect users to failover when there’s a problem related to Temporal Cloud. Please contact Temporal support if you feel you have a pressing need. You might consider initiating a failover under two circumstances: + + * The “break glass”  (fire alarm) scenario where Temporal hasn’t responded to an outage or is unaware of an outage in the customer environment. + * To test the failover functionality and ensure it works properly. + +You can still choose to initiate failover if you have issues sourced from your side or your dependencies. + +**Q: Under what circumstances does Temporal initiate failovers?****** + +Temporal Cloud initiates failovers when there are incidents or outages in the cloud provider. This includes failures of databases, storage, etc. We trigger failovers any time we observe increased latencies or an increase in service errors that causes us to violate the SLA that is in our control. + +**Q: Are there any other types of failover not listed above?****** + +No. There are only three types. + +**Q: Can we control the hybrid failover timeout?****** + +No. The timeout is not configurable outside of Temporal. + +**Q: Can a failover get stuck? What is the maximum amount of time it can take?****** + +There is typically no way for failovers to get “stuck”. We follow the hybrid failover method where we try to do a smooth handoff. If that does not take place within 10 seconds, we initiate a “forced” failover. + +**Q: Is the 10 seconds maximum unavailability window configurable?****** + +No, it is not configurable by the user. Extending the wait time is unlikely to increase the chances of graceful failovers during extreme incidents such as when a source region is down. + +**Q: How does the client detect that the failover has occurred?****** + +We do not send real-time failover notifications. Users are notified via email and audit logs. + +**Q: Can the customer determine the resolved failover region?****** + +Users can determine the failover region from the Namespace endpoint’s CNAME (\.tmprl.cloud). Whenever Temporal Cloud triggers a failover from the Temporal side, we update the CNAME to point to the new active region. The CNAME points to a Temporal Cloud regional endpoint. For example, a Namespace active in aws us-east-1 points to aws-**us-east-1**.region.tmprl.cloud. + +Replication Lag and Latency + +**Q: What affects replication latency? What can cause the replication latency to increase?****** + +Slowdowns in the standby cell, such as capacity issues or outages, can increase replication latency. Otherwise, it is typically a matter of seconds or even less and can be monitored through  [external metrics](https://docs.temporal.io/cloud/multi-region#metrics-operations). + +**Q: Can Workflows execute events a second time in the standby Cluster due to replication lag?****** + +Yes. This is explained in the [conflict resolution](https://docs.temporal.io/cloud/multi-region#conflict-resolution) section of our documentation. + +**Q: Is it possible to see whether a failover was graceful, forced, or hybrid?****** + +No, customers cannot normally view the method used. File a support ticket if there’s a specific need to review a process. + +**Q: Is replication lag emitted as a metric?****** + +Yes, replication lag is a [metric](https://docs.temporal.io/cloud/multi-region#metrics-operations) that we expose. + +**Q: Can we see replication information by Workflow type or ID?****** + +_[Answer in progress]___ + +**Q: Is the data replicated in order?****** + +For a single Workflow, events are replicated  in order. There's no ordering guarantee for replication of events between different Workflows. + +**Q: What happens if both regions become active simultaneously?****** + +This only happens when there's a network partition or delays in the Namespace replication queue. Normally, when cells can talk to each other, only one region will ever become active. + +If both regions have become active and both have active Workers, Workflows will run independently based on their local History. Workers fetch tasks from their assigned region. With global Worker setups, Workers fetch tasks from the ‘true’ active region as known by Temporal Cloud. Eventually, when the network partition heals, History is merged via conflict resolution and one side wins. + +**Q: What if DNS is still updating during a network partition between Clusters? ****** + +In this situation, the now passive Cluster can’t forward requests to the new active Cluster. However, DNS normally points to the correct active Cluster without forwarding. Workers configured to point to the standby Cluster can be reconfigured to point to the active Cluster. + +Conflict Resolution + +See [this Notion Page](https://www.notion.so/temporalio/Conflict-Resolution-Example-83e9dec0f8f246ee8584995ae2e408f4) for an example Conflict Resolution. + +**Q: How are conflicts resolved?****** + +Each cell has a version number, which is used in Event History metadata. Failover operations increase that number. Events with the highest number win during conflict resolution. + +**Q: What happens to Workflows if conflicts can’t be resolved?****** + +This can only happen if there is a bug in the conflict resolution.  If there is a bug in conflict resolution, those events are placed in a dead letter queue to unblock replication. Temporal will resolve the issue and reapply the events. + +Customer impact is limited to the affected Workflows. The rest of the system continues as normal. + +**Q: How is History affected if conflicts can’t be resolved?****** + +Same as above. + +**Q: How do customers detect unresolved conflicts?****** + +Unresolved conflicts are not made visible to customers. Temporal directs unresolvable conflicts (conflicts that require Temporal on-call intervention) into a dead letter queue and makes sure those conflicts are resolved and their events re-applied. + +**Q: How do customers manually resolve conflicts?****** + +No manual resolution by customers is needed unless Temporal cannot handle a specific scenario. + +**Q: Are non-selected event histories deleted during automatic conflict resolution? ****** + +No. They are hidden but not deleted. We do not expose access to non-selected events to customers. + +Data Loss + +**Q: Under what circumstances would a Workflow Execution be unrecoverable if it was started but not replicated before failover?****** + +The normal time difference between the two operations is typically measured in single-digit seconds.  So this scenario can only happen if the Cluster is healthy enough to accept the Workflow start request and fails to replicate this event. This is very unlikely. If it did happen, the started Workflow is recovered after the Cluster is itself recovered. The only possibility of data loss would require that Temporal lose contact with the previously active cell after permanently completing an operation. + +Metrics and Observability + +**Q: What information can be pulled from MRN metrics?****** + +This is [documented](https://docs.temporal.io/cloud/multi-region#metrics-operations). + +Always check metric replication lag before initiating a failover test or emergency failover. A forced failover when there is a large replication lag has a higher likelihood of rolling back Workflow progress. + +**Q: What warning signs Signal that a failover may be arriving?****** + +You should always be prepared for failover. One could happen at any point in time. + +We notify customers when a failover occurs. There is no time lapse between discovering failover prerequisites and the failover itself. + +Other + +**Q: Can Signals be sent twice since multi-region doesn't provide at-most-once delivery?****** + +During conflict resolution, a Signal could be applied twice. + +**Q: What happens if the active region is unavailable for an extended period and the standby region does not have the most recent Signal?**** ****** + +Workers cannot process the Signal as it won’t be present in any available region. + +**Q: If the active region remains unavailable for an extended period, does the active role switch to the standby region? ****** + +If Temporal Cloud initiated the failover, it will “fail back” to the original active region once the incident is fully resolved. Otherwise, the active role remains with the newly active (formerly standby) region. + +**Q: ****[Is there a way to determine the region an event ran in via the UI?******](https://temporaltechnologies.slack.com/archives/C04V0LSU5S6/p1717092196761469) + +Not at the moment. + +**Q: ****[Can we show branching in the UI?******](https://temporaltechnologies.slack.com/archives/C04V0LSU5S6/p1717171982044329) + +Not at the moment. + +**Q: What should customers worry about in terms of Signals and events synchronization?****** + +Signals are cherry-picked during conflict resolution if there is replication lag and conflict. Workflows can theoretically revert multiple steps. + +Customers should decide whether to add logic to handle this or manually fix affected Workflows if they believe the risk is low. Other known limitations have been [documented](https://docs.temporal.io/cloud/multi-region#architecture) around causality and so forth. + +**Q: How much time does it take to reconcile data after an incident is resolved? ****** + +It depends on the distribution of Workflows. If evenly distributed, data can sync quickly. If concentrated in a single partition, it could take hours. Do not “fail back” your region (revert it to the original active region) until the data is fully reconciled and the other region has caught up. diff --git a/docs/production-deployment/cloud/high-availability/guarantees.mdx b/docs/production-deployment/cloud/high-availability/guarantees.mdx new file mode 100644 index 0000000000..f4361e7325 --- /dev/null +++ b/docs/production-deployment/cloud/high-availability/guarantees.mdx @@ -0,0 +1,61 @@ +--- +id: guarantees +title: Guarantees and availability +sidebar_label: Guarantees and availability +slug: /cloud/high-availability/guarantees +description: Temporal Cloud's High-Availability Namespaces offer automated failover, synchronized data replication, and high availability for workloads requiring disaster-tolerant deployment and 99.99% uptime. Use Global Namespace for self-hosted. +tags: + - Temporal Cloud + - Production + - High availability +keywords: + - availability + - explanation + - failover + - high-availability + - multi-region + - multi-region namespace + - namespaces + - temporal-cloud + - term +--- + +:::tip Support, stability, and dependency info + +High-availability Namespaces are in [Public Preview](/evaluate/development-production-features/release-stages#public-preview) for Temporal Cloud. + +::: + +
**Maybe these could just be parts of the FAQ?**
+ + +## Multi-region Namespace SLA {#sla} + +**What guarantees does Temporal offer for multi-region Namespaces?** + +Multi-region Namespaces offer 99.99% availability, enforced by Temporal Cloud's [service error rates SLA](https://docs.temporal.io/cloud/sla). +Our system is designed to limit data loss after recovery when the incident triggering the failover is resolved. + +Our recovery point objective ([RPO](https://en.wikipedia.org/wiki/Disaster_recovery#Recovery_Point_Objective)) is near-zero. +There may be a short period of time during an incident or forced failover when some data is unavailable in the standby region. +Some Workflow History data won't arrive until networks issue are fixed, enabling the History to finish replicating and the divergent History branches to reconcile. + +Temporal Cloud proactively responds to incidents by triggering failovers. +Our recovery time objective ([RTO](https://en.wikipedia.org/wiki/Disaster_recovery#Recovery_Time_Objective)) is 20 minutes or less per incident. + +:::info + +During a disaster scenario in which the data on the hard drives in the active region cannot be recovered, the duration of data loss may be as high as the [replication lag](/cloud/multi-region#replication-lag) at the time of disaster. + +::: + +### Regional availability {#regional-availability} + +Multi-region Namespaces are available in all existing [Temporal Cloud regions](/cloud/service-availability#regions). + +:::tip + +Namespace pairing is currently limited to regions within the same continent. +South America is excluded as only one region is available. + +::: diff --git a/docs/production-deployment/cloud/high-availability/how-it-works.mdx b/docs/production-deployment/cloud/high-availability/how-it-works.mdx new file mode 100644 index 0000000000..9311b6794e --- /dev/null +++ b/docs/production-deployment/cloud/high-availability/how-it-works.mdx @@ -0,0 +1,197 @@ +--- +id: how-it-works +title: How it works +sidebar_label: How it works +slug: /cloud/high-availability/how-it-works +description: Temporal Cloud's High-Availability Namespaces offer automated failover, synchronized data replication, and high availability for workloads requiring disaster-tolerant deployment and 99.99% uptime. Use Global Namespace for self-hosted. +tags: + - Temporal Cloud + - Production + - High availability +keywords: + - availability + - explanation + - failover + - high-availability + - multi-region + - multi-region namespace + - namespaces + - temporal-cloud + - term +--- +import { RelatedReadContainer, RelatedReadItem } from '@site/src/components/related-read/RelatedRead'; + +:::tip Support, stability, and dependency info + +High-availability Namespaces are in [Public Preview](/evaluate/development-production-features/release-stages#public-preview) for Temporal Cloud. + +::: + +
**No audits, updates, intros, re-org**
+ +In traditional active/active replication, multiple nodes serve requests and accept writes simultaneously, ensuring strong synchronous data consistency. +In contrast, with a Temporal Cloud high-availability Namespace, only the active zone accepts requests and writes at any given time. +Workflow history events are written to the active zone first and then asynchronously replicated to the standby zone replica, ensuring that the replica remains in sync. + +
**Needs new images**
+ +| Before failover | After failover | +| :-------------------------------------------------------: | :-----------------------------------------------------: | +| ![Before failover](/img/multi-region/before-failover.png) | ![After failover](/img/multi-region/after-failover.png) | + +## Failovers {#failovers} + +A failover shifts Workflow Execution processing from an active Temporal Namespace region to a standby Temporal Namespace region during outages or other incidents. +Standby Namespace regions use replication to duplicate data and prevent data loss during failover. + +**What happens during the failover process?** + +Temporal Cloud initiates a Namespace failover when it detects an incident or outage that raises error rates or latency in the active region of a multi-region Namespace. +The failover shifts Workflow processing to a standby region that isn’t affected by the incident. +This lets existing Workflows continue and new Workflows start while the incident is fixed. +Once the incident is resolved, Temporal Cloud performs a "failback" by shifting Workflow Execution processing back to the original region. + + +:::info + +You can test the failover of your multi-region Namespace by manually [triggering a failover](/cloud/multi-region#triggering-failovers) using the UI page or the 'tcld' CLI utility. +In most scenarios, we recommend you let Temporal handle failovers for you. + +::: + +## Health Checks {#healthchecks} + +**How does Temporal detect failover conditions?** + +Temporal Cloud automates failovers by performing internal health checks. +This process monitors your request error rates, latencies, and any infrastructure issues that might cause service disruptions, such as request timeouts. +It automatically triggers failovers when these indicators exceed our allowed thresholds. + +### Replication lag {#replication-lag} + +Multi-region Namespaces use asynchronous replication between regions. +Workflow updates in the active region, along with associated history events, are transmitted to the standby region with a short delay. +This delay is called the replication lag. +Temporal Cloud strives to maintain a P95 replication delay of less than 1 minute. +In this context, P95 means 95% of requests are processed faster than this specified limit. + +Replication lags mean a [forced failover](/cloud/multi-region#forced-failover) may cause Workflows to rollback in progress. +Lags may also cause recently started Workflows to be temporarily unavailable until the active region recovers. +Temporal event versioning and [conflict resolution mechanisms](/cloud/multi-region#conflict-resolution) help guarantee that the Workflow Event History can be replayed. +Critical operations like Signals won't get lost. + +### Failover scenarios + +The Temporal Cloud failover mechanism supports several modes to execute Namespace failovers. +These modes include graceful failover ("handover"), forced failover, and a hybrid mode. +The hybrid mode is Temporal Cloud’s default Namespace behavior. + +#### Graceful failover (handover) {#graceful-failover} + +In this mode, replication tasks are fully processed and drained. +Temporal Cloud pauses traffic to the Namespace before the failover. +This prevents the loss of progress and avoids data conflicts. +The Namespace experiences a short period of unavailability, defaulting to 10 seconds. + +During this period, existing Workflows stop progress. +Temporal Cloud returns a "Service unavailable error", which is retried by SDKs. +State transitions will not happen and tasks are not dispatched. +User requests like start/signal workflow will be rejected while operations are paused during handover. + +This mode favors _consistency_ over availability. + +#### Forced failover {#forced-failover} + +In this mode, a Namespace immediately activates in the standby region. +Events not replicated due to [replication lag](/cloud/multi-region#replication-lag) will undergo [conflict resolution](/cloud/multi-region#conflict-resolution) upon reaching the new active region. + +This mode prioritizes _availability_ over consistency. + +#### Hybrid failover mode {#hybrid-failover} + +While graceful failovers are preferred for consistency, they aren’t always practical. +Temporal Cloud’s hybrid failover mode (the default mode) limits an initial graceful failover attempt to 10 seconds or less. +During this period, existing Workflows stop progress. +Temporal Cloud returns a "Service unavailable error", which is retried by SDKs. +If the graceful approach doesn’t resolve the issue, Temporal Cloud automatically switches to a forced failover. +This strategy balances consistency and availability requirements. + +See the sections on [triggering a failover](/cloud/multi-region#triggering-failovers), [Worker deployment](/cloud/multi-region#worker-deployment), and [routing](/cloud/multi-region#routing) for more information. + +## Architecture {#architecture} + +**How do multi-region Namespaces work?** + +Multi-region Namespaces replicate Namespace metadata and Workflow Executions across connected regions. +This redundancy, plus the added failover capability, provides measurable stability when dealing with outages. + +A multi-region Namespace is normally active in a single region at any moment. +The passive region assumes a standby role. +An exception to this only occurs in the event of a network partition. +In this case, you may elect to promote a standby region to active status. +Caution: this action will temporarily result in both regions being active. +Once the network partition resolves and communication between the regions is restored, a conflict resolution algorithm determines which region continues as the active one. +This ensures only one region remains active. + +### Metadata replication {#metadata-replication} + +Updates to multi-region Namespace records automatically replicate across regions. +This metadata includes configurations such as retention periods, Search Attributes, and other settings. +Temporal Cloud ensures that all regions will eventually share a consistent and unified view of the Namespace metadata. + +:::info + +A Namespace failover, which changes the "active region" field of a Namespace record, is an update. +This update is replicated via the Namespace metadata mechanism. + +::: + +### Workflow Execution replication {#workflow-execution-replication} + +Temporal Cloud restricts certain Workflow operations to the active region: + +- You may only update Workflows in the active region. +- You may only dispatch Workflow Tasks and Activity Tasks from the active region. Forward progress in a Workflow Execution can therefore only be made in the active region. + +These limits mean that certain requests, such as Start Workflow and Signal Workflow, are processed by and limited to the active region. +Standby regions may receive API requests from Clients and Workers. +They automatically forward these requests to the active Namespace for execution. + +Multi-region Namespaces provide an “all-active” experience for Temporal users. +This helps limit or eliminate downtime during Namespace failover. +There's a short time window from when a standby region becomes the active region to when Clients and Workers receive a DNS update. +During this time requests forward from the now passive (formerly active) region to the newly active (formerly standby) region. + +As Workflow Executions progress and are operated on, replication tasks created in the active region are dispatched to the standby region. +Processing these replication tasks ensures that the standby region undergoes the same state transitions as the active region. +This enables replicated tasks to synchronize and achieve the same state as the original tasks. + +Standby regions do not distribute Workflow or Activity Tasks. +Instead, they perform verification tasks to confirm that intended operations are executed so Workflows reach the desired state. +This mechanism ensures consistency and reliability in the replication process across Temporal regions. + +### Conflict Resolution {#conflict-resolution} + +Multi-region Namespaces rely on asynchronous event replication across Temporal regions. +In the event of a non-graceful failover, replication lag may result in a temporary setback in workflow progress. + +Single-region Namespaces can be configured to provide _at-most-once_ semantics for Activities execution (when [Maximum Attempts](https://docs.temporal.io/retry-policies#maximum-attempts) is set to 0). +Multi-region Namespaces provide _at-least-once_ semantics for execution of Activities. +Completed Activities _may_ be re-dispatched in a newly active region, leading to repeated executions. + +When a Workflow Execution is updated in a new region following a failover, events from the previously active region that arrive after the failover can't be directly applied. +At this point, Temporal Cloud has forked the Workflow History. + +After failover, Temporal Cloud creates a new branch history for execution, and begins its conflict resolution process. +The Temporal Service ensures that Workflow Histories remain valid and are replayable by SDKs post-failover or after conflict resolution. +This capability is crucial for Workflow Executions to continue their forward progress. + +:::warning + +Design your activities to succeed once and only once. +This "idempotent" approach avoids process duplication that could withdraw money twice or ship extra orders by mistake. +Run-once actions maintain data integrity and prevent costly errors. +Idempotency keeps operations from producing additional effects. +Protect your processes from accidental or repeated actions for more reliable execution. + +::: diff --git a/docs/production-deployment/cloud/high-availability/index.mdx b/docs/production-deployment/cloud/high-availability/index.mdx new file mode 100644 index 0000000000..7fda043014 --- /dev/null +++ b/docs/production-deployment/cloud/high-availability/index.mdx @@ -0,0 +1,113 @@ +--- +id: index +title: High-availability Namespaces +sidebar_label: High-availability Namespaces +slug: /cloud/high-availability +description: Temporal Cloud's High-Availability Namespaces offer automated failover, synchronized data replication, and high-availability for workloads requiring disaster-tolerant deployment and 99.99% uptime. Use Global Namespace for self-hosted. +tags: + - Temporal Cloud + - Production + - High availability +keywords: + - availability + - explanation + - failover + - high-availability + - multi-region + - multi-region namespace + - namespaces + - temporal-cloud + - term +--- + +import { RelatedReadContainer, RelatedReadItem } from '@site/src/components/related-read/RelatedRead'; + +:::tip Support, stability, and dependency info + +High-availability Namespaces are in [Public Preview](/evaluate/development-production-features/release-stages#public-preview) for Temporal Cloud. + +::: + +
**General note, we should be using "Replica" vs "Region", and probably note "Zone". Discuss. "Replica" is the corpus of Workflows. High Availability is a _capability_. Features are "replication" and "multi-region". Features achieve HA workflows in your applications. See messaging guide.**
+ +Temporal Cloud's high-availability Namespaces provide disaster-tolerant deployment for workloads where availability is critical to your operations. +When you enable high availability, Temporal Cloud automatically synchronizes your data between a primary and a fallback Namespace, keeping them in sync. +Should an incident occur, Temporal will [failover](/glossary#failover) your Namespace. +This allows your Workflow Executions and Schedules to seamlessly shift from the active availability zone to the replica in the fallback availability zone. + +## Availability zones and replicas + +An availability zone is a physically isolated data center within a deployment region for a given cloud provider. +Regions consist of multiple availability zones, providing redundancy and fault tolerance. +In some cases, the fallback zone may be in the same region as the primary zone, or it may be in a different region altogether, depending on your deployment configuration. + +High-availability simplifies deployment, ensuring operational continuity and data integrity even during unexpected events. +Regional disruptions or other issues that affect the data centers within a specific availability zone may occur. +High-availability allows processing to shift from the affected zone to an already-synchronized fallback zone. + +This synchronized zone is called a "**replica**." +The process of duplicating all Workflow data ensures that your replica, which serves as the standby region, is always available and ready to take on the active role. + +In the event of network service or performance issues in the active zone, your replica is ready to take over. +When necessary, Temporal Cloud smoothly transitions control from the active to the standby zone using a process called "[failover](/glossary#failover)". + +## High-availability and business continuity {#high-availability-intro} + +For many organizations, ensuring high-availability is critical to maintaining business continuity. +Temporal Cloud's high-availability Namespace feature includes a 99.99% contractual Service Level Agreement ([SLA](https://docs.temporal.io/cloud/sla)). +It provides 99.99% availability and 99.99% guarantee against service errors. + +A high-availability Namespace (HAN) creates a single logical Namespace that operates across two physical zones: one active and one standby. +HANs streamline access for both zones to a unified Namespace endpoint. +As Workflows progress in the active zone, history events are asynchronously replicated to the standby zone, ensuring continuity and data integrity. + +In the event of an incident or outage in the active zone, Temporal Cloud will seamlessly failover to your standby zone. +Failovers allow existing Workflow Executions to continue running and new Workflow Executions to be started. +Once failover occurs, the roles of the active and standby zones switch. +The standby zone becomes active, and the previous active zone becomes the standby. +After the issue is resolved, the zone "fails back" from the replica to the original. + +## Types of high-availability + +Temporal currently offers the following high-availability options, which you select when upgrading your Namespace to use high-availability: + +- **In-region replication** - + Data is replicated to a separate zone in the same availability region, such as "us-east-1". + This option offers near-instantaneous failovers but does not protect against regional disasters like hurricanes where both the primary and the fallback . +- **Multi-region replication** - + Data is replicated to a separate region on the same continent. + This option offers the greatest protection against weather events and other possible external causes for regional outages, as the regions are physically separated by large distances. + Failover may experience some minor latency. + +
**Did we want to go into further detail about lag and conflict resolution? if so, where?**
+ +
**Would be very nice to have more compelling explanations here**
+ +:::tip + +As Namespace pairing is currently limited to regions within the same continent, South America is excluded as only one region is available. + +::: + +## Should you choose high-availability? + +Should you be using high-availability Namespaces? It depends on your availability requirements: + +- High-availability Namespaces offer a 99.99% contractual SLA for workloads with strict high-availability needs. + HANs use two Namespaces in two deployment zones to support standby recovery. + In the event of a zone failure, Temporal Cloud automatically fails over the HAN Namespace to the standby replica. +- Single-zone Namespaces include a 99.9% contractual Service Level Agreement ([SLA](/cloud/sla)). + In single-zone use, Temporal clients connect to a single Namespace in one deployment zone. + For many applications, this offers sufficient availability. + +Temporal Cloud provides 99.99% service availability for all Namespaces, both single-region and high-availability. + +## Summary + +The following list reviews the advantages of high-availability Namespaces: + +- No manual deployment or configuration required—just simple push-button operation. +- Open Workflows continue in the standby zone with minimal interruption and data loss. +- No changes needed for Worker or Workflow code during setup or failover. +- 99.99% contractual SLA. + diff --git a/docs/production-deployment/cloud/high-availability/monitor.mdx b/docs/production-deployment/cloud/high-availability/monitor.mdx new file mode 100644 index 0000000000..4f3211dabb --- /dev/null +++ b/docs/production-deployment/cloud/high-availability/monitor.mdx @@ -0,0 +1,81 @@ +--- +id: monitor +title: Monitor and observe +sidebar_label: Monitor and observe +slug: /cloud/high-availability/operations +description: Temporal Cloud's High-Availability Namespaces offer automated failover, synchronized data replication, and high availability for workloads requiring disaster-tolerant deployment and 99.99% uptime. Use Global Namespace for self-hosted. +tags: + - Temporal Cloud + - Production + - High availability +keywords: + - availability + - explanation + - failover + - high-availability + - multi-region + - multi-region namespace + - namespaces + - temporal-cloud + - term +--- +import { RelatedReadContainer, RelatedReadItem } from '@site/src/components/related-read/RelatedRead'; + +:::tip Support, stability, and dependency info + +High-availability Namespaces are in [Public Preview](/evaluate/development-production-features/release-stages#public-preview) for Temporal Cloud. + +::: + +
**No audits, updates, intros, re-org. Add information about unhealth vs health, and trigger failover button disabled, opting out of temporal-initiated failovers, "Unhealthy replica error". After moving things to enable, this seems really light on content**
+ +How do you trigger failovers and observe Workflow Executions? +This section provides how-to instructions for the following operations tasks: + +- [Triggering failovers](/cloud/multi-region#triggering-failovers) +- [Metrics](/cloud/multi-region#metrics-operations) +- [Monitoring and observability](/cloud/multi-region#observability) + +### Metrics {#metrics-operations} + +Replication lag refers to the transmission delay of Workflow updates and history events from the active region to the standby region. +A forced failover when there is a large replication lag has a higher likelihood of rolling back Workflow progress, so always check the metric replication lag before initiating a failover. +Temporal Cloud emits three replication lag-specific [metrics](/production-deployment/cloud/metrics/reference#temporal_cloud_v0_replication_lag_bucket). +The following samples demonstrate how you can use these metrics to explore replication lag. + +**P99 replication lag histogram** + +``` +histogram_quantile(0.99, sum(rate(temporal_cloud_v0_replication_lag_bucket[$__rate_interval])) by (temporal_namespace, le)) +``` + +**Average replication lag** + +``` +sum(rate(temporal_cloud_v0_replication_lag_sum[$__rate_interval])) by (temporal_namespace) +/ +sum(rate(temporal_cloud_v0_replication_lag_count[$__rate_interval])) by (temporal_namespace) +``` + +### Monitoring and observability {#observability} + +You can view and alert on key cloud metrics using the Web UI, the 'tcld' CLI utility, and Temporal Cloud APIs. +For example, during the process of adding a region to a Namespace, you can see the progress of Workflow replication. +Errors -- if any occur -- will also surface in the Namespace Web UI. + +:::info + +You may notice that multi-region Namespace shows twice (2x) the Action count in `temporal_cloud_v0_total_action_count`. +This doubling happens due to regional replication. + +::: + +### Auditing operational events {#auditing} + +Temporal Cloud provides several ways to audit events: + +- When Temporal triggers failovers, the audit log updates with details. + Look specifically for `"operation": "FailoverNamespace"` in the logs. +- You can set alerts for Temporal-initiated failover events. +- After a failover, you can check that the Namespace is active in the new region using the Temporal Cloud Web UI. + diff --git a/docs/production-deployment/cloud/high-availability/work-file.txt b/docs/production-deployment/cloud/high-availability/work-file.txt new file mode 100644 index 0000000000..fa4d5b28b0 --- /dev/null +++ b/docs/production-deployment/cloud/high-availability/work-file.txt @@ -0,0 +1,67 @@ +
**STOPPED HERE. Considering whether this should be its own page**
+ +AFFECTED COVERAGE: + +- ALL LINKS ON NEW PAGES MUST BE AUDITED, REDONE, TESTED!!! + +THESE PAGES MENTION MULTI-REGION: +./docs/encyclopedia/nexus.mdx +./docs/encyclopedia/nexus-use-cases.mdx +./docs/evaluate/development-production-features/temporal-nexus.mdx +./docs/evaluate/development-production-features/index.mdx +./docs/evaluate/development-production-features/high-availability.mdx +./docs/evaluate/development-production-features/multi-region-namespace.mdx +./docs/evaluate/development-production-features/cloud-vs-self-hosted.mdx +./docs/evaluate/temporal-cloud/pricing.mdx +./docs/evaluate/temporal-cloud/sla.mdx +./docs/evaluate/temporal-cloud/legacy-pricing.mdx +./docs/evaluate/temporal-cloud/security.mdx +./docs/production-deployment/cloud/metrics/reference.mdx +./docs/production-deployment/cloud/gcp-export-gcs.mdx +./docs/production-deployment/cloud/audit-logging.mdx +./docs/production-deployment/cloud/multi-region.mdx +./docs/production-deployment/cloud/tcld/namespace.mdx +./docs/production-deployment/cloud/nexus/index.mdx +./docs/production-deployment/cloud/terraform-provider.mdx +./docs/production-deployment/cloud/service-health.mdx + +## Pricing {#pricing} + +**How does adding a multi-region Namespace affect my costs?** + +For pricing details, visit Temporal Cloud's [Pricing page](/cloud/pricing). + +## Manage your multi-region Namespace {#management} + +**How do you create, enable, and manage your multi-region Namespace?** + +Temporal enables you to create and manage your multi-region Namespace using the Temporal Cloud Web UI, the command line 'tcld' CLI utility, and the [Cloud Ops API](/ops). +Use these tools to create, upgrade, and discontinue your multi-region Namespace. + +- [Create a multi-region Namespace](/cloud/multi-region#create) +- [Upgrade a single-region Namespace to multi-region](/cloud/multi-region#add-regions) +- [Discontinuing multi-region service](/cloud/multi-region#discontinuing) + +:::warning + +Only Account Owner and Global Admin [roles](/cloud/users#account-level-roles) and [Namespace Admins](https://docs.temporal.io/cloud/users#namespace-level-permissions) may create an multi-region Namespace (MRN), upgrade an existing Namespace to MRN, or trigger an MRN failover. + +::: + +:::info Support, stability, and dependency info + +Temporal Cloud’s Terraform provider does not support multi-region Namespaces. + +::: + + + + + + + + + + + + diff --git a/sidebars.js b/sidebars.js index 5ba006c643..cecd9b1ff5 100644 --- a/sidebars.js +++ b/sidebars.js @@ -338,6 +338,22 @@ module.exports = { ], }, "production-deployment/cloud/multi-region", + { + type: "category", + label: "High-availability Namespaces", + collapsed: true, + link: { + type: "doc", + id: "production-deployment/cloud/high-availability/index", + }, + items: [ + "production-deployment/cloud/high-availability/enable", + "production-deployment/cloud/high-availability/how-it-works", + "production-deployment/cloud/high-availability/guarantees", + "production-deployment/cloud/high-availability/monitor", + "production-deployment/cloud/high-availability/faq", + ], + }, { type: "category", label: "Temporal Nexus", From cf6c2c1caa42cf218cbb22e35b8ab7cb7837ec19 Mon Sep 17 00:00:00 2001 From: Erica Sadun Date: Mon, 3 Feb 2025 13:33:55 -0700 Subject: [PATCH 2/3] EDU-3661: Monday Feb 3 Updates with Nikitha feedback - Some significant shifting of content - Info should be more readily consumed in order - Landing page is still not reconciled --- .../high-availability/best-practices.mdx | 217 +++++++++++++ .../cloud/high-availability/enable.mdx | 297 ++++-------------- .../cloud/high-availability/failovers.mdx | 79 +++++ .../cloud/high-availability/faq.mdx | 171 +++++----- .../cloud/high-availability/guarantees.mdx | 61 ---- .../cloud/high-availability/how-it-works.mdx | 97 +++--- .../cloud/high-availability/index.mdx | 137 ++++---- .../cloud/high-availability/monitor.mdx | 81 ----- .../cloud/high-availability/work-file.txt | 38 --- sidebars.js | 4 +- 10 files changed, 586 insertions(+), 596 deletions(-) create mode 100644 docs/production-deployment/cloud/high-availability/best-practices.mdx create mode 100644 docs/production-deployment/cloud/high-availability/failovers.mdx delete mode 100644 docs/production-deployment/cloud/high-availability/guarantees.mdx delete mode 100644 docs/production-deployment/cloud/high-availability/monitor.mdx diff --git a/docs/production-deployment/cloud/high-availability/best-practices.mdx b/docs/production-deployment/cloud/high-availability/best-practices.mdx new file mode 100644 index 0000000000..752cf60652 --- /dev/null +++ b/docs/production-deployment/cloud/high-availability/best-practices.mdx @@ -0,0 +1,217 @@ +--- +id: best-practices +title: best-practices +sidebar_label: Best Practices +slug: /cloud/high-availability/best-practices +description: Temporal Cloud's High-Availability Namespaces offer automated failover, synchronized data replication, and high availability for workloads requiring disaster-tolerant deployment and 99.99% uptime. Use Global Namespace for self-hosted. +tags: + - Temporal Cloud + - Production + - High availability +keywords: + - availability + - explanation + - failover + - high-availability + - namespaces + - temporal-cloud + - term +--- + +import { RelatedReadContainer, RelatedReadItem } from '@site/src/components/related-read/RelatedRead'; + +This page collects best practices related to high availability Namespaces. + +- [Preparing Worker deployment](/cloud/high-availability/best-practices#worker-deployment) +- [Set up secure routing for failovers](/cloud/high-availability/best-practices#routing) +- [PrivateLink routing](/cloud/high-availability/best-practices#privatelink-routing) +- [High availability Failover testing](/cloud/high-availability/best-practices#testing) +- [Monitoring replication lag metrics](/cloud/high-availability/best-practices#metrics) +- [Viewing operational events](/cloud/high-availability/best-practices#events) + +## High availability Failover testing {#testing} + +Regular failover testing ensures your app can handle disruptions and continue running smoothly in production. + +Microservices and external dependencies will fail at some point. +Testing failovers ensures your app can handle these failures effectively. + +Temporal recommends regular and periodic failover testing for mission-critical applications in production. +By testing in non-emergency conditions, you verify that your app continues to function even when parts of the infrastructure fail. + +:::tip Safety First + +If this is your first time performing a failover test, run it with a test-specific namespace and application. +This helps you gain operational experience before applying it to your production environment. +Practice runs help ensure the process runs smoothly during real incidents in production. + +::: + +Trigger testing can: + +- **Validate high-availability deployments**: + In multi-region setups, failover testing ensures your app can run from another region when the primary region experiences outages. + In single-region setups, failover testing instead works with isolation domain. + This maintains high availability in mission-critical deployments. + Manual testing confirms the failover mechanism works as expected, so your system handles incidents effectively. + +- **Assess replication lag**: + In multi-region deployment, monitoring [replication lag](#metrics) between regions is crucial. + Check the lag before initiating a failover to avoid rolling back Workflow progress. + This is less important when using isolation domains as failover is usually instantaneous. + Manual testing helps you practice this critical step and understand its impact. + When there's no real incident, the switch over (recovery) should happen almost instantly. + A switch over within a single region should also be nearly instantaneous. + +- **Assess recovery time**: + Manual testing helps you measure actual recovery time. + You can check if it meets your expected Recovery Time Objective (RTO) of 20 minutes or less, as stated in the [High availability Namespace SLA](/cloud/high-availability#sla). + +- **Identify potential issues**: + Failover testing uncovers problems not visible during normal operation. + This includes issues like [backlogs and capacity planning](https://temporal.io/blog/workers-in-production#testing-failure-paths-2438) and how external dependencies behave during a failover event. + +- **Validate fault-oblivious programming**: + Temporal uses a "fault-oblivious programming" model, where your app doesn’t need to explicitly handle many types of failures. + Testing failovers ensures that this model works as expected in your app. + +- **Operational readiness**: + Regular testing familiarizes your team with the failover process, improving their ability to handle real incidents when they arise. + +Testing failovers regularly ensures your Temporal-based applications remain resilient and reliable, even when infrastructure fails. + +## Preparing Worker deployment {#worker-deployment} + +Enabling high availability for Namespaces doesn't require specific Worker configuration. +The process is invisible to the Workers. +When a Namespace fails over to the replica, the DNS redirection orchestrated by Temporal ensures that your existing Workers continue to poll the Namespace without interruption. +More details are available in the [Routing](/cloud/high-availability/best-practices#routing) section. + +- When a Namespace fails over to a replica in a different region, Workers will be communicating cross-region. + If your application can’t tolerate this latency, deploy a second set of Workers in this region or opt for a replica in the same region. + +- In case of a complete regional outage, Workers in the original region may fail alongside the original Namespace. + To keep Workflows moving during this level of outage, deploy a second set of Workers to the secondary region. + +## Set up secure routing for failovers {#routing} + +
**This section needs fixing for regions vs isolation domains**
+ +When using a high availability Namespace, the Namespace's DNS record `..` targets a regional DNS record in the format `.region.`. +Here, `` is the currently active region for your Namespace. +Clients resolving the Namespace’s DNS record are directed to connect to the active region for that Namespace, thanks to the regional DNS record. + +During failover, Temporal Cloud changes the target of the Namespace DNS record from one region to another. +Namespace DNS records are configured with a 15 seconds TTL. +Any DNS cache should re-resolve the record within this delay. +As a rule of thumb, DNS reconciliation takes no longer than twice (2x) the TTL. +Clients should converge to the newly targeted region within, at, most a 30-second delay. + +## PrivateLink routing {#privatelink-routing} + +
**This section needs fixing for regions vs isolation domains**
+ +:::important + +Some networking configuration is required for failover to be transparent to clients and workers when using PrivateLink. +This section describes how to configure routing for multi-region Namespaces for PrivateLink customers only. + +::: + +PrivateLink customers may need to change certain configurations for multi-region Namespace use. +Routing configuration depends on networking setup and use of PrivateLink. +You may need to: + +- override a DNS zone; and +- ensure the network connectivity between the two regions. + +![Customer side solution example](/img/multi-region/private-link.png) + +When using PrivateLink, you connect to Temporal Cloud using IP addresses local to your network. +The `region.` zone is configured in the Temporal systems as an independent zone. +This allows you to override it to make sure traffic is routed internally for the regions in use. +You can check the Namespace's active region using the Namespace record CNAME, which is public. + +To set up the DNS override, you override specific regions to target the relevant IP addresses (e.g. aws-us-west-1.region.tmprl.cloud to target 192.168.1.2). +Using AWS, this can be done using a private hosted zone in Route53 for `region.`. +Link that private zone to the VPCs you use for Workers. +Private Link is not yet offered for GCP multi-region Namespaces. + +When your Workers connect to the Namespace, they first resolve the `..` record. +This targets `.region.` using a CNAME. +Your private zone overrides that second DNS resolution, leading traffic to reach the internal IP you're using. + +Consider how you'll configure Workers to run in this scenario. +You might set Workers to run in both regions at all times. +Alternately, you could establish connectivity between the regions to redirect Workers once failover occurs. + +The following table lists Temporal's available regions, PrivateLink endpoints, and DNS record overrides. +The `sa-east-1` region listed here is not yet available for use with multi-region Namespaces. + +| Region | PrivateLink Service Name | DNS Record Override | +| ---------------- | -------------------------------------------------------------- | --------------------------------------- | +| `ap-northeast-1` | `com.amazonaws.vpce.ap-northeast-1.vpce-svc-08f34c33f9fb8a48a` | `aws-ap-northeast-1.region.tmprl.cloud` | +| `ap-northeast-2` | `com.amazonaws.vpce.ap-northeast-2.vpce-svc-08c4d5445a5aad308` | `aws-ap-northeast-2.region.tmprl.cloud` | +| `ap-south-1` | `com.amazonaws.vpce.ap-south-1.vpce-svc-0ad4f8ed56db15662` | `aws-ap-south-1.region.tmprl.cloud` | +| `ap-south-2` | `com.amazonaws.vpce.ap-south-2.vpce-svc-08bcf602b646c69c1` | `aws-ap-south-2.region.tmprl.cloud` | +| `ap-southeast-1` | `com.amazonaws.vpce.ap-southeast-1.vpce-svc-05c24096fa89b0ccd` | `aws-ap-southeast-1.region.tmprl.cloud` | +| `ap-southeast-2` | `com.amazonaws.vpce.ap-southeast-2.vpce-svc-0634f9628e3c15b08` | `aws-ap-southeast-2.region.tmprl.cloud` | +| `ca-central-1` | `com.amazonaws.vpce.ca-central-1.vpce-svc-080a781925d0b1d9d` | `aws-ca-central-1.region.tmprl.cloud` | +| `eu-central-1` | `com.amazonaws.vpce.eu-central-1.vpce-svc-073a419b36663a0f3` | `aws-eu-central-1.region.tmprl.cloud` | +| `eu-west-1` | `com.amazonaws.vpce.eu-west-1.vpce-svc-04388e89f3479b739` | `aws-eu-west-1.region.tmprl.cloud` | +| `eu-west-2` | `com.amazonaws.vpce.eu-west-2.vpce-svc-0ac7f9f07e7fb5695` | `aws-eu-west-2.region.tmprl.cloud` | +| `sa-east-1` | `com.amazonaws.vpce.sa-east-1.vpce-svc-0ca67a102f3ce525a` | `aws-sa-east-1.region.tmprl.cloud` | +| `us-east-1` | `com.amazonaws.vpce.us-east-1.vpce-svc-0822256b6575ea37f` | `aws-us-east-1.region.tmprl.cloud` | +| `us-east-2` | `com.amazonaws.vpce.us-east-2.vpce-svc-01b8dccfc6660d9d4` | `aws-us-east-2.region.tmprl.cloud` | +| `us-west-2` | `com.amazonaws.vpce.us-west-2.vpce-svc-0f44b3d7302816b94` | `aws-us-west-2.region.tmprl.cloud` | + +## Monitoring replication lag metrics {#metrics} + +Replication lag refers to the transmission delay of Workflow updates and history events from an active to a standby Namespace. +A forced failover when there is a large replication lag has a higher likelihood of rolling back Workflow progress. +Always check the metric replication lag before initiating a high availability failover, especially when working with multi-region deployment. + +Temporal Cloud emits three replication lag-specific [metrics](/production-deployment/cloud/metrics/reference#temporal_cloud_v0_replication_lag_bucket). + +- `temporal_cloud_v0_replication_lag_bucket`: + A histogram of replication lag during a specific time interval for a high availability Namespace. +- `temporal_cloud_v0_replication_lag_count`: + The replication lag count during a specific time interval for a high availability Namespace. +- `temporal_cloud_v0_replication_lag_sum`: + The sum of replication lag during a specific time interval for a high availability Namespace. + +The following samples demonstrate how you can use these metrics to explore replication lag. + +### P99 replication lag histogram + +``` +histogram_quantile(0.99, sum(rate(temporal_cloud_v0_replication_lag_bucket[$__rate_interval])) by (temporal_namespace, le)) +``` + +### Average replication lag + +``` +sum(rate(temporal_cloud_v0_replication_lag_sum[$__rate_interval])) by (temporal_namespace) +/ +sum(rate(temporal_cloud_v0_replication_lag_count[$__rate_interval])) by (temporal_namespace) +``` + +## Viewing operational events {#events} + +You can view and alert on key cloud metrics using the Web UI, the 'tcld' CLI utility, and Temporal Cloud APIs. +For example, during the process of adding a region or isolation domain to a Namespace, you can see the progress of Workflow replication. +Errors -- if any occur -- will also surface in the Namespace Web UI. + +:::info + +You may notice that high-availability Namespaces shows twice (2x) the Action count in `temporal_cloud_v0_total_action_count`. +This doubling happens due to regional replication. + +::: + +Temporal Cloud provides several ways to audit events: + +- When Temporal triggers failovers, the audit log updates with details. + Look specifically for `"operation": "FailoverNamespace"` in the logs. +- You can set alerts for Temporal-initiated failover events. +- After a failover, you can check that the Namespace is active in the new region using the Temporal Cloud Web UI. diff --git a/docs/production-deployment/cloud/high-availability/enable.mdx b/docs/production-deployment/cloud/high-availability/enable.mdx index 839b0fba79..1a59bf83cd 100644 --- a/docs/production-deployment/cloud/high-availability/enable.mdx +++ b/docs/production-deployment/cloud/high-availability/enable.mdx @@ -1,8 +1,8 @@ --- id: enable -title: Enable high availability +title: Enable High Availability features sidebar_label: Enable high availability -slug: /cloud/high-availability/choosing-high-availability +slug: /cloud/high-availability/enable description: Temporal Cloud's High-Availability Namespaces offer automated failover, synchronized data replication, and high availability for workloads requiring disaster-tolerant deployment and 99.99% uptime. Use Global Namespace for self-hosted. tags: - Temporal Cloud @@ -19,278 +19,111 @@ keywords: - temporal-cloud - term --- -import { RelatedReadContainer, RelatedReadItem } from '@site/src/components/related-read/RelatedRead'; - -:::tip Support, stability, and dependency info - -High-availability Namespaces are in [Public Preview](/evaluate/development-production-features/release-stages#public-preview) for Temporal Cloud. - -::: -
**Some audits, updates. Needs intros, re-org. Suggest breaking down into "opting in", "setting up (worker and privatelink)", and "testing" because the content feels really mixed up right now and too long and the metrics section is now a little too short**
- -You can enable the high-availability Namespace feature for your existing Namespace by [adding a second zone](#add-zones) to your Namespace. -After adding the second zone, Temporal Cloud begins data replication for your new standby replica. -Temporal Cloud notifies you once the replication has caught up and both Namespace zones are in sync. +import { RelatedReadContainer, RelatedReadItem } from '@site/src/components/related-read/RelatedRead'; -**Advantages of using a high-availability Namespace:** +You enable High Availability features for a new or existing Namespace by adding a replica to the Namespace. +When you add a replica, Temporal Cloud begins replicating ongoing and existing Workflows. +Once the replication has completed and the replica is ready, your Namespace is ready for failover. -- No manual deployment or configuration needed, just simple push-button operation. -- Open Workflows continue in the standby region with minimal interruption and data loss. -- No changes needed for Worker and Workflow code during setup or failover. -- 99.99% Contractual SLA. +- [Create a high availability Namespace](/cloud/high-availability/enable#create) +- [Upgrade an existing Namespace to high availability functionality](/cloud/high-availability/enable#upgrade) +- [Discontinuing high availability replicas](/cloud/high-availability/enable#discontinuing) -### Create a multi-region Namespace {#create} +## Create a high availability Namespace {#create} -The following sections explain how to create a new multi-region Namespace (MRN). -MRNs provide multi-region deployment backed by Temporal's data replication and active-standby features. +The following sections explain how to create a Namespace with a replica. +You can create a replica within the current region or deploy the replica to a different region. -:::tip +::: note -While reading through this coverage, remember that pairing is currently limited to regions within the same continent. +While reading through this coverage, be aware that replication is not supported in all regions. +For multi-region replication, pairing is limited to regions within the same continent. +For more details, refer to ["Regional availability"].(/cloud/high-availabilityregional-availability). ::: -#### Temporal Cloud Web UI +### Temporal Cloud Web UI -During Namespace creation, specify the first region for the Namespace. -Then, select the “Add a region” option. -Adding a second region enables multi-region Namespace capabilities. +Follow these steps to add replication to your Temporal Cloud Namespace: -#### Temporal 'tcld' CLI +1. During Namespace creation, specify the first region for the Namespace. +2. Select the “Add a replica” option. + Adding a replica in the same region enables Replication. + Adding a replica in a second region enables Multi-region Replication. -Start with the following command to create the new multi-region Namespace: +### Temporal 'tcld' CLI -``` +Enter the following at the command-line to create a replicated Namespace: + +```sh tcld namespace create \ - --namespace . \ - --region + --namespace . \ + --region \ + --region ``` -Include both regions by specifying the [region codes](/cloud/service-availability) as arguments to the `--region` flags. -Before pressing return, add your authentication credentials. For example, `--ca-certificate-file `. - +Specify the [region codes](/cloud/service-availability) as arguments to the two `--region` flags. +- Using the same region replicates to an isolation zone within that region. +- Using a different region within the same continent creates a multi-region Namespace. +Before pressing return, add your authentication credentials. +For example, `--ca-certificate-file `. -## Upgrade an existing single-zone Namespace for high-availability functionality {#add-zones} +## Upgrade an existing Namespace to high availability functionality {#upgrade} -You can upgrade existing ssingle-zone Namespace for high-availability by adding a standby zone. -The following sections show you how. +Upgrade an existing single-region Namespace to high availability features by establishing a replica. +The following sections explain how. +You can either create a replica within the current region or deploy the replica to a different region. -
**The following material has not been audited for MRN/HAN**
+### Temporal Cloud Web UI -#### Temporal Cloud Web UI +Follow these steps to upgrade an existing Namespace: -To upgrade an existing Namespace to a multi-region Namespace: - -1. Visit Temporal Cloud [Namespaces](https://cloud.temporal.io/namespaces) in your Web browser +1. Visit Temporal Cloud Namespaces in your Web browser 1. Navigate to the Namespace details page -1. Select the “Add a region” button. -1. Select the standby region you want to add to this Namespace +1. Select the “Add a replica” button. +1. Choose either **Replication** (in the same region) or **Multi-region Replication** (across regions). + If you select Multi-region Replication, specify which region -You will see an estimated time for replication. -This time is based on your selection and the size and scale of Workflows in your Namespace, -An email alert is sent once your multi-region Namespace is ready for use. +The web interface will present an estimated time for replication to complete. +This time is based on your selection and the size and scale of the Workflows in your Namespace. +An email alert is dispatched once your highly available Namespace is ready for use. -#### Temporal 'tcld' CLI +### Temporal 'tcld' CLI -At the command line, enter: +Enter the following at the command-line to upgrade a Namespace for replication: -``` +```sh tcld namespace add-region \ --namespace . \ --region ``` -Specify the region code for the new region to add. -Before pressing return, add your authentication credentials. For example, `--ca-certificate-file `. -An email alert is sent once your multi-region Namespace is ready for use. - -### Discontinuing multi-region availability {#discontinuing} - -Disabling multi-region removes the high availability and automatic failover features that provide Temporal's highest service level agreement. -To disable the feature and end charges, users must contact [Temporal Support](https://support.temporal.io) directly. -MRN-specific charges for replication will stop once this decommissioning procedure completes. - -- When making your request you must let us know which region you want the Namespace to land in after removing the standby region. -- If you cease services in the middle of the month, your Namespace will be converted to a single region Namespace within 1 business day. -- Temporal won't retain replicated data in the standby region once multi-region has been disabled. -- After disabling multi-region, Temporal Cloud cannot re-enable the feature for a given Namespace for seven days. - -## Triggering failovers {#triggering-failovers} - -Failovers happen automatically in Temporal when a regional outage or disaster affects a multi-region Namespace. -You can also trigger a failover based on custom alerts or for testing purposes. -This section explains how to manually trigger a failover and what to expect afterward. - -Regular failover testing ensures your app can handle disruptions and continue running smoothly in production. -Whether responding to incident warnings or conducting tests, follow the steps in the next sections to move your active Namespace to its standby region and learn how to handle failovers effectively. - -For details on how Temporal detects conditions and triggers failovers automatically, see [Failovers](/cloud/multi-region/#failovers). - -:::warning Check Your Replication Lag - -Always check the [metric replication lag](/production-deployment/cloud/metrics/reference#temporal_cloud_v0_replication_lag_bucket) before initiating a failover. -A forced failover when there is a large replication lag has a higher likelihood of rolling back Workflow progress. - -::: - -**Performing manual failovers** - -You can trigger a failover manually using the Temporal Cloud Web UI or the `tcld` CLI, depending on your preference and setup. -The following table outlines the steps for each method: - -| Method | Instructions | -| ------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -| **Temporal Cloud Web UI** | 1. Visit the [Namespace page](https://cloud.temporal.io/namespaces) on the Temporal Cloud Web UI.
2. Navigate to your Namespace details page and select the **Trigger a failover** option from the menu.
3. After confirming, the failover will be initiated. | -| **Temporal `tcld` CLI** | To manually trigger a failover, run the following command in your terminal:
tcld namespace failover \
    --namespace \.\ \
    --region \ | - -**Post-failover event information** - -After any failover, whether triggered by you or by Temporal, event information appears in both the [Temporal Cloud Web UI](https://cloud.temporal.io/namespaces) (on the Namespace detail page) and in your audit logs. -The audit log entry for Failover uses the `"operation": "FailoverNamespace"` event. -After failover, the Namespace is active in the new region. - -You don't need to monitor Temporal Cloud's failover response in real-time. -Whenever there is a failover event, users with the Account Owner and Global Admin roles automatically receive an alert email. - -**Failbacks** - -After Temporal-initiated failovers, Temporal Cloud shifts Workflow Execution processing back to the original region that was active before the incident (a "failback") once the incident is resolved. +Specify the added [region code](/cloud/service-availability) as an argument to the `--region` flag. -**Reasons to test failing over** +- Using the current region replicates to an isolation zone within your existing region. +- Using a different region within the same continent creates a multi-region Namespace. -Microservices and external dependencies will fail at some point. -Testing failovers ensures your app can handle these failures effectively. -Temporal recommends regular and periodic failover testing for mission-critical applications in production. -By testing in non-emergency conditions, you verify that your app continues to function even when parts of the infrastructure fail. +Before pressing return, add your authentication credentials. +For example, `--ca-certificate-file `. -:::tip Safety First - -If this is your first time performing a failover test, run it with a test-specific namespace and application. -This helps you gain operational experience before applying it to your production environment. -Practice runs help ensure the process runs smoothly during real incidents in production. - -::: - -Trigger testing can: - -- **Validate multi-region deployments**: - In multi-region setups, failover testing ensures your app can run from another region when the primary region experiences outages. - This maintains high availability in mission-critical deployments. - Manual testing confirms the failover mechanism works as expected, so your system handles regional outages or disasters effectively. - -- **Assess replication lag**: - Monitoring [replication lag](/cloud/multi-region#metrics-operations) between regions is crucial in multi-region setups. - Check the lag before initiating a failover to avoid rolling back Workflow progress. - Manual testing helps you practice this critical step and understand its impact. - When there's no real incident, the switch over (recovery) should happen almost instantly. - -- **Assess recovery time**: - Manual testing helps you measure actual recovery time. - You can check if it meets your expected Recovery Time Objective (RTO) of 20 minutes or less, as stated in the [Multi-region Namespace SLA](/cloud/multi-region#sla). - -- **Identify potential issues**: - Failover testing uncovers problems not visible during normal operation. - This includes issues like [backlogs and capacity planning](https://temporal.io/blog/workers-in-production#testing-failure-paths-2438) and how external dependencies behave during a failover event. - -- **Validate fault-oblivious programming**: - Temporal uses a "fault-oblivious programming" model, where your app doesn’t need to explicitly handle many types of failures. - Testing failovers ensures that this model works as expected in your app. - -- **Operational readiness**: - Regular testing familiarizes your team with the failover process, improving their ability to handle real incidents when they arise. - -Testing failovers regularly ensures your Temporal-based applications remain resilient and reliable, even when infrastructure fails. - - -## Worker Deployment {#worker-deployment} - -Enabling the multi-region Namespace does not require specific Worker configuration. -The process is invisible to the Workers. -When a Namespace fails over to the standby region, the DNS redirection orchestrated by Temporal ensures that your existing Workers continue to poll the Namespace without interruption. -More details are available in the [Routing](/cloud/multi-region#routing) section below. - -:::info - -- When a Namespace fails over to a standby region, Workers will be communicating cross-region. - -- In case of a complete regional outage, Workers in the original region may fail alongside the original Namespace. - To keep Workflows moving during this level of outage, deploy a second set of Workers to your standby region. - -::: - -## Routing {#routing} - -When using multi-region for a Namespace, the Namespace's DNS record `..` targets a regional DNS record in the format `.region.`. -In this format, `` is the currently active region for your Namespace. -Clients resolving the Namespace’s DNS record are directed to connect to the active region for that Namespace, thanks to the regional DNS record. - -During failover, Temporal Cloud changes the target of the Namespace DNS record from one region to another. -Namespace DNS records are configured with a 15 seconds TTL. -Any DNS cache should re-resolve the record within this delay. As a rule of thumb, DNS reconciliation takes no longer than twice (2x) the TTL. -Clients should converge to the newly targeted region within, at, most a 30-second delay. - - -## PrivateLink routing {#privatelink-routing} - -:::important - -Some networking configuration is required for failover to be transparent to clients and workers when using PrivateLink. -This section describes how to configure routing for multi-region Namespaces for PrivateLink customers only. - -::: - -PrivateLink customers may need to change certain configurations for multi-region Namespace use. -Routing configuration depends on networking setup and use of PrivateLink. -You may need to: - -- override a DNS zone; and -- ensure the network connectivity between the two regions. - -![Customer side solution example](/img/multi-region/private-link.png) - -When using PrivateLink, you connect to Temporal Cloud using IP addresses local to your network. -The `region.` zone is configured in the Temporal systems as an independent zone. -This allows you to override it to make sure traffic is routed internally for the regions in use. -You can check the Namespace's active region using the Namespace record CNAME, which is public. - -To set up the DNS override, you override specific regions to target the relevant IP addresses (e.g. aws-us-west-1.region.tmprl.cloud to target 192.168.1.2). -Using AWS, this can be done using a private hosted zone in Route53 for `region.`. -Link that private zone to the VPCs you use for Workers. -Private Link is not yet offered for GCP multi-region Namespaces. +An email alert is sent once your multi-region Namespace is ready for use. -When your Workers connect to the Namespace, they first resolve the `..` record. -This targets `.region.` using a CNAME. Your private zone overrides that second DNS resolution, leading traffic to reach the internal IP you're using. +## Discontinuing high availability replicas {#discontinuing} -Consider how you'll configure Workers to run in this scenario. -You might set Workers to run in both regions at all times. -Alternately, you could establish connectivity between the regions to redirect Workers once failover occurs. +Removing a Namespace replica removes the high availability and automatic failover features that provide Temporal's highest service level agreement. +To disable these features and end charges: -The following table lists Temporal's available regions, PrivateLink endpoints, and DNS record overrides. -The `sa-east-1` region listed here is not yet available for use with multi-region Namespaces. +1. Navigate to the Namespace details page in Temporal Cloud. +2. On the “Region” card, select the option to “Remove Replica”. -| Region | PrivateLink Service Name | DNS Record Override | -| ---------------- | -------------------------------------------------------------- | --------------------------------------- | -| `ap-northeast-1` | `com.amazonaws.vpce.ap-northeast-1.vpce-svc-08f34c33f9fb8a48a` | `aws-ap-northeast-1.region.tmprl.cloud` | -| `ap-northeast-2` | `com.amazonaws.vpce.ap-northeast-2.vpce-svc-08c4d5445a5aad308` | `aws-ap-northeast-2.region.tmprl.cloud` | -| `ap-south-1` | `com.amazonaws.vpce.ap-south-1.vpce-svc-0ad4f8ed56db15662` | `aws-ap-south-1.region.tmprl.cloud` | -| `ap-south-2` | `com.amazonaws.vpce.ap-south-2.vpce-svc-08bcf602b646c69c1` | `aws-ap-south-2.region.tmprl.cloud` | -| `ap-southeast-1` | `com.amazonaws.vpce.ap-southeast-1.vpce-svc-05c24096fa89b0ccd` | `aws-ap-southeast-1.region.tmprl.cloud` | -| `ap-southeast-2` | `com.amazonaws.vpce.ap-southeast-2.vpce-svc-0634f9628e3c15b08` | `aws-ap-southeast-2.region.tmprl.cloud` | -| `ca-central-1` | `com.amazonaws.vpce.ca-central-1.vpce-svc-080a781925d0b1d9d` | `aws-ca-central-1.region.tmprl.cloud` | -| `eu-central-1` | `com.amazonaws.vpce.eu-central-1.vpce-svc-073a419b36663a0f3` | `aws-eu-central-1.region.tmprl.cloud` | -| `eu-west-1` | `com.amazonaws.vpce.eu-west-1.vpce-svc-04388e89f3479b739` | `aws-eu-west-1.region.tmprl.cloud` | -| `eu-west-2` | `com.amazonaws.vpce.eu-west-2.vpce-svc-0ac7f9f07e7fb5695` | `aws-eu-west-2.region.tmprl.cloud` | -| `sa-east-1` | `com.amazonaws.vpce.sa-east-1.vpce-svc-0ca67a102f3ce525a` | `aws-sa-east-1.region.tmprl.cloud` | -| `us-east-1` | `com.amazonaws.vpce.us-east-1.vpce-svc-0822256b6575ea37f` | `aws-us-east-1.region.tmprl.cloud` | -| `us-east-2` | `com.amazonaws.vpce.us-east-2.vpce-svc-01b8dccfc6660d9d4` | `aws-us-east-2.region.tmprl.cloud` | -| `us-west-2` | `com.amazonaws.vpce.us-west-2.vpce-svc-0f44b3d7302816b94` | `aws-us-west-2.region.tmprl.cloud` | +The replica will be deleted and your Namespace will no longer be highly available. +You will no longer be charged for this feature. -:::tip Learn more about multi-region Namespaces +:::note -If you have more questions or feedback about this feature, reach out to the product team. +After removing a replica, Temporal Cloud can't re-enable replication in the same region for a given Namespace for seven days. ::: - diff --git a/docs/production-deployment/cloud/high-availability/failovers.mdx b/docs/production-deployment/cloud/high-availability/failovers.mdx new file mode 100644 index 0000000000..425a17490c --- /dev/null +++ b/docs/production-deployment/cloud/high-availability/failovers.mdx @@ -0,0 +1,79 @@ +--- +id: failovers +title: failovers +sidebar_label: Failovers +slug: /cloud/high-availability/failovers +description: Temporal Cloud's High-Availability Namespaces offer automated failover, synchronized data replication, and high availability for workloads requiring disaster-tolerant deployment and 99.99% uptime. Use Global Namespace for self-hosted. +tags: + - Temporal Cloud + - Production + - High availability +keywords: + - availability + - explanation + - failover + - high-availability + - multi-region + - multi-region namespace + - namespaces + - temporal-cloud + - term +--- + +import { RelatedReadContainer, RelatedReadItem } from '@site/src/components/related-read/RelatedRead'; + +A failover shifts Workflow Execution processing from an active Temporal Namespace to replicated Temporal Namespace during outages or other incidents. +Standby Namespace replicas duplicate data and prevent data loss during failover. + +- [Triggering failovers](#triggering-failovers) +- [Initiating manual failovers](#manual-failovers) +- [Post-failover event information](#info) +- [Failbacks](#failbacks) + +## Triggering failovers {#triggering-failovers} + +Temporal automatically initiates failovers when an incident or outage affects a high availability Namespace. +You can also [trigger a failover](/cloud/high-availability/failovers#triggering-failovers) based on your own custom alerts and for testing purposes. +This section explains how to manually trigger a failover and what to expect afterward. + +:::warning Check Your Replication Lag + +Always check the [metric replication lag](/production-deployment/cloud/metrics/reference#temporal_cloud_v0_replication_lag_bucket) before initiating a failover. +A forced failover when there is a large replication lag has a higher likelihood of rolling back Workflow progress. + +::: + +For details on how Temporal detects conditions and triggers failovers automatically, see [the failover process](/cloud/high-availability/failovers). + +## Initiating manual failovers {#manual-failovers} + +You can trigger a failover manually using the Temporal Cloud Web UI or the `tcld` CLI, depending on your preference and setup. +The following table outlines the steps for each method: + +
**Need to update the CLI instructions**
+| Method | Instructions | +| ------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| **Temporal Cloud Web UI** | 1. Visit the [Namespace page](https://cloud.temporal.io/namespaces) on the Temporal Cloud Web UI.
2. Navigate to your Namespace details page and select the **Trigger a failover** option from the menu.
3. After confirmation, Temporal initiates the failover. | +| **Temporal `tcld` CLI** | To manually trigger a failover, run the following command in your terminal:
tcld namespace failover \
    --namespace \.\ \
    --region \
Temporal fails over the Namespace to the target region. High availability Namespaces using a single region will failover to the standby isolation domain. | + +## Post-failover event information {#info} + +After any failover, whether triggered by you or by Temporal, event information appears in both the [Temporal Cloud Web UI](https://cloud.temporal.io/namespaces) (on the Namespace detail page) and in your audit logs. +The audit log entry for Failover uses the `"operation": "FailoverNamespace"` event. +After failover, the replica becomes active, taking over and the Namespace is active in the new isolation domain or region. + +You don't need to monitor Temporal Cloud's failover response in real-time. +Whenever there is a failover event, users with the Account Owner and Global Admin roles automatically receive an alert email. + +## Failbacks + +After Temporal-initiated failovers, Temporal Cloud shifts Workflow Execution processing back to the original region or isolation zone that was active before the incident once the incident is resolved. +This is called a "failback". + +## Disabling Temporal-initiated failovers + +When you add a replica to a Namespace, it becomes a high availability Namespace. +In the event of an incident or an outage Temporal Cloud automatically fails over a high availability Namespaces to its replica. +_This is the recommended and default option_. + +If you prefer to disable Temporal-initiated failovers and handle your own failovers, you can do so by navigating to the Namespace detail page in Temporal Cloud. Choose the "Disable Temporal-initiated failovers" option. diff --git a/docs/production-deployment/cloud/high-availability/faq.mdx b/docs/production-deployment/cloud/high-availability/faq.mdx index 85b75a94b4..a3bef904da 100644 --- a/docs/production-deployment/cloud/high-availability/faq.mdx +++ b/docs/production-deployment/cloud/high-availability/faq.mdx @@ -19,201 +19,224 @@ keywords: - temporal-cloud - term --- -import { RelatedReadContainer, RelatedReadItem } from '@site/src/components/related-read/RelatedRead'; - -:::tip Support, stability, and dependency info -High-availability Namespaces are in [Public Preview](/evaluate/development-production-features/release-stages#public-preview) for Temporal Cloud. - -::: +import { RelatedReadContainer, RelatedReadItem } from '@site/src/components/related-read/RelatedRead';
**Repurposed material. No audits, updates, intros, re-org. Converted to markdown automatically from Google Doc rich text, so there are many errors**
-Failovers +## Failovers -**Q: What is a failover****** +**Q: What is a failover** -A failover shifts Workflow Execution processing from an active Temporal Namespace to a standby Temporal Namespace during outages or other incidents. Standby Namespaces use replication to duplicate data and prevent data loss during failover. +A failover shifts Workflow Execution processing from an active Temporal Namespace to a standby Temporal Namespace during outages or other incidents. +Standby Namespaces use replication to duplicate data and prevent data loss during failover. -**Q: What failover modes does Temporal use internally?****** +**Q: What failover modes does Temporal use internally?** Users cannot configure failover modes. The following descriptions explain Temporal Cloud’s internal failover system: - * ******[Graceful failover**](https://docs.temporal.io/cloud/multi-region#graceful-failover): Replication tasks are fully processed and drained before transferring control to the standby region. Temporal Cloud pauses traffic to the active Namespace before the failover, minimizing the rewind of progress and avoiding data conflicts. The Namespace experiences a short period of unavailability, defaulting to 10 seconds at most. Under most circumstances, the actual time the Namespace is unavailable is much, much shorter than that. - -During this period, existing Workflows stop progress. Temporal Cloud returns a "Service unavailable error". State transitions will not happen and tasks are not dispatched. User requests like start/signal Workflow will be rejected while operations are paused during handover. This mode favors _consistency_ over availability. - - * ******[Forced failover**](https://docs.temporal.io/cloud/multi-region#forced-failover): In this mode, a Namespace immediately activates in the standby region. Events not replicated due to replication lag will undergo conflict resolution upon reaching the new active region. This mode prioritizes _availability_ over consistency. - * ******[Hybrid failover**](https://docs.temporal.io/cloud/multi-region#hybrid-failover) (Default mode): While graceful failovers are consistent, they aren’t always practical in certain circumstances such as when cells experience outages and/or a critical database is unavailable. Temporal Cloud’s hybrid failover mode limits an initial Graceful failover attempt to 10 seconds or less. If the graceful approach doesn’t resolve the issue, Temporal Cloud automatically switches to a forced failover. This strategy balances consistency and availability requirements. This strategy balances _consistency_ and _availability_ requirements. - -**Q: What is the difference between a handover and a failover?****** - -They are essentially the same thing. It is the process of transferring control from the active to the standby region during outages or other incidents. - - +- **[Graceful failover**](/cloud/high-availability/how-it-works#graceful-failover): + In this mode, replication tasks are fully processed and drained. + Temporal Cloud pauses traffic to the Namespace before the failover. + This prevents the loss of progress and avoids data conflicts. + The Namespace experiences a short period of unavailability, defaulting to 10 seconds. + During this period, existing Workflows stop progress. + Temporal Cloud returns a "Service unavailable error", which is retried by SDKs. + State transitions will not happen and tasks are not dispatched. + User requests like start/signal workflow will be rejected while operations are paused during handover. + This mode favors _consistency_ over availability. +- **[Forced failover**](/cloud/high-availability/how-it-works#forced-failover): + In this mode, a replica immediately activates in the standby Namespace. + Events not replicated due to [replication lag](/cloud/high-availability/best-practices#metrics) will undergo [conflict resolution](/cloud/high-availability/how-it-works#conflict-resolution) upon reaching the new active Namespace. + This mode prioritizes _availability_ over consistency. +- **[Hybrid failover**](/cloud/high-availability/how-it-works#hybrid-failover) (Default mode): + While graceful failovers are preferred for consistency, they aren’t always practical. + Temporal Cloud’s hybrid failover mode (the default mode) limits an initial graceful failover attempt to 10 seconds or less. + During this period, existing Workflows stop progress. + Temporal Cloud returns a "Service unavailable error", which is retried by SDKs. + If the graceful approach doesn’t resolve the issue, Temporal Cloud automatically switches to a forced failover. + This strategy balances consistency and availability requirements. +**Q: What is the difference between a handover and a failover?** +They are essentially the same thing. It is the process of transferring control from the active to the standby region during outages or other incidents. -**Q: What situation triggers a graceful failover vs a forced one? Who or what triggers it? What are the differences in results?****** +**Q: What situation triggers a graceful failover vs a forced one? Who or what triggers it? What are the differences in results?** Users can initiate a failover, but they can’t control or configure the failover mode. The three failover modes are internal operations on the Temporal side. They are explained for user education. -**Q: Under what circumstances would I want to initiate a failover?****** +**Q: Under what circumstances would I want to initiate a failover?** -Normally, we don't expect users to failover when there’s a problem related to Temporal Cloud. Please contact Temporal support if you feel you have a pressing need. You might consider initiating a failover under two circumstances: +Normally, we don't expect users to failover when there’s a problem related to Temporal Cloud. +Please contact Temporal support if you feel you have a pressing need. +You might consider initiating a failover under two circumstances: - * The “break glass”  (fire alarm) scenario where Temporal hasn’t responded to an outage or is unaware of an outage in the customer environment. - * To test the failover functionality and ensure it works properly. +- The “break glass” (fire alarm) scenario where Temporal hasn’t responded to an outage or is unaware of an outage in the customer environment. +- To test the failover functionality and ensure it works properly. You can still choose to initiate failover if you have issues sourced from your side or your dependencies. -**Q: Under what circumstances does Temporal initiate failovers?****** +**Q: Under what circumstances does Temporal initiate failovers?** + +Temporal Cloud initiates failovers when there are incidents or outages in the cloud provider. +This includes failures of databases, storage, etc. +We trigger failovers any time we observe increased latencies or an increase in service errors that causes us to violate the SLA that is in our control. -Temporal Cloud initiates failovers when there are incidents or outages in the cloud provider. This includes failures of databases, storage, etc. We trigger failovers any time we observe increased latencies or an increase in service errors that causes us to violate the SLA that is in our control. +Read more about [automatic failovers](/cloud/high-availability/how-it-works#healthchecks). -**Q: Are there any other types of failover not listed above?****** +**Q: Are there any other types of failover not listed above?** -No. There are only three types. +No. +There are only three types. -**Q: Can we control the hybrid failover timeout?****** +**Q: Can we control the hybrid failover timeout?** -No. The timeout is not configurable outside of Temporal. +No. +The timeout is not configurable outside of Temporal. -**Q: Can a failover get stuck? What is the maximum amount of time it can take?****** +**Q: Can a failover get stuck? What is the maximum amount of time it can take?** -There is typically no way for failovers to get “stuck”. We follow the hybrid failover method where we try to do a smooth handoff. If that does not take place within 10 seconds, we initiate a “forced” failover. +There is typically no way for failovers to get “stuck”. +We follow the hybrid failover method where we try to do a smooth handoff. +If that does not take place within 10 seconds, we initiate a “forced” failover. -**Q: Is the 10 seconds maximum unavailability window configurable?****** +**Q: Is the 10 seconds maximum unavailability window configurable?** -No, it is not configurable by the user. Extending the wait time is unlikely to increase the chances of graceful failovers during extreme incidents such as when a source region is down. +No, it is not configurable by the user. +Extending the wait time is unlikely to increase the chances of graceful failovers during extreme incidents such as when a source region is down. -**Q: How does the client detect that the failover has occurred?****** +**Q: How does the client detect that the failover has occurred?** -We do not send real-time failover notifications. Users are notified via email and audit logs. +We do not send real-time failover notifications. +Users are notified via email and audit logs. -**Q: Can the customer determine the resolved failover region?****** +**Q: Can the customer determine the resolved failover region?** -Users can determine the failover region from the Namespace endpoint’s CNAME (\.tmprl.cloud). Whenever Temporal Cloud triggers a failover from the Temporal side, we update the CNAME to point to the new active region. The CNAME points to a Temporal Cloud regional endpoint. For example, a Namespace active in aws us-east-1 points to aws-**us-east-1**.region.tmprl.cloud. +Users can determine the failover region from the Namespace endpoint’s CNAME (\.tmprl.cloud). +Whenever Temporal Cloud triggers a failover from the Temporal side, we update the CNAME to point to the new active region. +The CNAME points to a Temporal Cloud regional endpoint. +For example, a Namespace active in aws us-east-1 points to aws-**us-east-1**.region.tmprl.cloud. -Replication Lag and Latency +## Replication Lag and Latency -**Q: What affects replication latency? What can cause the replication latency to increase?****** +**Q: What affects replication latency? What can cause the replication latency to increase?** -Slowdowns in the standby cell, such as capacity issues or outages, can increase replication latency. Otherwise, it is typically a matter of seconds or even less and can be monitored through  [external metrics](https://docs.temporal.io/cloud/multi-region#metrics-operations). +Slowdowns in the standby cell, such as capacity issues or outages, can increase replication latency. +Otherwise, it is typically a matter of seconds or even less and can be monitored through  [external metrics](/cloud/high-availability/best-practices#metrics). -**Q: Can Workflows execute events a second time in the standby Cluster due to replication lag?****** +**Q: Can Workflows execute events a second time in the standby Cluster due to replication lag?** -Yes. This is explained in the [conflict resolution](https://docs.temporal.io/cloud/multi-region#conflict-resolution) section of our documentation. +Yes. This is explained in the [conflict resolution](/cloud/high-availability/how-it-works#conflict-resolution) section of our documentation. -**Q: Is it possible to see whether a failover was graceful, forced, or hybrid?****** +**Q: Is it possible to see whether a failover was graceful, forced, or hybrid?** No, customers cannot normally view the method used. File a support ticket if there’s a specific need to review a process. -**Q: Is replication lag emitted as a metric?****** +**Q: Is replication lag emitted as a metric?** -Yes, replication lag is a [metric](https://docs.temporal.io/cloud/multi-region#metrics-operations) that we expose. +Yes, replication lag is a [metric](/cloud/high-availability/best-practices#metrics) that we expose. -**Q: Can we see replication information by Workflow type or ID?****** +**Q: Can we see replication information by Workflow type or ID?** _[Answer in progress]___ -**Q: Is the data replicated in order?****** +**Q: Is the data replicated in order?** For a single Workflow, events are replicated  in order. There's no ordering guarantee for replication of events between different Workflows. -**Q: What happens if both regions become active simultaneously?****** +**Q: What happens if both regions become active simultaneously?** This only happens when there's a network partition or delays in the Namespace replication queue. Normally, when cells can talk to each other, only one region will ever become active. If both regions have become active and both have active Workers, Workflows will run independently based on their local History. Workers fetch tasks from their assigned region. With global Worker setups, Workers fetch tasks from the ‘true’ active region as known by Temporal Cloud. Eventually, when the network partition heals, History is merged via conflict resolution and one side wins. -**Q: What if DNS is still updating during a network partition between Clusters? ****** +**Q: What if DNS is still updating during a network partition between Clusters? ** In this situation, the now passive Cluster can’t forward requests to the new active Cluster. However, DNS normally points to the correct active Cluster without forwarding. Workers configured to point to the standby Cluster can be reconfigured to point to the active Cluster. -Conflict Resolution +## Conflict Resolution See [this Notion Page](https://www.notion.so/temporalio/Conflict-Resolution-Example-83e9dec0f8f246ee8584995ae2e408f4) for an example Conflict Resolution. -**Q: How are conflicts resolved?****** +**Q: How are conflicts resolved?** Each cell has a version number, which is used in Event History metadata. Failover operations increase that number. Events with the highest number win during conflict resolution. -**Q: What happens to Workflows if conflicts can’t be resolved?****** +**Q: What happens to Workflows if conflicts can’t be resolved?** This can only happen if there is a bug in the conflict resolution.  If there is a bug in conflict resolution, those events are placed in a dead letter queue to unblock replication. Temporal will resolve the issue and reapply the events. Customer impact is limited to the affected Workflows. The rest of the system continues as normal. -**Q: How is History affected if conflicts can’t be resolved?****** +**Q: How is History affected if conflicts can’t be resolved?** Same as above. -**Q: How do customers detect unresolved conflicts?****** +**Q: How do customers detect unresolved conflicts?** Unresolved conflicts are not made visible to customers. Temporal directs unresolvable conflicts (conflicts that require Temporal on-call intervention) into a dead letter queue and makes sure those conflicts are resolved and their events re-applied. -**Q: How do customers manually resolve conflicts?****** +**Q: How do customers manually resolve conflicts?** No manual resolution by customers is needed unless Temporal cannot handle a specific scenario. -**Q: Are non-selected event histories deleted during automatic conflict resolution? ****** +**Q: Are non-selected event histories deleted during automatic conflict resolution? ** No. They are hidden but not deleted. We do not expose access to non-selected events to customers. Data Loss -**Q: Under what circumstances would a Workflow Execution be unrecoverable if it was started but not replicated before failover?****** +**Q: Under what circumstances would a Workflow Execution be unrecoverable if it was started but not replicated before failover?** The normal time difference between the two operations is typically measured in single-digit seconds.  So this scenario can only happen if the Cluster is healthy enough to accept the Workflow start request and fails to replicate this event. This is very unlikely. If it did happen, the started Workflow is recovered after the Cluster is itself recovered. The only possibility of data loss would require that Temporal lose contact with the previously active cell after permanently completing an operation. -Metrics and Observability +## Metrics and Observability -**Q: What information can be pulled from MRN metrics?****** +**Q: What information can be pulled from MRN metrics?** -This is [documented](https://docs.temporal.io/cloud/multi-region#metrics-operations). +This is [documented](/cloud/high-availability/best-practices#metrics). Always check metric replication lag before initiating a failover test or emergency failover. A forced failover when there is a large replication lag has a higher likelihood of rolling back Workflow progress. -**Q: What warning signs Signal that a failover may be arriving?****** +**Q: What warning signs Signal that a failover may be arriving?** You should always be prepared for failover. One could happen at any point in time. We notify customers when a failover occurs. There is no time lapse between discovering failover prerequisites and the failover itself. -Other +## Other -**Q: Can Signals be sent twice since multi-region doesn't provide at-most-once delivery?****** +**Q: Can Signals be sent twice since multi-region doesn't provide at-most-once delivery?** During conflict resolution, a Signal could be applied twice. -**Q: What happens if the active region is unavailable for an extended period and the standby region does not have the most recent Signal?**** ****** +**Q: What happens if the active region is unavailable for an extended period and the standby region does not have the most recent Signal?**** ** Workers cannot process the Signal as it won’t be present in any available region. -**Q: If the active region remains unavailable for an extended period, does the active role switch to the standby region? ****** +**Q: If the active region remains unavailable for an extended period, does the active role switch to the standby region? ** If Temporal Cloud initiated the failover, it will “fail back” to the original active region once the incident is fully resolved. Otherwise, the active role remains with the newly active (formerly standby) region. -**Q: ****[Is there a way to determine the region an event ran in via the UI?******](https://temporaltechnologies.slack.com/archives/C04V0LSU5S6/p1717092196761469) +**Q: ****[Is there a way to determine the region an event ran in via the UI?**](https://temporaltechnologies.slack.com/archives/C04V0LSU5S6/p1717092196761469) Not at the moment. -**Q: ****[Can we show branching in the UI?******](https://temporaltechnologies.slack.com/archives/C04V0LSU5S6/p1717171982044329) +**Q: ****[Can we show branching in the UI?**](https://temporaltechnologies.slack.com/archives/C04V0LSU5S6/p1717171982044329) Not at the moment. -**Q: What should customers worry about in terms of Signals and events synchronization?****** +**Q: What should customers worry about in terms of Signals and events synchronization?** Signals are cherry-picked during conflict resolution if there is replication lag and conflict. Workflows can theoretically revert multiple steps. -Customers should decide whether to add logic to handle this or manually fix affected Workflows if they believe the risk is low. Other known limitations have been [documented](https://docs.temporal.io/cloud/multi-region#architecture) around causality and so forth. +Customers should decide whether to add logic to handle this or manually fix affected Workflows if they believe the risk is low. Other known limitations have been [documented](/cloud/high-availability/how-it-works#architecture) around causality and so forth. -**Q: How much time does it take to reconcile data after an incident is resolved? ****** +**Q: How much time does it take to reconcile data after an incident is resolved? ** It depends on the distribution of Workflows. If evenly distributed, data can sync quickly. If concentrated in a single partition, it could take hours. Do not “fail back” your region (revert it to the original active region) until the data is fully reconciled and the other region has caught up. diff --git a/docs/production-deployment/cloud/high-availability/guarantees.mdx b/docs/production-deployment/cloud/high-availability/guarantees.mdx deleted file mode 100644 index f4361e7325..0000000000 --- a/docs/production-deployment/cloud/high-availability/guarantees.mdx +++ /dev/null @@ -1,61 +0,0 @@ ---- -id: guarantees -title: Guarantees and availability -sidebar_label: Guarantees and availability -slug: /cloud/high-availability/guarantees -description: Temporal Cloud's High-Availability Namespaces offer automated failover, synchronized data replication, and high availability for workloads requiring disaster-tolerant deployment and 99.99% uptime. Use Global Namespace for self-hosted. -tags: - - Temporal Cloud - - Production - - High availability -keywords: - - availability - - explanation - - failover - - high-availability - - multi-region - - multi-region namespace - - namespaces - - temporal-cloud - - term ---- - -:::tip Support, stability, and dependency info - -High-availability Namespaces are in [Public Preview](/evaluate/development-production-features/release-stages#public-preview) for Temporal Cloud. - -::: - -
**Maybe these could just be parts of the FAQ?**
- - -## Multi-region Namespace SLA {#sla} - -**What guarantees does Temporal offer for multi-region Namespaces?** - -Multi-region Namespaces offer 99.99% availability, enforced by Temporal Cloud's [service error rates SLA](https://docs.temporal.io/cloud/sla). -Our system is designed to limit data loss after recovery when the incident triggering the failover is resolved. - -Our recovery point objective ([RPO](https://en.wikipedia.org/wiki/Disaster_recovery#Recovery_Point_Objective)) is near-zero. -There may be a short period of time during an incident or forced failover when some data is unavailable in the standby region. -Some Workflow History data won't arrive until networks issue are fixed, enabling the History to finish replicating and the divergent History branches to reconcile. - -Temporal Cloud proactively responds to incidents by triggering failovers. -Our recovery time objective ([RTO](https://en.wikipedia.org/wiki/Disaster_recovery#Recovery_Time_Objective)) is 20 minutes or less per incident. - -:::info - -During a disaster scenario in which the data on the hard drives in the active region cannot be recovered, the duration of data loss may be as high as the [replication lag](/cloud/multi-region#replication-lag) at the time of disaster. - -::: - -### Regional availability {#regional-availability} - -Multi-region Namespaces are available in all existing [Temporal Cloud regions](/cloud/service-availability#regions). - -:::tip - -Namespace pairing is currently limited to regions within the same continent. -South America is excluded as only one region is available. - -::: diff --git a/docs/production-deployment/cloud/high-availability/how-it-works.mdx b/docs/production-deployment/cloud/high-availability/how-it-works.mdx index 9311b6794e..8dc196ffbd 100644 --- a/docs/production-deployment/cloud/high-availability/how-it-works.mdx +++ b/docs/production-deployment/cloud/high-availability/how-it-works.mdx @@ -19,19 +19,12 @@ keywords: - temporal-cloud - term --- -import { RelatedReadContainer, RelatedReadItem } from '@site/src/components/related-read/RelatedRead'; - -:::tip Support, stability, and dependency info - -High-availability Namespaces are in [Public Preview](/evaluate/development-production-features/release-stages#public-preview) for Temporal Cloud. - -::: -
**No audits, updates, intros, re-org**
+import { RelatedReadContainer, RelatedReadItem } from '@site/src/components/related-read/RelatedRead'; In traditional active/active replication, multiple nodes serve requests and accept writes simultaneously, ensuring strong synchronous data consistency. -In contrast, with a Temporal Cloud high-availability Namespace, only the active zone accepts requests and writes at any given time. -Workflow history events are written to the active zone first and then asynchronously replicated to the standby zone replica, ensuring that the replica remains in sync. +In contrast, with a Temporal Cloud high-availability Namespace, only the active Namespace accepts requests and writes at any given time. +Workflow history events are written to the active Namespace first and then asynchronously replicated to the standby replica, ensuring that the replica remains in sync.
**Needs new images**
@@ -39,22 +32,21 @@ Workflow history events are written to the active zone first and then asynchrono | :-------------------------------------------------------: | :-----------------------------------------------------: | | ![Before failover](/img/multi-region/before-failover.png) | ![After failover](/img/multi-region/after-failover.png) | -## Failovers {#failovers} +## The failover process {#failovers} -A failover shifts Workflow Execution processing from an active Temporal Namespace region to a standby Temporal Namespace region during outages or other incidents. -Standby Namespace regions use replication to duplicate data and prevent data loss during failover. +A failover shifts Workflow Execution processing from an active Temporal Namespace to a standby replica during outages or other incidents. +Standby replicas duplicate data and prevent data loss during failover. **What happens during the failover process?** -Temporal Cloud initiates a Namespace failover when it detects an incident or outage that raises error rates or latency in the active region of a multi-region Namespace. -The failover shifts Workflow processing to a standby region that isn’t affected by the incident. +Temporal Cloud initiates a Namespace failover when it detects an incident or outage that raises error rates or latency in the active region of a high availability Namespace. +The failover shifts Workflow processing to a replica that isn’t affected by the incident. This lets existing Workflows continue and new Workflows start while the incident is fixed. -Once the incident is resolved, Temporal Cloud performs a "failback" by shifting Workflow Execution processing back to the original region. - +Once the incident is resolved, Temporal Cloud performs a "failback" by shifting Workflow Execution processing back to the original Namespace. :::info -You can test the failover of your multi-region Namespace by manually [triggering a failover](/cloud/multi-region#triggering-failovers) using the UI page or the 'tcld' CLI utility. +You can test the failover of your high availability Namespace by manually [triggering a failover](/cloud/high-availability/failovers#triggering-failovers) using the UI page or the 'tcld' CLI utility. In most scenarios, we recommend you let Temporal handle failovers for you. ::: @@ -69,15 +61,15 @@ It automatically triggers failovers when these indicators exceed our allowed thr ### Replication lag {#replication-lag} -Multi-region Namespaces use asynchronous replication between regions. -Workflow updates in the active region, along with associated history events, are transmitted to the standby region with a short delay. +High availability Namespaces use asynchronous replication. +Workflow updates in the active Namespace, along with associated history events, are transmitted to the standby replica with a short delay. This delay is called the replication lag. Temporal Cloud strives to maintain a P95 replication delay of less than 1 minute. In this context, P95 means 95% of requests are processed faster than this specified limit. -Replication lags mean a [forced failover](/cloud/multi-region#forced-failover) may cause Workflows to rollback in progress. -Lags may also cause recently started Workflows to be temporarily unavailable until the active region recovers. -Temporal event versioning and [conflict resolution mechanisms](/cloud/multi-region#conflict-resolution) help guarantee that the Workflow Event History can be replayed. +Replication lags mean a [forced failover](/cloud/high-availability/how-it-works#forced-failover) may cause Workflows to rollback in progress. +Lags may also cause recently started Workflows to be temporarily unavailable until a Namespace recovers. +Temporal event versioning and [conflict resolution mechanisms](/cloud/high-availability/how-it-works#conflict-resolution) help guarantee that the Workflow Event History can be replayed. Critical operations like Signals won't get lost. ### Failover scenarios @@ -102,8 +94,8 @@ This mode favors _consistency_ over availability. #### Forced failover {#forced-failover} -In this mode, a Namespace immediately activates in the standby region. -Events not replicated due to [replication lag](/cloud/multi-region#replication-lag) will undergo [conflict resolution](/cloud/multi-region#conflict-resolution) upon reaching the new active region. +In this mode, a replica immediately activates in the standby Namespace. +Events not replicated due to [replication lag](/cloud/high-availability/best-practices#metrics) will undergo [conflict resolution](/cloud/high-availability/how-it-works#conflict-resolution) upon reaching the new active Namespace. This mode prioritizes _availability_ over consistency. @@ -116,32 +108,34 @@ Temporal Cloud returns a "Service unavailable error", which is retried by SDKs. If the graceful approach doesn’t resolve the issue, Temporal Cloud automatically switches to a forced failover. This strategy balances consistency and availability requirements. -See the sections on [triggering a failover](/cloud/multi-region#triggering-failovers), [Worker deployment](/cloud/multi-region#worker-deployment), and [routing](/cloud/multi-region#routing) for more information. +See the sections on [triggering a failover](/cloud/high-availability/failovers/#triggering-failovers), [Worker deployment](/cloud/high-availability/best-practices/#worker-deployment), and [routing](/cloud/high-availability/best-practices#routing) for more information. ## Architecture {#architecture} -**How do multi-region Namespaces work?** +**How do high availability Namespaces work?** -Multi-region Namespaces replicate Namespace metadata and Workflow Executions across connected regions. +High availability Namespaces replicate Namespace metadata and Workflow Executions across connected Namespaces. This redundancy, plus the added failover capability, provides measurable stability when dealing with outages. -A multi-region Namespace is normally active in a single region at any moment. -The passive region assumes a standby role. +A high availability Namespace is normally active in a single isolation domain at any moment. +The passive replica assumes a standby role. An exception to this only occurs in the event of a network partition. -In this case, you may elect to promote a standby region to active status. +In this case, you may elect to promote a standby isolation domain to active status. Caution: this action will temporarily result in both regions being active. -Once the network partition resolves and communication between the regions is restored, a conflict resolution algorithm determines which region continues as the active one. -This ensures only one region remains active. +Once the network partition resolves and communication between the isolation domains/regions is restored, a conflict resolution algorithm determines which region continues as the active one. +This ensures only one Namespace remains active. ### Metadata replication {#metadata-replication} -Updates to multi-region Namespace records automatically replicate across regions. +Updates to high availabillity Namespace records automatically duplicate to their replica. This metadata includes configurations such as retention periods, Search Attributes, and other settings. -Temporal Cloud ensures that all regions will eventually share a consistent and unified view of the Namespace metadata. +Temporal Cloud ensures that all isolation domains and regions will eventually share a consistent and unified view of the Namespace metadata. + +
**Needs correct field name**
:::info -A Namespace failover, which changes the "active region" field of a Namespace record, is an update. +A Namespace failover, which changes the identifier for the active element field of a Namespace record, is an update. This update is replicated via the Namespace metadata mechanism. ::: @@ -150,36 +144,37 @@ This update is replicated via the Namespace metadata mechanism. Temporal Cloud restricts certain Workflow operations to the active region: -- You may only update Workflows in the active region. -- You may only dispatch Workflow Tasks and Activity Tasks from the active region. Forward progress in a Workflow Execution can therefore only be made in the active region. +- You may only update Workflows in the active Namespace. +- You may only dispatch Workflow Tasks and Activity Tasks from the active Namespace. + Forward progress in a Workflow Execution can therefore only be made in the active Namespace. -These limits mean that certain requests, such as Start Workflow and Signal Workflow, are processed by and limited to the active region. -Standby regions may receive API requests from Clients and Workers. +These limits mean that certain requests, such as Start Workflow and Signal Workflow, are processed by and limited to the active Namespace. +Standby replicas may receive API requests from Clients and Workers. They automatically forward these requests to the active Namespace for execution. -Multi-region Namespaces provide an “all-active” experience for Temporal users. +High availability Namespaces provide an “all-active” experience for Temporal users. This helps limit or eliminate downtime during Namespace failover. -There's a short time window from when a standby region becomes the active region to when Clients and Workers receive a DNS update. -During this time requests forward from the now passive (formerly active) region to the newly active (formerly standby) region. +There's a short time window from when a standby replica becomes the active Namespace to when Clients and Workers receive a DNS update. +During this time requests forward from the now passive (formerly active) replica Namespace to the newly active (formerly standby replica) Namespace. -As Workflow Executions progress and are operated on, replication tasks created in the active region are dispatched to the standby region. -Processing these replication tasks ensures that the standby region undergoes the same state transitions as the active region. +As Workflow Executions progress and are operated on, replication tasks created in the active Namespace are dispatched to the standby replica. +Processing these replication tasks ensures that the standby replica undergoes the same state transitions as the active Namespace. This enables replicated tasks to synchronize and achieve the same state as the original tasks. -Standby regions do not distribute Workflow or Activity Tasks. +Standby replicas do not distribute Workflow or Activity Tasks. Instead, they perform verification tasks to confirm that intended operations are executed so Workflows reach the desired state. This mechanism ensures consistency and reliability in the replication process across Temporal regions. ### Conflict Resolution {#conflict-resolution} -Multi-region Namespaces rely on asynchronous event replication across Temporal regions. -In the event of a non-graceful failover, replication lag may result in a temporary setback in workflow progress. +High availability Namespaces rely on asynchronous event replication across Temporal isolation domains and regions. +In the event of a non-graceful failover across regions, replication lag may result in a temporary setback in Workflow progress. -Single-region Namespaces can be configured to provide _at-most-once_ semantics for Activities execution (when [Maximum Attempts](https://docs.temporal.io/retry-policies#maximum-attempts) is set to 0). -Multi-region Namespaces provide _at-least-once_ semantics for execution of Activities. +Namespaces that do not participate in high availability can be configured to provide _at-most-once_ semantics for Activities execution (when [Maximum Attempts](https://docs.temporal.io/retry-policies#maximum-attempts) is set to 0). +High availability Namespaces provide _at-least-once_ semantics for execution of Activities. Completed Activities _may_ be re-dispatched in a newly active region, leading to repeated executions. -When a Workflow Execution is updated in a new region following a failover, events from the previously active region that arrive after the failover can't be directly applied. +When a Workflow Execution is updated in a new Namespace following a failover, events from the previously active Namespace that arrive after the failover can't be directly applied. At this point, Temporal Cloud has forked the Workflow History. After failover, Temporal Cloud creates a new branch history for execution, and begins its conflict resolution process. diff --git a/docs/production-deployment/cloud/high-availability/index.mdx b/docs/production-deployment/cloud/high-availability/index.mdx index 7fda043014..3b8b999b52 100644 --- a/docs/production-deployment/cloud/high-availability/index.mdx +++ b/docs/production-deployment/cloud/high-availability/index.mdx @@ -22,92 +22,115 @@ keywords: import { RelatedReadContainer, RelatedReadItem } from '@site/src/components/related-read/RelatedRead'; -:::tip Support, stability, and dependency info +Temporal Cloud's replicated Namespaces provide disaster-tolerant deployment for workloads where availability is critical to your operations. +When you enable high availability, Temporal Cloud automatically synchronizes your data between a primary and a fallback Namespace, keeping them in sync. +Should an incident occur, Temporal will [failover](/glossary#failover) your Namespace. +This allows your Workflow Executions and Schedules to seamlessly shift from the active availability zone to the synchronized replica in the fallback availability zone. -High-availability Namespaces are in [Public Preview](/evaluate/development-production-features/release-stages#public-preview) for Temporal Cloud. +Advantages of using Temporal Cloud’s High Availability features: -::: +- No manual deployment or configuration needed, just simple push-button operations. +- Existing Workflows resume seamlessly in the replica with minimal interruption and data loss. +- No changes needed for Worker and Workflow code during setup or failover. +- 99.99% contractual [SLA](#sla). -
**General note, we should be using "Replica" vs "Region", and probably note "Zone". Discuss. "Replica" is the corpus of Workflows. High Availability is a _capability_. Features are "replication" and "multi-region". Features achieve HA workflows in your applications. See messaging guide.**
+## High availability options -Temporal Cloud's high-availability Namespaces provide disaster-tolerant deployment for workloads where availability is critical to your operations. -When you enable high availability, Temporal Cloud automatically synchronizes your data between a primary and a fallback Namespace, keeping them in sync. -Should an incident occur, Temporal will [failover](/glossary#failover) your Namespace. -This allows your Workflow Executions and Schedules to seamlessly shift from the active availability zone to the replica in the fallback availability zone. +Temporal currently offers the following high availability features, which you configure at a Namespace level: + +- **Replication**: + Workflows are seamlessly replicated to a different isolation domain within the same region as the Namespace, such as "us-east-1". + Choose this option for applications architected for a single-region. + You will failover within the same region to a separate isolation domain. +- **Multi-region replication**: + Workflows are seamlessly replicated to a different region that you choose. + Choose this option when your business requires multi-regional availability and the higher-level of resilience that separated locations offers. + You will failover from one region to a separate region. + +:::note + +Please note that replication charges apply when enabling high availability features. +For pricing details, visit Temporal Cloud's [Pricing](/cloud/pricing) page. + +::: + +## Replication and replicas -## Availability zones and replicas +High Availability features in Temporal Cloud simplify deployment, ensuring operational continuity and data integrity even during unexpected events impacting Namespace operations. +It uses a process called replication. +Replication asynchronously replicates Workflow Executions from an active Namespace to its replica, which is physically located in another isolation domain within the same region or another region in the same continent. +In the event of incidents in the active Namespaces, your replica is ready to take over. +Temporal Cloud smoothly transitions control from the active to the replica via a "failover". -An availability zone is a physically isolated data center within a deployment region for a given cloud provider. -Regions consist of multiple availability zones, providing redundancy and fault tolerance. -In some cases, the fallback zone may be in the same region as the primary zone, or it may be in a different region altogether, depending on your deployment configuration. +## Isolation domains and replicas -High-availability simplifies deployment, ensuring operational continuity and data integrity even during unexpected events. -Regional disruptions or other issues that affect the data centers within a specific availability zone may occur. -High-availability allows processing to shift from the affected zone to an already-synchronized fallback zone. +An isolation domains is a physically isolated data center within a deployment region for a given cloud provider. +Regions consist of multiple isolation domains, providing redundancy and fault tolerance. +In some cases, the fallback domain may be in the same region as the primary, or it may be in a different region altogether, depending on your deployment configuration. -This synchronized zone is called a "**replica**." -The process of duplicating all Workflow data ensures that your replica, which serves as the standby region, is always available and ready to take on the active role. +High availability simplifies deployment, ensuring operational continuity and data integrity even during unexpected events. +Incidents that affect the data centers within a specific isolation domain may occur. +High availability allows processing to shift from the affected domain to an already-synchronized fallback domain. -In the event of network service or performance issues in the active zone, your replica is ready to take over. -When necessary, Temporal Cloud smoothly transitions control from the active to the standby zone using a process called "[failover](/glossary#failover)". +This synchronized domain is called a "**replica**." +The process of duplicating all Workflow data ensures that your replica, which serves as the standby Namespace, is always available and ready to take on the active role. +When necessary, Temporal Cloud smoothly transitions control from the active to the standby using a process called "[failover](/glossary#failover)". -## High-availability and business continuity {#high-availability-intro} +## High availability and business continuity {#high-availability-intro} -For many organizations, ensuring high-availability is critical to maintaining business continuity. -Temporal Cloud's high-availability Namespace feature includes a 99.99% contractual Service Level Agreement ([SLA](https://docs.temporal.io/cloud/sla)). +For many organizations, ensuring high availability is critical to maintaining business continuity. +Temporal Cloud's high availability Namespace feature includes a 99.99% contractual Service Level Agreement ([SLA](https://docs.temporal.io/cloud/sla)). It provides 99.99% availability and 99.99% guarantee against service errors. -A high-availability Namespace (HAN) creates a single logical Namespace that operates across two physical zones: one active and one standby. -HANs streamline access for both zones to a unified Namespace endpoint. -As Workflows progress in the active zone, history events are asynchronously replicated to the standby zone, ensuring continuity and data integrity. +A high availability Namespace creates a single logical Namespace that operates across two physical isolation domains: one active and one standby. +Replicated Namespaces streamline access for both domains to a unified Namespace endpoint. +As Workflows progress in the active Namespace, history events are asynchronously replicated to the standby zone, ensuring continuity and data integrity. -In the event of an incident or outage in the active zone, Temporal Cloud will seamlessly failover to your standby zone. +In the event of an incident or outage in the active isolation domain, Temporal Cloud will seamlessly failover to your standby replica. Failovers allow existing Workflow Executions to continue running and new Workflow Executions to be started. -Once failover occurs, the roles of the active and standby zones switch. +Once failover occurs, the roles of the active and standby domains switch. The standby zone becomes active, and the previous active zone becomes the standby. -After the issue is resolved, the zone "fails back" from the replica to the original. +After the issue is resolved, the domain "fails back" from the replica to the original. -## Types of high-availability +## Should you choose high availability? -Temporal currently offers the following high-availability options, which you select when upgrading your Namespace to use high-availability: +Should you be using high availability Namespaces? It depends on your availability requirements: -- **In-region replication** - - Data is replicated to a separate zone in the same availability region, such as "us-east-1". - This option offers near-instantaneous failovers but does not protect against regional disasters like hurricanes where both the primary and the fallback . -- **Multi-region replication** - - Data is replicated to a separate region on the same continent. - This option offers the greatest protection against weather events and other possible external causes for regional outages, as the regions are physically separated by large distances. - Failover may experience some minor latency. +- High availability Namespaces offer a 99.99% contractual SLA for workloads with strict high availability needs. + They use two Namespaces in two isolation domains to support standby recovery. + In the event of an incident, Temporal Cloud automatically fails over the Namespace to the standby replica. +- Namespaces without high availability include a 99.9% contractual Service Level Agreement ([SLA](/cloud/sla)). + In this use, Temporal clients connect to a single Namespace in one deployment domain. + For many applications, this offers sufficient availability. -
**Did we want to go into further detail about lag and conflict resolution? if so, where?**
+Temporal Cloud provides 99.99% service availability for all Namespaces, both single-region and high availability. -
**Would be very nice to have more compelling explanations here**
+## SLA guarantees {#sla} -:::tip +High availability Namespaces offer 99.99% availability, enforced by Temporal Cloud's [service error rates SLA](https://docs.temporal.io/cloud/sla). +Our system is designed to limit data loss after recovery when the incident triggering the failover is resolved. -As Namespace pairing is currently limited to regions within the same continent, South America is excluded as only one region is available. +Our recovery point objective ([RPO](https://en.wikipedia.org/wiki/Disaster_recovery#Recovery_Point_Objective)) is near-zero. +There may be a short period of time during an incident or forced failover when some data is unavailable in the replica. +Some Workflow History data won't arrive until networks issue are fixed, enabling the History to finish replicating and the divergent History branches to reconcile. -::: +Temporal Cloud proactively responds to incidents by triggering failovers. +Our recovery time objective ([RTO](https://en.wikipedia.org/wiki/Disaster_recovery#Recovery_Time_Objective)) is 20 minutes or less per incident. -## Should you choose high-availability? +:::info -Should you be using high-availability Namespaces? It depends on your availability requirements: +During a disaster scenario in which the data on the hard drives in the active Namespace cannot be recovered, the duration of data loss may be as high as the [replication lag](/cloud/high-availability/best-practices#metrics) at the time of disaster. -- High-availability Namespaces offer a 99.99% contractual SLA for workloads with strict high-availability needs. - HANs use two Namespaces in two deployment zones to support standby recovery. - In the event of a zone failure, Temporal Cloud automatically fails over the HAN Namespace to the standby replica. -- Single-zone Namespaces include a 99.9% contractual Service Level Agreement ([SLA](/cloud/sla)). - In single-zone use, Temporal clients connect to a single Namespace in one deployment zone. - For many applications, this offers sufficient availability. +::: -Temporal Cloud provides 99.99% service availability for all Namespaces, both single-region and high-availability. +## Regional availability {#regional-availability} -## Summary +Multi-region Namespaces are one of the high availability options you can choose. +They are available in all existing [Temporal Cloud regions](/cloud/service-availability#regions). -The following list reviews the advantages of high-availability Namespaces: +:::tip -- No manual deployment or configuration required—just simple push-button operation. -- Open Workflows continue in the standby zone with minimal interruption and data loss. -- No changes needed for Worker or Workflow code during setup or failover. -- 99.99% contractual SLA. +Namespace pairing is currently limited to regions within the same continent. +South America is excluded as only one region is available. +::: diff --git a/docs/production-deployment/cloud/high-availability/monitor.mdx b/docs/production-deployment/cloud/high-availability/monitor.mdx deleted file mode 100644 index 4f3211dabb..0000000000 --- a/docs/production-deployment/cloud/high-availability/monitor.mdx +++ /dev/null @@ -1,81 +0,0 @@ ---- -id: monitor -title: Monitor and observe -sidebar_label: Monitor and observe -slug: /cloud/high-availability/operations -description: Temporal Cloud's High-Availability Namespaces offer automated failover, synchronized data replication, and high availability for workloads requiring disaster-tolerant deployment and 99.99% uptime. Use Global Namespace for self-hosted. -tags: - - Temporal Cloud - - Production - - High availability -keywords: - - availability - - explanation - - failover - - high-availability - - multi-region - - multi-region namespace - - namespaces - - temporal-cloud - - term ---- -import { RelatedReadContainer, RelatedReadItem } from '@site/src/components/related-read/RelatedRead'; - -:::tip Support, stability, and dependency info - -High-availability Namespaces are in [Public Preview](/evaluate/development-production-features/release-stages#public-preview) for Temporal Cloud. - -::: - -
**No audits, updates, intros, re-org. Add information about unhealth vs health, and trigger failover button disabled, opting out of temporal-initiated failovers, "Unhealthy replica error". After moving things to enable, this seems really light on content**
- -How do you trigger failovers and observe Workflow Executions? -This section provides how-to instructions for the following operations tasks: - -- [Triggering failovers](/cloud/multi-region#triggering-failovers) -- [Metrics](/cloud/multi-region#metrics-operations) -- [Monitoring and observability](/cloud/multi-region#observability) - -### Metrics {#metrics-operations} - -Replication lag refers to the transmission delay of Workflow updates and history events from the active region to the standby region. -A forced failover when there is a large replication lag has a higher likelihood of rolling back Workflow progress, so always check the metric replication lag before initiating a failover. -Temporal Cloud emits three replication lag-specific [metrics](/production-deployment/cloud/metrics/reference#temporal_cloud_v0_replication_lag_bucket). -The following samples demonstrate how you can use these metrics to explore replication lag. - -**P99 replication lag histogram** - -``` -histogram_quantile(0.99, sum(rate(temporal_cloud_v0_replication_lag_bucket[$__rate_interval])) by (temporal_namespace, le)) -``` - -**Average replication lag** - -``` -sum(rate(temporal_cloud_v0_replication_lag_sum[$__rate_interval])) by (temporal_namespace) -/ -sum(rate(temporal_cloud_v0_replication_lag_count[$__rate_interval])) by (temporal_namespace) -``` - -### Monitoring and observability {#observability} - -You can view and alert on key cloud metrics using the Web UI, the 'tcld' CLI utility, and Temporal Cloud APIs. -For example, during the process of adding a region to a Namespace, you can see the progress of Workflow replication. -Errors -- if any occur -- will also surface in the Namespace Web UI. - -:::info - -You may notice that multi-region Namespace shows twice (2x) the Action count in `temporal_cloud_v0_total_action_count`. -This doubling happens due to regional replication. - -::: - -### Auditing operational events {#auditing} - -Temporal Cloud provides several ways to audit events: - -- When Temporal triggers failovers, the audit log updates with details. - Look specifically for `"operation": "FailoverNamespace"` in the logs. -- You can set alerts for Temporal-initiated failover events. -- After a failover, you can check that the Namespace is active in the new region using the Temporal Cloud Web UI. - diff --git a/docs/production-deployment/cloud/high-availability/work-file.txt b/docs/production-deployment/cloud/high-availability/work-file.txt index fa4d5b28b0..b95c4f0da7 100644 --- a/docs/production-deployment/cloud/high-availability/work-file.txt +++ b/docs/production-deployment/cloud/high-availability/work-file.txt @@ -2,45 +2,8 @@ AFFECTED COVERAGE: -- ALL LINKS ON NEW PAGES MUST BE AUDITED, REDONE, TESTED!!! -THESE PAGES MENTION MULTI-REGION: -./docs/encyclopedia/nexus.mdx -./docs/encyclopedia/nexus-use-cases.mdx -./docs/evaluate/development-production-features/temporal-nexus.mdx -./docs/evaluate/development-production-features/index.mdx -./docs/evaluate/development-production-features/high-availability.mdx -./docs/evaluate/development-production-features/multi-region-namespace.mdx -./docs/evaluate/development-production-features/cloud-vs-self-hosted.mdx -./docs/evaluate/temporal-cloud/pricing.mdx -./docs/evaluate/temporal-cloud/sla.mdx -./docs/evaluate/temporal-cloud/legacy-pricing.mdx -./docs/evaluate/temporal-cloud/security.mdx -./docs/production-deployment/cloud/metrics/reference.mdx -./docs/production-deployment/cloud/gcp-export-gcs.mdx -./docs/production-deployment/cloud/audit-logging.mdx -./docs/production-deployment/cloud/multi-region.mdx -./docs/production-deployment/cloud/tcld/namespace.mdx -./docs/production-deployment/cloud/nexus/index.mdx -./docs/production-deployment/cloud/terraform-provider.mdx -./docs/production-deployment/cloud/service-health.mdx -## Pricing {#pricing} - -**How does adding a multi-region Namespace affect my costs?** - -For pricing details, visit Temporal Cloud's [Pricing page](/cloud/pricing). - -## Manage your multi-region Namespace {#management} - -**How do you create, enable, and manage your multi-region Namespace?** - -Temporal enables you to create and manage your multi-region Namespace using the Temporal Cloud Web UI, the command line 'tcld' CLI utility, and the [Cloud Ops API](/ops). -Use these tools to create, upgrade, and discontinue your multi-region Namespace. - -- [Create a multi-region Namespace](/cloud/multi-region#create) -- [Upgrade a single-region Namespace to multi-region](/cloud/multi-region#add-regions) -- [Discontinuing multi-region service](/cloud/multi-region#discontinuing) :::warning @@ -55,7 +18,6 @@ Temporal Cloud’s Terraform provider does not support multi-region Namespaces. ::: - diff --git a/sidebars.js b/sidebars.js index cecd9b1ff5..e79006cc83 100644 --- a/sidebars.js +++ b/sidebars.js @@ -348,9 +348,9 @@ module.exports = { }, items: [ "production-deployment/cloud/high-availability/enable", + "production-deployment/cloud/high-availability/failovers", "production-deployment/cloud/high-availability/how-it-works", - "production-deployment/cloud/high-availability/guarantees", - "production-deployment/cloud/high-availability/monitor", + "production-deployment/cloud/high-availability/best-practices", "production-deployment/cloud/high-availability/faq", ], }, From bdc9f2c43f2847a19a614e41c04443c849400b2c Mon Sep 17 00:00:00 2001 From: Erica Sadun Date: Wed, 5 Feb 2025 12:21:22 -0700 Subject: [PATCH 3/3] EDU-3861: Landing page prototype phase 2 - Adds inline definitions - Adds "under the hood" - Simplifies and focuses content - Adds explanatory images --- .../high-availability/best-practices.mdx | 2 +- .../cloud/high-availability/enable.mdx | 113 ++++-------- .../cloud/high-availability/failovers.mdx | 4 +- .../cloud/high-availability/faq.mdx | 4 +- .../cloud/high-availability/how-it-works.mdx | 170 +++++++++--------- .../cloud/high-availability/index.mdx | 166 +++++++++-------- .../cloud/high-availability/work-file.txt | 18 +- sidebars.js | 2 +- .../definitions/ExpandableDefinition.js | 58 ++++++ .../img/cloud/high-availability/failover.png | Bin 0 -> 50651 bytes .../high-availability/logical-namespace.png | Bin 0 -> 46386 bytes 11 files changed, 286 insertions(+), 251 deletions(-) create mode 100644 src/components/definitions/ExpandableDefinition.js create mode 100644 static/img/cloud/high-availability/failover.png create mode 100644 static/img/cloud/high-availability/logical-namespace.png diff --git a/docs/production-deployment/cloud/high-availability/best-practices.mdx b/docs/production-deployment/cloud/high-availability/best-practices.mdx index 752cf60652..5d8e4708f6 100644 --- a/docs/production-deployment/cloud/high-availability/best-practices.mdx +++ b/docs/production-deployment/cloud/high-availability/best-practices.mdx @@ -65,7 +65,7 @@ Trigger testing can: - **Assess recovery time**: Manual testing helps you measure actual recovery time. - You can check if it meets your expected Recovery Time Objective (RTO) of 20 minutes or less, as stated in the [High availability Namespace SLA](/cloud/high-availability#sla). + You can check if it meets your expected Recovery Time Objective (RTO) of 20 minutes or less, as stated in the [High availability Namespace SLA](/cloud/sla). - **Identify potential issues**: Failover testing uncovers problems not visible during normal operation. diff --git a/docs/production-deployment/cloud/high-availability/enable.mdx b/docs/production-deployment/cloud/high-availability/enable.mdx index 1a59bf83cd..44787af14a 100644 --- a/docs/production-deployment/cloud/high-availability/enable.mdx +++ b/docs/production-deployment/cloud/high-availability/enable.mdx @@ -1,7 +1,7 @@ --- id: enable -title: Enable High Availability features -sidebar_label: Enable high availability +title: How to enable high availability +sidebar_label: How to enable high availability slug: /cloud/high-availability/enable description: Temporal Cloud's High-Availability Namespaces offer automated failover, synchronized data replication, and high availability for workloads requiring disaster-tolerant deployment and 99.99% uptime. Use Global Namespace for self-hosted. tags: @@ -22,105 +22,56 @@ keywords: import { RelatedReadContainer, RelatedReadItem } from '@site/src/components/related-read/RelatedRead'; -You enable High Availability features for a new or existing Namespace by adding a replica to the Namespace. -When you add a replica, Temporal Cloud begins replicating ongoing and existing Workflows. -Once the replication has completed and the replica is ready, your Namespace is ready for failover. +Adding a replica to a new or existing Namespace enables Temporal Cloud's high availability features. +You can either create a replica within the current region or deploy the replica to a different region. +Once added, Temporal Cloud starts replicating ongoing and existing Workflows, along with their metadata. +After replication is complete and your replica is ready, your Namespace is prepared for failover. +If an incident occurs, failover transfers control from the active Namespace to the replica. + +This page shows you how to: -- [Create a high availability Namespace](/cloud/high-availability/enable#create) -- [Upgrade an existing Namespace to high availability functionality](/cloud/high-availability/enable#upgrade) -- [Discontinuing high availability replicas](/cloud/high-availability/enable#discontinuing) +- [Create a new high availability Namespace](/cloud/high-availability/enable#create) +- [Upgrade an existing Namespace for high availability functionality](/cloud/high-availability/enable#upgrade) +- [Discontinue high availability replication](/cloud/high-availability/enable#discontinuing) ## Create a high availability Namespace {#create} -The following sections explain how to create a Namespace with a replica. -You can create a replica within the current region or deploy the replica to a different region. +To create a new replicated Namespace, you can use the Temporal Cloud Web site or the `tcld` command line utility. +The following table explains how: + +| Approach | Instructions | +|---------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------| +| **Temporal Cloud Web UI** | 1. Visit Temporal Cloud in your Web browser.
2. During Namespace creation, specify the active region for the Namespace.
3. Select "Add a replica".
  • Adding a replica in the same region enables Replication.
  • Adding a replica in a different region enables Multi-region Replication.
| +| **Temporal `tcld` CLI** | At the command line, enter:
tcld namespace create \
    --namespace \.\ \
    --region \ \
    --region \


Specify the [region codes](/cloud/service-availability) as arguments to the two `--region` flags.
  • Using the same region replicates to an isolation zone within that region.
  • Using a different region within the same continent creates a multi-region Namespace.
Before pressing return, add your authentication credentials. For example, `--ca-certificate-file `. | -::: note +:::note -While reading through this coverage, be aware that replication is not supported in all regions. +Replication is not supported in all regions. For multi-region replication, pairing is limited to regions within the same continent. For more details, refer to ["Regional availability"].(/cloud/high-availabilityregional-availability). ::: -### Temporal Cloud Web UI - -Follow these steps to add replication to your Temporal Cloud Namespace: - -1. During Namespace creation, specify the first region for the Namespace. -2. Select the “Add a replica” option. - Adding a replica in the same region enables Replication. - Adding a replica in a second region enables Multi-region Replication. - -### Temporal 'tcld' CLI - -Enter the following at the command-line to create a replicated Namespace: - -```sh -tcld namespace create \ - --namespace . \ - --region \ - --region -``` - -Specify the [region codes](/cloud/service-availability) as arguments to the two `--region` flags. - -- Using the same region replicates to an isolation zone within that region. -- Using a different region within the same continent creates a multi-region Namespace. - -Before pressing return, add your authentication credentials. -For example, `--ca-certificate-file `. - ## Upgrade an existing Namespace to high availability functionality {#upgrade} Upgrade an existing single-region Namespace to high availability features by establishing a replica. -The following sections explain how. -You can either create a replica within the current region or deploy the replica to a different region. - -### Temporal Cloud Web UI - -Follow these steps to upgrade an existing Namespace: - -1. Visit Temporal Cloud Namespaces in your Web browser -1. Navigate to the Namespace details page -1. Select the “Add a replica” button. -1. Choose either **Replication** (in the same region) or **Multi-region Replication** (across regions). - If you select Multi-region Replication, specify which region - -The web interface will present an estimated time for replication to complete. -This time is based on your selection and the size and scale of the Workflows in your Namespace. -An email alert is dispatched once your highly available Namespace is ready for use. - -### Temporal 'tcld' CLI - -Enter the following at the command-line to upgrade a Namespace for replication: - -```sh -tcld namespace add-region \ - --namespace . \ - --region -``` - -Specify the added [region code](/cloud/service-availability) as an argument to the `--region` flag. - -- Using the current region replicates to an isolation zone within your existing region. -- Using a different region within the same continent creates a multi-region Namespace. - -Before pressing return, add your authentication credentials. -For example, `--ca-certificate-file `. +The following table explains how: -An email alert is sent once your multi-region Namespace is ready for use. +| Approach | Instructions | +|---------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------| +| **Temporal Cloud Web UI** | 1. Visit Temporal Cloud Namespaces in your Web browser.
2. Navigate to the Namespace details page.
3. Select the “Add a replica” button.
4. Choose either **Replication** (in the same region) or **Multi-region Replication** (across regions).
  • If you select Multi-region Replication, specify which region.
The web interface will present an estimated time for replication to complete.
This time is based on your selection and the size and scale of the Workflows in your Namespace.
An email alert is dispatched once your highly available Namespace is ready for use. | +| **Temporal `tcld` CLI** | At the command line, enter:
tcld namespace add-region \
    --namespace \.\ \
    --region \


Specify the added [region code](/cloud/service-availability) as an argument to the `--region` flag.
  • Using the current region replicates to an isolation zone within your existing region.
  • Using a different region within the same continent creates a multi-region Namespace.

Before pressing return, add your authentication credentials.
For example, `--ca-certificate-file `.
An email alert is sent once your multi-region Namespace is ready for use. | -## Discontinuing high availability replicas {#discontinuing} +## Discontinue high availability replication {#discontinuing} -Removing a Namespace replica removes the high availability and automatic failover features that provide Temporal's highest service level agreement. -To disable these features and end charges: +Removing a Namespace replica disables high availability and automatic failover features. +Follow these steps to disable these features and end high availability charges: 1. Navigate to the Namespace details page in Temporal Cloud. -2. On the “Region” card, select the option to “Remove Replica”. +2. On the "Region" card, select the option to "Remove Replica." -The replica will be deleted and your Namespace will no longer be highly available. -You will no longer be charged for this feature. +Temporal Cloud deletes the replica. +Your Namespace will no longer be highly available and you will no longer be charged for this feature. :::note diff --git a/docs/production-deployment/cloud/high-availability/failovers.mdx b/docs/production-deployment/cloud/high-availability/failovers.mdx index 425a17490c..dd43fdcea5 100644 --- a/docs/production-deployment/cloud/high-availability/failovers.mdx +++ b/docs/production-deployment/cloud/high-availability/failovers.mdx @@ -1,7 +1,7 @@ --- id: failovers -title: failovers -sidebar_label: Failovers +title: How to trigger failovers +sidebar_label: How to trigger failovers slug: /cloud/high-availability/failovers description: Temporal Cloud's High-Availability Namespaces offer automated failover, synchronized data replication, and high availability for workloads requiring disaster-tolerant deployment and 99.99% uptime. Use Global Namespace for self-hosted. tags: diff --git a/docs/production-deployment/cloud/high-availability/faq.mdx b/docs/production-deployment/cloud/high-availability/faq.mdx index a3bef904da..e1d3957ba0 100644 --- a/docs/production-deployment/cloud/high-availability/faq.mdx +++ b/docs/production-deployment/cloud/high-availability/faq.mdx @@ -87,7 +87,7 @@ Temporal Cloud initiates failovers when there are incidents or outages in the cl This includes failures of databases, storage, etc. We trigger failovers any time we observe increased latencies or an increase in service errors that causes us to violate the SLA that is in our control. -Read more about [automatic failovers](/cloud/high-availability/how-it-works#healthchecks). +Read more about [automatic failovers](/cloud/high-availability/how-it-works#health-checks). **Q: Are there any other types of failover not listed above?** @@ -235,7 +235,7 @@ Not at the moment. Signals are cherry-picked during conflict resolution if there is replication lag and conflict. Workflows can theoretically revert multiple steps. -Customers should decide whether to add logic to handle this or manually fix affected Workflows if they believe the risk is low. Other known limitations have been [documented](/cloud/high-availability/how-it-works#architecture) around causality and so forth. +Customers should decide whether to add logic to handle this or manually fix affected Workflows if they believe the risk is low. Other known limitations have been [documented](/cloud/high-availability/how-it-works#workflow-execution-replication) around causality and so forth. **Q: How much time does it take to reconcile data after an incident is resolved? ** diff --git a/docs/production-deployment/cloud/high-availability/how-it-works.mdx b/docs/production-deployment/cloud/high-availability/how-it-works.mdx index 8dc196ffbd..60e1a2c59d 100644 --- a/docs/production-deployment/cloud/high-availability/how-it-works.mdx +++ b/docs/production-deployment/cloud/high-availability/how-it-works.mdx @@ -21,126 +21,87 @@ keywords: --- import { RelatedReadContainer, RelatedReadItem } from '@site/src/components/related-read/RelatedRead'; +import ExpandableDefinition from '@site/src/components/definitions/ExpandableDefinition'; -In traditional active/active replication, multiple nodes serve requests and accept writes simultaneously, ensuring strong synchronous data consistency. -In contrast, with a Temporal Cloud high-availability Namespace, only the active Namespace accepts requests and writes at any given time. -Workflow history events are written to the active Namespace first and then asynchronously replicated to the standby replica, ensuring that the replica remains in sync. +## Health checks {#health-checks} -
**Needs new images**
- -| Before failover | After failover | -| :-------------------------------------------------------: | :-----------------------------------------------------: | -| ![Before failover](/img/multi-region/before-failover.png) | ![After failover](/img/multi-region/after-failover.png) | - -## The failover process {#failovers} - -A failover shifts Workflow Execution processing from an active Temporal Namespace to a standby replica during outages or other incidents. -Standby replicas duplicate data and prevent data loss during failover. - -**What happens during the failover process?** - -Temporal Cloud initiates a Namespace failover when it detects an incident or outage that raises error rates or latency in the active region of a high availability Namespace. -The failover shifts Workflow processing to a replica that isn’t affected by the incident. -This lets existing Workflows continue and new Workflows start while the incident is fixed. -Once the incident is resolved, Temporal Cloud performs a "failback" by shifting Workflow Execution processing back to the original Namespace. - -:::info - -You can test the failover of your high availability Namespace by manually [triggering a failover](/cloud/high-availability/failovers#triggering-failovers) using the UI page or the 'tcld' CLI utility. -In most scenarios, we recommend you let Temporal handle failovers for you. - -::: - -## Health Checks {#healthchecks} - -**How does Temporal detect failover conditions?** - -Temporal Cloud automates failovers by performing internal health checks. -This process monitors your request error rates, latencies, and any infrastructure issues that might cause service disruptions, such as request timeouts. -It automatically triggers failovers when these indicators exceed our allowed thresholds. - -### Replication lag {#replication-lag} +Temporal Cloud automates failovers with internal health checks. +This process monitors request error rates, latencies, and infrastructure issues that could cause service disruptions, such as request timeouts. +Temporal Cloud triggers failovers when these indicators exceed our allowed thresholds. High availability Namespaces use asynchronous replication. -Workflow updates in the active Namespace, along with associated history events, are transmitted to the standby replica with a short delay. +Temporal Cloud transmits Workflow updates from the active Namespace, along with associated History events, to the standby replica with a short delay. This delay is called the replication lag. -Temporal Cloud strives to maintain a P95 replication delay of less than 1 minute. -In this context, P95 means 95% of requests are processed faster than this specified limit. +Temporal Cloud aims to maintain a P95 replication delay of less than 1 minute. +In this context, P95 means 95% of requests are processed faster than the specified limit. -Replication lags mean a [forced failover](/cloud/high-availability/how-it-works#forced-failover) may cause Workflows to rollback in progress. -Lags may also cause recently started Workflows to be temporarily unavailable until a Namespace recovers. -Temporal event versioning and [conflict resolution mechanisms](/cloud/high-availability/how-it-works#conflict-resolution) help guarantee that the Workflow Event History can be replayed. -Critical operations like Signals won't get lost. +Replication lags are a concern because: -### Failover scenarios +- Replication lags may lead to a [forced failover](/cloud/high-availability/how-it-works#forced-failover), potentially causing Workflows to rollback in progress. +- Lags may cause recently started Workflows to be temporarily unavailable until the Namespace recovers. + Temporal event versioning and [conflict resolution mechanisms](/cloud/high-availability/how-it-works#conflict-resolution) ensure the Workflow Event History can be replayed. + Critical operations like Signals won’t be lost. -The Temporal Cloud failover mechanism supports several modes to execute Namespace failovers. +## Types of Failover scenarios + +The Temporal Cloud failover mechanism supports several modes for executing Namespace failovers. These modes include graceful failover ("handover"), forced failover, and a hybrid mode. The hybrid mode is Temporal Cloud’s default Namespace behavior. +The following sections describe each style. -#### Graceful failover (handover) {#graceful-failover} +### Graceful failover (handover) {#graceful-failover} -In this mode, replication tasks are fully processed and drained. +In this mode, Temporal Cloud fully processes and drains replication tasks. Temporal Cloud pauses traffic to the Namespace before the failover. -This prevents the loss of progress and avoids data conflicts. +Graceful failover prevents the loss of progress and avoids data conflicts. + The Namespace experiences a short period of unavailability, defaulting to 10 seconds. +During this period: -During this period, existing Workflows stop progress. -Temporal Cloud returns a "Service unavailable error", which is retried by SDKs. -State transitions will not happen and tasks are not dispatched. -User requests like start/signal workflow will be rejected while operations are paused during handover. +- Existing Workflows stop progress. +- Temporal Cloud returns a "Service unavailable error". + This error is retryable by the Temporal SDKs. +- State transitions will not happen and tasks are not dispatched. +- User requests like start/signal Workflow are rejected. +- Operations are paused during handover. -This mode favors _consistency_ over availability. +*This mode favors _consistency_ over availability.* -#### Forced failover {#forced-failover} +### Forced failover {#forced-failover} -In this mode, a replica immediately activates in the standby Namespace. -Events not replicated due to [replication lag](/cloud/high-availability/best-practices#metrics) will undergo [conflict resolution](/cloud/high-availability/how-it-works#conflict-resolution) upon reaching the new active Namespace. +In this mode, Temporal Cloud immediately activates the replica in the standby Namespace. +Events that weren't replicated due to [replication lag](/cloud/high-availability/best-practices#metrics) undergo [conflict resolution](/cloud/high-availability/how-it-works#conflict-resolution) when they reach the new active Namespace. -This mode prioritizes _availability_ over consistency. +*This mode prioritizes _availability_ over consistency.* -#### Hybrid failover mode {#hybrid-failover} +### Hybrid failover mode {#hybrid-failover} While graceful failovers are preferred for consistency, they aren’t always practical. -Temporal Cloud’s hybrid failover mode (the default mode) limits an initial graceful failover attempt to 10 seconds or less. -During this period, existing Workflows stop progress. -Temporal Cloud returns a "Service unavailable error", which is retried by SDKs. -If the graceful approach doesn’t resolve the issue, Temporal Cloud automatically switches to a forced failover. -This strategy balances consistency and availability requirements. +Temporal Cloud’s hybrid failover mode (the default mode) limits the initial graceful failover attempt to 10 seconds or less. -See the sections on [triggering a failover](/cloud/high-availability/failovers/#triggering-failovers), [Worker deployment](/cloud/high-availability/best-practices/#worker-deployment), and [routing](/cloud/high-availability/best-practices#routing) for more information. +During this period: -## Architecture {#architecture} +- Existing Workflows stop progress. +- Temporal Cloud returns a "Service unavailable error", which is retried by SDKs. -**How do high availability Namespaces work?** - -High availability Namespaces replicate Namespace metadata and Workflow Executions across connected Namespaces. -This redundancy, plus the added failover capability, provides measurable stability when dealing with outages. +If the graceful approach doesn’t resolve the issue, Temporal Cloud automatically switches to a forced failover. -A high availability Namespace is normally active in a single isolation domain at any moment. -The passive replica assumes a standby role. -An exception to this only occurs in the event of a network partition. -In this case, you may elect to promote a standby isolation domain to active status. -Caution: this action will temporarily result in both regions being active. -Once the network partition resolves and communication between the isolation domains/regions is restored, a conflict resolution algorithm determines which region continues as the active one. -This ensures only one Namespace remains active. +*This strategy balances consistency and availability requirements.* -### Metadata replication {#metadata-replication} +## Metadata replication {#metadata-replication} -Updates to high availabillity Namespace records automatically duplicate to their replica. +Updates to high availability Namespace records automatically duplicate to the replica. This metadata includes configurations such as retention periods, Search Attributes, and other settings. Temporal Cloud ensures that all isolation domains and regions will eventually share a consistent and unified view of the Namespace metadata. -
**Needs correct field name**
- :::info -A Namespace failover, which changes the identifier for the active element field of a Namespace record, is an update. +A Namespace failover that changes the identifier for the active element field of a Namespace record is an update. This update is replicated via the Namespace metadata mechanism. ::: -### Workflow Execution replication {#workflow-execution-replication} +## Workflow Execution replication {#workflow-execution-replication} Temporal Cloud restricts certain Workflow operations to the active region: @@ -151,12 +112,6 @@ Temporal Cloud restricts certain Workflow operations to the active region: These limits mean that certain requests, such as Start Workflow and Signal Workflow, are processed by and limited to the active Namespace. Standby replicas may receive API requests from Clients and Workers. They automatically forward these requests to the active Namespace for execution. - -High availability Namespaces provide an “all-active” experience for Temporal users. -This helps limit or eliminate downtime during Namespace failover. -There's a short time window from when a standby replica becomes the active Namespace to when Clients and Workers receive a DNS update. -During this time requests forward from the now passive (formerly active) replica Namespace to the newly active (formerly standby replica) Namespace. - As Workflow Executions progress and are operated on, replication tasks created in the active Namespace are dispatched to the standby replica. Processing these replication tasks ensures that the standby replica undergoes the same state transitions as the active Namespace. This enables replicated tasks to synchronize and achieve the same state as the original tasks. @@ -165,7 +120,16 @@ Standby replicas do not distribute Workflow or Activity Tasks. Instead, they perform verification tasks to confirm that intended operations are executed so Workflows reach the desired state. This mechanism ensures consistency and reliability in the replication process across Temporal regions. -### Conflict Resolution {#conflict-resolution} +:::note Failover downtime and DNS updates + +High availability Namespaces provide an “all-active” experience for Temporal users. +This helps limit or eliminate downtime during Namespace failover. +There's a short time window from when a standby replica becomes the active Namespace to when Clients and Workers receive a DNS update. +During this time requests forward from the now passive (formerly active) replica Namespace to the newly active (formerly standby replica) Namespace. + +::: + +## Conflict Resolution {#conflict-resolution} High availability Namespaces rely on asynchronous event replication across Temporal isolation domains and regions. In the event of a non-graceful failover across regions, replication lag may result in a temporary setback in Workflow progress. @@ -190,3 +154,29 @@ Idempotency keeps operations from producing additional effects. Protect your processes from accidental or repeated actions for more reliable execution. ::: + +
**Needs new images**
+ +| Before failover | After failover | +| :-------------------------------------------------------: | :-----------------------------------------------------: | +| ![Before failover](/img/multi-region/before-failover.png) | ![After failover](/img/multi-region/after-failover.png) | + +## The failover process {#failovers} + +A failover shifts Workflow Execution processing from an active Temporal Namespace to a standby replica during outages or other incidents. +Standby replicas duplicate data and prevent data loss during failover. + +**What happens during the failover process?** + +Temporal Cloud initiates a Namespace failover when it detects an incident or outage that raises error rates or latency in the active region of a high availability Namespace. +The failover shifts Workflow processing to a replica that isn’t affected by the incident. +This lets existing Workflows continue and new Workflows start while the incident is fixed. +Once the incident is resolved, Temporal Cloud performs a "failback" by shifting Workflow Execution processing back to the original Namespace. + +:::info + +You can test the failover of your high availability Namespace by manually [triggering a failover](/cloud/high-availability/failovers#triggering-failovers) using the UI page or the 'tcld' CLI utility. +In most scenarios, we recommend you let Temporal handle failovers for you. + +::: + diff --git a/docs/production-deployment/cloud/high-availability/index.mdx b/docs/production-deployment/cloud/high-availability/index.mdx index 3b8b999b52..6074e7a4d0 100644 --- a/docs/production-deployment/cloud/high-availability/index.mdx +++ b/docs/production-deployment/cloud/high-availability/index.mdx @@ -21,27 +21,100 @@ keywords: --- import { RelatedReadContainer, RelatedReadItem } from '@site/src/components/related-read/RelatedRead'; +import ExpandableDefinition from '@site/src/components/definitions/ExpandableDefinition'; -Temporal Cloud's replicated Namespaces provide disaster-tolerant deployment for workloads where availability is critical to your operations. -When you enable high availability, Temporal Cloud automatically synchronizes your data between a primary and a fallback Namespace, keeping them in sync. -Should an incident occur, Temporal will [failover](/glossary#failover) your Namespace. -This allows your Workflow Executions and Schedules to seamlessly shift from the active availability zone to the synchronized replica in the fallback availability zone. +Temporal Cloud offers "Reliability-as-a-Service" to support mission-critical deployment when applications must be highly available. +Data loss and disruptions to Workflows can severely impact business. +Replication, the critical component of highly available Namespaces, protects applications against outages and downtime. +High availability creates a fallback "replica" that can take over Namespace duties during service incidents. +This keeps your Workflows running and your data available, even when your usual Namespace is unavailable. -Advantages of using Temporal Cloud’s High Availability features: +## Replication and failovers + +Temporal Cloud’s replicated Namespaces provide disaster-tolerant availability for critical workloads. +When you enable replication, Temporal Cloud syncs your data and Workflows between an active and a replica Namespace. +If an incident occurs, Temporal automatically **fails over** your Namespace. + +During a failover, your Workflow Executions and Schedules seamlessly transition from the active Namespace to a standby domain. +This standby domain is called a **replica**, as it replicates the Workflows and data of the active Namespace. +Once the incident resolves, the Namespaces reconcile and control returns back to the original. + +
+ Data and Workflows replicate to the Replica. Failover transfers control to the Replica that becomes the active Namespace. After resolution, control fails back to the original +
+ +A high availability Namespace creates a single logical Namespace that operates across two domains: one active and one standby. +Replicated Namespaces combine access for both domains to a unified Namespace endpoint. +As Workflows progress in the active Namespace, Temporal Cloud replicates History events to the standby domain, ensuring continuity and data integrity. + +During an incident or outage in the active domain, Temporal Cloud seamlessly fails over to your replica. +Failovers allow existing Workflow Executions to continue running and new Workflow Executions to be started. +Once failover occurs, the replica becomes active. +After the issue is resolved, the active replica "fails back" and the original Namespace resumes being "active". +Temporal resumes replication from the original active Namespace to the replica. + +
+ Data and Workflows replicate to the Replica. Failover transfers control to the Replica that becomes the active Namespace. After resolution, control fails back to the original +
+ + +
+ + +Dive deeper... + + +An **isolation domain** is a physically isolated data center within a deployment region for a given cloud provider. +**Regions** consist of multiple isolation domains. +Isolation domains provide redundancy and fault tolerance. + +A **replicated Namespace** consists of an **active Namespace** and a passive, fallback **replica**. +Depending on your setup, your replica may reside in the same region as your active Namespace (standard replication), or it may be located in an entirely different region (multi-region replication). + +After a **failover**, the replica takes on the active role until the incident is resolved. +After, the replica **fails back** and the original Namespace resumes the active role. + +In traditional active/active replication, multiple nodes serve requests and accept writes simultaneously, ensuring strong synchronous data consistency. +In contrast, with a Temporal Cloud high-availability Namespace, only the active Namespace accepts requests and writes at any given time. +Workflow history events are written to the active Namespace first and then asynchronously replicated to the standby replica, ensuring that the replica remains in sync. + +Each high availability Namespace replicates its metadata and Workflow Executions to its connected replica. +This redundancy, combined with failover capability, provides measurable stability during outages. + +A high availability Namespace is typically active in a single isolation domain at any given time, with the passive replica in a standby role. +The only exception occurs in the event of a network partition. +In this situation, you can promote a standby isolation domain to active status. + +**Caution:** This action will temporarily make both regions active. +Once the network partition is resolved and communication between the isolation domains/regions is restored, a conflict resolution algorithm determines which region remains active. +This ensures only one Namespace remains active. + +
+ +**Temporal Cloud’s high availability features:** - No manual deployment or configuration needed, just simple push-button operations. - Existing Workflows resume seamlessly in the replica with minimal interruption and data loss. - No changes needed for Worker and Workflow code during setup or failover. -- 99.99% contractual [SLA](#sla). +- 99.99% contractual [SLA](/cloud/sla). -## High availability options +## Types of high availability -Temporal currently offers the following high availability features, which you configure at a Namespace level: +Temporal currently offers the following high availability features. +Configure these from your Namespace: - **Replication**: - Workflows are seamlessly replicated to a different isolation domain within the same region as the Namespace, such as "us-east-1". + Workflows are seamlessly replicated to a different isolation domain within the same region as the Namespace, such as "us-east-1". Choose this option for applications architected for a single-region. - You will failover within the same region to a separate isolation domain. + Your Namespaces failover to an isolation domain within the same region. - **Multi-region replication**: Workflows are seamlessly replicated to a different region that you choose. Choose this option when your business requires multi-regional availability and the higher-level of resilience that separated locations offers. @@ -49,73 +122,31 @@ Temporal currently offers the following high availability features, which you co :::note -Please note that replication charges apply when enabling high availability features. -For pricing details, visit Temporal Cloud's [Pricing](/cloud/pricing) page. +Replication charges apply when you enable high availability. +For pricing details, visit the Temporal Cloud [Pricing](/cloud/pricing) page. ::: -## Replication and replicas - -High Availability features in Temporal Cloud simplify deployment, ensuring operational continuity and data integrity even during unexpected events impacting Namespace operations. -It uses a process called replication. -Replication asynchronously replicates Workflow Executions from an active Namespace to its replica, which is physically located in another isolation domain within the same region or another region in the same continent. -In the event of incidents in the active Namespaces, your replica is ready to take over. -Temporal Cloud smoothly transitions control from the active to the replica via a "failover". - -## Isolation domains and replicas - -An isolation domains is a physically isolated data center within a deployment region for a given cloud provider. -Regions consist of multiple isolation domains, providing redundancy and fault tolerance. -In some cases, the fallback domain may be in the same region as the primary, or it may be in a different region altogether, depending on your deployment configuration. - -High availability simplifies deployment, ensuring operational continuity and data integrity even during unexpected events. -Incidents that affect the data centers within a specific isolation domain may occur. -High availability allows processing to shift from the affected domain to an already-synchronized fallback domain. - -This synchronized domain is called a "**replica**." -The process of duplicating all Workflow data ensures that your replica, which serves as the standby Namespace, is always available and ready to take on the active role. -When necessary, Temporal Cloud smoothly transitions control from the active to the standby using a process called "[failover](/glossary#failover)". - -## High availability and business continuity {#high-availability-intro} - -For many organizations, ensuring high availability is critical to maintaining business continuity. -Temporal Cloud's high availability Namespace feature includes a 99.99% contractual Service Level Agreement ([SLA](https://docs.temporal.io/cloud/sla)). -It provides 99.99% availability and 99.99% guarantee against service errors. - -A high availability Namespace creates a single logical Namespace that operates across two physical isolation domains: one active and one standby. -Replicated Namespaces streamline access for both domains to a unified Namespace endpoint. -As Workflows progress in the active Namespace, history events are asynchronously replicated to the standby zone, ensuring continuity and data integrity. - -In the event of an incident or outage in the active isolation domain, Temporal Cloud will seamlessly failover to your standby replica. -Failovers allow existing Workflow Executions to continue running and new Workflow Executions to be started. -Once failover occurs, the roles of the active and standby domains switch. -The standby zone becomes active, and the previous active zone becomes the standby. -After the issue is resolved, the domain "fails back" from the replica to the original. - ## Should you choose high availability? Should you be using high availability Namespaces? It depends on your availability requirements: -- High availability Namespaces offer a 99.99% contractual SLA for workloads with strict high availability needs. +- **High availability Namespaces** offer a 99.99% Service Level Agreement ([SLA](/cloud/sla)) for workloads with strict high availability needs. They use two Namespaces in two isolation domains to support standby recovery. In the event of an incident, Temporal Cloud automatically fails over the Namespace to the standby replica. -- Namespaces without high availability include a 99.9% contractual Service Level Agreement ([SLA](/cloud/sla)). + High availability Namespaces' 99.99% availability is enforced by Temporal Cloud's [service error rates SLA](https://docs.temporal.io/cloud/sla). +- **Namespaces without high availability** include a 99.9% contractual Service Level Agreement ([SLA](/cloud/sla)). In this use, Temporal clients connect to a single Namespace in one deployment domain. For many applications, this offers sufficient availability. Temporal Cloud provides 99.99% service availability for all Namespaces, both single-region and high availability. - -## SLA guarantees {#sla} - -High availability Namespaces offer 99.99% availability, enforced by Temporal Cloud's [service error rates SLA](https://docs.temporal.io/cloud/sla). Our system is designed to limit data loss after recovery when the incident triggering the failover is resolved. -Our recovery point objective ([RPO](https://en.wikipedia.org/wiki/Disaster_recovery#Recovery_Point_Objective)) is near-zero. -There may be a short period of time during an incident or forced failover when some data is unavailable in the replica. -Some Workflow History data won't arrive until networks issue are fixed, enabling the History to finish replicating and the divergent History branches to reconcile. - -Temporal Cloud proactively responds to incidents by triggering failovers. -Our recovery time objective ([RTO](https://en.wikipedia.org/wiki/Disaster_recovery#Recovery_Time_Objective)) is 20 minutes or less per incident. +- Our recovery point objective (RPO) is near-zero. + There may be a short period of time during an incident or forced failover when some data is unavailable in the replica. + Some Workflow History data won't arrive until networks issue are fixed, enabling the History to finish replicating and the divergent History branches to reconcile. +- Temporal Cloud proactively responds to incidents by triggering failovers. + Our recovery time objective (RTO) is 20 minutes or less per incident. :::info @@ -123,14 +154,3 @@ During a disaster scenario in which the data on the hard drives in the active Na ::: -## Regional availability {#regional-availability} - -Multi-region Namespaces are one of the high availability options you can choose. -They are available in all existing [Temporal Cloud regions](/cloud/service-availability#regions). - -:::tip - -Namespace pairing is currently limited to regions within the same continent. -South America is excluded as only one region is available. - -::: diff --git a/docs/production-deployment/cloud/high-availability/work-file.txt b/docs/production-deployment/cloud/high-availability/work-file.txt index b95c4f0da7..b8b18d332a 100644 --- a/docs/production-deployment/cloud/high-availability/work-file.txt +++ b/docs/production-deployment/cloud/high-availability/work-file.txt @@ -1,8 +1,24 @@
**STOPPED HERE. Considering whether this should be its own page**
-AFFECTED COVERAGE: + +- Body doubling / parallel play + - https://www.deepwrk.io + - https://www.fastcompany.com/90632591/focusmate-body-doubling-virtual-coworking +Co-working is considered a good "body doubling" practice because the simple presence of others working nearby can significantly boost focus and motivation, acting as a subtle accountability mechanism that helps individuals stay on task and reduce procrastination, especially for people with conditions like ADHD who might struggle with staying focused when working alone; essentially, seeing others working can subconsciously encourage you to do the same. + +## Regional availability {#regional-availability} + +Multi-region Namespaces are one of the high availability options you can choose. +They are available in all existing [Temporal Cloud regions](/cloud/service-availability#regions). + +:::tip + +Namespace pairing is currently limited to regions within the same continent. +South America is excluded as only one region is available. + +::: :::warning diff --git a/sidebars.js b/sidebars.js index e79006cc83..173557f2a6 100644 --- a/sidebars.js +++ b/sidebars.js @@ -347,9 +347,9 @@ module.exports = { id: "production-deployment/cloud/high-availability/index", }, items: [ + "production-deployment/cloud/high-availability/how-it-works", "production-deployment/cloud/high-availability/enable", "production-deployment/cloud/high-availability/failovers", - "production-deployment/cloud/high-availability/how-it-works", "production-deployment/cloud/high-availability/best-practices", "production-deployment/cloud/high-availability/faq", ], diff --git a/src/components/definitions/ExpandableDefinition.js b/src/components/definitions/ExpandableDefinition.js new file mode 100644 index 0000000000..e502a9594d --- /dev/null +++ b/src/components/definitions/ExpandableDefinition.js @@ -0,0 +1,58 @@ +import React, { useState } from 'react'; + +const ExpandableDefinition = ({ label = "Definition", definition }) => { + const [isOpen, setIsOpen] = useState(false); + + const toggleDefinition = () => setIsOpen(!isOpen); + + return ( + + + {isOpen && ( + + + {label}: {/* Dynamically change the label */} + + {definition} + + )} + + ); +}; + +export default ExpandableDefinition; diff --git a/static/img/cloud/high-availability/failover.png b/static/img/cloud/high-availability/failover.png new file mode 100644 index 0000000000000000000000000000000000000000..d538d17be57a185354bdd7d7d375b0f568f938b9 GIT binary patch literal 50651 zcmeFZRa9Ng+BJv3qgWQ2oN;5ySuvum*Bo}Pk@bYT>i!R&inN_ zJw{*kW%mUGcCn~c^;Fd(bIu}MMM(w~2_Fdx3JO(DR#FWL3Z@JS3OX9`4bWn&LbeEe zgLYMu5r?XpAp8q_khIW|vs6@sq6dCPgn|yWfr5K|1o*-SzM!CBbD*JMf$z|-Z8fDBDq*1J%M25b_O{#rBq(Virp ztC9nMxGvuLqE^6^uzdpO;qmqaItcz_z|tW2=|BM7F|pq9aNW7qtn5#dJ>Ge`-b&G( zDsaPNHIZR6%F7mxfdL~fiiikJMhf>W0HcH*u1!S-6&G-7~+0 z@OQSzhLPW$g`I-~(_yPL5=xOV@M$FC@$Mjdg(61KR9Og;;Cs6j$_=bjT=6s_W*L}U zoT)rgJckqU3yUJ>4=F^%KvT@yY=m8o0_ z@CZ$ILs3ytP?lYS=zAm2+^6g2?#7d~_LcfcVLN1&4(H+@!k>3#+qb?G9FOyCG`z*8 zZU`^8n#^iiacp1S3MFy0tL}79_r9+sg*olK$s+RllY9NQINNL0dI;-v=|zR%3RK5K zqfg!jGE08C`ZJwX5Z8a||Kjcae0P8>53X!pzu3B6QqJUW%(23!Q|YlfS#7<(UUBkh z`4dm0+_V!Tvfg#mdAf|PELaOAN42e6Z@C=Bvu^Eu(pvlksrGMFpq|~jI@}Tied)pr zdN}Rgh}RhPdH^@CG}z1@<}T3(1AEvoEP%7j5V0x`7bb)ps`P%sqsz=={cAB(!<2Z4 zFtvm2FIk-T81a$5*ULjGhv;<+dUnfb0>_^|GS+_MVF-v7db*nBc3kBp+0A}=ERG@) zSa^QE=?;~_KSabuAok2*xui-qFkP{)ABWjH0Hs=G{e62_lW)Z_m-0{VwNa6Tndb%D?>^i+57nrp=Kcqwp zv9!9ouR(U)evh~A4zvg3>B*U+Xn*UMjv$N1%^n-RJO&p{jf`(1;Jqbq2|Hh&A4B_%x6hw%cVbq2E2g}nDDu|X>^vd!+MRL| zKGpRHAt0?CO=tl#ToW9%(^l5!!}4mu!B0$>FycAa7*lip;Q7?g2BC-*O(5GQV2^{8 z{VuSTb!xObkE+`FJ4}hT`(T(Ff_+MYQHTYXNC#3rHv) z911Pj3Ik(kx|;d8Qq_KP_=y!oUjM^MJIlv!-Zv-U&JV0i>y`RE?RWd> z;Hz2X8<%+Mv{6h7=_QZu2SZ?Gz}xoblVpPh<6pcK20h*9HMJX^e0B(Q(}#~GG6jEW z-6m=qjSSWe5qq0YUS@vM_UXrz@pMA@W|ph+cnnzlR1uDqqNi~_KSucQJnX$UB?e<9|5x3clP?;nH=O9NU` zl_A0jGLK=DZ9lFXJ1#4)x(KE?q(rnZ57cMohzwwJgC0Z|zRj70wL}z~+I8%Hy1u^5 zepziEb2gL#d_~!NXw3P;N`k$LUl_c9a)KYX31wm6HVfjFE=cUdjM!jWANE-MQ(dLF zP7)_XUVM}XNnynzK0aE@l9eJ#9{pgb8h*ZhzvjOiLbqO1M{uK45vJ_L7|@X6I>DFq zrQZ6DDPeV3sL!Wz&)&B*x0;u7l~nj629)1@&KU8H5Q)l9c5H59C!>k!{7Iz~aMDM` zE#eiqN3b-*bQrCN=|n(U7pPrwVs;9dIYPTjtE(9(rdFA5S_Cj3U~&vYKGo&CO;*-- zY_(|7bnd)>lEg!`Wj5(?y;<|{j=G!IHU^K-FN&v!XG8Y|P%LJ9pYZvoL!%1^V7}oV z;V<%Km(5q)!y1(_NEteA*@<=tf{P5_xfo+<7J0dwbhr@x)bP~c&-ojriV_IN8X_>{ zW%I@|qb9Iq0Zm9z#Ivi`4Lbe73w0`65-6BR(SMqkOpzZzi(UI%vDQhX{ES{-_fxHo zfu(9%{GFStAWw>2F}iM%Y-s-kS-b%?0M2WMR7wm6I`D&7!(0EO%4V+%xfK7GhmGX{ z6NP+iG_0{v&tc{Rj5m)7gIEncOiWB!v9SmZCb2TIC?LM9r}=8Xy;GL1+wJomB4LCZ z(lMrH0|IiAs$!yKJ=@Y_CI!M7xHx$}G~P|97zB`<^W%`nvxzn5wBF0Hl}P=|L-&gv zHvF^6v^)hvWkVd1@7474*WYiark=OZ{m{bI{E{{niDSNSxUeEind@cl4Ifw?q`ex+2>KZVsknA*;}7jGO1y& zqe?4*(~f!?Ooeg@#;H{?QrD3hlfK8x*jb;TK5s;~GmRR^g1$GCK54DBP;I-#4_-t| zG=&}Ux}c3&=NJl;WJ7md3UrJl9a5R4z97_~>pBfd90yB`8D?qI4BGdg*+5u?{YV&c zuitD#=BI)1)EILDlSba)#`Oj=>$GXomlKcf%Pq2|IsFXcek@mLpDuQc*8Ws`Lhs|+ z<=dTjAINAxpqMym(pD>uw!w5fKZpz;qH~0lDn(i@CL7$O71?pti@XetOnKRHIZ+?Q z4WcE02ixABZLOO_MNV~?g5o}2{+7@-`da8@Mh255V>d2}WRgRfhd(u~im1T?vsT7q zH=2rJtpHckjZy$tiS3>tHk*i!i3pd*2{%r?%NDEQKmCO8x6hFi@jBZErZsXzMXx3M z`Knkq4!^hGy;vbTKQjWp4rdtS4vo8p4CU^72bG*{%Va0%h|RTgC&iG?SSW7Qhj8tC z36S^IpP!d57sT;%7>ELUeP{xVkM?E^5lW_}cPWga{ zJo5unkQmNH9SS#L59>GW4CR9Opx#)rkr&+q;(4CQIOKiNg2?S=u>I|A!Iak?UPDq! z+$ZLMQpA0DXb**trl__V(lV1~^%0zr)-t~e0>1Q+Cg;>nQq}HfDiZzMhos$CKl7`h z5xj#wKCaOoXiAX!c>E`rpnx-6Fxx5nefe}T$+Z5KAxW+UUbA#i zR>99>-e&`soV7vR#1LH)-z(MiU#fI2G%V+$48{*pW3kdnt~MER6pqP-6C&pyfpUXZ z2=H@-LmF&IfpMe(o%9R`wfg2;7!eN6=n2#&&^R|c^N+2)W<2+RhjecZ6)4RO)n za~*@PW9MJkaLE*`Vidp#aicJF&gjU6;;TRw`pM{F5u=LYy4(eCdf&7@{iEs1j8EE; z!Q^2YkQp|H2H0!ZRZk@Bypae9k!3^^BTcn&|EfTTpomojQ%g|#fn7@*p>z;ki27|{ zjo93|d@QBquY6Hj8K1~T`&x-7UWKm96ey9i9$4L(v}=?!Wn5mxG|mwOv7^peC4oZt zW*{S{ffyN7rIZY+Z<%;6&dKt7!gd#l=^Q6gmc~vlX>or;9a`!W>vAUZx%hxR%Ng0z z23m2dGq`60>VyLf3{^0ppO+qD0({`c^L6w3d)u!5nla=E71Gj*6D6dk=;KlBZ$!am ziYfT$>T^qa+xv954Zq$S`?&a&j(@e!4$HKw4Vql={DmfohaKrjFWmo@&mxyVz=;gy zooRds2D5M-xya(X`@RV?*ebG-tAaPPbbPNyB6NX0>MHYEb|8?yaewu+n zQK6hdh3*+Hu6tpKveAa?q)>gb^Jt6~7hMY_GICqV0Vg{A$ibC9P-%grMAcs!^*u(v z0s(oP*xUw)`Zryq64UX!;gt+zT*nize@*D$Ckrc_)?>dzn;nWFIVF+#=rK$d`mLmF z|AT?{Pvc{1m*Rozf>AT+`<7T zTIL~Ifu@2MXFZQKwy&}zr{^xA|eA8VMH!AS>rl22!$B3EvMm-gZK>E9mj z7rPx7aRs#b+o*w6bs!_n)6<6(u5z=&9833~NbtI$V-?-^Imy2Jh7i;!Z*9AVk?)>G3!I#pzb79y#=updeiQt$-n_&5 z*Rt-9>yd31_oZzX#Na~DpV}8+do#TbOV`aq9KNOEgw3L#pAlxEn`#?vWkmyt!W;<< z;90x*HN$P;pta)j>Goia!*HWbx8|MzS9wKcFyU~XYDf0YB^NWeCWP(^>a8AoR%pur zci`%Qk@Hg@vzARzr?qo!0xp^ZX;WV84^AdaR`swQWScrzEOYo?oa#6kfza>o5N^q; zUyE9@rr@R8kQmeE8Wos|oa$w~#$p@-n-E*yDE zbx4D(`%)*q%%_Aex%l&n{4O{5<1fnZYTW0mdOl<9LZqH8Zfc+45{Rl?ev z(T9A!;J9(o$j@6Jq$P;hs}37X`#APFhF^XG-Nhvn-hP+N9cNHzJEv;8?D-a!A(d`fMpCi zUq4fJjwb4}n#=F(rdjo{U>j?#LK-=Z7LsVjieG-5#=J#2wKkERnBNZiel|==?=v9i zsrvMbx1}1P?Go@vrLgUSoJsqY?`qxP%!b`Qt7;)?5C8P3xyUN@C~8S|g$;>KzRr>% z@LF?Qy!(W~6vk`Fq7Isy41k99;Um=zL7IIMRL-3CBaO!BG*Gkd9Go;oj{G~!KE^m} z$)IzH66Gow5SER6@F|*r?Lx9Ml;qoH`nlQRpZ7wIUIPBjHlL)T3s z&JA)P)V(+W08n`X%4jWrBD&DmJ;Dpq8fF{yUT*pQEAEv54;1am@Rn5MuPEC;$d?Ku z5dH?pi-6Mz_!uS9SSP(U?CC?*f59q-f2}}z~2U7$cl+da>0Kv#Qjg?t%UJ)^D;im z8~!io_X;Y&1C`mkPhWKZvoWXcbwarQh0Oms=-8luFk2+nHpc&KRM7>}CWeS^wSUd_ zA2jj*X#nJ9i37%^$k4Cz{I@j0Eo2m|2OOZE3>YtR~`Lh)xa`|O#o@BGk)BDf4v>S z(v;r3zi)XoogdoR(7@=tt9MZiATE5>E<&q2vK?Epgpd2x`pu5dEW!tR7apL-Gaq&J zcou@8+l+`*12RsLlFv#R43^REw~~-vDq)NRu6n6xMkZ+(4c{#CH1;K{>!xnObe(?cQ`tBq6I)gH>>KI zsoGVeKY$aWyT$BpT5GAA>2-I`qWR-+?qf*7cBvbjcy<$j3at$Pq$n@qHTnJ>Wq^a{ z{4{o1<-7AP|G^f^H@kkb9&-RoXjaXa<|2oqcZGpoE}@1hUCf5em4-$uWZu6te8i?! zC|N8j|NJ&E12IRcl3LYY0|ZLpbuqP}(6HxvJ)z@ThAw#8#YTiijktQ5UX}AH7c07O zHFrsKEHLrZie($UrRGv(q&FH;zckwe=SvqI*Kpzh(}m;|utL5C+&6fYjl++zTZ|^a zvmDAx0LK$VI$x#K*~I|#=DS!e-%JW~IE&j~KwMQ*f~5MiAmNZygaXtUziJQ7rau}( zH0N?r%G6IW?7d3RFoq98>(PEJ)=2%?==SeRGK8Ou0>e& zYd#?EUNWsd&myT-B3}KSE!L|q^3g;DoD&kX@rn6-X)?k2MM1Grrywsw;-Q3S@abWB zyxMMw$LV@un6%3r=Ia7#q~86=QxYQKZ}e8q^a3wreqQH;^^cf^+a)f5&$8=) zK_2hohlu|yt8_k`vn%gPSeYTq5w85AqouBJ-n#(|S6h7{AsOj4Q6t+N%dFsse5NqU zx|%CZMzAVW^F;*6i0*s#LP9tEK+iKFv%yHEp<+Z^5@33u>b@ZP9Hmau6kFz4en?t5 zyG4s6qV*XNo5~r}JkOY1i|ETUkSt^*8^wE)1$5mG_oO=n|CXw=pijsb-eXK{w%;EX zyyN@v&;T5zL~{87)g8bkFL=V^ruhP^O0KEDHCMW*?;YcEfjS#y@1WDZ+bX4_NEc5j2&*SSW`D7fmWcFPkpsJk=9E}J@~cj zPlaxjV>2c0aE*ul(wSb6#h;&-xB%lF;Ttdz*(oUPJXg-#sof_^072mMQZko?4aq2y z$1;|Bu@QGkqDtbcl`L@7f{tsF$g8v~D`}xi+%Q7IY>6~X$)v)JpT&C_6d)9UP416m z`HuNtSCHSH0moJFj2az?Yh~ew^4gM(ZHM_|pYxSE$8sul`VZ$A-gIJB!MqMxj4FW_ zc|=$dDG3YKAb&c%WWjm>C*8sAaDZIvX=}bz```J)yJ%pu_tx4L#FCdqX{^xO8Il3- ztSVCv!GuAL5v^0U^Av>(kP(&CL={7vqlMAK)X&E$woitG9>+?A8!Y=lxm;WtU0&uz zpM-}w|3j?$TL@5q$=Z_4m3p8l3%%B37MQZvHj0XhiQ}1ts&E6ec654;=R5} z2u=k+QyHo;FSEG6t-y)mTkztXfJD6nTWy7HD{j?d3empHh(&&4$ws?OHPOluC6n^+^Bc2sQ9}_$)U;N4^Mw74a-4 zZUrXAd|dS=z0)~>)ACT(+h+e4J2!jA(*0Ox=rVwF4KR3|U4~7roZlYZ3Tti=>+aXR zPnlvB_-2errMSfAcqNU3OQMN|_+?68T*H42^&+!mxVrJ_RU37B z>9U4w^5~er>PO}{DC&1kh(-wJbjqg>X7Rg~y&^(MI;P?FM)HY%Q=<7lQT7xDQQr=_ za}F10``Jb?HR;&iVpP48IsD$OsckslwC-gmA@Tg1W72bm-*%2xd4lXub;rezHUF2V zbw3;j#-?7o%ElxJuz%O=6Y7=I1;5r!qQgDr1fz}aM=eADCYXB82AJGI#vy>=j_cgO zp}BIE{fzr*07v@@-_|}5*b3Ua#R)Fd51gC}03*GDjym%OFhw&09m(D0q{s$BFp9zF z5h}3I%*oTNvd9`hzjAqBjL^JCj2Og?n@OB)L8jFEjtl{nBE)|OP=#i(pQX1XkDCZh z=Ys?fB8T2DPdCkMW`pK^VuVvOM4yMruIJR1mrX*6_QNMZcGV8n&efgQAp0dBnx!f% zkzIP+VH}swPP2+U5C^0vQuh?Zua5u=c3gawie$xxD>IA=k~H7$5BK|ti7#Wm+Tzj# z@VpNqqi>+)K4uRXqjXr06-*`wP{!5`0%i8&fOO=Ct&LD1Av`cL3r~gqdbiNrWx-+j zJ1p2^nQwOMl}|BN2tj2y{R{9tmaqQ7cfD9S`(zxJ*yl`4qR$H;!#rmLfn0DRUhkw> zj_RHh_aT=L)qmsZ!~<;na-ZD%iz*vHo^6(E6t{!)SwRf@I5xq_yBzb^?M4p`wtcM& z7x`QAmGe3Xm-^vKLQLh>D#G`FC|+_+@y`Ih^;%swa~ft7AAt#*Ve~FwNG)Qv#N~N* z0yY=<=1A<}4B*D@VuUP$bl}A}-ik?%jrPC>$s2M|Aw!4!$uWEcNH8H)&y&^zl)7N) zpHgbp*tKkrBg%f;jf(ByCjBt}1I%=J;WUg{fT0);-NK*DoervrMQ=!o)xb3rOG%{* zXvtBeG~{gm{XvwHth~XuN=&VG#6mB_W=B*nIEh#LeivR1WdwzqqeG@2U91(bc4riA zss$Ui5e{i+tVbB@*Eh-Dr=5#_*Y(-U0JWc5e|*P`MdOdc9q&JK)DhF5>Wm_@EnPk) z-Wr0dH@t|vOW;$I4~Qv#(Zp0zEl#23Sq!=Mr94qcHct;N!HxwL$hgYu(e_%> z_#v+pZZb4@3D9KNGq!zV8v(f2#Af>th2|nD4CS4?!mXVlV`h^y7KgN!&KSk2+qOf! zI(3n|-S~14vjcDp&&)YCp>vU$BWRJAgs+#({C*+L_5^ld5_4e6(x=UcJpENBLsG`o zP4r*TwQ6it^06j5Kf-JHXx%=wYa-L~iuf-0b6@n2IQd-?D64ranIH%}9M{)dQpbwF zkw*k6akg2j^#MJyWWYoN|wa!sUKVU_F6GL$06U%s!SNs#FJ!-o8zn} z0ZLzE_|@w=Hguj&JPzH<6JXliiDjPvA>69#gDQwZznYeLu+eCc`O;reW90t? z>68wHC*Cq6N&DrtXlTXwXG`KUASHMh3-PlTC*0%DdkUjUUY79bo;eU}$Gd>Gkw*f* z>xEB^1WV8nxJe0wh49~lo&1luLeAYL$lP6?i2ZK2vNpwV$vPWC#NnCf9Z&-~`<0H# zlmkV@sI3=kMU_Y~8h9)*tRDuDV3|035%;3T1~i0Ep(x={8PaA+Q?TITM!95}KS-cV zKcAzVcLBuZ*3U8l@lRx{zqmQfS(Bw%R~c@8{^I!>xgwrh54Fo>-O~-|HV9xX4x>_Y z0o_qQAa39m@fO`km~?-Sjg_Sca>u6eWpwy{9~vMG?3=D}##aD0lKlZH*CGiD8|IDX z)oh0ohOQuAf`qsT@eERM=b8S{;HzAW?WohV5Dlau3rm7;k8mep)|)`-hihiz)Rsu( za=Rq_@_6ZYh~dp!#1}5GR4_PO0tDe@4#hD}#gb?Pz^ zp+%GnI;zyqEx5+YSv03K*%7BhH{z>zRUxdg?I{L15V7+9&0u&sYeh1Z1TyMUg>#O= zn3@$;HB#1Z;UVY-6}8*MHfH)rsun#OD5@4UaBnIKEG2?C$$kSur9X|PyKlEe?}B`7 zXF65&1V@g3HNan{q=aFi!Sv=)t;1@058aqma+=_M5EmpZ-Gvuq)lXqH30j20e!mpb zz3U{)>~Da6VACyEjUncZq&tX=sMoJE%%UgMaBf{@7QUk9)5$RDQVV(5co9yAgN$)K z_e4T##hO|{dq%ql-68HtmAuDIsr?h#g`K`+3Rm}5j>-vTE*K9v*nqojlPHJz>w6m> zmlf&0v4Bj!IyFhIa@9g~*L{&SgOx;1Vjv+!RXgxepl6D_WxEOu`(*ZE7qR)nH#%DW zZAw)Kz~*Q~b#e%P06>B70(O0{4d7XuPgUn}oiR+Je7bQRer#900s=``P%2O0?5AILfp*+8U zUiBx-IwQd9L*S(oqabmXT~~e};?#{57?w-3K?EoM)t9~FUa{YCV4J1{ww=FnL|e|$ zR&bQB9di(FLzK0V*_PoL4Tb>-^&%LV3mcpVn zHB1gLUsmZI=Z;gt!K}lINWP?!iMj!w0eO(63H?h>(1tD3Lz%^FXjA(V|AhZ!2g6R@6^r8R*ZB!<1G{#>RveQ)AO5-lL>+U4;{ z*kRowL?Ior7RNiQC8a|w5Ky>b&sT{Q?SXyEvfmp04W5MaphF#lwkUa8BbYAD`Ww7D zV{|@Yc}1Q4dK;SC;?IsRSE-e2b6?M@$F^BO^!fuc3)U23w}!`zP?LiW)cphAQLM-+ z6>u>hayyHKCK+M%G!H|i{FcP?b7lDXkzCnm79I5;Gbb zv}udH7V*l4`%H3F{19G=FFuSQ(c3>4ThZ_i_(TGF^EtlsU)s2$i(_5w8&bpvuOt4l zW{l3#nMn-K(MO=fM*G<#jY!b-?jFy5#B_n*={fl@&RY_$f+-ogR+b5y31XEfd#15aOJasWGO-f_^9FFko zd+Igp2x8j!?T(Ue9A)Ccl({(8R~{XE*thc7qcTRi*08F*XD@T{*dluG9x*jm|cItD{FmXqSFL@~9ffMc$y%Tze20hqPr-4ZyYRA^W_ zO;qXZVOWO8+6$EKs2IVp;Z}r0R;C~%9(~(0C1)Ck(Ig%zFm5W$eK*6j*U9VOpRP=y zJ^H1cotZPl(?381$Tj^Tl8IN=9835TSS}|0rr2Tl5#m!Yxlx^aN%|mj)Nvw^*Dzd~ zh5-jzMGcyyH6Yu(L6)vr+%fB%;6GX-c|HrRXq6-+TZH{f&PehFL=A^eAbKI@vl|RH znMPp`7G0U7+ResUizNh6SeWL(nM_E@%aTNK^y52F_T#J!Z_b2=umq;;%2#JBrj+({ zzDrZ*42&fk_8~Rgocx42&dEW`z7)HE-x%Jb;W*YM8ikji9a;3* zT0-vaVvrW+)n0=TqO~v;960s5S3xtahC9G3Hzr~nzoh*Kw&o)7ep?ggS|qNY`+evls_K=ds-KISWPV9J@vkR+6U@)!>5 z8vgd%J>*%3Eicf|5Ik$A+pxS}eaQ?Y0TdZgcCQ<^_tMa}LNo8jbw=qmr$|(Vs3kPw z?)MnB-KZKAdeRDviA_HQ(>Gs(KMcZbNzI0>(7z#TU^r$=)f?=#!DuioxB*nREAc7~ ze_jEEY_B3kfZ@cTsfTF^3L-@S?hq66I0gj0lH_)KZ=A~hZ!e$f;r*h-`U>;BOuSJ$1G2vGYd>qjpE*&`{ip{il z3^weJoJO^3{8qX!_*R(tQd2>E3(X*khnO_FovV=2ctM2To;>TN5{jYU-+)`X3 zkn6fL-%NzMs)ymp$k#3Mxlj&nxc29byETt48PhmD@+rePoLu(VG(QT-NaBVqwla*i z*c`xT(JjJ{5*{X-^;8)A_249G-lW+vxZCwzRm`hLT;$F4bk198dGwL(l8e3i0!@ED|pni{oajOFbrD@!mKuOoodj(+YPcLKMnr zApx&>Mv^160g`N{EnGW9%GIPEd_q4@^kVQhi$(Dg= zykg&R-g->`@ydcpX`Fjn9?uEhYO2&%cw$H4ExY5V)4mIZN8y$b{pYs&opGOsm5m8r zkrKROi<`ymCS{mhr!TOlk*nqmYF($!L_Cifxm3DE!{pSX^=TIWoA%$yK*^sf;OC=lC3mh3 z4Kq{H)|`t;xpAeUolHVfipX4gu?M@TYOR|xAz9yZ_$XeaAD&L9!Kr1Umone1uw_>K zXPqO6kyK$EO9WX1gA-5g?{I=Edvw)(XTWuCVPSb3y-i07ZG%-&fEK&LQePduC~4C-GB7Ek~7{6N9fd0jJ( zqupiK;cDHIdW~j;kTN>N2^~e8CR!uf{xS6e0)00lx6Ys`IO6I0gr*Z}#*PBAQ3#tF zG|LNmCxG0;;fdB0{T|_Q>3k-XjVk8UfOAspEHh#OU*P#)Xj|ENf|zGb41APOQ*}x~ zO$7ea8wV&R)_P1vtb%9O*52q8k=09=D68P|F5ZbMewlcJZ`{NRVqUz-!LS+9E%U9c zcJ3Smu&Y5!;Z@x-WBfgd`RVngiF0f)nxaXGAf4NDf`#GClpwAj1nQ-boIPP@eTW|~>z z$9y)Np6cNcsz#Yey<42bI(l;t!;Wl>zf)?da}f%j)#MS^oDqXoZ{2rpnb$kum#zFCu8$URRvs;KqCS5jBrBg?rM+PwU;NE-+t&en#P<9dAY>5;CriTEsO zCY-;-5M$nS8C*1XKQXNo^L<|9*5x7K0e3yH@AzND|Bh76XRkN%<>HvI$t(ttUg{l7 z8%juoD7I9&8S(m=l$i#Iy`(rw#|G8gZ|)N37wig61Br=?hIeX@ALdKUuMNR?&FWaCLS0xjmjJW>JbI3DhiusMAj&=dDy?X7qWVsGH)}tr~Y{De^7oh|$ zV>~4Ra6?5WPZol;xi+Z99F~?L{-EdjV3P?vS8`1i(Tlk1ezs%1L zrDXI&>Jt?0qY)qniaYy^so{#WKuYRWZ98T(*u=(yZWc=f+dU2yVNE z$s5+5)KG|9I;DbZ5thQs&F3r*N=T^a-qOb>6J0kdud}`(huXEIml))DYE8F!O%)oB zImhDlky}U$Mu{cdVffq@9?V8Zt9w}4K0mG7`nE8W+Zp({^`X_lB0Y1z3%K0yIQk;2 z?_i;x+5Rha!bnfeA2F4_F|>(Q^dAEM6zncV`v_W+qAyW* zvJca^=a1(xxy{y`$`TOLu_t*&%!df4-MMyqtd~ZZ_O%7TSxX4)9uy@R4eP#+{<^#FE4yD zz1DCH;IUxP0zdgPvmPP@wkUs`FHtkFa-8Y-lgB;f;32f(P$@X@mG)C}8Sz9hD|dOl zOiUGtiUt<8YGu6i*?qFp%HRBFa6^eUXon=GZu!#kAx@EmV|jmaq}y9AQ*_FCAWXmw zbN+kz^LG~Knj?Ep(!+NF?hl+YTJ7yooNtJI8^&Fct=A6y``~hx9nga0dNSC|4&S8? zG9&@ol*PM;c;Pw@>CA)T`_m7X0hLW>ql1($v0V{5uIlxQJ{KJ6BCbXQ+9iv;zcsU; zIzXXilb!6ea6e>nvWlJ~SbU#ER8_L~hi%ahc-*IIhd>6+X7d`&uAoJ+P(t5;5x zZj@$d4YPkgSNa|^I|wPz79hDAL3SmscXrzO_<5lfZ4QykMs5=TC-kNHT(Vo}n)QHj zp~%2Jb=+;`R^FBX;4SgXyK5`6L8;L>Stku87;|Dm2%r%HpRs5{mmJ@@`*5MV5!a@d z8X^0M4!)+7Z3_E|cUCZ)vgP~`{PI8K_4>}mM&=zz%y4Bm0L!gqH~vM?xRy^wHP}f; z%exI;^H?p!_b{?Jy9pqsoGxjUudmB5yKeJ#QfK0EUPrsFsiN~{Q1PL6!si4=;A#2t z*j7db;p&eSO|^dmQRBid_W+8bNx%EJwZ{perX%^4l|#pSk)3C7*9@8?$iGTiG}Ft3 zcTueb&Ev39Lv~LSv49k#`6Yq0&A<$?-%amobH#yv79e(>q@n6JJo_v=FMdEg{Im{Q z$7%Jd+b8ywGjz$f$)gVo$Rr4ce?o~8HUneIl))gC#MICcZ5wP;~Hd>B+M43HKxTry~l*ge|QQ@X| z4hUn4QMGYi_TUox#^$(5yMeb3WVhxv&WCJ_7)@yNgwEUJgrf4L z!z(8bDDJf0JwJvUzbrUhA13F6p$}U8#|pU{;ui!!ra_~#J4OB`$sJyqv)L2P5?UKW z*fTs`UMo$;j;{I*H=lUGFWH=Pr3XW8xTo%I>eU0{43pxU-LD*EcM-;-O(?=f>^vY+ zh+J`W12Hx0#IKM-aRe+7I9;W~Ty1ze*&VL6KlhaHV1<`xne27(K681W1rHI4`SvCe zv?Sc}V=x71!(56xF(!`|cHojU&_xPLo0o zhn3BGJThtzCr#r=pFHFjh=jnK}i^h6aVr`qmO&*kO|{x+TuH(MSzuYnw2`jvCZ6y&U>-R(WV) z&LhQx#Z^28q8HWPHKx$N$BNaIQDK8kaX-jh@U9=P*r;j;&9$Fg;A^hxh*?w3$Mggp z7vEnRB=bGm4obFfLkPg2W=BXw&b6@{&x%*7Vx{)Ysyn|`w{z_p>phx)eU$HU&ir+Q zFjFU3tT^z=c3<-0#7jo^h1hpxvp}g6mC)^+;$%)r<0wyFGmIP~48c}Puqa0}Qq+S- zk)(jq7vFpyQ)3tqIGi>K>pT8^s!-}a6|U9=gSio7fg3O{b~y9di)|BI48|QcEHVCl z(QdPk0?f^cO#6h_#SY&57vT0|U^t^?&%s(NWAPe2trI_2yVf3-4kId#Icpq>xd3f} z@nyIGM9Gfi5S=)&WCBwqTgxAZ@WzIUtV|S6cg+-ejJFO+3|tRcSg=}gZ~)XYw|`%Z z(jE2uMHh6#0XnomJlO^9Ax8IsRez{*^>lfaf=gSOPoz}4lWpcfR;a)7%(!Gxv>977 zzEk9mf`H>Mtbpllpi(->Dez#vlm$v|-1twNQs=&^Dl|SkG{gp?J!`m7%vU#hU1qIS zEOMJeQ`X6|v0Ak55}G=@eGPK%yN9ihmZ!|qqe^Vu&e@CX_UUymP4053%YX+ZB*6<3ob9t3K<^qJz)29#+Dn zwChh^3_o$PaBO5f99yioFONHXntV3uOJhvphftC?zO;W7fPg|}s7hP`Wy8EN{Ln5Z zaaeLiL98YZcwh7BTCwxaG97X;M)gq|Gx1f{H?zHLQJZkUKBDLHo@Mgb%pmVCgW?*2Uc8Yz^kP5g(t>lN*56{=%t9rbAEXM zuj(MsOvcpw6kGw^XM4fb;foLW{X2ER>j|>3v0QPj<2HW}+x-0_Wt45A_BtPnfvI-) z*LtGHeKGelTW>4v+fe(M?B_&Zezy$p(W)-wV@m;&32&Kt%ydc2WgVa|f=-rKwR+Xm zSm7N0m@Qv?NnkaU*9e^Wsp(T%S7LkLhpvA^mE5o|gd(d5iQ+&wQyXrM zKLNtV&YDVP_ifsDg0e2AmlZTFzbmqX-E6uSXrYI_`7~fks^!3xJb)>s%d677I_3Bu zkv~*_3WK}N8j&+k3*RqN{B@n|qwv_;dI;5fAgXnOgDJ0EBqkU`{iUVgJ7C zbE7Yaf1;~v(2v{ar)IrlN!+C1-;9?@!1;32*3UlQ~F`YC%cjoC#xQMP(qSw`!@7i_m+7mV6bUY2E7Wpml}IHOnHHo6Z8Nz!xf zRgD0F%~*uT`kW=-%-55T$;)PD*|ikXgrCw`8UrE2^&fGXN(oR6Z$#SFxw!VC^fSGT zJZr34^pk_v7wzj6QhUjV6qoHr*8^Uva`e>;7WW%71GH23Urd<>AYxg)a$=16PIR@jX<3ifP@f&c@HDIu6l4mQ8r=;n;mHdKq~9RENBN7ZL?BsN>~t#!x6UZJPaTH*oHNDNDmG#?ohRwC<%b-G%JaPg ztTc#`vX9H5sX(h9akbrmQ0KlOO$z=>EVx z@EYKDA1FsGxXwQq0sY!U{B{ZbOV#Jbrl$QigLON{5N4XP_o%N-9jM`-?7Q zzGk57?bM|E^>U`C6v$v|E{q>OHyA$Oqc(YNkgcQQsNb!n*?Rcu>lJ4m_wS&Ie7VTw zKpz4skw2>v#l-wqwuf_nzx`bzC{wTAY%eGvKo1;aZvTB7|Ah%ARe}0r5ul1S5N@8fTNyqPQbU7xQY>&6`p0Oog?y| zxOgGmmOMh^e3qgau8}WO-`~_%I~T+t`ClGP@;vVYx6PlBLHh5YwyLo_U#}A<%5qz! zDJn&*I=wt~meY4YebKijO9xanSryj?R+$--e@y=Zia{o#B>17Ly~wxj>mRk3)Ja6F zL_oKmr>12xu8SY6%GI4Bm3%{q+-X~$=yEVhEZ~4rnp1xnOnlX{auKH+arWMljhA|| zk1P6|QKluU1J_U6e(x)jwn|n9hdQq1+y{dg5|Q;mzR>$@|L4{hfZd!t#%?BI&Q`1fH?~1z<1Z51Rj4ZJRixEl!PX$-10wLoRrPuJro-OkpxII5! zH5O+2Z()B-Pe(Ym!w(L=)C1Wjbs6tG{>_@cI{j*-q8UI=s^_I}Ie6P_HQ7iYyV)+s*xD3ab%(5Z_pV|#0t{nx+v^DWsXc^V z`?-_RO#F&ayG-P=^o`b`CPQ})fiDnWd(p`kKAx&y+qD0yv&TnkE?0AcP*m-5ak{xD?N9zCe@yLCod{NK8dH%17-LVw| zs@_CedYfa2Ki-c5%71Pca5BS0XSkE_&%GgXOK@{le^H!x=#4zcsxNg6oPU&e(o(T% zBnQN|IQJ+0=otiMcE`aZX5b^cW4qVnzs)1QK7{$Z;q*Tj6#nSJk(Q4!n!$EMDqpO% z$jTkJE*jcT*zL8c3W0kUjhwvuk$d$vJK)FU>IN%e-(P4(4lFA^rKpuMt}z)VZmR#6 z<=Nl}b-}gUKe+$2>~w(3T-EGm0X2%H;Du}yPJvfFLmo)Vjl=Ew|4{V}jClr2+p%rC zL1Wu?V>^xQG`5Y#Zp_BEn#MMpv~e2yyK~;t_Iy9!xu2cc(az3wE!!FGbwZuuvK{S- z;eYBFyI=F-Wi;A<5es=_Fhc6U$Q6_^)E1WZ$F-(b75lDbGl54N@vkcO5s{^CN-_Gc z3ny-yk8vwzvUY}Ht(Y%CUdSxXl2X`TiY@-A93~)=bZf*)@!)RL3tZDhDO1q%Yooq% z%=oAKACVu+^wCo5OtI(oe2Dps$v3-ZLxBWBc$)RV$ZuGxe&-H45vlxN*@Wgvwwh{B z8W*(pZ*m$JtZN!ubA1m-tvH(@t;gP1sLFF#)05AXrnU^_xZRWjHEvMuzM$~{XwyR< zt@VX!1hcx!ALK?k8tBO3!;y|)Go(%ZY~=(I|4MsR#^to6TKc;7NOHIpto5&J#?7Xv zZ2WR$DnA?O)vHINgDH~`xP_Op2=B@ZGopO{OfY9f> z2W^TJ`J*X$GC<>-gzgP*Ers=U1$7&!EuYGXOpT<$GSy;nRayV1s3l{XT&F1oz6>{b|un+n09LcbUFY1K^g4mX8C_P9*=g zS=Z4*mm=`X{`TJepX^(|jC-GQXL@gkaJU~YePVEu{ze)0O6Y%4fs1~UxL93(Am}w{ zt#qs7%6Y;!4@z(y{%1X*!aXPY^F9&pY~FfQPh~2$ZzXjzD7a1jnfGuTHyZ6|48R@J z>L}gGBgndG_vw+rYiC!f>A2_1aX9MZ>hGY`y8Yt)Idd1$zb=ac2DH7#;Aoffc4O=H zfog9SAYIY6Z|P;JH;2#qKPZuOyZ38Y+%o@};IUh#l^gBTZGB^Cr5*ixQJO-(?BYLX zg8>D)O4W6thKy0c{YPhTXAaDkF7VqTng6)mZyU9Pb^|E zJ5i7(t?bJQUJjsJkHD!Oltl&NT_f{-plQVtQqh;?8`0w(u4qNuvy;wJv8){lpctnA zQB0l#MBDX=bx_x%UZ$W=Tcf^Trz?;L9*93#Y@^OOO36PTl&!f_cG+`wIyY05RRHnq zgtUE*&*E3=e~k!~bcz(>ysqtThvR23dgZ~2^=c73Lv8w`z-oT!;Rz|~d@^1(pLOJ0 zEyiKDg6%t82EL9}b|-IZhh|C>|LYH^3HXlEpI|>_3)3?Mk~G1;xZiQBFKJSdc`(qX zT>d+j8F@)ENwAKlUfZ_)BS$l8R5hhv8I9)ueB=Wdgo94w1kVOK2!-qKLF`o^AIhLO zo=A`uWE7s}JnVUTEz{?GYwj zd-nY{rbgO$uE7k26%7l0vCQ1UeKOqN* zgg=cTGTPj4GeN&U4~!r46s&aYB`)}F`m}S+A4cnov6p{NcQ#*cwBOjlRvgdbT-`~|=Q%BIJLNcxJ3MN4Q0{5qtCbU6 zL5+zz|EfMz0W_21Vts}_xu4la-4YtNSGQze+x}7!SNl@=4>*Vz0+IuwuD=XLHddVx}2zRUYVc9I=Y!$gRFl0L>njj>VQ_#$vOK?;}0l$dJR>?>-VL#e`G30 z0c~ypZeh$^f#9bF__a>`Wi!v#R!{{~p@$7~=Y#P^Ml<3SZb;Pva;LCq2^&^i7v z8Ng9A+1J$nTSqJRw9~3tBE|XNo}Z$eo6g(Tuk3I@rd^%+t#;IOmAA9(zyBbLK7TRW z{gu|Bd-{P}XO7X#`{QGETLnFfgWc9zWoFd|Hs~(0>7a&Pw2{fN~1#ikz zzRBXw+FBHR8cus$FMq*$Y`JD_zMNyiuNz^KzyDl!LGzC!E)ND`3BfSaw(`~YQR_%e zL+c2U6XU|c7Rpu=e;x>Q^BYfxg!pUD@#EQolWv-JvPZKs?Htxht^X}{0dera0t{<- z+}F4=J%(@>OqQ6AxV=5rQ#(o7%hAmgzK$7m_hOy$j|il(EZk^+H0S)$om~8M)RbOf z_-AJhp+ObZ6}fsdu=Rm3=@9VIY_dAH@14v{yx4WlEwS6mtp3g56>L-Ds;5;IskH~X zKLp1E&41(U8&>yFv}d`ZiLO2xk?WW#6BxSPuvNSM_?3;@imTw^?c&Bcx6?-UTBmQn zmFBgks-^1MRpI}0n^IhejLfP{fk(|vJ@-cGgDewP0;4KPOX>n1>kXUsn<|~>36;q> zulXk3PbNA>TPzD}Rjouv>L35n=pEvK%7I|<{3@y(E5DzZv;qjRQ0FCw1#DRlCK_7P zTbrcLLTn22k2EJw`H!j;)QJ9Bf-uTY4nu!LzMb&bmdduB$TnKnew2B|Vqd#vUUyYm z-pskT4DqbKK8|%#AI{$D_aCOW|AKJmD8O1OR~FKl+iZb~9%$qMOjsFo657T1p`{zw z8=`gxW#VtOv+dGWip^3VEq87i;!!qj75~qFFxn3$Fl^jj^C1d+7lM=^zzF*I-BxP`E3x_%e)F6jt>}D2hx+C+5F=6iJC^It}9D;W&Zv#0L4~>G35lqVHu#z z_3IkXWy;@=(pVI8eaOOfyT<2SK7_BaDI_pj{}^IxUUDzhHC*`u1|R$eh>&5|t_|AgGs*a)uX+9>ct;_uq2 z>A`FuY7cT2obMbmov!CO4zk^4@cq(FDs&2XJk={=ydM>Gyy~`ZzG2(1toa2Aqr?VE6pE+EWcRE3rHvi6x=CW-XR z_KrH^?S)n}q%M&%e^-G5rWWq6qd5=syDAqY@RzTkiE4pLS|uc%j!xE9T6Ma3t19p2Jtf@-MU@;+d+f#j3qsqEX7Keipkf zQ_OAlrS~}jsyiE?(p~x%q)~h7bZU@eU=Bbd^G|b4nG4D0CZiEXm zv}}RlGaCon?zsiV|4X+bc~&S6VvQxc)a#$MhFDn5G>^tPSGS0t@~pmoPE9CKe_IIoZ6C zC9uGJ∨%TsKu&4=gY4^07@2e3@=TCe%*vz(})FQhB}91%13dv#;rTD1Y_Zjp00g zL!SeXyH$x8ypIQ2&I$pe3+A;vmse`JDMiYE%-BK|B1^rI)2+PvX+6u>@gWFVogYnH z{+UX4cJG)#-F?;hd5=o4PG>C^`VBF_0u-A$01VyLIYm?UcZf54FY)yU`k9rLLufX_ zQRJX}fNg6~UF##}(UF8#3|W1)^CwJPJF1p_Og>dMWh_@}aXbe=ztlUcVNQm+N0x>j?tKhx zxcNCvuKQIwiEaF_sW-dCuS;(z z%SXT#j*&;WTs|L+BgllsRTjR;D(bDLjA;S^>BcEiFisS>^Lusr` zmwWULaC4}$s$t;s5x_tcWdjs(g!099IlKU!XeW(2DR_i#$bP#UfHr&V*1g@e_H$7w zePdz0K3be429TrAcTE$r{york=I>Uxu04lv&6fo1tUpL!z|@wO13^u*VwZo!hEsN< zB64xTMaCx_DiKn>`*HmYL4FBK$7UOTycEq#{X1AJfX94g-hX-P9so?Tz<56(_sSc~ zrYQJ%A@TD_7x0@;GwuMM@@{$tAd&kBfCJ;t=DIIp?3ZL2JNI_`1`L6HXdX?XTJ=20 zKG=nO&Gvh_7Z7Mr3fVJ?*133N8686}uWDNPMj{=;C)p~Z`MYa`ENV&GZW#KUxdT=Gl5BW~XGZ6E%+#McSv=OrOd;GtA>5<63$d0R4$=w#J`d9odPk1mpgcgT$NI>CE(*SpTperyb+Klp$HAj29a%1rzs9*QMgEfd-E8lD;wRYrY) zinzZmE~LPG@<_eyLTP9hh%dHM>XC0d$Vvk;ytMIISQU! z_B&!go!BDZeb#kcWyMFMUHC0|79i~N*4qD`zH1=I1{u9YnhYa`1m-o)zUUD{yhu_P z%~aEV%EJpVe?D7;i)&d>_G3`tsX+V=Z;BU}_S%u@9SwjcvdFaWYJCPm4u|@p{gIpI zmI-mO%&$Cq01s0$iqB#i{s!MY+3%G+erX4kVGCf5t_9&eW0hDvyAR7NY_>zNZzi(= zguzXT;4_Wba{RFE8q5g*n7o2;20(;WwDem5Z1H531k-Eu-DjMSC#6pS_{X|_3ke;1 zh5HOsrQc-;03c@Ma_6bMocANg1mK^c>%XIz4*+>KKrwSe61dwsXFW)0`?7f$J$K2w zfU7%`^9&GeTE1~k*q5}eJa7R4&T-`&?@FRhXbBHrI{`rVHibhv?>U_&52O0h-tGC)^RfzgEA zL8kvWCaujCX^#vIFMAPE7R8B`+a_PNZO#__O}r1XMBfDvUcBI~X!77m)X9xI42W%& z7RLU*T^$QT-g_ZG3n+4`>O7c=WJ>HeC5czfA+L#1P)#p4!^%7Tm78Ek!GVOV@9d5EM7xaGMqfvO##rAw?A%}Rk(xUJrOVwnVwoU1aLc$;7cu%?P86$_VbTo_N`A}jN-lg8; zi1tHjy|b@ldS+xkTCO)=Pf$HfNe@;YvG;q_5&S^TaExC!tuTwgbk`efT$1hf`vfsC zx^>LG>oQx}RNN6D1h{^Powsmp80CP&j6f-c1?Vhm5^&oJBCNiGrJee-6>y7>MTDaD zjRfMb<;iL7qZKo0VPF=Y?lQ$2a z5WxCP@KMZ}YkJuoBF4dqJeX@jCcCh8h%!xWPk4$)^rkLi;LLg|FzoR@b|6e5D*ytI z0&DsQT7%yO2)PDM?c05cH&+AE;2!yaWM&>+i<}9mt^f@?*8Pl3`|Z!dxQD@zHS@Mt z0A(xxu8b5(zc zX3GJ{V!fG-o}@Eye0XU*9MDBjOgg(pE3+8(1x=EU`jRm^fLIszl>9U()p3t~w;>a^ zc}8#-#(Zem*=$YbJi7ku(kP42+6>69m)lwEmZ6(=pz1?D;jE{S6D?{lxRPe9ueW}$ zpB<}fSnOO4h7!n$CnUFCoGa{D0V1)c?k)h)Y))_w*9`ZqU)#DWW!%}LHok*bM>w~5 z*5%$w&-eZZt~vCNnpSZm(sBmDwm~xic?ap@Bc89(_3$^hy^E;*(K;Zhj$(q3(d5wS z;7D8~yHf5}`g&ofEP+DHqs1u#tCn9B{Y`?3HW=Bk#A;{UK6n`d!OxBZjq;klE*S7D zq&SJ)T=bIX&+OHg2gL{2Qml)R=9w8W!^VB3*ZRAdeiOqW&m zKD4_X>ZZGth@2n3w_rBn-X@F|@A6sRF$MqV9DCP5UYnoYG8ktY$i+&sDS6(-!;pR}p?J{?Bois;|UoYt&>{b|BxG!^UmSl4Fyk6l(KhMUSXW^;=ThiQ1jF7Eo z8tO~=xv0~G*zd`agQ(eoeX;OG+k)1PIk%?9pHcE5D2&+GBfwnr^+>J)s|DM@fnxOb zaX>sbCmArSj-(0IAwOb~uJyh#Qx?c)H{b4hk5*Sy?p)~`!uy$$js~ADal5TLA@5k1 zd$$uCqxxUUP@K+*jz_dCb_+u$&6SU-#q*t)1xb#q8MY6Ol5?yR;zh6ft*(Rub}On+ z{L!o`u{_iqG4X<&4&zS%kDrZ-0FzU_BJKufvlM+)w$1<{(QMrW>PQuuEa|CS=fO-9 z*YCrV`l)i33^TT$c;(Me2ZFa5>XHi2;c;bh(F>?-b!l+5TyV`qNapquJ%MO!{w=;c zp2a-4A6Z0Pisl6m$Q@ppGlu9fIDDzH97HlvO-VfsDe&XWBAVk5YQ;s{2NZPESeXjM z>tzKJFG4y}AH&_Cqc8LdVjJn4HWV34aJbhCe&U z#~KGLkQ?Ic?1Xb@wFyhXqeE~qNeS$m9S-L_AaR55#JuI1G+({fsO9LM1oqQd%?~$8^+(q=l3(r zbO5O=);Nm!vcDGuBCn@pAlzyJbqr-+j9;nz1}TrSME+yTa{J;p>t_GIxmx zL}ol-FvL~wJ%m3v!BWIvrp8M`xITaRtbcgIP3np=evTG$aAER|m;xMDr`~D81u3fT zXKDI=BsPPiAk0nKUB-9z`89na0N0Eo#(UuM+T}TJ(RL&4u2EFog^{JFXg9(P8z~<50(| zb_jW&SQ}h=78Djc@9t(P-oZd6$~+n-Km;ep;{68o%%`sd4Vi`exN1yb&a5c}W$)%i z%9=9dT-hdRQNK+-!Db;hveS@wAU6+56I%Kkp%^is^tZquu=#a)QSR*VGRJKfeq+h3 z*P|rp_4q5|9bfqsSQkA@bxsElMr#9#0`a3}(r%C|iY|&!1@;?IOSUhI#pGgklhOm| zZ8^rf_|w&&GJ8=p2Sq-0bla&@P~FG&@AFbA(z%RL#XkxW^x$rH7SPvy%?cnZf=7!m zhY?sq0Bem~22VUAyaZjjNB$y2QikBaRvAup*iqA>8hAd%K%#K9iQ)<2WwrK78ZVAD zM;US(FMz+zy9Km7g=H>Ip}<66!S&ND>q=8z+6ZR!9~^VBbCWnR@bdM7c)uale(AzR znBl;+fR^qOM5wGaNA7D6_!=e<>9LH-@@tY4SCacMuh7yTJE{PD$kH|`K$#qN!{Fy_ zqAM>kR#6<(+285Q>Xr3&k)9<^gq|uQ+;Z)3`<(Sz2%nlM!bMI&(bZfw~G#L0LNL z@#`GyUVa0<+c{y*A&TRezS!%@UGA&m8@rVghVTQ5zA~ZxCPpTs!SB+;rEp8J#es+pP@LId_O8S|XZDDIE=5qlNfv3S^JA2l zLZXtk@*omHG72&s%s6!Djwe48#+myPV%`)2ihn|sJA;5@3GJP1LW=y|jVWI2PP1__ zY-qA1JIU?5PsE2+2h;l5gOOym`)-i&X$yIr_G7`M2UXpl! zg5neyB$=FMag7KSe$KARctMNij?y0RBtb5r8jsgP7V&#G8e!USCd5=(oZFk!Q+9NE5l;+B-1)*^Dvy~q;s#WY_~lVHNgM2_D39Ryb~9EoA! z$f^=9J~M(8hl0M zIO?1aC{i5z1i~`crv-Xv<-IB+)?Fr&vvH0G*|O*k!+Vr87nc?tvwWfR35SZMEsSOo<4QV-8ftm!ZA+^G zTzDJD2Aw!oGAoACr(jVK743N;?>K9pZDwH9lx(keeW$}u2VoGp9b zu9AZ;i9RW&5EGERk+s0znvM)VzG3eO+DPxda|oaRS9f$ejdvY96Bp)+K5uZc2fRWUMGuC+_69 zg2S2FT9mRSv2!?7b%o}GeJX&ItB6F!C;`q0I8gLK=j+3#v=m0cDn5BzFkUpqW+h2> zpq&px=cM$wxJ7sUl1q;@yCUWjYWNHlm3k-J#x5f;eC5+h;So>54?|1%X{}g`Vi>GO zS>_2z>GtBvJ4>N5Ahz)~xECo4^VPcrhb`XS27OwLJPn+paow>k#n;`3a_otCpRN}Q zJqhl7#FEZ%lYLpPQn?pW&QQ z?i2Om%e72$CgUP)Op9Rkv;34q^*?c5a>b~@n22jMXc(?UzL}+S5!X@VzpRpo8sm(4 z$q>)Dn!U2NFhZ3>V%pt$bOk!RVaNBvz|Cw@=hB39v)k#BTM$`~sxXQh>``;@EbWEyQ5IWNIm1p4oEasS7M$#mhvs>P6cxA% zw|tJ$)p%IHPOsSVW1b#J5@#?1A#Ru%eY9BaPv$fw(a*vR6R%ID1Wa_I2$hx<&NkEX zV^#Cf72@~W0l$vTUV1V!E^qR?oZ3FYwC$qnS!r((0}L!#C*pe0hvF&hDargi;n_%_ zZ$)yI-n$E1G-pe<_j;}>OpNN)o*#oDt1PW3hKWG|Pl6($zGk>Brb&$oU9??50|TG7 z<8{G3K)qxptgO|ViBVj+v_&cpkg|0j)8|Zn9F_qo9@2!(brZywPM~5Y9Y~S!`lt{^ zgr6hSL=q|IH^AuhM6eIqf&!~jN5|2xz?5oy6fy0u1M+$b2_Q!9lrHR}v79 zEiu=+BGqCr!W>^zRCsqzry?P&qe1r)ug3=7TC7*5!?UEhN-%RAN&W*S2-2o(+It{S zg5uXKS68mvM5Sacx8!cZ+|-~gxqi`ZIi-X`h&XY+h*hnrm5+?HIpz;!n{buIs}Vm#o*1m*t(*E)@K zkrgv-Zu}D-;Ha7~>o?6Q!VslZ1h2oX7rRQah)V(~4riNY4+4>LhI;xkl^Xg9ev`V% zXejf;UB90L&JvpW@kG*QNM!K~ym)zUpRn0dN~98xSZJBFZ{dP@uw7c|L@MLGCV)?7|>NEW$IbEm}H7BOiecaVX2;}NicqTHKDfdFH9 z&*j`Q zkCu|&S$*f%kaB-e2ysYAr8-F_viD20>c>N9G16?sR2gQ*b9?p?c;d%|btLbx2B5Go zlps(sY2e6oG?(<;ZvDDE*Vf0FYEHU*>^C~jj_uu6vUzWP-8ZuBC-@I^qgl2K3C}DG z%YuI?aQz6wZDpP$UVXMCldE)DIaJroR|^rvdZ5*{?~W7FPJqt& z}tH{}Qe43|4gQ?Q%+BuQlc~bc6FbS^sBH z7-$|bR+E@T@w%dmHijG%YV6PoTYUJ(PfJt17C135fdL9I!QU)6jvFF2n*2&{I8IH@ zR#z{QL9@wMuqe>nBEkpFcIuYz(=_Av$G-dc9S42rD@Iuoat{5R^(Y=@4l9WAC~3M0 zEr>D)`4PluAh~UOAXe0JadQL7y=u{M4zbQ6f82i-gb%+C2XwoMU46f05ig)Ke_6~& z=t@Z4Fe!XcRw!_L(6WK$l-fA7RM~0Y}GmrM2xk|aq^Y3J`Y3-EYi#KOk$?LrBarWs)`N365|6<|xk#rAmbakVwh!A0oF}Pr?WIu8@sE=uz|?Y9o3? zLqKLwT)zGUF^Z0YTZIo@HAZEHq!q~(ViL2`|Fm5l_fEdW+JqnJ1dZp>VI%c!+P}dB z2m405rw0iX6DxNvxM5=Y-2BQBm{=4=M~@)QC9lX(3TFSv_#RN zZAoD262y!Gh~I+^7utmOY!ls4U-V6^ZCWGNNOq_S(R~nT2C*Md8yG`mEOnrm*Tq)k z-Ny6A0I^@|uZEupa0yBd498)GBp7z~N9JgTLVLVJ+_EP4buq&%D$;^voTF&?W8O^) zTvzxdKN}j?6~20~9!f{R9e2$#sScUGW%@(frc&kY;0VMB8BBClki#)qqS`?uDs zToR8ly%PwNrH`J>`Bvwv1D+_9&1K-{gPrHPJqh%Qh7XEQIi%En4Odc&)0=i=0t_FJ z$VZTs5YL~&j>u2C$XA~8ReLXy$qZL@Gaik?vCd#SXfFtM`?0!NmoSXW%`(?MAto3X zffCz)An2*bE5`cCnZF#!0!e|rqE4>wNl}Qn;*6Y(d)?1V{9L(ywnZQixaEq(5^@V! z8+pqa<~<`nh5V_punf93qm&#stqT`Um>T(9E+3i02PDVnQfJuFyJ06iED)RkNd}$z z`CI{H$fF5(2Mt~l)zq)Dz>f=ft1(Mwm*~a|{0GFn_$%l{FiXCokRQ7hI(E-U$q9_j zAsm8kL*9kwR3C>DzY?W&^s8pe&Z-@j#w_Nv8U?cq` ztQH99{A;YZ3*RAo2`spMWfjt6gEEM+QqMLYeS$h+JGE_KUO}GZSy_Ww5tE1eerjgj zz{G+fS;6PXi*_o7D}_MkmPtR{TTF;A1}~WJv#1iKNf(sEiN{3``u$#C_jxX zhR)}F<7{(@xo5N+)=?YhUi`=z=e2zAvCgSY9P0~tvim}~MR?qmtMQ^gY)tepjE&G> zr-?wzA=<^vyh9z4_tDvUr9gKlYL)|Qe~c=jMXZ9NAyz*_Qa6lo^5ywaa;$)z$)PCN z!q3~&D1ze7I7ostk%SR}(C{KrGdIYiYYHs_@*Bdq2G@9=Ms@*V5Z3Gxj<_uTFYn3| z(}RGv9szbDVSKN~`g?wlADGphoijT)$A%_#kBII*EU}`Q77ZyUoy2VD`u+P;h8{l% z$LtX-^IV6}6yhmo&Y0F*Z!R?bDsI@9Vh@_(tIWO5cDri4HZ8K(hEJY7PKi(NhaISo zVac}3Gn^;D3uhT{bq+O{bT1LV8vY~2j(f7dOJS_FBV8hu7QW3#yH zfUw1qZTHm6v>Tq{T*?@8_`#Xs?YyxMp})Ax4{BSYGj(7XLE)hxj&?<>Ul{BG-K=3u zS>6)h`SUy;B+Wrk#@#5{i1)n@!f0qFKd9))UA}P031lU&q8G8C143bM5SFMVjsm-A zT-_Re=uezBvHLC9D5&5|l(|*AfLYMc{d41Clnm069D5?gXeRb>(9)|zM5R4uDE%8GI4xLiek6q+2R+P~ zXAE~TI2T{JIS*PW!G47b>55^x72Rd@$3bp0MX#rqv2_dy?b!qn!Y=V6*piRu0Hf`6 zOf1daP{$n|O|Bz)T_3#QmJSW~!IvJkjxeFRrzA@8cn>>`RD!M?kicO>u;2M!)sil0KPCvjc)0ChdD4eQkiG z07)mS`GgCWJ6=7qj@lB6O1RJml<=t1l>rP<96Qif@Q6l8eG`YipFfXb-EwX?8fvBn zGDumxWBOR}gE1v~amr&|>>KkPrQSz}3UkG@WP{I1e7l%YTSY$b9jYa|hU^XTH}u*z z{=Jt0g91t>1i^v~N-ct-4W~f+Ms1Acn8v&YckrHCs6);QB^Eh0)S)O)6w?gG3I03z z7Kw?MN&BorE=&Vi9x*>69uc!fip^mW*?8MtD?zr-Y_SJ9Ljhh@<+(lrCn9OX7siS( zJ{t=mB-9DGK+RafI?*7#T+*@;WNjU9&y{>ZvUPlIO_+{ca#&-tz#`)nA>KQ$B5RM> z|65o%2*!JlVZ#7!JP9IZQ{;d$T&#{}GFUlCriU;{qjAIc{_)}AeA5O#s5`h3(JsU} zV;~T016vaFr;9%&6(K3}rbgsj@d$fr#Mpl`k_(?e#+XT(E5c}C#7pWJ3c~NYFTDsA ztm9h(7mg7WFxj91gmfYQ{bwOB6c9_uU%^@gST>U6Wv7q+St8QlI_g}miA8OwJHBEN zs+n!VCHZ#h?Usa4pLr0)(@OSfd{ct0RQuIfHkQ%jkqHu`9X(sT9SJ5Bry?`BQqz_p zK~rK_)f&7qaYCfR(m(gX8V8ex;t*7#bYs@vfA~MU@PAuCyaJ4c_j|1xRyeGRW#dkT z+W0Ttw2TuYjH8wI%53G7s!%pU%&w$nEs<mWvySTVu}OQ$^y zm!_d8hWm>(tg8Vz%Vx!la>NLo-Zt9$J~$A z_laZx!|a)f*(8DLTL@ceKQOX&@_h!5?#6C}wJehUVYeAe&^R*`!N>%sOGL?T=- z4}tXHy~V8rp49E>x{mbwn|zuk>2hv~L^ulSKYtJ|#6tpNTfWQn{NEF4}RJTHqbE+@ZObUzn8*IL4bhEe>%QxsCPZ-0-?{J7iw;&!M?MWD7_-R?%W ziJ?m=4mIQpAz_L#as`Fcww-DE6Qzk50D$Q;QmJoE?8hZdz-Zm z`IW$n#x%IB5{G8rQ4nsiV+oeE+>fuk!DUqg)&{Z_T|!?<7%~*9tV#rBIza!4Ap^lQ zVBmAsnYh~B&E|8PJu(!d5dW&0T2^?A#HK{J;T(PpkUR0U{`9m`<9#SP|Md62djMXa z&wlIhj8d~1?1`)G&2y8vj6V*t$hm>hZhB{gckpaV;?I*fpE}S1V~Q(@uN*oTVuJnm z*Jd!KdWWH{&L8_T?PaDnyv9Pgor|tU$lv*uWY1KR>0oi5|3~s~FFv=)=7N6G6WZyE6aA`OWist|QKV;79)Ju(3`Vy^cNml9?iQFb-^O66JrRe9+NU;_!o`;@Ks z&(g|jieKB7(R&F0{oEF{=U9aunU6$kd3i=xk3KQw-Irm4q^4^}Y(DbvL$?(0z0wop z1dY#K?+jFZ-pSL;+Yz)V{ta^|0aqeIC(7H~aUWE<`9Dd`9?mQKE@)ZWf41Tjkw8&X zy}Rq({le8OW%sxxP*z>j$h^hzSAHRg8FD7ukzpuiN>!axRU-g92Q8>M%k*@JCiccC&$rsD_Y4yZKt34 z$}Nwl0iq-(;qSFi!-Y*u4L|yMZY5EDDF&Vl>ER!*MEUOu0`kc7C|q|x8_10N+(!K5 zI97&tUzLyayDAh~-~%4z@Kxx_;yuRDQhKbC%egI`<8T-%7ZRAS#Q5`j60*Dj47XC4 z!*0iWC(G)Z&uaocMOWeEftorc3b^8`O1nM}#|$P9uOD)Bet)H_(jpzF9}lqARgeA~ zKVXo+=#LgrmW`Dzzh z;lz)!_RovG!$J6-T=&ba>&>Ly9P9R6jwpYiQ^_bSRtwPC>_jFfh&B4%bHtj=uh|PE z{wmkf>vj~~LHVfgKfzau0_A*bDO-vb@zso-l5J z^bp!Sm}@1E^+A&9cf=Vp4FOc-5Bi<%NwZ-Dv^9SGl4cA)uF7%m zWeval$wozZpzq4?Khr0bx1T1u&X&l&+^otjTb4*qcRd!jnbNciMjTT9j3e?+LpbI5 zK=fo9S!sshpUfZM!TiS}qs0Kg<&SSD(moHCG&nq}-zIrvBX3q2$+wwOD3#WO#l(fv zvr1+w7#4r2*?SzgNq{mb!Yux^1)&2nkVN2x8)bgWMAl2y-U(jW`_==Q>6b_MQ=eH9 zcQmz%gqvf2kqKN5;SKGZYB?RgAaqd8C5NQHcJLDdLP%Mo#!bo!7+ja`tu-P|%-GIQ z4PuKZ7dD=KC_O<+SZ~6kjHCBf@2dWfLsVn@k2!}6fuVdzTrrh8B#)EI56yWmc-9f! zB<)%#Q*bC!L{Xzj1;#ZEtL8fEckv`(Cqg|I$|*(a0PhtB z4R;mSt2fP>NgO~8d@L)a+TU7kk8ITtf@fhZ;#Mhn5CMb7l&7)u#>ks; z)U6$we=(f^Oh%crT9Z@_FR=Up@J7>P&9atrkx_g;t~E@)YTz>xo(K}phW6%>^PH)j z>Wyt18erHq}tqj%dM!1x1^e`l)#Uu{NdzRRET*}k%UaOXM9J7qRfB#j*krxh(qFf2&t0M8-YnanZb;)RD&B2Ih zF5kN&iyEu`e%rrdfxu8p>j_$?R(3wSS>9c#&}4Ecz4v`GNvNdTL8k(1bu6B|+pE>k z%hR4O!4hN?ZL?=2_d@>MaoiC-90CGZ$$!&#q>xHdrPqy?aP(Z|lQ)gv5giRKXY5o1 zI;fTEF2qEg2BA~s-of5j_9apByz3@wmsJ)H3-=#q7a{`!7KH1`IH#<_?de2vDMcBf zwI$lT+ZJc3lfQ1kg&KCvvvRfH0Y{_8NAG#Jo)OtU#}f!>K%I|s+Yf@T31Hc9jk;kE zQ8&v}T-UNn@?W)8m0ivnx=+yIOK6lj*VC=8`%ulf>NW4Hl@D#HJ*_f6Xi_VGASvia znc&IXr0%k&#efl%JUMRC z8pj-mrSH?tWn$Mgs5-@ZTS;wId8bl$p<@!h%#WEKnlp~v=ryN^HuHagb522c5J%#X zG98K@k7*uvhaI7nvUaQs$MYIXkPKzKc3q6om8*w#2=pLTs(-dxLaDww=CyHFJ*IA6@1B@;JK>7_#?Q_^3C4ceJYkCh~+tOTT zkEJ?gV{(F3X#RC^i9@#jMsR6pg>FBKm$CZ0{#@mVCayTp@vYpQ@Vy4 zLb|&6UJI4`=@OdGF-|vtz}swf5R^)()0B#|-xS8`2a>A#uoR z37AnICb_(bLyft{JobYM>Z&?${L5+5SF4M|atIgGaPPo`^hJR_<;tFK+& zxGD+nZrOjbl92Y>u7*vda#DB+RUo*rIsGD>Z_B6cdP6zF!tvL0(taMsADX#gcbi#T z2Mo+u`~%()a1bXmNgsC1IBU(k_GWz%Ze-{z68iY<)03NjtpaChPky#*Y7cq389F{? zhajG$FGA|hpKlp{uu2}sDmr*TCD?bLXgp5yp^i%!CRp8SlQX-Rn z{Hpe;zWdAmYe|El3%Ytt#p}xQw8+!#p?{WOSPwCM(C! zrsm__Ol#w?wo|Pho1ECE8EK24>5-uTK#dOs)?ZU4jIJM6)2%1 zr$gB;xkxRP_&2!XQYO|#I##ctz?)^c=KOIf416anz)rI2_ShcIBj^m{Bn z{uCst@ND8+z*A0@hP-pN97ZRh%A!lJH%!{L8%YJXc|W*&9_A2C>2P8(V?M<10 zNIzh}Qv5z~P4)e}S1o0VD%JMGMY<4uB9w3c9q!Yb!0` z=dTTgS1N-t_8XP;OL_&f$RBqFT7!!QdG=TnWX{`kJkyJ=@r};umj~PGwkxRrT%S(~`U_S(Q{(uh zIwg=oj-oMtJjbnw)0CM)@Bv#qgv=6EkV9S}ts7WUzs;8KapTreM*JuD$EmcG@o#`C z!Hn+xuaT)?b-bEdY$9(9uc=0HE0S$)|@k6l{V6 z_QH7%4S^b7o19d9(>uFmaVk%b(VJ_kU+33!z-&(QlKX?cQ1{{Ms66|>&%%qmR>NL@ zkZvecqm_U0k?QJwWkkKpE`w!__!q%Pj;Yc7|0t9Fn*#AkRj36a0{9C99?;iw=HqIk zrj)(^@roI@w)n{v$g7U$Eh=>B1c;rr1w~eES2kmFN*Tu_Q!YzRBkkX#qM7Zv`My{` z(jTe|P;9M{OHjAHc6f-MrEPIA69;Um$uzg#W1CV{+jS*4$poCh>kdK{W~~B7Hop34 zgnm?_Wo-CHK6+o~UXU>Xk&flKm-S#^5RtB3{>*9dRKZv-H32WdMz=a>I7Iqk0N+T3 zOYgXuMupbY)J#|u6AiA|OA-teo~XdxY80+c8i#&5erF!hN>91s_~_E^ZjrOaf-S1R zg*(;DJXf_VdiJZ6__xb@o@kR~Q4I2eM6$Mzx3e^_D-KJ+i?6^CF%8)O9zsPHK>W8? z`{Rs5Y^Dykt6*Bgxx8?Q3lhM8kS0p$c34VRq#1iEiXjpAWuf4Qg9Scr72kuGSgCH5L%A|} zA$EDefFi{QEMe`+8Y$`RZF=M2KNmv5S->Np*_PuP*?%*FD@BK#F;*~t$8n|Sjv3?K zp*Pi$#fV2 z@dI9urXnmR0YK~8Ct5cgXjjmiJYxn7beKR4+XJ)6FZ=Sg0$Fs`aBV{~v`A0o|_vJ%?L5Nc4-F^C_#(I`E z{&FyLsMhv9hkhqaE>d~sd*q~G3{h36&U#0IS%?#ZN4vWzX>9(%tsUh0HOlS&$*0EL zw?7(&H#A*k+3Ic@6pXd6>FX|a!cdeJc8qn93Ym`-y+q0tdOa^`xUOuw=B+yPl!Hz- zAy}@hc6d#Kh!hG{>z?C)eECspr;Rc~76M)ij?#H|x$vm@>n*bkW0U!*RI-x8XIRG_ z_k&QWYX17b)>_&foS&>HO=7a%4G-CDzVk=Xw&wWE&yVvqcoAIsUqBR6dERtaq0hjJ2i*ud=kmQ1%6zIZ`%% zGgpj95{Cqy+VOwVkBjTj#K}2)j%Ay(FKhF-aE)s9DUX!WXiEgz?T6l!*5Nn-5n3#NyXaxI`y|sKoOb$n%S!*~|BWQ2hyJVMPH20;5h$9!LCV zt6S>XsD>Ihit?^tDdVZQ*CqR}f%XiT9dmzA^;_U}0ULM+vr5>mOd?%sbX{8AmXg+S zhbu$iN{T4sshU!_0`^|Na{o00=ZF@Qd!627X0Vb0JrPItdhKo@+05a4tT*Q;I={cZ zg^DL6Brfe*%#(X~Y#LNsmjq{hm=nd&*EeH*Ph&K_q6EdGrZ+197@LVy3*GV+<-#Vy zyO<_W&z!2$@p*Q?!V*xYmQ}NYz&@G#QH*X_%^zU9E;~nf-~X|dDX{lBKie%7g=)WD zt-sy9c~LJSp^?nXHNEWn44V?`zZ+8CjE&(+K5|c1&cmDe>L~rqvyaP#$ueIU33y91 zNlO|TycyeDpPm@es}|amS&8imyDSSRc*Urn z`57dBAhEQR92(155Z84lRWibT*-c;cq&adL=q&Ey_rb#8kRZf#QDL7miJwcol{Su2 zn4+0+Z|hSowULJvQW_1q`r}9})<mY_7V`0gGEYLaY89Uai=RQ9O6^a9wj`^ge|y zJ~%z*-`bcJrw;A2vqXS5-Hzd#FUv8x>%!G@I`h2_H^P24Vk;gWu4P8+1(|`C_<{`$#{<$cpH4-D)B3e@k{5ImqyHOY#+GT1sH&;J)&Q_CYC*Jd-OD@nmevOIa}XZ6<4i`z{rRueA`C% zmO(W)CA@OU_%w5F<2i?R69!F1eeX8#!CpOombF~tZ+>FKHBrQn2kmX zu-R4MCME6F#YtQQnTS2DE8lilB?KidsFRqWf!KO{6ilG(xJy=J9DCgQdd^Ycq5Z64 z_ec=Rc80xc+WkWMS8_ZRF)-mS>CWwNg$|2i{5DbUz9Qx!`SxaEqIofkx~g>cteVkv zCH^zuVdX67!gAA{kIyT~s{PeI@Z2(@G}*{{$|$!hLnwIZ!x8%x@Kjy0(dcKlAkSUA zf3IZ$&n_;@z*9g)OPl4%j0Gzt7Fao3h2y2-npb9@F31w|_!HIAyPx@{w^(xLNSh(* zCiWH>eVL3v(}euSJ$(E#Y31Hn*EqywTpioA z$m%<+581=2u!1l?=ZKdXY%3|yJPSx!ki@aPutl7It#;MZ@DoEF#r58IE{{`jxyj`}xQzk2qbAc}L?R90oY z*2bqQb=Ohi>PO>VSG_E#1Q{ErY5~au1ul)8-Ry^Nj|%E8t!Wf~VTu{w_ms|*5K|b= z`z|T#u+dPDsJWQ@VcEO7-(LmmH4fpZI8$5f;6RQgriQreF8Dl+H76>0l5pp-m&=2# zoEhd|$83Rjzwze#^q-pK3f&K3QE>?x#BfP&Rd!-$dRz3Bm9sV|0(V38OPpW4{7iwl zy)&Z9^=!XXpY048Vm*<}w4-1Cz@D>zJLbJWK;yNV%f3NB`3F&K?@-TF7L>z`RluE- ze)nVMS0(ZeV;j&TpQqd}D;^-%AGPWuPeklcR$oUv1UDEnkAnW}%)jc8N}b4vFBx!m z&Ahh~@R`T^GA`ALQa9E9-Wqp*l;G$p?tjIO_m8J#7>J!3deqp|Df>MH$5_YA;twx& z1v+enLj{A@acb?&9!2>5v9#@TF-qd3O z19*;}(6CcGOYL=4L_HvO~Sbggq+G9Kqq^7h)SrW4TW1@A|C&E9u^bi8xvWVgE) zDE2u1dgXP;0c(!Qw?G{1$C!0Pr5Oc@}Y5+71!&ctBqgp>iP@#jR>0pkT!l zt|&V_Ot4J|8}e%`8J*~5v*HFp#9VX~+&Iu7%DM97(NFP!xo zA>?Cu1SH?`6V`X3RouT1%?xj@N%~bE=QhJI8NDxdYGrO;c{j9|&Amav^IWLir^=16 ze$*B?d-MOLSsX{D`yL;Dd0&iYs=RTS)nWLip_%B@x%*pl5vngtJNbKO5X!5w)zn>W zpSTAb$ABB1&I6#KQ3b`3ug!lpiv@c?N^bpHExz$0RAWrucyaL=m4h)cm2A5yr1?6r zJa3eP!}`K*+>9^KDrgXEI<7Wsl8DB1v5sS*pdWIFL&4#mSy-&l8&BtEBC({cnHKyi^V>~3sH7-sTqeD-X( zd&J7N-`RR7a@t8xOKW}p=^Mnel&INa>YEvf^CR*U?!l#~!itaHV_&aDc~*jT+J(DR zfTj5G&-JphAIb3rbE;?QGiQH%eh5@;XPM{-sfU!?IFL(caBAfy*RCX(s2wzi@r zKf0rNBfRkEVWjg9X^j)WpVjCadr|5Sixzcz?PH6#FJ0dyx9j-^vj!aUaOmA7^xV>M ze7hgObJXb09$Cn*A&5l@($_Vqr8 z#N8uLVd5lQ|JbqDg{w>qFT^a|LW4AWD#Y;lx0jHSF|1 zr?b26pQ-FP6(3s(l;L3V{rbm&om?(;{nh`)u~OjT!{&<4aU7P2C`u{o^2Y~ERN3-R!j@g{dRo<9s>ARlAm41!%f}nt0L?!=c<>vY+0V3<@EzCO;xy@s^Q#f>h&u-D_9UIDeRWe3HJ;3M*;0isk zcP=vOX2f?TK}g~#*?4 zbnG_V@vBh3gG@b`^3RN?honMZV~F`(y$nPR61Dn|)0gIl1m~HP5}dDC;nZJw+phMy zkuLaQQ0_J@JzuBucg#jTfy#P6xQl(94+*MFD-j@mFzg99Vw&&ES2^eB3R9`lLHfC$ z=I5qZ_e5*!hF$Ob5o%geQgi(AeU?->HBHs778;sECc91T-CUa@$Os#qng zd5O$1Rp#K$RT-VI-X=%+`S}5RGsO2_6*`RLiM~t`4mcT3y&hS|wi?>`G$YvCWW4Y7 z0vl`So;+5#c-1$?&olFzcd`FIf~6FfPEX0;ZtYn&nL?>j95WUXcMq`f6h%Mi+cHcX zK5cc}6Qx_{QzA(vsVtJ%vaXSK+9k@^aElcQ;uVI38;5KFKI2vnLohBj3QC>?bC@3g zF@3=FNldgWn zEcNJCQDwwu5Hi8JM4k|{2phP4RIkYANPqNLc2p-n6i1ZI|5(Q&=#NQZx}D!W%(^U32c@QQoU zja4f$QOft1CbO%oZ0u#RuRW2p(o2GJtskWtW^snop8|PvNVRYLBSSFH$tMN{v}t{1 z@fpuLc&`6^yVoSR(EYw78tAyFFqj7unevL`Pu34N_!Zc)INH+>89ff?U%D~<&E+19 z?$V6Pn=-U~jgvLk6tEo*#jd1XC(%9Moo6dTkvMZco|SF>^nqWPZ$Zyj$JN%ev_kMw z#_-w9+sAc>pZim0olOi4NFXM{TwmOL#m)djE$Q-W5sl>2t;^!ZYiHFmgOJRK(ty4{ zz9S`CJRPbSp(=Rc%4B=7gNX50ChVe3g*u}s#V4A9j-D#P1Mc5+n!oK9WX(rN;Xg+< zggyx3^~4jUko|QZo)kR*=LY<_Sp&}QTo_(%d-2v9A-P00A4xrH9wzo)qp|le>~krS zx^bw!_e(^}XN?q+D||UZmB2J<;_xwFwqmfW$IMva&e+cK-B_P51pFb-C^#uo!qxaq z9;z$0UAFxrD9?c8;Y4BWKiG9)f6n&^!~2Voa=MT!-~%xEf{YDP?5FfXGvAWio2JQz zQ|O)lX2d%q;A>;{4wH8ROOWC~HqSbefvpynCLB!#O9Ib>=hz9ji-tNwZSul4A>&iq zl7C|;hhP&0&69r$%lfQYV8X9`Oco^l2vvJBTAjNGghr`gS}neU5JMbK+iY<>vgh7w zq!BpG{30a9Oy{sRw|0wq73nNsr)MU)=@A^R)#^BPAwtCN^a9`6utE~b#W z)(6MG_~bninVM>`Yt3~BAJ|Un40-ZY;Yd(XQD3rxMOxZmy8);5vec#JH|ElRLuH?3 zM96T4zUVlxGfT|tMiJIPAUnN78I-jx?iQbu}@eTCu zpyf1re6sjcnb&n9htwKF^F?X8ox}%y7PbllvsFJPPv6WsVM$I$ncBOnyr}Ii{Ipre zW8eYhQ`G%;j%m!~0h|eOLEp7(R+p$-O5gj{96cujCr1gKx_{WB4n7S~V+4e$t)9>tDlEQ{xo6 z(_?uVjZ*rrsWKbtL~OBt92U_17eMZ;meYxWElHg%s_4i3-4PV+Xt&xwIaI=F#(rqB@Rjqt&&)_+KmR~Y$ zH0CJ=o0H>_2&LAc#J@*@V^7K4$^ly_{_Bw)r&mzWD6XmOXK@3vru3U0^Ba0&ix0z9 z>^g7%?N7jSb*tyHSf@nGb~P9E|C6rWlzzmOW4Sbx%%=8=Y zXwW7Se0+oZmJ@h4C|4$R$1pfz9Qa1AGy^y)!MlU?J-qbg4N1H;16yQn2T+b?Z9YYJH@=+7Pojob5rxY#fCNo=y&!mRc) z-bs-_rHX3vox~J;#=AN1+ZTlARq{87^xRg@aa$^bsE%#hFa$UR`;wMgk}EvgUo`yv z$E2X$VteTFuYw@ngxeD5;1Z@!9A{CzIS<27xR*>Si)*bLm3Obo4yeUUrG76`xM{56 zmn;6Wb0W-(MIXV@ajBT{mWVt`T;sVw9CAF^`ON~o@Z5cbf0%*2r_nS9kDQJe`P z#i=}Dep;Kph`f8%MepAWRBTQ%cUJWD)WlZ`ZA*wVJ-ai;xd_;-xN~ITCT_m+nM8HR zW-=kx(>C}eV$ZOP>)BXr2Hh{R#k^E|zTvZPP)FpZ5UdXhWVxJtW7Op*#UsaNKP1i? zdpzm~RM%{3kSHS0{y)A(Egl{3d5O;b3PP{F#c3Cc5}^NiJ&8tnSr!dqN-Oe?j}3x; zM#uJL3hB*uW(N`bpU#tJ^eu{o4%9N8f#K8mt&a#Z~6b>YX`BTb0et)ewcWH4 zM-pGWcu~0`6Jq@SzgP8q83}{E{oNqU5FxOoc4$^>Nx?`>ZTL|rW9p%humvDq^=E_( zkDYOutK3INNq@)cn_sCOd=UMDdBdRSgFRzdr0mfs*;A?UepZRsWuxyNvsJnZ(NqoFr z*_$U>4cxcI0+-0%{<1MY~nEuZuN zx7{7_N1YA?wQHwl-l!!c@SA+YS$Juxb~4Xno!oGxw3O^1$Y%aH{>6e?V0~W~KF5CN zz@Ftz93x>ovk106w2x0E@$UHY2AT7O&L3->w_3o3r1}&7lm0OaM!L>cQ{Ch^svo}@ zA`Y5W1x0y3esZ^|uMcsJG&^|Y>d89QTlku2Y6Ce{y^T{9$SX`XkkuG}?o8_3&0Mm_2FQ;uI7zZ zSin01w?*eIhMHtzg=LOzmsVK_em%l6;Ylp(hf+nV zi!+6)z%?!jMj_xD7nSTOgz+K1=pF1^FX+{PD_}R(A=9@(&8|l((BA|(*uU7uG$k5B z2Nr(Tx3j%)tgj{XHEn?v+SmflTE2(eJ*gs`*z+%2HXD;If3Y2x=2aON!fAb{w^a43 zJ&}pJ%VS(7(KXI(_4KQ@8sSzFBxYCJ5I*jK?}KNqlkId0*7%)60k*?G&8#hYYAGII>x-#69wH)$#hq>i;|7@q@R)haT6L z;Vbw?J(+V>P@!@`oDc4O70HM5K(CuZk_ofsPH zic&N=q!V9|rc*yHuGrKcx|3Q^Ot3yR`exD^7kDaTGu~ zhpGxS`aUF}NpC+tB2eVM0|x$2YJ(tfvo@}`h5!E?oPdDgdHb*ahk*wCM**CKH<-To zzb@e4<_pI3|Nmb$@nQo9^MGbP(l0Ch^#3rqdHTZaR3@3#kW0!*3L~W>)i>9*&@(ds3`CJO zw$`^HI#OK=GfQiI8ykH+QchcIJN^HG-9||rleX2@<)%duc2Fn`ewCVQ>2%$mFc)|B z83HCjAyJaI0Vop~W#*@pk1qA2N9RS7{nio7HcnY~Dc==ascaf!U<{;)ax|73m^Ie2 zo44s-cS$ViS1Yo&WwoAateEQ4??0Y#H`FRmO+03lX150kGlR4YYD6zD*uVMKJ1#-L z2rtgNU$Hv1Y=8i)t8f{P6q}vXK@gI?{R*AjSf!@5=#k#Bo%q@61D*LdcdI1?LRf4w z`;4EQF5Oq@8{{j6lJS@u8hRE)*858DKLmyhm^K?NF9ruB`LePp#?_2lz*H8bH5+p6 zr;iSZ#3GPau3!LWu*H5e_+LW|Yl4~ZKh@h6b6=ANLVJ)s!bxZZ_wrTEL=XWEp&hO< zO07+S$?N58I`1^m8PuZjelPlVNtX6SfH5TcKL?0|wA?wqNQnZ&9F`zB4EAlvb#t>R z3@+4cf-o=s#ME?j5vtl#S)|Go&o$>s{{@8BEkh$wRl>nmg)vQQ^iKyD*E!0x-{wGJ zrl$KN>y8oo$9kx)Jd624Xsm*m7>zs#qeBIf zVRJ0@h)wMS!${}P6p3J;LYV`8Xv`yQgJZ!A)^sjQx9REB&4Occg!vrDB1i#g5z-PA zRoUjm@e71vuA~m*YWT<~z(9i_h1)t0_O|OU1cOF_*b4DO8_}c&TaR=uDjWI!^)&2l zZso==QVhOdbVc_#x)0gVY_?VVt5)r7Y(ZZ7jAMGDmBmN)H;0B`Hq;$~;Bjr#;<39n@E>3n2W1Y(9?1-A{qVa@6+tQuJ=F!X zpnF`Omzw)xB{o3_Cbk@C8knKOpGrx7297Nl3&PyNgPQib$)O~m1cU?C1eCd}V^;d- z3Bnx2(3=Fa;Gjf9+19ENwrFuR5V!k>mVD+YpS>&x5L!Q3Q)}Y7FW8UTof}MGWu#v) z8Bn+QjIc(>nT6@V0MzgH2MRSPMeA+m6kViROB=t6pmEV35JHb-5J74edM(x#0>)u` ziNAtW~kTszvbDn|n*rT$NOo`Q1)%%Huy&LejyNkxGK1OVk3isYjPWAG#@613c~-o~NO zIe3(2!3Mqmvap<%jF4(H2*otli)@FKIq;N_UFnmta|v~D&$(Ca4jg5JNCrc$+0^jR zEvTALkv+&23bUoOmpPa@1JRjz!5`fXXLR(SW+6`$6h|nF7Q%X!+MD_h!XVclVZ9~m zY=Ryf2Gu?1Yi8q;$LMP5HS)}Z$n0tcfZ73y3S*N4CS!exIu;Ey)ouS!4NHW1Da&k} zlJ#)W9e+;wO2e zd`KkZpFA0BP6RhYL}7TE1NEg=Y#~Sq%0ADIupX2|_Qj*)GAiVX%Hd#GUy1sAHUB#I z#%&A=cLft+EG7nG4Cr7x!XQoo%+xOhQ`nS#P=bmb%EBBv!vYduanj%mr`8c$Fl*X;kbCw5^YzB#vmSOs+j#{7xd%E9!4@F6?%~+w|Fi^C_`DQrsk_D|%Iljt0_jPY z9VuyDEC_O7>ja@$Xk!af{U$-fyA**SEk;ij@bAeW1iS}j?1GjM7{}~U zY&bY>pe(vl93Ux}K2wdbBmxN0)mB>y);MS&Y#qg)~P%%0=j%2PqgZku(?Ak4~-KmoT z5`rxv5L1_o8hK7&lw}HmQ2&bp)ej6t`OQQX()qJQ4OAg8W*QE%e2^h5L_ZY_uq^-O zv6dHBl3%%+iOyGEvJ>LUR4@rGj0Tkg6lt}FCM$>@T2L5&fl6gmTZ_1?BZ#jnCO(v% ziH0fjRY4ep_f{`Rb}2+!ar<@!*@H0bwK51KP)wI$qZJA;1zM*LqUbLuAZ(ZH-*=fw zproyQOr*eE7_)?izlo~b{}evlWT3@2?{GQQ&#NBUTyXOJBI1AQ-7l=z;iRf_ZZuUn zDo(;Gr3pl5+Kh>mMk{%9ayDivCPy^{UZ!nVsVJ#Yu*5ypuboE8W*?8VWwVgIt2+Lt2$iBa#yD`yWlGq=oy9`J-_No zf@F}QmEf;_6&Q!Xz!($Q8!$Lo4r|Qhrcw?_iv>NIu&zjIqk%SyRK7Ei_<<2t4FnP< z-b^DGQ=oeJIqf1P+So;$+C2AN%w zz=;Nm6hrfZgB+Q186o|2BNgRM=$2rL+e?0H!9i4S|M38!|AoXd0kdUydWJ4L{OTzmIYLUx8;pn~%4-wf7sVF)BET6)ms`CsYTHB5TC>ee~ih63fJzd@Q(5eWV4|CJTp_XZ?a z;*3flHI8+#3~&T#u#qIIZXo%0UCJtz-dYw=j-agiU@ya11SO%x9Ey8G^GDaJQF5h} z;c=Ph$rIKK5@19A<}Yo5bT=`70u>Ytoa150)w+luf1~`f(n7NtB1}P+x`N5oP{!)& z{ip_;2oRaYI;qBcU%pJA{{)0!g8w0LwMrl2j+ZV|p4(@NJM&C6Iyj;Mfte1A+r%Sm zQIk2!3@P5iiDdq0<)>3*=**l@*f65$VbX9M*2#xbRX~PJYxZ&8zN zhMdP-ZiBRtXnl{QMT|IGx68<5ze_7R9qa)HROwv2BZHLcml8TrB8A*S;! zYPz=06Vuj)r`Fr1t9qLPKgG@hhS36ro@`iJdm>d#`U3wq;X;EH{*DiWWWipBvLejA zy>`F^bW6i16~#PhtBtO~0zs<~q>aB%503lx0%+M=Y}9DkZI?lv1A`ue5Ugy<&0Ur; z@;VwC2`1<|>EA4jiPb^j*E`?Qy>Boc;B{7}o;ZDMh@Ot%%^CeOAX66C{_R!TRJd70 zl=xtW6gV3~;clyb=!w?A2NR4sf7YQ*T3 zVYLl}8jY=47#-I3nmUBqDcG*AuJp>msv2|ipEG6XRWe%dH)P$FK#rSRf_$6li-!w6 zCLBNL9HScDspt7#ubN1RAphrs(hs?_#~>|+1{Vk6KOaZPZ_aka|L5!7ObBRAiOgE2 z^8e#>B4VY@|KmJ3zWpyEhm)tXK6d`+=z|1(rQhUh8I)n4V$+~uOr z;YOP0K@{HkX1Y(kx}JN-07oB_&B}K=mg^>$V;+Y!naf|#Ue#W2BoBpw6wf=|yl?lp zFE3_o8(3TJMioEIgVb_&@=n%0%_8gj&k4ewNtfn_o81!E!v&k41yBZh18s4d&ll741lm^1;5obY>Ye-m%--^LyUGEuW4g~n9Vw4p#$$M&yS}%i2y)8pPGRYs z$~)%u-R$<~{QcG~_ZL_l4twQY^}t1@M(Q@}x3WUwQq!6)X8Ptmc8hH~iOroN1TQqE z4{Js-YUUh58n&UybELM7%gy%}vl^pauYA1pxozs!<7|FCB@7iDQ*gwoJ?UkfACZ2w z?|pwh8TEp*9TQX=^vZk6`B1NJut0gX$MzsUJN#r>`2K9nrk6gCiC$OpT;kC-_M_VI zeXHClYWp(^w~TzFY~{KSBsB}W@)?Z#;^!aqQ^#oHIZ)Z^Ls~Qe0fJ3^qhR>sosglq^_$oMFu+2}8%ova2aZFI5WxTG7{o~DKi1cl1{o}*o z;~cSh$`6dl0v0E+jTA@4$x3r2VeiW(~g6A>WswcpV)U^Q4eNew!ays_Yk+u4V z>27h&g#dwu0fK86xoggbSI0|rr@T~`q{DHJzRXl8I~4IpHmb>&R={lzY-*QLge6@A zUaEDAgm;<&H*H;Xcn6N-C~kSU(R;$7KOSq5b91)63M>grVddM!$A?MQ%wI#A=BevK z*K5%l+9roPGdzgkB?)w3lts^hU(kA}KfI*OuVbrK9Flr)Luv5{b zwr!JK_Jk+bo3aTl%Rjlys4r<>1N>sFAl7=L4=lQw>gldrO?5&kjKKlU zuQt;>Z6ptOa(8eHtT972*f}lC)7;H|w)UOIs_RWFja9wZ;gcfNu|tS3Il(a)NJ3r^ zuv=q7WS0Bl9HNF@Z9(uUZVx!3a~?31kQd)fKdKy_+i`2X#m$qVZ7#>F|j zp(=Gi^MN6LgNg86tiK-hh{Ploc&hc#OEyYT=-Q;q4EYaRwsgdWrGSQ&@Tc+oEK!b* zU|lqEKhoW>cB%NPZmZufNnCo3ji0+H(k^helYF1l^DNGRP2K+P(9(4&GW)5{JdhnB zv_s)@XaDLcc`bE>Kt=vU>L7*cQhyl*<3LXN{XxT_0QQOLSC^RvS?)U^_U3u>>2URN zg$A8?A2wZ*jn%&UPVH5b&@^7P#YXK7KA^9z#-jD2%Ussp@HAn|B5O7zst6G&#FZ(u z)X0{af0wFSVz^I#2rU+^D$9SgjzD*O*YHBvGDCPi`VK`kzIt2S631RtK0^bnSb)8H~n%EV@s31tex-NLZd$y zTN{76AE(7~y<>R8kXuAWt57rx)yGq;r1LrRantLMQT2&yrwpx0;c2`4IU?l0?CyDX z+{=mJ8${uePd!sG`Y1W`E8GGUId^Xq!aRY?HBnFy>f%0l9YT_*=$#b|qfUA~k89mF zj!SMCbNWsP2)_2xISL17&!3MZ6@`1>-#AO~OSwphDRv27&W+)Qfn6oEcwb3zI&)v@ z!2UJ~? zBv^@nIy+0WnRFFZ4kmd<91&_+WQO}ZoTe9eWubFzx@L!g(fycuD2mqyny@!j6wbu= z6D&*W91f+zocBH&lagnT<9-Y$9~rL-Z@J&M<$YXZ)VdtA-^vVfk%*Ptn?Iu6dF!&* zaM(;f5|cF70Az2~t}h5kd?oceKZh*`ZS3P!N`BZ);TF4Yf18-yXrr@^QRmdnryf;a zYwK{>DagC=Qe9BH(5b6wvEbeh2NEj#r5>m5oR2D@uG9NLW>}+vp4&g3)@XajexA(gr zaU+j$Mc(00k0uA;$*Ua5mrhKLM{VV<*SaSS*t_&I=sVjq@j4aMBuwuxN~pCq7Yhcn z!+BP|QS-gdEHSS7qt@3UPM&C%rSOzLi>r*+XB_)acB_%WPJD2nPJAygztQEl;1Kk$6i=}d!s)qo8CH# zm%8P<+d5sFOQY{D@zCKd{@BV#}>x zr?E8hB>b@?Uc>0S4)1%t@xBpEoW(p}-In~v)IDQOq8}Wcmx;{#jl#25mCj^#tq9~J zBI44pXJOdM&Dw7;XD!Pu>(0%2F`2R)R?Fc2uc6UXIC17ef@1MJNC7 zB@h|w8B1!jicaT=s!YY1Tt*Z?{*M48y4hCUlP4pA^^J6`NS^|L$fqUlt(T2?9D9-_ zKeeOD3ws*ZG%1kVAn)eYdpah(y5B_)t#L_YdMkP*3Ukzcy7?3Jhrb?f1eMV0Xad-+ralAVucb^a~Jr{edd zXaNU6c6NynIQh2wr!cX7-@K4uRD4(n7+4ZiVI=u5Nk5IhTZK6{5JP=-GI%zol#cg+p>UjeoR=y6 zCE%GDKba8_hJ=hSgHzr~O5ttQ<-S(F z&Ud7-Y-wzLv`Sp$J6Xm>e10uey%Y>N8U^=O;nDot!w_AZ#Fchhy|s@1oM^?JA;pa} z#cE>z^WovQlpkg}>t!5wCv+SY2qAu6<{{{@C^3ptiv0U)uNCY|vas7lkGhU17Iip1 zcJeyL-fw;;@tLsJ_t?%|wgh762C5xCMJg3aB~HebwX_yd3pA$xwOxg%N{(swda6gP zFCCtdo76lWDTMp3 z?)P}A(+!RZjz5k!b#aifKIX&}jRG%IZ}DW5eZZw*@{57q?^qK752Jf9db23iSB&&| zatq~G7`nw)$9OaCu2yByLWtHF=DgCN8J)%E&rLNzilU_>Itm*WS_%4PFe7pEVti5d zxgirO|%Q&R{H-SPJq@jsRsTbbt+%vk+nN3j9h}qJjf0W!UIACVb%RcBJ_5y2q z(jS2X(~qF~)Sd|3(r%#^;JQ*nIAC;o;c=$j<}}``w>2x-${AgBYbSivisZP+_1OVg ztNB?oZ!`Ih0m^>=8y{iJm7sfiZ_>TE2$cXKyx!b34qp?ZKl#EK?3DYlo$tdgeod`6 z$ZZWsL(%4lxyyQzY%1`$aN z!BaAy@R=$6N%`J&NQpgrPPP@`cq%OrI+G;Q|CW*ZBdWAn*0usm68h$t;im00sASC# zeM~BJpO~DL&+y2AlRX_gbr%rV?gas0_<_wr@^dztfmaH55GF)ylKFhiE zAq5(+094UUYLj|q;`q(D;q4j6>&d=c(u}!@UhBR5sNdF-dwDrOlv&83k@`&_WD@#- zjlT#OUR~tJ^l^=FcBQoTiC~2JI!h5~n&7^owOlM%!nGQ?cs&j!ZWi@7;=3klCX~A5 zrSK2zF=NPSV&ul{eu04UE8lY9XjUY$%(&x<*NhKN>X46_33%~H)K z1&>~9)i$qDmJBTJjo27fn!_CXqVcdh)zYBrin5~BAxsJso_1m#Oed@I4g&9tzC1jR% zEJd7$Y>lyf`EA>NGrKWa@~24nlzmH%OxOAqQ+hzjD!xa)c^KW7z+Mt}1`=q%^p-*9 zc%~=}mx8k{v(@sD-N<A5%kHjfy&%p%q1y>Gm0!Ib<`Rhio&=2+eEL|`Dw0J4_9 zI`o@=Ag+xkMZIyJpK%){_2h);yPLu_Au?n@0(oz3ST4$Jj5D^3`I3bd_q!knk_5DF7Ghv zLXh@;Mo^3di8BDRa+K*n?1AC4R9(R46*$XwEVh}7)LF|X{BXqVI6y!{$}n(UJi+5! zn`uEy9|M@gm1-fZp~P}x^u~+lU-#q~i%n|E8Sn8`(MJwVx<~KNhdP7Hw3?jU^K*oC;+4LXV+eMJ>NA^}TgI@xxXnQ(Tn2wO@`$Pk5E+EJ2iM zxF!rC`oY@4mL;8=z-dSmLE!5aO?%cpgTNUC0p<-FNtwfTPgv}WrYm1+eDByC$jRI& zX4B+(Ik=D78{TVyE625rpJ$;O(OA09NUA3i(k8=$!VJzHU;BgX34I=L>=xm3kMY!n zb0YJi^Db7a+oUUE+E3KY1Nvi)i%5$J z>wW@LPa>NN3sFtJKLLFjmDiXI96N1Wq&igO`tt!&8ow797kz`Ix^u`!>;_|38T>xY z2-Adg?zC7$Lt$NYGz2|C*BWo`KYA!78$svBo)wj>+zWlaSu;EQprRPtXuMt08S8r> zhD1|TYWkDUugd~sWTKMOG6@1L;JNYgyD|KNS+-O<>KK`LqMhF){g!>eoQU92B&>P*do*lD{-7Nz0p`}AwZ{4S(dERRzVD9u!N8`KnIebz>jpf^- z+#zmH?5DUbcsG`^s>L^zIDT@g`BJx${kAhr{tKK83R{6QudH8b@o$_vW`Mhh)J$f~apB{Wc=rGeZR=`5&(fxD2l{1+Q(c zigY|xwwO2X%lwSAf?$^lvk4bWYc{Gi;>H7VQj z$3$_;a2=jwme!NAvlWAU_xGd8gT;_2A(r=1?qf};CIKSNi((qWK`rZc3^oqbF>(o0 zOK%S2ZWDIx1zO9|tbgd?&)q1;_vx6Sh|@i8VpB!i zS98+x$0qP*S~L_%IM0P4h-bRWoyQZPT=!mD=U&@NEJ6!Ei=(_snD?DXsmSzc zjiScv!|I>Hw>fBEv<3;tSw=MY-27Ulb%2LB%u>DYaljOXT5 zD1UPvRhaguf|zZTO!NXstoya|ggb}Uppgs1qgN;R<+6J-E>+s}4Z~jMQcqJN=RlDl zMrrO9V@~Sxb`E}b$?TwuNlal2Zzqc<+hyhS>dcrgd3W{MibzfQ+~_+wM>cAPyKOTI zh76}Q%sEpiq^}@j6U=%gUfkE5^#Q#hRd&laZ$@RMLvV%sC1G8LtHhntA3d-330A}f zsIp;)HOdqXNs3LZh{X~abBka9@UF3I9y)O?uHTikI|XgORlEGr=|vLD)?c!bn{qi~ z!$A{2&DTF8_T=q2!t?g$&s5sMxls;3*HYP-1NIE3uJ2y%J}c8ZYQ8-<_54vrFp6R` zjUP3ydku6bPsvmvkZ3vfB6pbXJ;rsHzMy>e4cPEj3$kbjH0%cVX}-E7H@~p#x3+zP z>@(Yn9=CiLcWcx#FL%rAuWPBgH#BmK5=)q~^FW$;pKSKx z<`+fyr@-tkS)Z_a^WbQ-#=V(P%1$oMy)W^$`)XUII{cRNR(Wkk+GjzAvx5P(zXS_D zW8P~SQ`9F4?yXIY;4Gi9fC-FKb(t}fJraC*Nwv$vfA>c#32nPJz;3ttOvvm6IcIS+%deJx1t2fjO_acNy-EnCdsF9XAfIi zO3S<=6{2~nC)MB1g?t6VQ=AoNBz8&bSIT6J#iSlIBeE-#o1cqPNB-zC2W~$I43KZG zSWdNjCWPl98{sUlh0I=DGlQZ~g*Jx?V^g4?uEQLQBY1ohEx(dF@>@#-@-RA=aANIl z!r;@a08P4|Rp?8elgRjp$`&vvbw>QbgnO-o2o`N+5I}$Kni|TQGSS6Seh$_xm}Dv@ z3o{O^G;7DQ=@)DqwJC=XCMy{xEO+2Kn=!yKI90RVj!kp+Rd5wn+<`;M-nH*Wq=Mi! z-V};jN{#kULpdiNjmQg%yyxSVM=-|E$uL;5HqR=(^=iqdGprGu+3RxHF>0fQ7rOvP zi~Zcrh%Hr`88~yeH+&5e_)BCgjxHUP{R{dM^dzQq!2m@&jxjw zmEqj|vo}R~N~lMSm81Hxq-<#X#qQpVXW4a`q77PmI@)-n?h&~A4w#Es)<*%YeJsAR z?PsZe*g`y;{WmPcxH!H$xCXx$psnn+QVqt*TLI;4GHSsh@~K9u?(gn(gN_ZbT_5uF zC=WQ>&xCjWcnys$(03Fwo+kwimA|%9GY%A$jAP0DGlp7krQ<$pUE_X;i4xeSXjhbz z!r&fM+a|gMa|xk#v!bTHQ#V>_{X)B|JBGO*4Z*c;ui*5N<@RTvDUNKqHvE!u*bw6! zexayds=GalAIwjdAOcF`{If%vHBjY^h>oD`7)&q~K3Y+`J7Qa2n{+OxKa4VP6;|fw zSgVeHF9$}e{8bF|Qg%%+O!a(w#i5(_z}7{y4m!^lW(>g-5=6`nFNK=3bQfLAk#q+sqo?5Kt0vq*46{NWqdVg4?(iD?eX|&@5O!*Ye$(e z*H|5K`TDa#cCG+_AwE>jQI|+CzlZ}S4&s~Cc(JkY&;BFAE`l2xlnl<8TJIPrr1rO3 z*%1fELC6huy=bvs?98RuY&SJP|IL#bY-yXw(K&7i_Af(8O?}gj@O?F$e=s6vbYzWR zWNcvCDl~?DKXqWxz|_2roxC5MRK|m$OZ@%~kTrQ6IdLrA?2PdruT~>2D>2an5RAj6 zTFcOgw~hQ0(Nm9jqe~(UckpBBRGqH^6HD5YK1nUhPF79o?aGg}cCTcJ`k_C~Gk;Bq z_!6gVk*bAjLd^jTCI^Mzt<-#v^y`eR_VegQAwHE1pv7XwLGtVHo+JnyxLKlLrZew# z3?)WN#AI?1Al+pv7%FCgJ{Xt5H{SXn8BC2OIt^N-nsejJ7RxyJ`J2oA?tA0q{k29J zvD30_dC$R>eJl+*2Oc_b?5mjHVfd4+4QrnYX-GWi`MISPb8)O5vA_mtcDnehAlt!& zGRO;ll-X8?X0D*?m5o~owAM=jt0^{N4RO*T*N=w}O6oV@>ysc)4JlN~Gfx8<{eVOV zxW70{+)^ku(0SjWL!cA)aWJ0bo-p#s>x->I_HQzev5DUOQ1KrINI`cOV|ojj*JM0Bkiw5Lq)an-xy(+g6ZgSlZrh(S;UwNn14I6)ZMFx*yFUzcnBSTZ6$NGv zKu5NAw75Cn-Q<^%XU)no`AO*m2^9UBXqH63|32@K=FF)T;n|h=D6w^(k)mQd^caSF z|HiS+(9ZB^>6dq_?}+^BAjsd54-A$I)#kwD<3)XNiX?_*8Y+NG8lPcII8Ar^a1@!; z3dH03GU1}O91rV-37p&-RZ^OjDL=lRQpuym@_Thr6kBExyU&va|-?`e|ossz8u;ff5%S6MlSiqn}ZDRM@ zD!vC#OebmC)7T&{4sZmwkUGu8X|4gEO!Le~vGuRQJJe9{A+RpImjWN&teXUSzdYA& z7WLZE@b*M)fR1OR!Rn(p%xF4bHZch z!>-qigjA`fxtK?kr315k@*qiI7f7n;>PPf5?IDB1VP2z|cyCrK`8 zWdX@9rju3-QC`VG?780{svj(JxQX?(5^AZGv(XgN5ahEXpwT~m*@cA%wBx=m$~N>X zuuF;&lr9qhb_5#jh!1QuKh?2RjjMQ<`OEQMk|mRrBXU>d_s$9nBfE9d10UeO>QOR8c3df=@qU3JThXolRS`X$Z|)~n7tEMPx)}2G5widG))Z${p>-z;0i81vbM>R# znC&J7FI&tpTN+@^CR;5)EMY5A**~+tQY-eB@hly;tx>Mis3imSRDe+^uEG;-6v=xj zh3_?cJ2CxqWyPCXMQ|$THEzC;i8S@}i=hnTB#a;?%VK0l-Bf-+A+V*fKeFMPdG=0x zrb?T0X6DC*h|M`&Yd)9dk5Psju>wP<1oTNhA&7UPUN6kd8}oyr6HgR*!kT9&1yv(W zz;=!>)TrzNH2odzK9k1a1I91TbVFQW;bF0B0}@nK-2#DjXX?hjE%&z>0mqR(qcRUY zL%?FA5Mr4PzrY<}ANwKuu0V_hmwnVMd(!C$hwYkJMaW-PgEQ@eO)k-BYlA#Flq!{; zKN@8j*d`>Q7-2h*~?$tBGNoMjuLfjG$VZT1m-878WWqv zy_|?Ji0^(VY2b5s;dyZfxV`kyACkx*MjduJy2bXs&x+w z9XGrd^5&K=#^8rsu(Uu>*rLCt2yV)yO6w}7hEw`I!JAktvZog8n4w?EddQ~8+0Su| zi=$mBqQz+IQ!hO((h(Qzn@*CSg%;&E6qPV@7pe5oW7b5u$}q}9YOV>GnqtuIznbU= za%2^=jH{!L`KP-0@t{Vyd_7^lP>W6uv#Y zPPBuI?B<<(r`LuhJk~~^_rMe@%8}-k8 zwRIBIqggxv8|eQG{L{NH-O+8l3RR&z?VNr7b~6{NlTH-1>x(YKrn{$v zI!jz_RRao^fhAl`#hNy9&LMaPtH|IYhH5I+5OdWY!haYN6}4;`pY({GnYev$=EYK{1y#58%;G{+ z96HKO?m=5{N3ZHSmA&Ee#8b zyT4caKTZKW(Ml1KmJV7y91?y!Q{N+U#IBqy)~=ajc6k73VPrm4w<*swfsf6^0Vb?w z_Au4?FPCDVE?e}ztph6jx{CO7wJ|&*fJ!qjNvOM4z2LD6RVsY(7BLrRiy$m1j97@V zqvr0PmV|Nn0=%+0W`J*IJ&-}pAq=pG8f;tctH~>V3REK?Ks;tMB4Z{0YfO?&~vFvVn@{KJQxU$Ytg z{-z@j0Plfm(;gJx`^&|}RHsqd>F!VmDnuXL;e&x2{D01Z94(u}T6ffaukkLN%jmXC zqeV~P`x+pc5+LMs{nI&((e94gv5jp_`pA4sTORlv-Bki5{*%G#8-;cUR4+8Qv0W3? zM+4l8Fukl;Biy%QB4W0-;|_-pBr>X)z6CsAJ>B;o-YwDt zj3h4fCyihrX>64VN{j`r3>C0Eo(8A<^Daiw?f|CcHuhQ)8sbbo6JAHY{2O3FC_EV{ z{^!ZO5dkupva#u>&TbD%C1z2>J9p@&yK^5%O_&<^(}3pNffP2 zKolk7AK=0@d}J^G5ddIMeE~--WcEm`67`akGS{zYV;jz9Y7EoeEj@)e;F@y-67KPItSRg;RG%xK zZJKVk+9ob zjNZK1`ctU>nfim(Yh`i3HkPRBqMLL_$6c=)G|duo3y>%73YhPE06NI_)8o;_?&={p z;Sq~MfM+)G!vTC=G6qmN#|#{kGfOR9XwD{H*1Rrc2UyTtC`C(SXqcqnqECpdBt5_h z7cJ#S!jl5=U2nxPAX@TcGJ*8FTU&O^ui7lHq>+XM)gKaOVqvRNL>$IwbB;rTfcyO` zo_8a|W4Hu6!N6~1JbC5u2!U@Rg>cg{(8Mk~$tm~iEk7sc8!xWtWti?uaCr0A@sz7X zEikuO~TFH!NWZ3ik=Ej%FaI=j^1AgK$T4JlSJnkG9(DO}14Ry!ad1~{|_54VS)UK7_2}1=CYWb}^3yIbao5t3XWO7XOq1av;6t`)jDkk7&3BO#DTt zpOv^SxSQ7ioZSW`;d_`NXU&umMh(DesvT40FE)eloc4p}4TH!x0(O%mRo)m>txG=H zwtt5UOqWVWZ0hfyuLl%0q3aUs$x@Ow@L@(`A%@|5_&i*jV*^g&V@1^8L8w2cz_*=a zb5OTgqDca0XurAu=-DcjJi$+ksvC)@1Z`;$=e6m}4THrX=l;N+VL{blbvgYYJ^17Hd zXAuaei%-PIuykoZ{WMY?P9ey*F~(WW-qm6=g&BN>UVJU7I)J~gG8RLi#5^Y^TJxv3 zjz9q4%mtp26*a3rD7MK}<-RES9VbCKcVC`ExF-8|SQ)0MFgemeGv%TdQ<4*4`R{K4 zB(j?G3GD{Dbgv4$U45VXKjhBi?-z!m-mz@Md9*cvIqSF=y~dcUr;Kcht0sDz+C66L zHj?@T0lF(MHxm$KnrFRE2kt>D)C*jbI%9$WMPOl3UH|iXxbTBp&A7_DIyhyKxw4<* zp}vPUxpuidsH^SMxWe15YB^r+;Mf&wX2hh0Ij!~B#BpF+org%`j@>V}hNqAhD6=`Z5pPBai zDg+?bSSxoABy^OiSa+V$E9g?=DBcSydiP`OxOU7Wv3y5)6lM9%FxzXfcwIl@S4(CDwJqhO~+o+mi6KgNQRe)hYW>9 zD}zF}R;&W2&s7tIua=M=`Ls4A2{eCOx;41I8he@-CQ*0?F#q^C$3Q~@j^+8EZUn4K zwKG#J(ZzkugTFwz&42P}(}BW@WSl$o_gkWsx2&tr<*J67jNoY1*9yXDUT|B*^08iPiTqd@??}x{e=tE!sbISst4XUe z*G&&fIlk#}I z26x*aRZggtcM`PYujbH89_>z7=J{Y{z7&lEd5vLeEdeyaHp{Px-~kj(O!;*CPGZOc z>4BQu0%@8IjR>;bn~d!Nny*ncJ1=A`{eO~*I|g-Ldjo;UVVpiTF3DAn}_;;?Gd7{P+`r0&Bsz{}ql6n^yTs$T^->s`#SVX@SOa^I)m_D?bI zZ|C2h1lJJ2jq&wo8p@^7S9>9sodihaf1p%@VYGHB`Rmba9?Zv zet$oupS=$z8c&rxILR~1TGiBf$3kB?M5tPxxThBdbC*Gqq1;FcZ!uj(_ z{X=e8g@cLO$Gd|FWM<@i9S(xwXsQft_c zMIOKRCYZa{O8onrt^-_?YvKH@sm`L050@?X;ZacLbtYk{uYjP(wB;D3L3sg43I~AR zV)JDej6tdZEn0yi>6b>kqL!1HEhy&O=wF=thk6(jAolE!4$@*J%cOdp^c*A@huf3d z)Mkno_QFj4y8$ZtfZ*fZ>BGwzEvT@uMo`~-v00mj;>GPmmkCT#%@MUA6l?oYoLA;I z74#DT5pm-d6}1KkQ>BYVJAipyi?E^h3ve9;d@mwG&?w9{=@%=+>uf~&vzvwA_w2O2 zX-zFDNlyvs%p&!zNvb6c;r~YDJj2ZMLYr|ee`6O?$E6^RTbM)f1bj^)ba{$(8 zDxrt3TDEDRiNYN_A7a1v#}x|R}Xpe!^J-E+T{RXrn@c&5PBH80je5TtDo+G za^~unJw_=Ydg`-A-@Umx)urfcLT{UJo+~Cxc?%;L^Mv7t@V_7l>w&8{*{1tH^eFNeSsgFJkEP> z`4Aa=`1fI|gV^Q=Mq&u@#5xuM)5W};_c!N`aYAtt?50V^5&a1SNt%WN(#qAVVcf$|N5k0|HP+7uC zHboyvme)9-uKR%6Aq-KEOrlxbB&Dra#~b@hFs4HRMYA!r2tY;tLLI0fJ5H z@njV?2;U=jd!OW86EhrD3`k}C0k#~j!id=0bOE)`L#gK_`@Hd(6H8euVtA9l%m9Ef zYhDbwc+;GjHFzb1T?bgsc7Snz=F1#rRIr`Y*s+F(6$>yTEk0Pgos1FQxj(c(7OMKr&=`g4FJ_zV z|2p+pUFH49HF~SJx!c94nR+kac@atzDXCyP)@EbS)>hr<+oL@GS{Yu*6@Gp18}p75 z-=ptyP2q!c=VGm{q{i82h)2llsAqpKs0rZWYO9ghSUa>P)8(j5TDoCot@NppHOPk( z4w)qYX*yv5EdL>5^W_Q}9}uZGZEnhOeB86aEB@o(c=NAAm{#D~G$kHDF)noDm&;dv`jt**h~@TKKPOYxe~apq*>D zJ!trhG9G3%@6aFbpeoEQ0v^)WDEyQ9QD*DIBvhqldYWG((x)-!EO&)RFrY-s5uUj({7 ze41ur0lZNo-4j$F`3`BrWBo)Id3j~eEW3$gl(#Ge9i zFeN~RDJ`gsne{+hNFt*SILP|oq{nIpI0z$goY?1BB?kKMURERxU|zEV&eKiFq=h@H zvz&8y>D0v0ClOppd#3Uheu+F{^d`-bHGuX0_Zq_s$Z{_|c|eUfPamx{%Oh`Y0U2@P%~(A#h~Y-p>Cxt(7yF|kryKfSa4zL@^XJ3pa+zM z(y|AC@m&(KTh`~r1jJ-+l z^MYXLwKxED4A_D_R4)2F3dWAh8d(EPXUv zn{3<6>lD=_i&m!e9bhGf0hMG?tVPIL(0M7xG!4+?XfQ2If%gUM256O5mC8C+5Y zk*zEM>Z}R`K!p}@0F-RbDeu$G;LVISmmtU8F<1-$_M&fr;%@Bgw>tZ^B>?-s1e8ov zf;kwEkWS92i#3sCk2+4XXzM-rJm%RB^zy>0_fB@0%OQC{=|aF@eqhfyt3*6 zC<+^;@LgTLdjqtsIhR@MO$zrl+)1ERNe>4W(G0oQ(J#2-(2mrq%r}CN>IT@yb+QPD z3y?j1P^XceJ$JGQ!!^RKb@$#(b*+bsa=U=urEMt!^nTX!u&K#~xCOQO_xD9O0B2us zFds_O^BXXD(zdq}!g>H#fnEY-jfY{X8&Iw~)C}_NuDRjTB7*&}kx(%0;9$~k7fyqQ zCVgV40peVP%4swPXPL|vmO*zCF%YRisxif}xC*v5#U~8hRhh%dWK^u&OxOQv^jokX zu7C276+BD<>Qx&6`#Hh&!GuJ-S#Q&)0VP{wCxPf6e=8_k_Ag1wyQ3sF8;O5^7eG7q zd_QQ2%#+-jQH%lcq#av$k1MZc-X&QaBlt@Q3y0Pqg^-HQc0R0>Q4Twub*0yZ`+NFo27t8{^&wBACEG#Vu^3{5r z_8>o=u-JJYYs(%>{nl&NjJ6G+P3#03*6XVW?uHM~&HW2b=$GI! z>n?hDzvY44Ume*TbVW0mBihaF8*lzQf|lSE@PHe}N1Ow3U&0zdzLNHisnjBv9e@iq zOhUc>kTp9-I$@!*-@<53Xq5E>$UsjW^ggsW{=3*$n*>t+)k8oe>{zqt?ODQ(ac{60 zDh$P7r}@Uk!x2Vpk)*jo%AWPRpphZ!Qz1N9gOoxn^bka`#3{Rlnd5K`4N_M$~{0k^W6 z*%oY}t5$7Uq(GiwFf}rDOqQl;F9KFpkY<42qclhc!LnAFun&cbNoG^2SvbiIfOp8 z{BHmN_d&qf6*E8&;;|d9kNX$eE1)HyRE^4LL#Rb?=UZ z+IC4Si3gZV#OWm2y zI16^`4K;SWMt1fMaMi8SkYsnznnJhv^WkyltZ*1h%SiGu#y!65O*f^ zeNxBNW}Phf+j>@(J`o}5HX_0|2>JwqHCQ=x^w&6?7#{|yqcu@sO;?JEBrO!h zn1rlT<2#9aHe0b(LDpl{T{cr3!}XDVf3qszmP@GaD(eiA3M79sy@w!f z+zz8hwhYHKI7>vqY5|B-MT<_W09WNS41|SFJ)Dv)ZET%vc=aXAC*^yP)D4iFFG=;w z(!Ps*G-SbYxlLi^^tVX1gCUSHn(GDINqvJ~xMzTP?H7SmU z9KokX=Ca`V)-nicgObCAzJ-N6Yx9-Sqq$|^QV(wsS)B-3g32jgV3;mIi~e>1Zw^rM zBez`?G1eLR(G3>{W`=vAsda_`eG5RYyKKJKMY{6eF3>D!GG~kN?&brWO+Bd&Z zaaL73RtnbWM&ViwP=2WhkNdR%e0KyTm~vo=i`35WInUA)?w&W%SH|%~I9>9g7G2^&X!x_vnHHF}HlsXQ`OI1y1c4%jw?<}c7B?*41=!Nz+YA&W(LML3wk~vy zfaN`;xn&t6^JU@c9#U}e@m>Po;m|~#6ty;=jXM**WhL^+MDp?6=DptpJFudhC8%4- zN3Pu#!hEQu!}q>P1@fCEn#P3#UH@BE^wz;0yd2m-ENRMqcrJ4cxCug;yM8S4FNHhaDr34}px?;|vksjv;~t=wxxkEWB4F~$`cxi@Qb0(<`c;Dd*doLy|#`Yo39ITZB-enfXiBb6! z=gNECY6{D+sddLiQIaoBEh%SQZ`84v_rg3psN%-vE;MB>su18PkZFB~Or z5stukn$UZgYn#xUyM4$RYi|2ColXd9tbybh=VL)FkFt~7S~(sdT2vUh^6Lp;N;jD# zUyfLa`B2$6ymRQzJnmunRyjL-@u=njwp-&>{cd8y$$^vYCBDWeggBQAK@DVvx>;R4A=t zGL!;!_=88v%v~J1b&t~zX|td2%EbTS?2P1(3th95X^3?$nfrI2TY(Y5uj{4`SC2qc{Ukk& z-b}#`?@HFg0YdNWU*WA{LF-*NaWR>!=m?mDX1j_12Kz-$-MIVNbeYGm1TSy;wc#`Z zeYn58i;wA7rmHIRS@Ui*Sagpz^ z35p{>nG~VOz=dMcNlj={7YzzouX5Z0z$iU;&?~1u3A9tG*rR$TUmppjmB#O;J#*d# z8U!2ucqg3Iw_-Mj?LR$)?=;2&ay!A6uY`*?fworfYw%aRxYb=irTr+sNPrpu7;@NS zp4tGJD1EdRB!SJHls#RmkF3-Q_2}fVZ>dH@z76VnU3MLwZ{~X1@N@=mOJ;M`9m{I)D?-nb z?B(~M)YLM{(f?$vrzc;{zQre564`NJyF~<)K_3lxPa33xRfJzLI4~Db6K@a2T?gTt z1M$76*oQk&6Zkw56gR|N@8YV20~QzwFOn6~S#+y&ky>$NSw;8efG>}?jVkxA&S9?V z*F3)h?p8t$-leL{#SaM|TN!+E;@ADu+Z+!8IIH6pzp7<6tbVmKP;CC9t0Wr1_UQ}z z!|!4GDr}(b)(nc#-2`fkDHw)knRyrJucl14(bQD0eB&%mAS(maaj0`9=t&Jzek#aV zIt6&{m?g>;Tc((Bj%KG{D~Ly&&F#s>T4`7yAQ*ud)c_UAGlb6qP>@E+dyNtwq+|Zq zZ3Eoidx*pD0MyN~YjCJ)0v3OrLPs{+%Ow=RgtM` zgMBo~?h?oZ>j+?$+Oj&lWUQ?5L?8g*BQ~KvAwt&vQwkBot))BbIG;|bHiFiJ;d#w; z!AwpxXqv=QOFPiY&DeJrt5dXM{ePM9L9&wj8M=3nBFT`56;wSLhu%nONST2Jw7E~2 zPns4lv*o_hkN%4PG-fWw>sbwS6kWTZ1wEDj{2RyjKj@RM(XvW!c?LMc3KIh_8^Zs} zG+&WW0WNZ=%H*Fg5ujIa737q9WihjIqL+$dsoIXRJ3q0HRBqCOn>I?y_Ef6v-!u}cSNR*DKvbpTvj#a)p8T0^84M^-l#m^$zV%w zKarx#ygGAwnzd-wnBd`mEay)ds}0N9W-|HxUuz-)cZ9|28#R=eG6qrO)|>f>!%ugC zQ_s;EXv>C~fB>1-8B%Sf#J2GaB{Swf+XhLD_s3I4Ih(Hos`Zd|!`F#wwAMy9ILjS? zV4YuDAQl?3NK4f9;$15v27h4<$bf3=`*W?700Q+jjBoNJXxqIi-C|Y#uliscE&%{F zEH_U8QCnZajt8)$87BzO0Q9%lu=Zo<5|CEMwLhm+37q*KT} zxBbW1X4uaeZv?(! z{R?2e9kO<-M+6F=28bM;2Mk-BeJpedHs}&T?mv1yN0fZIMSgGOM`-?cj39?qrvZe) zql+pC-JDnBhKRZzEGZyWF&xx>4#LPO6}Wk?5+nga?GzSy=nlL*>+Mc37-0kEGM{z8D_IUM%YV$o$dqXEAjX{5C*0- z=jKwGzOJIqJho0rZ64s5#txI;Ft4Y#4awWJcQp_buM3<1EWo;SalHiYo+a zh00Mw!GEb-eZNvyK)zIv%VGIG8Vm`fWdNpX@6%2hsDS)O5EM)*QTbx;>|o7EF00Ht zwK{||D6;0bJc80q%~kE)fq}t1VD$-declV3jb_ce2Vey$?<}^Xm%rjazfoTt9B#ht z!akjekn{e!Ip}0OXJ@y?uAIQ@7V>HCtf%sYp!=Bv;Pj5Xp~k1G^`JMWx^9)`?_B_j z70BI>XW|^#>wTb5hp=4h#@#gpR02%SF$=!^a+cd} zKY@4-2o4~CKrsXsE(j1`z(Rzx90tAF7H%9u7uHfN%0|_l!2f0knVsojLQ|+nOdB|; zn*bS&PC8C^Ufcv4o4ce`>%XdI=yw(_EHKtf#`kIk{b15Lux6Zl)XJC}10-8?QJ@D% z=#goWXv3)k2SmHo>(&64rIFe0E(1=-lN!p{e}3YzMd~~O9!tW{7mT3q0y-pHfCR{I zFhT>S$biLRf{AGg@xgKd{(K8SF=9}A*YA)EFxKa51^>P$7_5zFz8>4206RDF?!+>^ z?FZ)x7ou1}4UY+P4mkndu8JxOMk(SdI8Rq)DAK<(!04h_l93-cjk378?20n5QlyS?jTgVe3>Dw!J!oWy4(q>uM4FMbfZ2*y;rfc(5j5+bwV`+W&hEb>M(n0kLK9v>_iH;Z zOj&sG>Hs$3g&t&oYN*~z1DnwXYN@?xHNyx3wBaJ-#5G{{$owbX{Pj=)vk3=t#Xw@~ z03@zmIr|HZwfG)D?R1>fX#FB99V%MZ$5j^%;UEM7why!s(#!)idt^zVqLY zWJp905>;Gyg4|%xrw&vW3p(z*t03BU{ka?|bh#cS)eL?tp6i{e1!&LPn}&{pnv2YS z0_?=Umxq%I=8R1Am>;NQ)sDW0;QXrfzQ1F0UI-E4hxBXm`K3)xlGIhf5C{gdYG&DiO$R^=m>mvZV6WM11DNFMMff+6Pu6f6QJUn&k1F#RT z=|@h&VuEhvzVCdNih9x~)qw9S3;NL14+d{&#*yAzlj6D9y*UQ5P)A_CV=~8J_aCXh zuPhGmz`}1jxCZOcb2MA{n(VyX`*z@Vne0BKejBh{5cBs~TG2HGt5-Fd3|NaKPeCz?~{5lmjm?0yIfR!g?b( z<-b6TKaeTd{Jx-8_)&gND(H5i*3(;Xr;HkGtpUvoJZ2Gjy2Q)<$-w^^_h%HeZO7S9 zuAlM~P|}V6d(y$#rK8uN~BKT0S~i* zlq3Ig??;int8&2qIgNo$_NdHSK z+uY9gc4w}Ev2MYiUB>~ulz)Wh=WwnfLGpE-Dt7?Qt3a=hX_Sx+Lp!)=NKLm@)fCseQ?&S;Vz+cyJui?LD zeVBZqD$l|Tv;nR73#JVh;$1C5dec**aaM)d@5~c#l}Bw zZcF+n7(F=CSOURc_%Vxx2bRY@aHs8tPD=#DHIF^by0C`>$&ano`utO2Mt{9Nh$F5| zCW)jE{{;>GwNfemZGdmy4AhmMLK=cZ1A$8*lJG*2gGdr=rjAht1FD0o?Q-{_Jz!X9 zC4UEXVs84*AnZr{&w{%mp%SupzIH8mjh_NJM?sB@H4t|~d)K!CDjl6yLPwE2KKU$CzxKYhneIZ6k%C$U6J z#PTh;^IOg}u&pK{0G26f01+E=+ND1t^vC{rRN-K_>-cu7C+5Lj5z6dC8l##P&=Z71 z-+fCI+@T;HLnMhW#Ed>_()Bo$7)Bca8*5$afvUQ3JQVW&I@T%WaPa;+k%2!L#Xh(d4_E@0oKF+J!8}B8`Q;2EbriHjpC!N&)x3O0^4e$N z3eySdEaYE2)^X8i*-@(O`{zZfXb54yHY|NnDUkjF@0ZbDF4Fj*XM37Z?*~Q@X z;Ftl?;0y+N>HvWa*5_<0L*HNKPynBy4C{k z#y>m7As&RvGVICgJz(m4KDPiwsD(Ia0or$NCmUm%$t3@LXmhD;K!v+isMLVvHGFzl0@TZ8@UPeK46c75XOh4{|0?%4_k)uf4|p@ zEKFZuK=1?76+<8W!L>t9X4T-OgE9gLb;fP)oXbn5jE9&loMTNu&OB}31Lnr(D^!2f z1OHwq5p_67@c?~??nQNw6&66LvJb$I%0KlC0k~8b(q=vFdl27TcbT`D#X1NZ&77R) z%nSe2#Q)ey8`!>J@@AgQ)&Yq9dfI1!hVh1ZBvzLF6|^xwMu8>wFk0C|ejiCRw*aWp zEF7$9HSz;!kN>}w`Yh^mYY0^s=O8L}`th$N8O{TV%oC`F$^!)wr0CxR5B@l10V*|c z{=_;PC9d;%duKPAKZ2}()$}k}Mex2L4Na4}{PeBNx<23Ehcx?gu91pkaxvgLQAcyP zn#MlkY=njlXt5tL*Ja^5scM+#%a8ms>4CggL&N%RlCE{&7>pHShK16PfCCnl3dW%R z1Zj}Gr<{KxSG<}6p>i|EH1DwDb*7LPQI*r+M9)75^+r+Qbi8jJok6Epk4Wahnm_`+ z%&*ZQU2x>Kl;4NPWiQ+URfqv6FjQX))SljT+FukrlZ9IF|NM-a!TKX&I05G!dqzTF zm-UAuED-1^sy+u0IXCk?U@#UQf&Fo9O^X7_WqyE^umFM~(3}3(_I{C-jFd-eTMeMD z21vbWp15xE>{EB1gqV$C-E+=q03*z$1APf3SlEw?#8cH8d(;@l_~m7xsolSq_)qg> z^0W<0Ds2}k`?_O#B~??czN!z(J>P{w0C7Ato3_K%2*C-3T2(k46G=0mWdDC_Tq>sb zy+Yp_bTrODuVc^L2X%fxY*Z6#KpUJ_%7_id=;}b(w#(IM`6ixIS)Mn4X2))=>>q6l z6p;ek-l}S^qVukNRC9um0B69I)qkK#p<63UwFIdob@#{-)Q1r>dtRUrO1JCJ@vd$J z<@45wEc$;I87w$JRv`m^XTaap0X2p|0^a=h4#@&Rs8loK1qf<~P9r43bMB|0(y~;*FL_9oT2#950Kw2~J*ebdvRn`{Za*CE=@IQtHGbwX_ zT%`0zbHG~om1$~E3;2)}X6=tEo@9~AOB=YFTu4oz#kL0i_ou?cd2Nt!KEGETBB{g2 zuLk7^mVp%iI5i*5VL@G+Gjp~F1S<6&U+8x%tA^HqNqkV6CffjLT#}F3L5tm>m@B!u zXKc<06eh%N-k_hb1!!1NBY;34S56uEZy={^aWz8Sc%k$6yin3)LVc*U|L1|A%X}~Z zkBCOJGCCQww&=d^2&81H8*%T1!4@I4@E>Zrzq_^ukF!Qon;0S+(!a3)oD))-pH~VU z^-@8l&og>>$ijT-?H<^B;g=xwwsL<8#rkK70(QGY#Gm>YdX<72`N=ZWHY91z4FPr` zaRL*dQ9rN}WDSu4>_d8)`-{sLKf;n8b{~3UgL5=L8068WJNWWy3b^dsI0J13|8Bol zkJy_x|2zfBv=yLdyrdstB;-6^iGAGw{JZO+Iy6nGoHpGdM$;%-;-(77#`~?n8hyH3 zQ4$b&ZUO&D{%Oj`X9ykAMj?c>u}X0gw5^ZwGS{=J(f6WV~b=6AdTH1WWg z1ES*S5Y*t+9+*V(uDx#x6Fm?FN3Emh%S6m0{4NK2EITb=n1}aXEkW1Im1PIiBftNr@Bj~FGVu?2 z7Rk3&q?QM%+Y1e&4nDr|N_Uzww+2xlthqS#4;Cuk4avSV4I{Yh*r1-u8dwANEu_MK zBN*BP@!C}Wu3u$+r&j>y}HC zTJUC!wLRlcf*_4^8QcH_r;9tO0hATmBHqw`(vi6|#1NkCB!BKC(%2rv8l zxj=sq0WUlO9nCy2J2ybq0PXJr0JDhf4F&fh8Qua-+Iz9!_p+I~1wxQs5sioAH_l#> z2k4TWuw9a`M=&Vq51_X+;juHLE>&i5Y%{;Nl;!XUyzdz z4sjj--y4{~{%N_iQs4BoGkRHycl#Waa|fHI$_(7KF@IL(t^ zzmaP}U%Qr(4#KZBfQp6}29$_FVP4z`nipM1i_#hu=4udM=OK&fyE{m?hDh>WWREZb zVsN5%0{HoR1pe8vK?P}+`JV;}qzaoqhOpky*cGG(JSt$R_r-gn{(~PBx%J}cywvA9 zb8D*g!C&44f<639?*^RNo5~kv_G~-fJwPE>kFVqMZ5IIUf6|c}W)N2Be|2@>(^4`Y zc)Edt%>;D3_8>2nX&T4B_W@p~}3Lk!?9rGk^5 z$)1pfBU@U&J797eKm$@eQD!CUxQ3JiW{REJRGR;kKDYfzLj z@qPsoYwdR1zUmdl|0=$>#C_D$22Y?l@&$6{}_50DBVrimwg zrD32HbAr*U#3BpO<1S0FX}LV=J_5pAzQ>KiAT&c=gEJJWhW9}_UQfCr{~KjWNAkEX zwO{hy&*{=1jsU}p!h6Pmv;r>(GW#(->^&w@yF81l%rleQlT@wTLNIRxmRcj|0qh}o zmaAngSMOTQ-_|3qUr;4);fxCWKuHFsyRqG{WbyGaMJyV0G$}SM$8$ zsqJ|(zxsUa6{3*fO4pI06^&}U(b}Xa;wpwettI*8W}#ugt3{c+U$h$j>%)ThnqF0R zSHqqczX(L*S!K<*_&0q=Mzamo@&h{;p8z5EJpX>4An9^Y1+iRUI3N1(?a@He(Q2PX zdF8e7-TBE2ICe>&W?Lv(Ck)};I^Q)&IQna=&wU4cH1tX&mvgb?<%@ALuZwT)n-|6C zsO8_IsbW;;rK4F;y0ptpA<=&$nDE@fOOKLDSwyit-=1MPl@r{j+RuLk9-;zJIYbEyZL#2QvfkqvMHNx?GlyyIvNZL`8=8#lgr*91SbG8W0QJkH zbcS=zNT(XfiM{%9pE=JIU3Wz9Gb!#0V8k%(6XZ0%!(ypWN+H*1H%-3B*w?1B(fV*D;UJ(a*h%le0Yh3#$n9vdG_le2G4nIYoPkDjrR`QdV)6h?WH&d zFFt8{B8LY9joBK>{g?MGH+BP0wR`4|Ed%kok@7AVYo7zJi0}ty=f9@fkr$3;9|ihn zwf+vhTnukqPOL6Wrq1LPUQ-F-7)j7w^myDN*+V8t`k}Jq{#th(l$ZznZqI-7 z(3eSihU!(MHqfzs*cYZ2WVPbk4w@}1F)vG`f3;ESgcvQK?fAfARMcl(k8NmTiY5Wr6R5)spCr zwEpJ8AXdea%NlzDY8R41u+qNi_`F-Oy%{-M%8r;fV4$Q;ySwKy#gX0$7 z>TwTbd*L)3L;(jHvYb6P*?lWuh?XW+TPByAV=A14SMt5belwdGhhO<^px?<(6@Typ z0>=|2w22x-)c3>F-G^meeN%g;sYASQ<3$rm?_Rp+F&z`F>wGk#2=K*JA*fZ+tD!@E~#HHZmF(W29v=xHexd>-7I&KVB^ zLIsD(d?w`#woYRYt4cpQpOq0$8|209gkUz^x4s)kwBHxLZVy|b;6L#(_r!!wM^(ki-%R-aLr$al%yzVbqPjo8orY9qgkytRJysP1DWUahOJ zg^GAQq~$7_y{2sE+Xb}Lunq!~6p-P(;4VjME|KVdVX^%)mf!_58)X>* z%FM|*V&!}xtCUtu_kUf=0xwKSQyf;`|MM&Z7u-Tt){g@DRF~gu0aXqBj{Yr~1s|Mp zK6KMOZ^fh%sWVh4#xZFZ7fH?)8XdSt zS#iF&P?|b5i1|{q&BXzQm9)O(G?LeT4X*U72Y^G zob0BRg5evHP?MrZj~R=y;0DGqS>aG&#Iv5kaYjwEzk9o?-lC5+YV}UqMs263g%by} zr{z&N>4Uk@62zs-hdH@;(}o&mK}00u+vONKd3Xev00Kl)OTsfo;(Q2Gdh*U|0d*@p z6kI~!3Ze|T888i<74 zOOzw)&zc*KlYP|S#+~xV(#t1c4H{Frib^qIh$o)G)xxM-(0pD-Be$xi9!!}U*k4PP zN-bZVMuoF(u94~)OdiVWDPo0#rO0xv&Tx73rU_4(4K{@(c$z_Ov$W1OHKTdYgKyEH zAwo!8PXR7G8$DEuKVxD!DYDl83j2-Ffe1?_y~I-y3&n9TF^1(nBFIEJ_mT@PNUF`{ z!!>2P0^`?V`NN)zA+f_6crx zoezcT=9E}=Bx2rJ=x0nA=%Sx6%?29SkT$%^mLn)ekqA-1kwc4DoFGd`*4g09uiY$t zn;|q8r|w}jPnQ`csE<^Q4sPGE$q|pu4+pC#RtnmT9h)u&aytgwm_=2`HRy~~b@oW# zlcwuw(yMjps_}T~Qj`nNLN#5D*Aq0E&Bj#J9>A$+*QoYZo5P?~QKI^qP=WxNyvvUI z7E*k%Mq!*<((7JKUk0@|L}OUw*MbdBHTFDbOQfy4l1~9cwcyD_d%TADD@3xL`FF*< zwcik$jVri5>Pe`CwDr;Q*RE#_@7AaW2C7X50nh;)p`KK!Pl}vKux^^GhARwk{zjZ_ z9QD%Wj&(2!R%29#RODZan&+<+}j?^Ua7VVJ0VFcc}3bN=KK>4DNFn3G6l`U!-=xd+N4r&;7xAN&Ks9*hp(-Y ze}j@!SlD;fwBi|x8_C3BXlRw?L#iw^*RO6*Sua%5{=_nIN<9zPmk)^V>ej|0GB#@> zzaM09K5&K3IL`y0v)-4O7qcT?Yb{q(<&x6-(7ZNi?+KkNbX)2n6Cl$4J>=Ieewxk;j=5LGn(>m~E)S!;o7~qpGG|$WM-sp+s<(ad z)olQHk%jraTPmfStY&-KgPKp9;VR;7AX3sC_VToHy39q1qzg1m%Gyk1b*G-Z_Bz6O z?s;|c%I`Ap)aXvj}H#t1mz+ z?0KGz;n~a7d@{Zl<$YtwX%U_9wbv0>B%t{s+!P<~TlH1#TM^3`*uyRsFE;j;lIn5k zy{9a$ugJD$TFz?rgzqn-+?R=$HiQm*F;2>(_>KVXCCQO{=pi0Y#^U_(9@5bmt{nm} zH=;rSKxQN|8N7eq%nIf*J4%6?cMn{+5@K4K_n+)HnYJm+@SFiJi5IwJKuK{sYk}~G zmVr>AF~$QJM3SF?d)vwG`66Jp!4Uxe=&`IcN?DZWj^P53oT-Opn_#m&yc-J==(d0K zuN1y33Svy1z+#35LB2WCWM8g5aHqN)f2PpmXyV#>7-925V)^dMXn*s-pX}ymTi#%X zAOv1HSwpYZ1x(pio<{>YlKkh7SGn6pi)j z8O4MKnwO<*LXM=gpW}Xqhz~wikxHW*Amb@s_UswMJ6+M6y#Np}{?MJ!_3Vffi`!Tj z%D(87p@wovFWoJR_vHYo^ge*`(oq;>rU9{YYUU^-Fb2x91&A9C_;8SU4qyUfvAIzE zqqgrzXgim3#1`v=$v8~Pxe)zmQa++)Ch{@Ov&@B?Q%f}~I7 z&sVRBE^OULG0mu8Q>1F6y>fYpVM?9j(yM!idGGfOk7724XK-P#Qd%#EY$Wqqx3%}} zSX{n6BJor42;%-^jbt@QLV=lpjHE;q=6bxw%cRwH+yJ*Wxq0URk#<5 z~XaLsQcJt z2@=GODGu{w3=CHsTA7`7LCEz0@UYwOog9I&kW^LEs3UkEZ{5<}S#9L%s*Otr@%(Yd zd6Stg>%OxJK^|$<5%H9uEB0|jSEX22n+H%SWj}vZi_Ox8QSY!3Jq3o`>=f!peJ5NDaJoLTEY+nR7{T1=*6ek zp3gLxIu?!2D(x4YmppUT-O^R34nbC!a!if8=bGU;uj}F6hZ%B}8O_r)_hu!ijJs29 zy_(8f1|(ioc(0u2^YJZ?t?lsRaC+laf5i!`)pgGY7f&gMo6no$jck&MV68;5Ub!K< zWQ*Ti^awjaag8AmgLfNj44LZPPvd^?dbNy5L(xl5zS_BRR0%MC(*&b{q*VfgeB_vr zHI)%EcAK3YON-dx!zo24LNUG7mUtHQW4rmmZZW}gXJg&oOM-V>Ebpk z!Ane%tb0iPXQJsUoCR-u5`ZHg@;j#GGlx~LZ%&tc4{gMEh6p=EtDO(44X}TddF`HOB9P&xbcY1B)wOjp#gGqHcY6 zbGZk^)~1{Vy?1tV1c52=)C^+$VI*NtfpMz-nL2n;v0t7&q#rz-k0x9>uD}?7?(>Q>*E{emyJrm zu*vz-&7lvLj7OfJ&7~7^o794hBwVk1ca#`C?yK)F^bv2v`DfTSi8!63P7(iXI0>9W zQ8iRWJ;2nXoln=c)O>{XJqjN87?Z%4=-&oH7Xte7CX6q^JfA$yLfN;agbO|TU-0UC z3xo<@`*(#<@#qm4&|s|+*|82;?84}m_R-A09w(=<@(H9T<*oFM^6KOV+N_fZ7 zcv4*`SSoz?o=nPby_Yd@CS@7izVBQvX!61GEQMzTiEw<=2S87 z2wF*W?C$k%Zv>;Rbx=I_EHppCIFyX|(uT^$(bV3V1uy;!djAE1IGa)sQXGS*S@eB6 z-}QL`S<|=Nmu4|60hA{}@)LBzNI5%h>t2sheQhPO=-m)6sA7{?vVXbftc@A3M(fLr zymi}tjJb?wMB|rCV$zdz`q5`bIYB4~vnPI0AUi^x0T(Au$0P#lrV5s$4=H;vPycjG znnjGepH=J<@Z-419)s;{$9~FB4*3R9CqdakI<&{Se9e>aX$sR}UEEdggIVmuIx0OG zy3ZoyE{Rr%XRZ@1Cm66YzHAz-sV{}aMVWBa(tX;)$jeN<6|XKbg1r?$F`B`nNpquW?U_S}Z zPy|9{o`G*w*fZxfu(Dd{<1f>M!fnOQjgNAhlUAwQc#=sYXG+n7P`nP?lKgC8x^VM# zV3(2mE#_GzHN8Ouk3hwYQ5i1L!U{~ZAV5^-_4)k)mg!b@+%#R$>C&3T*B00C16)F; zJ*L8Mb;sO?x8Xwmz8Hu`1t;$uBr#Qs*>eb&u+_O|@x?y83l}iYy&=Cij65I~{;Iv> zDbBjsoZ0Taevp&ONEv1F|o|i-3cB4*f+0%6X*Pt(BzA=?tZdV8S0VvI# zltde}^GWlWmQCwfc8f=163%Fb)t6@OV}EPKZu2pFhPR`$EZCX$meQ;*=r zAe4@u4j2xuK-XhT7LSf=J8A?&HZ&S7;Y#h}l46S4a!2%TCVdnr_#B)bs&Ry@urdb7 zj8qR<;3Sjq{0ixs=7XJZ=@6c#&2Wlt8oN?Iu}>xWy~N7&Sp;<)eMcQP5MNf!AS~uX z2d4^N9NHi*&ie*xj@BE~_v`%C*q@cilaM8TivGldPhh8yGCnJj)VuW7VlyhxEL@r$ z!R+{X(S#X<9oJWIPP`Ri_qrDwB^M6wUXNOjvF*q6OhW^1d`7kv^T0&oK}Jm)YQkKC zA1`wsqP~dYm^UObU}2VK(G>5eYbJO|*m|L}->^d}hn#o0`P!SPl*orB9CPwDD@MJP z6367I$_lQ_=lYU(oqKqH{fBC3+p=;KZP+g8Ut#k5&`)sSeq3WtuOEXxwVqpk1Y9ocvG;g8TaS+*f+^biT&EgL#|eG7p#iish9ZwH}4A zdL=I zGYRRSraI?l%t)7Oj0>VhAgKq`i3MlEoLwxd!3*`c4=Z`xsF+bbGE~hNfgNAD#LH@p zKTTB(kc^XGWH}N7`T(k24ny7`L6!Q>Tk?$)8?WQkg>}$R_5MU&>Ku*8X0`rkFo0Yf zdeP0}*!44j4aS%Vb@mR|o2Gc^>-pl0?VMEd3T;zdRp+VWZ@Z0ab8~8|xnwr+BP=f_ z^VQb0dS=^j7xO2JRa+M4lEDvx+)<~;nX4m6lu+(t{SG8~VKw2(20#NZioAzWWm~IuVpwgJkA4}FU8`+%@VNRCW+y$yn^WQ# zsPmm*jMIU$VicxpWku9c!F>sPkIXai3R??HA+}5k1O5W*iY`>J*QIaDKd0Jzx~7S9 zg=u8F+D_0bf%97|wio+WD-E{ZCYg-BX|(ub|!h0 zU()dK4hb_9t3h3dz8If%C)-0W-#E}$XEsKXL;4X{-@{t@V9Jo6+b;Y z+P=wh{t#Bn$HWMyu;ir>X(Uj%h|)g-mcJB4jAlWhcGI(2@r!}|Tda;Z?y ztFl6LVJ^6(6k7?6Nn0wXmaS~uHK|``JqYt5J@+R^|Gm*u(re&hbc$iC;u4K`)ZI;N zLn1mNoebtJXn%v3MV4&ne?@T(OlD{Y+6Izzd+#{*nCA%VYsSdk*f3X4#Qp~iX81a9 zodt~&W;~G5masMUG3y;Jw8B;uq$vRgm5o%%5>YXwqFE2U+`cl15npn{*LAtFI%HH* zlAXWZ62Z?fI;hQd0Fr_ZT~(IH*ueXbfj)Y)FhVrg9qFgbl$FLPEM;f|QkPtA6`i0t zN+pu~QoP?_~% zS9w9f0NnQ6t{AH1Na*yD;*dUTs7u+xS(q49!_r)u=Srs=ri^wF<`P=@`-NcM0)Inc zBAKLkgH0YWj>wX{X4fFW-=i!X9a>bZ*9D|8SctUYq#pKZJ|e?MuW?EHjTmJfHSw3| ziPHDz`Vq(_sds#T(CsHIT@81HR(hcH)uiZE5PEj`scGqG;v2WHE3pT;Im45o7mRev zsy)f|0{LKt!99^@mJb`_LU%T0N^c zl_=bJY_qdY0{^CuBhk~Bk3z4!LhANBw6QQ1$<1Uy$7%b~Nv}0XMM06#xSj?j4=&*n z>-WI!YD-!^89PIFp%(Q)Bn4{@C&xC8=bQ5|PI%a;ez>#kt*>!Sr|SKtT>&Z{grn+S zufGHD+Ec2C56h#-HtTf$JY!-6l30d)ST2=$XOJBkge~NbY>H}w;quKE!;vXB z*8S)tR!dcn6v%__lqaU^{*)iPFjD>iR~ zFcUUT=lb3R@1m&AH~5CmrH7#`zK0qs$cvhaF7-W)&-S0(KZ3Uz4fRWVBxv*j z)z_M06z&OAOThP_JhYnw^|Kug6LWQ}Wa#$FZ0?5>b-^^cv8-b+Ow| zRmX4NN-2^j{SxyPQ6$PEeQ)VE?&BwV-*zCzpDRo1Fe>VKgmrOP7KUgN2WvvOiDL`z zH>+=Qt+;sMfi4~F@g?N$gII?U$C07;GZeVnFc=gGF8Wc^DE>X39#j<1gjI*Z?;Jny zS;z`0>B2qM5Nw8a&MD9wWSXl@ck+(JbPUOe{WuY?qZNl&?O%+7&gSp|Jbu?l@vnHe z`5YFE^eIwf4s?+O+~aAlvRn+zzEz?x<1AyC&kB9J@U{E)#0gerbu;w@i*)09hd(cZ z?Pkmwb7#}0x*q+6>!@=d<_^I{>bo*(kRH9Sz+rxyEJvmZITJ>%47F!K@OwVfa8dSBk_{u7x+&tpn7{D(mrE*U9Z_hIu{Tu&oB20h7R=HN4T5)@$rWHjG0 z3Wf7`jK@zoub$-BUc_@Fpudv0h|?iJd5qV@y$%oOo+e+h6Rq&VjEdBca#}aekg*5u zY&~nG6aL#q!F~3(q*J}Kz+_G#Wn2upmukV#;H&d6@A`@9QBuC46YbrnsEgybcQ(a7 zJRz`(Ur8@KP*rZgR1UvV-NwV6)-`1|BsOf!_X=uB<}X|&kobWTk5nd5F`@9BaoRzGPS(smXv=Ve^<3~c_;v^Er*?)BW6xUV=z1S06{lDhIMXq26$h}Tic zRuHzLpROKn#SHyayr(<(u@B{8I1U{V9h`afaj6!1+Chb?rhC-XY9)274X2@xe`QrD z0*RvKCKX==iAIZXq!PL(4`-|haJx*NHQM7?vy1R?>8 zMr6IzF#jjR_9NfH36IykJc^yBzSY4eXmSoQy1wjQ9HFChtLFPdc1+u{R|3B@uXuu( zRHC~gGzqe#;Z;gqJL>S=qy#j(D=Eaf&WH5leW(NLRLy)1i6I zF`?o*(fA|d09_je{}+4<@`w@kg!{;QoJBU7--66;+4k)Wv$5j}EjC4yrcGP|$1MUI zgx8&D&*xc1z2uO*R4SKRyb``3Q{&~pkeDx*{EYX1lU6nNZ6rg^1zxXx#7tt=or3RO zRrMV;r%RW9`$&r1C_eRt;BRd0ot?m3Z{kQr#L!_8ChQD6r}PG$1&xzzNn%8j_AK*k(#T21N9A)^3D}_Ye#t3aTA2C!@HlPE=zL(140jc(} zOvUv~2)T`mp^?J~u@vn>l`z2ASqWi=a&Mi&`kp_$GqgO^EX+AiupP50mKOMk&wGQ? zDCFoRM%l1Ni+?rq8_NME!_Y&5W{SdMr;ixt)~TOP*Ct?EottKl7V%nO&bxt&PTR}s z&EZGWO~wr6%eXl2o* zNg6UL=VYKr5;cAFfMd+#N6PKwZ{!-YJM+!ZrYiFh?#+hqA?;5itl!;4^Y7~h8P8bM zsPTNMb&*aOZYV0me4n1B=zhd%K~f_W!OVnF7kWf3N7xGM>c!DNN`R&D3qz9$N%yi} zHOJHIL_#I4QZ6(e|1m`4lfbtB=ojQlnS?9|3PBlymUgSf8HHZzTy5fNPsS7$jtj%o zndI}`Uj;#3nOt&iik-udOvpX_(udDI{ql@fT0r_a3NwP=Grwv_JlEhs9799|n6uSL z_oCSBuY~Rid4@*gEto|Z{mh}n=w+SG4}!Sjv~TqF0t^}@6+x`MMOnEVq7%=}RO+?N zxwyT!9}K>g^Wd+L%6R>$&j~N%o6qn?_o6FAyC{(O@~yJAUQts{dLZN3lvC1guZ-o# zEtVmLNDvuojc*K_O52oA*|v4aQe9Ir1&BV#kMKO)F%`1`*;_G+I3i!#| zFSeih)6YM`-0#J{KnP0!``=aFgU`vxy~nAFagwO!JG3U@uTX6Nue2) zYT@NAZD-aotb2V`S*OR^w(v$Wakj#f*KAeyHl;mZc`JM^SpXlU?)~}RRRf`&x%rp7 zIauv?b0=H8jf?cy|3viA37)_z3uNE)M6y({JN~q-WsHFZE2Mq@oDAiXhK#@>dsRl% zxN7K4FS}k+^Cz5_d_rbKlGaI&m@q-)3haR9#&r(+AsxlBSizPIwUQgp)dU%#e8CDI zO!wUAD-?_2v`#y@)vF)PouHxCcZB_xI#io273*5b`b`|kcA@-c&Cks%&YpHI<0D|8cn5fA0=j>wIUQt!14@@~UKIGj;@ ze(Xp4%ioszSq~O=oLYU_fj~I=w#+D<3dhJd-6|5?(kfCb+`Mxk5u{;O7h{vJxvIK&;bOuC(y= z#%e3>xoLoGkQYY1vdUzOEghDfmdfGCzrufU-?qJ%kTc|Zj9;to&xM6hELS*=Z5w%@ z-b=pF7Ci30)ak+QUeM@V&3^1CYlxyt-ZNx)BJaDq?hVe^zfo{s~#n~D%@)Mtye%f=e7Haz+8YcoOxY1OiOO}~ARWdw<-{w$2nEn({@DICqCrvw!*?;kR5j*7zPPGJ3 zGO9J=@Yy;~J+~7?7Nd{4BKHe(l@$r>39J%9^tgHGi!)5#FyR=aSK_3J9q*{bpM0;Lj(q{x*OA#MssV(;DOX@L+!25bvvIV}@m# zf<^f`1cRZ4-KLS&R_=;un#0F+S|~ArHoJ_spUHcOXsYp6JGZ;n==5H-n8PA0ta4LJ z%>@F-SVsMQy6|+51Za|W3(xd%QJPHbl_Bv{q7b9@nY)F4XrGVI+yWuJIJNdeTSh+- zCo8Bp#ikkDZXMLdR%N4m6s$I>Ay7#;8`}~*;x43&jiudQu*u+`J>Kt(7|4;z1g4|TZUc!y2;tD9#m+Mk4HRq*w}e9}ni$&tV%PydeP zb$5H*$*R!A$F55osmH5Mo!Su}qZrKIO3VB)c?1tjYvRLR#tr$U=oPk|O%XRjnyT#z zrG&w;ZEzwq^<^36-}l|NC*o6i!%ZLU^{r;twaOfFOkdm|n0&cYpOw*`z3ODmZ|1Kq zV|*1extg82^gvU%Tt7DU=JxdteOCHoWQ_GuaBo%4rX0_j+u|wAf4WcM%ATu-Y|XZGU8u%HC#Z`~0b_e%rpLv};qgaz``5*ak=F zWs*TTX0j>zthj&i{Ip`-?ugRU_4Km+`Z=lqU$WPi#%A%X+m$Xl+ZXPJ+)Fd{@OV5E zQN(+8xj+5kL8gj_(Z$NgSxzI6!lf0;XZm&~GKa{{&_}?8$eelxfv!DnH!u>vsa5#3 z%`KR$H9qBmpb7tF$HRlcGt4RVS6{akLxjamRXFgCh@@>L$yC4Jpmmancw7es1XB6R zz#9P^eA9s#LmJA6n$NE1u~nbLG!>nh5B~q9P>pkmZjG0OYFDG4%$;VyU&-x#N<671}mpgt~00O5@7^d)vG$6 za&2H>VtUr2NxqhE2Fvo2p4FH;Gtd)aiCj-L&_w)oVgb3nd@y5Fx72<9TW35WJy#F` zNc#Cqof@Q0B1h*cjHDk=eUmkbU}p;n^*P3(%AaAC=Xx|qsdo^Nj5I2h(hj5@6ckDh7 z51Ts1-P8L1724rPZWZf3K6w4K8Zy=#Wlww~A8hr|TbLo>u4o8el}&b@N4&WGjbS?-B(Ln@0q%m#c2d4G~N7s)H6u1-HHj^wQ=G<{v%E9N| z%yn=?^~y)tY|mfyh(4o^W3`7l5rE(23ay(7!NAjLf-p#LE4o^+M~FVxES`9-oa5=aOB2OiH0s?`Zd)B*EmD<_4*YtK zcvrDz{Lfy>V+My?xA{kVYL~#o>CtbC5;dY;IVH@KS&tJB1p2wp=`GG8cLbc)Ns=!T zKG}8)@*F!jVXwOUy{5d#SyKViSA}^(5GCeDYi=~LoD58J&#OKf&SINPLyYM7lY3cumFYks&2Y4R~XsQ-nbgex+6*NzNB>L*xOFea=H!eSFr-mcS z7d0oG2OKo|a*kQP>fsXIv3s8Km}fZHsC><8f-ofFRs3r!{c%D+O=V)PveKMgNo^c~ zqlw;apATTXt6D&gCl5GvFHwX4xujfv?C+I2)^-E6a69VLX z?4$c;xNm#l6>9`pdszOX%o1CQjWE^Ge=qW1#P@GR%YR4r_TOFzx=?mo?#Qi+p^+2& z4rKbef2_lfCL9fliFdd5e`q&1uKW<0xzZJW zhyJcXvH6|oXFVG4gKAwQR;75RDy@=n&OY^qgT!>bv1`6$?1iJZ6c^86(t{0mR3=T- z-+WNwIH6FVAnG_U+gp@2idP``Vxuv3~%YUx8; z&t}nbt8Xt2<~^!+Ah%oa8L|D#IJI+&rysgb3S@g%j2y$VVf}-7PDT zvu%xc*SV}Fg&ht#Q*?@0ipo76>I(IK)!S@zSC#CyeH0z+Xr3^_@j<9X$xuu!N#Nm- z_tvb(1F0d=X@%1g=XREArF%14*neg+B`hG8^hLsJ zhY;L<|D6Bi@Eu`?p1Fwpp0)fxbN@bPtZaAZWWACwvmXB`$V?sEzwL(G-I|sp!vB@~ z|19{sw@vwJd|)2;!e4)}-=oXx(oO8`Z7fYq=$3X=A!n-DC9jfwTIU(GygMa>&|t{C^qkOH8Rn_10835w$C zG&i$5b66lDG8_I*IlvFZMJ*@e=JG0&k_bjdI6yIJnp|kL+-$QDOBQ=M;VkRk?qS(y zmta4}%!LS&^oW%OK2*=$#FgeoG;Z=>b7kY#7);LwS;8=dnoN^0|Gd%$ljha|jp#we zRn+CRjyg9LiBt$uW4#>4Jc1&-)LR+SfB{p)mkyKoGoxWHnA-A2N|xBw2QsilG57mJ zFNH?Q>Fd*SpYf`4@|wHAuo!g<>t~R`06aKlBLP%P9ZX_IAk15kHU$aXNYye(n#|u- zvrwe06^Xm~0s;j*%}R_y-pi5of6e;N0D@?>9Od=3mT!;8t(}df6Cjib)3pgSlPL-j zrVLQK9kIj#!OJ`_*RUJNhE@tdXp%1nLbO3K4f2H|FNU2yuHnC#Mu&WT;CzKY!nCW> z-2L2nz=j!GDodLz0&!xt2`cJy>EUw)4GbkWD}m zF7ZnpbV$>6f=NyKwv~WJ3G=0y+5ujoA)r9<1>Q9eRv(27y%g{ET!E>T;-%+ds;xJU zA*=f(YF1667yNJtT4?>QMJ)L6+Q~eR*b8LIi#-i2uuZB#GTmt$4edrkjyIY1P&6b2vi9mwL8>cmn~K@_r91q|5rSdhl(36kVr zUC%3iGQ9jHy9DInP-`I+Y-E=9&dSYQyYLfpW3}tT&->T?Vfyzla@BA$5GJlHoi!9* z`B?E|W_Ix7Vvp){$Z9XzjUuY?MM-u-Gj%;8$uP#U<<%*N)z<}m zUjfDR<+P>6x)L#eqLF8Li|B4}TlOmA*_d;x5PF8kY+U|>MP!)vSE?YM4~4r+!0 z!ySY9h_6%rE)XAA7~~4HLuh4CV}L#b2@H6WI}&`Ni(sy@`!LKTK+q5fC_MM)RlLGL z9FrLZc!?sme=->tJ{E&4EZ*X}VjPP7FNhZZXxkR~^(z(eSHWdOgO!>KMFTRR@ zFnB%PX}rK7=26E$?DO%m@N7;~F)6?y%AYVoiaS4w4nI7oG(^{-%M2k3-Jq@x z#BhxlL5``?MV?f-m_mnNaiM@fG#gdPY<6~mL_v}iDt>2jy7Aqz(R_1loqA*Z60NI% z*E|Mff+8S-vp-DY1_V(~y&jC-+jtpWIjh`g|G`|8O&J7ulNHtY#>jM{lqDcQP}!Gw zkV7o>w+m^4po*g{mw-SS@SwfxBVC%BH)WEb%ja3`7r%qG)NLe1++S*E>7B`nXhjf zC6`%qtMP0??pwba+e@=2tY3tFx%rTp4S|k^NHA%6Rs;2xW-xGj5COF+T+CLrQrxRQ zX5t^Bb-r2TPB~1SRRij=-+9$Oo8fO4Weq(s-;{RL9|vTd;Rm-@!5}J%T$8`hLTD$7 zS1$nq-fDfNbEH7_6-W`E+GlE}b}{1+mI}(m;Cf9^_koXh-jJm8($pQhR(&onJ0%vx zUFx|2tw!P7N?+*BcF=#%ZM<#8nvnEh ziUlJaCSkbcqyaz9YHE?EKv(IdYbGIdy_7!d7vVEDRY7;vnKNiCmQ>7;^Dp!#SlxmXp>{n^>+$wMcYA~&b`y~Yz{;cb>)w7sxus9r@4hmzcb+63v*?@w_Xh^rar zgGtX~s+immffIBBg!>vBEFd}Vxgs|JbMZEXji9<24vYi{*BNZK zYGU^caAD*NLvB}(UCab2BFuHo50!RPirI1iqU4I_YCmmTiDfHxTYvz?!1To-yL$KG7c35=Ub)UquExx&wZ0-RY)Na6Z*b&wFAiLaI*JG4OFAPId00r9;Gi4(*` z%uyT0hw*>>nGV&jL39L1{tt7)awcJ;2 z_QLgU3tCdY1*ICb4y*~}`~m1D)eDUm2ZWfaXuLqj8AYFN?|2gk;46NLqx?jWCY1lE z=m0U%yJ>Y>n+wBrN#DTQ1lgX*cYrAgtu8Xp4VsPAwXN@Ervd$t-#s&AK=KSt(UTHu zg$ylvel`=&pGO-KRZD3>*AM70HIpMYH<*D^a5hp-sUx!GsN=u}1O~^iS|Q^SX%mbo z#wAG_lt4nCD(Rem8k(mSIi%(>@U&)VC@dDWYzQ|T>krdDy#um90)u;WI}ogjAR=n% zrf-%f+x5y;M!R@lP2KU3h~4yRe$5 z`qcFLymX|1cYS{BedHTHjOR_*r+j&_V?r?X4PY`!igsKOkj1Jn)SHTC5(b0-6_;Q2 zd9E1Ma;xqR>V@%9$YR;LxgN|q5b@RBIKFIY32$Hz80H)qGuX=Kb5Hg*+1zg=o-j5I#Sjq;#PN zh?)o~yp_74x3w1ey?*s7pg}mhNOYhvsJX+mc%k7EmK0{8Aeqd%+WJRURNE9R>I?s;C%$jTCP* z5F~>qB=L&ZzpWUizSRL%gMqGFwtns7D;PJc@`Jf_oYl^17RaSf18Jzhh9?<@L4#rN z$*9vL@CAw4MN0M)T>`yaorZc{ELJAEmS%>$_y^L;obZp%ncg!1(pc*z5Q=Jml5r9- z$^w?t@1b<#*MM${iICJ!5-p7UQu!Qx(Zg(g^vWt{BtHql^pw9x4Ujwqq=jL6lHD~B zC3ACY-h5uxp9Q=J2HW*k(H7Eh*!p>iFKl{#5~gkbxTMxUSz~1DB)d^v6=P1c(p&_R ze=4GfCL)ED{Vv1QH7BHl@KQV}7Xg{F{8|s01YpXIADhT4P$Fth5SdTj$1&;;i{9{` z#RZmXn^N?-ylBG&t6kyfTHs=@8)3h7K^P_>V=A``^)Li2US(gMYNmD!u2CB;C62V| z<|^3@Q_$Z;PAn)L54ja1eH|(I9ZV8TqiE)QPTsf#xFH~m*L(~@ioY%hNmhpQLDlX` z->3Ki7h&eI)~%4F&8zXRjl+lP-Ef>cpwgy9M#1>30Os?GqK?h