From 1b4f97b094930a1622e78ac49324e142b7be7dd3 Mon Sep 17 00:00:00 2001 From: RichardJECooke Date: Mon, 19 Aug 2024 10:24:57 +0200 Subject: [PATCH 01/32] start offboard article --- .../docs/operate/offboard/offboard.mdx | 32 +++++++++++++++++++ 1 file changed, 32 insertions(+) create mode 100644 astro/src/content/docs/operate/offboard/offboard.mdx diff --git a/astro/src/content/docs/operate/offboard/offboard.mdx b/astro/src/content/docs/operate/offboard/offboard.mdx new file mode 100644 index 0000000000..816877831d --- /dev/null +++ b/astro/src/content/docs/operate/offboard/offboard.mdx @@ -0,0 +1,32 @@ +--- +title: Offboard - Leaving FusionAuth. +description: Learn how to export your data from FusionAuth to migrate to another authentication service +navcategory: developer +section: operate +subcategory: offboard +--- + +import Aside from 'src/components/Aside.astro'; +import InlineField from 'src/components/InlineField.astro'; +import InlineUIElement from 'src/components/InlineUIElement.astro'; + +- [Introduction](#introduction) +- [Create A New Instance Of FusionAuth](#create-a-new-instance-of-fusionauth) + +## Introduction + +This guide explains how to export your user data from FusionAuth to prepare it for migration to another system. Reasons for exporting your data include: +- Swapping from FusionAuth to another authentication service. +- Wondering if you start using FusionAuth how easy it is to leave. +- Wanting to export your users into a data warehouse for analytical investigation. + +If you'd like to follow along with this guide, please install Docker. You will start a new FusionAuth instance with a sample user and see how to export their data. + +If you already use FusionAuth, you can export your data from that instance. + + + +## Create A New Instance Of FusionAuth + From 1f66a729684ee45476f71d1f8abcb34469fc364c Mon Sep 17 00:00:00 2001 From: RichardJECooke Date: Mon, 19 Aug 2024 11:26:33 +0200 Subject: [PATCH 02/32] save --- astro/src/content/docs/operate/offboard/offboard.mdx | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/astro/src/content/docs/operate/offboard/offboard.mdx b/astro/src/content/docs/operate/offboard/offboard.mdx index 816877831d..cd1656d777 100644 --- a/astro/src/content/docs/operate/offboard/offboard.mdx +++ b/astro/src/content/docs/operate/offboard/offboard.mdx @@ -22,7 +22,7 @@ This guide explains how to export your user data from FusionAuth to prepare it f If you'd like to follow along with this guide, please install Docker. You will start a new FusionAuth instance with a sample user and see how to export their data. -If you already use FusionAuth, you can export your data from that instance. +If you already use FusionAuth, you can export your data from your existing instance. +## Export To Keycloak Example + +```sh +docker run -p 9012:8080 -e KEYCLOAK_ADMIN=admin -e KEYCLOAK_ADMIN_PASSWORD=admin quay.io/keycloak/keycloak:25.0.4 start-dev +``` +- browse to localhost:9012 +- tenant -> realm. application -> client. + ## Database Export Command ## Online Migration Techniques From 371bd53da2bb91051be0a600d4f1f021b0d30291 Mon Sep 17 00:00:00 2001 From: RichardJECooke Date: Wed, 21 Aug 2024 16:23:03 +0200 Subject: [PATCH 14/32] options --- .../docs/lifecycle/migrate-users/offboard.mdx | 23 +++++++++++++++---- 1 file changed, 19 insertions(+), 4 deletions(-) diff --git a/astro/src/content/docs/lifecycle/migrate-users/offboard.mdx b/astro/src/content/docs/lifecycle/migrate-users/offboard.mdx index fe44a91e5a..6704d6bfa3 100644 --- a/astro/src/content/docs/lifecycle/migrate-users/offboard.mdx +++ b/astro/src/content/docs/lifecycle/migrate-users/offboard.mdx @@ -21,7 +21,8 @@ import Diagram2 from 'src/components/docs/extend/segmentDiagram2.astro'; - [Database Schema Documentation](#database-schema-documentation) - [What To Export?](#what-to-export) - [Export To Keycloak Example](#export-to-keycloak-example) -- [Database Export Command](#database-export-command) +- [How To Migrate](#how-to-migrate) +- [How To Handle Differing Password Hashing Algorithms](#how-to-handle-differing-password-hashing-algorithms) - [Online Migration Techniques](#online-migration-techniques) ## Introduction @@ -84,15 +85,15 @@ Some data, like users, applications, and roles, will probably be available in mo To understand the user-related data tables in the database, you need to be familiar with [FusionAuth core concepts](/docs/get-started/core-concepts). Below is a visual summary of the object hierarchy. ```mermaid -graph BT +graph LR Users --> Registrations Registrations --> Applications + Registrations --> Roles Users --> Groups Users --> Tenants Users --> Roles Groups --> Tenants Groups --> Roles - Groups --> Applications Roles --> Applications Applications --> Tenants ``` @@ -118,7 +119,21 @@ docker run -p 9012:8080 -e KEYCLOAK_ADMIN=admin -e KEYCLOAK_ADMIN_PASSWORD=admin - browse to localhost:9012 - tenant -> realm. application -> client. -## Database Export Command +## How To Migrate + +There are dozens of alternative authentication services to FusionAuth. This guide can offer only general advice to migrate to any of them. Each service allows you to import users (and sometimes other data) in one of three ways: + +- **Direct database import through SQL statements** — These services tend to be free or open source. Examples include Keycloak, the Janssen project from Gluu, or Authelia. While free services don't have the funding to pay developers to write user-migration plugins, they allow you complete access to edit your database and insert whatever data you want. In these cases, you would write a SQL script against the FA database that generates a SQL script that is a set of INSERT statements that writes users to the target database. +- **API** — Services that are more sophisticated than needing you to write custom SQL, will provide an API. Instead of writing SQL to generate SQL, write SQL to export a YAML or JSON file of your users. Then write a script in bash, Python, or Node.js that loops through the file and calls the target service API to create a new user for each existing user. +- **Plugins** — Some of the more powerful services provide dedicated import plugins that will import your users for you, given a JSON file or database connection. This case is similar to the API one above, requiring you to write SQL to generate a specific JSON file of users. Services that do this tend to have cloud-only offerings, that don't allow you direct access to the database. + +TODO add brand examples to API and plugins + +From top to bottom, these options decrease in complexity, but also decrease in customization. In other words, if you have direct database access, you can map a lot more of your existing user data into the target system, but also have more work to do, and more chance of making database entries that cause errors in the service. + +## How To Handle Differing Password Hashing Algorithms + + ## Online Migration Techniques From e5251feb03fcaf567d34f7349fc79c22923ea1d2 Mon Sep 17 00:00:00 2001 From: RichardJECooke Date: Thu, 22 Aug 2024 11:43:20 +0200 Subject: [PATCH 15/32] add examples of which companies use which import techniques --- .../docs/lifecycle/migrate-users/offboard.mdx | 37 ++++++++++--------- 1 file changed, 20 insertions(+), 17 deletions(-) diff --git a/astro/src/content/docs/lifecycle/migrate-users/offboard.mdx b/astro/src/content/docs/lifecycle/migrate-users/offboard.mdx index 6704d6bfa3..0dab52d9a4 100644 --- a/astro/src/content/docs/lifecycle/migrate-users/offboard.mdx +++ b/astro/src/content/docs/lifecycle/migrate-users/offboard.mdx @@ -18,11 +18,11 @@ import Diagram2 from 'src/components/docs/extend/segmentDiagram2.astro'; - [Start A Sample Instance Of FusionAuth](#start-a-sample-instance-of-fusionauth) - [Browse The Database](#browse-the-database) - [Online And Offline Migrations](#online-and-offline-migrations) -- [Database Schema Documentation](#database-schema-documentation) - [What To Export?](#what-to-export) - [Export To Keycloak Example](#export-to-keycloak-example) - [How To Migrate](#how-to-migrate) - [How To Handle Differing Password Hashing Algorithms](#how-to-handle-differing-password-hashing-algorithms) +- [Social Logins like Facebook, Google, Or GitHub](#social-logins-like-facebook-google-or-github) - [Online Migration Techniques](#online-migration-techniques) ## Introduction @@ -34,10 +34,14 @@ This guide explains how to export your user data from FusionAuth to prepare it f You can follow along with this guide. You will start a new FusionAuth instance with a sample user and see how to export their data. If you already use FusionAuth, you can export your data from your existing instance. -To learn about authentication migration in general, please see the [generic migration guide](docs/lifecycle/migrate-users/genericmigration.mdx) and the [migration overview](docs/lifecycle/migrate-users). The principles in these guides that explain how to migrate **to** FusionAuth can also be used to migrate **from** FusionAuth to another service. It is important to be aware of the different migration strategies to choose from, especially online and offline migrations. +To learn about authentication migration in general, please see the [generic migration guide](docs/lifecycle/migrate-users/genericmigration.mdx) and the [migration overview](docs/lifecycle/migrate-users). The principles in these guides that explain how to migrate **to** FusionAuth can also be used to migrate **from** FusionAuth to another service. It is important to be aware of the different migration strategies to choose from, especially online and offline migrations (static versus dynamic migrations). + +FusionAuth might also have a specific guide on migrating from the service you want to migrate to, [here](/docs/lifecycle/migrate-users/provider-specific/). If so, check the steps in the guide to see if there are any important differences between the two services that you need to plan for. ## Start A Sample Instance Of FusionAuth @@ -63,26 +67,24 @@ Below is an example screenshot of the FusionAuth PostgreSQL database from the ex ## Online And Offline Migrations -Now that you know how to access your user data, you can write a database export script to extract the values you want and import them into your new authentication system. However, you need to consider new users, and users that update their details, after you have exported your database but before the users have joined your new system. +Now that you know how to access your data, you can write a database export script to extract the values you want and import them into your new authentication system. However, you need to consider new users, and users that update their details, after you have exported your database but before the users have joined your new system. You have three strategy options: - Offline migration: You take your application offline. Migrate your user data to the new system. Switch your application from pointing to FusionAuth for authentication to the new authentication system. Bring your application online again. This process could take from a few minutes to a few hours. - Online migration: You use webhooks and the FusionAuth API to migrate users individually from FA to the new system. As each user logs in, a webhook event fires, triggering a script you write to migrate the user's data and mark them as using the new system for future logins instead of FA. This is massively more complex than an offline migration. -- Hybrid migration: Perform an offline migration without taking your application offline. Use webhooks to send any information that changes during the migration process to update your new system. +- Hybrid migration: Perform an offline migration without taking your application offline. Use webhooks to send any information that changes during the migration process to update your new system. This is called dual-write — where you update changes to users in both FusionAuth and the new authentication service, until the time you disable FusionAuth completely. -An offline migration is the simplest option to perform for most systems, but you need to warn your users your system will be offline for an hour at a convenient time. You will have to use a form of online migration if your database is massive and your application cannot afford to ever be offline. +An offline migration is the simplest option to perform for most systems, but you need to warn your users your system will be offline for an hour at a convenient time. You will have to use a form of online migration if your database is huge and your application cannot afford to ever be offline. Whichever option you choose, you will need to test the process thoroughly before running it against the live application. -## Database Schema Documentation +## What To Export? FA does not have any dedicated documentation that explains the database schema. The schema is discussed briefly in this guide, and hopefully most tables are relatively obvious from their column names. If you need help understanding something in particular, please ask the FA programmers on the [Slack channel](https://fusionauth.io/community). -## What To Export? +Some data types, like users, applications, and roles, are used in most authentication services. But some data is so specific to FusionAuth there is no point in trying to migrate it with a script. This includes settings for webhooks, connectors, lambdas, and user actions. There is also no point in exporting logs like daily login counts and FusionAuth instance settings like themes, because your new authentication service won't use them. You will need to manually reproduce actions and styles like these in whatever format the new service specifies. -Some data, like users, applications, and roles, will probably be available in most authentication services. But some data is so specific to FusionAuth there is no point in trying to migrate it with a script. This includes settings for webhooks, connectors, lambdas, and user actions. There is no also point in exporting logs like daily login counts and FusionAuth instance settings like themes, because your new authentication service won't use them. - -To understand the user-related data tables in the database, you need to be familiar with [FusionAuth core concepts](/docs/get-started/core-concepts). Below is a visual summary of the object hierarchy. +To understand the user-related data tables in the database, you need to be familiar with [FusionAuth core concepts](/docs/get-started/core-concepts). Please read that guide before continuing. Below is a visual summary of the organisation. ```mermaid graph LR @@ -108,7 +110,7 @@ Here is the full list of FA database tables you should look at for export: `appl `identity_providers_tenants`, `tenants`, `user_comments`, `user_consents`, `user_consents_email_plus`, `user_registrations`, `user_registrations_application_roles`, `users`. ## Export To Keycloak Example @@ -123,16 +125,17 @@ docker run -p 9012:8080 -e KEYCLOAK_ADMIN=admin -e KEYCLOAK_ADMIN_PASSWORD=admin There are dozens of alternative authentication services to FusionAuth. This guide can offer only general advice to migrate to any of them. Each service allows you to import users (and sometimes other data) in one of three ways: -- **Direct database import through SQL statements** — These services tend to be free or open source. Examples include Keycloak, the Janssen project from Gluu, or Authelia. While free services don't have the funding to pay developers to write user-migration plugins, they allow you complete access to edit your database and insert whatever data you want. In these cases, you would write a SQL script against the FA database that generates a SQL script that is a set of INSERT statements that writes users to the target database. -- **API** — Services that are more sophisticated than needing you to write custom SQL, will provide an API. Instead of writing SQL to generate SQL, write SQL to export a YAML or JSON file of your users. Then write a script in bash, Python, or Node.js that loops through the file and calls the target service API to create a new user for each existing user. -- **Plugins** — Some of the more powerful services provide dedicated import plugins that will import your users for you, given a JSON file or database connection. This case is similar to the API one above, requiring you to write SQL to generate a specific JSON file of users. Services that do this tend to have cloud-only offerings, that don't allow you direct access to the database. - -TODO add brand examples to API and plugins +- **Direct database import through SQL statements** — These services tend to be free or open source. Examples include Keycloak, the Janssen project from Gluu, Authentic, or Authelia. While free services don't have the funding to pay developers to write user-migration plugins, they allow you complete access to edit your database and insert whatever data you want. In these cases, you would write a SQL script against the FA database that generates a SQL script that is a set of INSERT statements that writes users to the target database. +- **API** — Most services will also provide an API, allowing you to manage users from a terminal. Examples include AWS Cognito, Keycloak, Gluu/Janssen, Auth0, FrontEgg, Stytch, and Authentik. Instead of writing SQL to generate SQL, write SQL to export a JSON, or YAML file of your users. Then write a script in bash, Python, or Node.js that loops through the file and calls the target service API to create a new user for each existing user. +- **Plugins** — Some of the more powerful services provide dedicated import plugins that will import your users for you, given a CSV or JSON file or a database connection. Examples include AWS Cognito, Auth0, and Frontegg. This case is similar to the API one above, requiring you to write SQL to generate a specific JSON file of users. Services that do this tend to have cloud-only offerings, that don't allow you direct access to the database. From top to bottom, these options decrease in complexity, but also decrease in customization. In other words, if you have direct database access, you can map a lot more of your existing user data into the target system, but also have more work to do, and more chance of making database entries that cause errors in the service. ## How To Handle Differing Password Hashing Algorithms +PasswordHashProvider + +## Social Logins like Facebook, Google, Or GitHub ## Online Migration Techniques From 6757752a7a4929dc322d675a93ba191a89683512 Mon Sep 17 00:00:00 2001 From: RichardJECooke Date: Thu, 22 Aug 2024 11:47:46 +0200 Subject: [PATCH 16/32] edit --- astro/src/content/docs/lifecycle/migrate-users/offboard.mdx | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/astro/src/content/docs/lifecycle/migrate-users/offboard.mdx b/astro/src/content/docs/lifecycle/migrate-users/offboard.mdx index 0dab52d9a4..5841a61499 100644 --- a/astro/src/content/docs/lifecycle/migrate-users/offboard.mdx +++ b/astro/src/content/docs/lifecycle/migrate-users/offboard.mdx @@ -126,8 +126,8 @@ docker run -p 9012:8080 -e KEYCLOAK_ADMIN=admin -e KEYCLOAK_ADMIN_PASSWORD=admin There are dozens of alternative authentication services to FusionAuth. This guide can offer only general advice to migrate to any of them. Each service allows you to import users (and sometimes other data) in one of three ways: - **Direct database import through SQL statements** — These services tend to be free or open source. Examples include Keycloak, the Janssen project from Gluu, Authentic, or Authelia. While free services don't have the funding to pay developers to write user-migration plugins, they allow you complete access to edit your database and insert whatever data you want. In these cases, you would write a SQL script against the FA database that generates a SQL script that is a set of INSERT statements that writes users to the target database. -- **API** — Most services will also provide an API, allowing you to manage users from a terminal. Examples include AWS Cognito, Keycloak, Gluu/Janssen, Auth0, FrontEgg, Stytch, and Authentik. Instead of writing SQL to generate SQL, write SQL to export a JSON, or YAML file of your users. Then write a script in bash, Python, or Node.js that loops through the file and calls the target service API to create a new user for each existing user. -- **Plugins** — Some of the more powerful services provide dedicated import plugins that will import your users for you, given a CSV or JSON file or a database connection. Examples include AWS Cognito, Auth0, and Frontegg. This case is similar to the API one above, requiring you to write SQL to generate a specific JSON file of users. Services that do this tend to have cloud-only offerings, that don't allow you direct access to the database. +- **API** — Most services will also provide an API, allowing you to manage users from a terminal. Examples include AWS Cognito, Firebase, Keycloak, Gluu/Janssen, Auth0, FrontEgg, Stytch, and Authentik. Instead of writing SQL to generate SQL, write SQL to export a JSON, or YAML file of your users. Then write a script in bash, Python, or Node.js that loops through the file and calls the target service API to create a new user for each existing user. +- **Plugins** — Some of the more powerful services provide dedicated import plugins that will import your users for you, given a CSV or JSON file or a database connection. Examples include AWS Cognito, Auth0, and Frontegg. This case is similar to the API one above, requiring you to write SQL to generate a file of users. Services that do this tend to have cloud-only offerings, that don't allow you direct access to the database. From top to bottom, these options decrease in complexity, but also decrease in customization. In other words, if you have direct database access, you can map a lot more of your existing user data into the target system, but also have more work to do, and more chance of making database entries that cause errors in the service. From 6deb8a0952719140e4c8f689324a599941728dab Mon Sep 17 00:00:00 2001 From: RichardJECooke Date: Thu, 22 Aug 2024 13:24:09 +0200 Subject: [PATCH 17/32] finish sql export example --- .../docs/lifecycle/migrate-users/offboard.mdx | 109 ++++++++++++++++-- 1 file changed, 99 insertions(+), 10 deletions(-) diff --git a/astro/src/content/docs/lifecycle/migrate-users/offboard.mdx b/astro/src/content/docs/lifecycle/migrate-users/offboard.mdx index 5841a61499..d4a1f50d3f 100644 --- a/astro/src/content/docs/lifecycle/migrate-users/offboard.mdx +++ b/astro/src/content/docs/lifecycle/migrate-users/offboard.mdx @@ -19,8 +19,10 @@ import Diagram2 from 'src/components/docs/extend/segmentDiagram2.astro'; - [Browse The Database](#browse-the-database) - [Online And Offline Migrations](#online-and-offline-migrations) - [What To Export?](#what-to-export) -- [Export To Keycloak Example](#export-to-keycloak-example) - [How To Migrate](#how-to-migrate) +- [Example SQL](#example-sql) + - [Direct Database Import Through SQL Statements](#direct-database-import-through-sql-statements) + - [API or Plugins](#api-or-plugins) - [How To Handle Differing Password Hashing Algorithms](#how-to-handle-differing-password-hashing-algorithms) - [Social Logins like Facebook, Google, Or GitHub](#social-logins-like-facebook-google-or-github) - [Online Migration Techniques](#online-migration-techniques) @@ -113,14 +115,6 @@ Here is the full list of FA database tables you should look at for export: `appl If you want to browse the diagram above in DBeaver, you can download the ERD file [here](../../../../../public/img/docs/lifecycle/migrate-users/offboard/databaseDiagram.erd). -## Export To Keycloak Example - -```sh -docker run -p 9012:8080 -e KEYCLOAK_ADMIN=admin -e KEYCLOAK_ADMIN_PASSWORD=admin quay.io/keycloak/keycloak:25.0.4 start-dev -``` -- browse to localhost:9012 -- tenant -> realm. application -> client. - ## How To Migrate There are dozens of alternative authentication services to FusionAuth. This guide can offer only general advice to migrate to any of them. Each service allows you to import users (and sometimes other data) in one of three ways: @@ -129,7 +123,102 @@ There are dozens of alternative authentication services to FusionAuth. This guid - **API** — Most services will also provide an API, allowing you to manage users from a terminal. Examples include AWS Cognito, Firebase, Keycloak, Gluu/Janssen, Auth0, FrontEgg, Stytch, and Authentik. Instead of writing SQL to generate SQL, write SQL to export a JSON, or YAML file of your users. Then write a script in bash, Python, or Node.js that loops through the file and calls the target service API to create a new user for each existing user. - **Plugins** — Some of the more powerful services provide dedicated import plugins that will import your users for you, given a CSV or JSON file or a database connection. Examples include AWS Cognito, Auth0, and Frontegg. This case is similar to the API one above, requiring you to write SQL to generate a file of users. Services that do this tend to have cloud-only offerings, that don't allow you direct access to the database. -From top to bottom, these options decrease in complexity, but also decrease in customization. In other words, if you have direct database access, you can map a lot more of your existing user data into the target system, but also have more work to do, and more chance of making database entries that cause errors in the service. +From top to bottom, these options decrease in complexity, but also decrease in customization. In other words, if you have direct database access, you can map more of your existing user data into the target system, but you also have more work to do, and more chance of making database entries that cause errors in the service. + +## Example SQL + +Let's look at an example SQL queries for each of the three approaches in the previous section. Consider exporting a list of users, containing: user Id, first name, email, password, hash, salt, and encryption scheme. You can see from the database diagram shown earlier that you need to join the `users` table on the `identities` table to find these fields. + +Below is the SQL to get this data. + +```sql +SELECT + u.id, u.first_name, + i.email, i.encryption_scheme, i.password, i.salt, i.factor +FROM + identities AS i + JOIN users AS u + ON u.id = i.users_id + +-- OUTPUT: + +id | first_name| email | encryption_scheme | password | salt | factor + +00000000-0000-0000-0000-111111111111 | Fred | richard@example.com |salted-pbkdf2-hmac-sha256|ULoj1fuENZ+QvRqoaOhZ2YX6vuI7uqi7pY0a1EcE32Q= |u8ikPE4m35czpQArp2lDLYDGpIIo+FC+wiNzCclLRbw= | 24000 + +00000000-0000-0000-0000-000000000001 | Dinesh | admin@example.com |salted-pbkdf2-hmac-sha256|VBSc35CHt/4udxuL+ctb+MY+inWUGr4gMvZSwhvJ8iI= |l5LWrb6/YBIR3USJTFdDHIGYBaDWvqN1uqSRGhDfQHM= | 24000 + +a58ebb5e-a207-4653-824c-b7f41a73c63c | Test | test@example.com |salted-pbkdf2-hmac-sha256|3JK/aB+MBHtFXvxCvoVqZ4cl5wTkiV843dwA/HKGKBM= |s1ElWordVlDuCjuy1rhHz5i2GPdGp9NcVCfx+jSFRic= | 24000 +``` + +### Direct Database Import Through SQL Statements + +To change this SQL query to generate SQL insert queries, use the format below. Here you assume that the database you're writing into has a single table called `user`. + +```sql +SELECT format +( + 'INSERT INTO user (id, first_name, email, encryption_scheme, password, salt, factor) VALUES (%L, %L, %L, %L, %L, %L, %s);', + u.id, u.first_name, i.email, i.encryption_scheme, i.password, i.salt, i.factor +) +FROM + identities AS i +JOIN + users AS u ON + u.id = i.users_id; + +-- OUTPUT: + +format + +INSERT INTO user (id, first_name, email, encryption_scheme, password, salt, factor) VALUES ('00000000-0000-0000-0000-111111111111', 'Fred', 'richard@example.com', 'salted-pbkdf2-hmac-sha256', 'ULoj1fuENZ+QvRqoaOhZ2YX6vuI7uqi7pY0a1EcE32Q=', 'u8ikPE4m35czpQArp2lDLYDGpIIo+FC+wiNzCclLRbw=', 24000); + +INSERT INTO user (id, first_name, email, encryption_scheme, password, salt, factor) VALUES ('00000000-0000-0000-0000-000000000001', 'Dinesh', 'admin@example.com', 'salted-pbkdf2-hmac-sha256', 'VBSc35CHt/4udxuL+ctb+MY+inWUGr4gMvZSwhvJ8iI=', 'l5LWrb6/YBIR3USJTFdDHIGYBaDWvqN1uqSRGhDfQHM=', 24000); + +INSERT INTO user (id, first_name, email, encryption_scheme, password, salt, factor) VALUES ('a58ebb5e-a207-4653-824c-b7f41a73c63c', 'Test', 'test@example.com', 'salted-pbkdf2-hmac-sha256', '3JK/aB+MBHtFXvxCvoVqZ4cl5wTkiV843dwA/HKGKBM=', 's1ElWordVlDuCjuy1rhHz5i2GPdGp9NcVCfx+jSFRic=', 24000); +``` + +You can now save the output of your this to a SQL file and then run that file on your new database. + +### API or Plugins + +To generate a JSON file to give to user import plugins, or to loop through in your own code that calls the new service's API, run the original SQL query from the start of this section. In the DBeaver query output at the bottom of this screen, click `Export data` and follow the wizard, choosing `JSON` as the output format. + +```JSON +{ +"SELECT \n\tu.id, u.first_name,\n\ti.email, i.encryption_scheme, i.password, i.salt, i.factor \nFROM \n\tidentities AS i\n\tJOIN users AS u \n\t\tON u.id = i.users_id": [ + { + "id" : "00000000-0000-0000-0000-111111111111", + "first_name" : "Fred", + "email" : "richard@example.com", + "encryption_scheme" : "salted-pbkdf2-hmac-sha256", + "password" : "ULoj1fuENZ+QvRqoaOhZ2YX6vuI7uqi7pY0a1EcE32Q=", + "salt" : "u8ikPE4m35czpQArp2lDLYDGpIIo+FC+wiNzCclLRbw=", + "factor" : 24000 + }, + { + "id" : "00000000-0000-0000-0000-000000000001", + "first_name" : "Dinesh", + "email" : "admin@example.com", + "encryption_scheme" : "salted-pbkdf2-hmac-sha256", + "password" : "VBSc35CHt\/4udxuL+ctb+MY+inWUGr4gMvZSwhvJ8iI=", + "salt" : "l5LWrb6\/YBIR3USJTFdDHIGYBaDWvqN1uqSRGhDfQHM=", + "factor" : 24000 + }, + { + "id" : "a58ebb5e-a207-4653-824c-b7f41a73c63c", + "first_name" : "Test", + "email" : "test@example.com", + "encryption_scheme" : "salted-pbkdf2-hmac-sha256", + "password" : "3JK\/aB+MBHtFXvxCvoVqZ4cl5wTkiV843dwA\/HKGKBM=", + "salt" : "s1ElWordVlDuCjuy1rhHz5i2GPdGp9NcVCfx+jSFRic=", + "factor" : 24000 + } +]} +``` + +If you need to map FA columns to columns in the new database, use SQL syntax like: `SELECT u.first_name as name`. Or you could do the mapping in the Python or Node.js code itself. + ## How To Handle Differing Password Hashing Algorithms From d503ad5c2e3a860ec34b9ccc96027302eeebfbd1 Mon Sep 17 00:00:00 2001 From: RichardJECooke Date: Thu, 22 Aug 2024 13:28:20 +0200 Subject: [PATCH 18/32] edit --- astro/src/content/docs/lifecycle/migrate-users/offboard.mdx | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/astro/src/content/docs/lifecycle/migrate-users/offboard.mdx b/astro/src/content/docs/lifecycle/migrate-users/offboard.mdx index d4a1f50d3f..89e6904627 100644 --- a/astro/src/content/docs/lifecycle/migrate-users/offboard.mdx +++ b/astro/src/content/docs/lifecycle/migrate-users/offboard.mdx @@ -117,7 +117,7 @@ If you want to browse the diagram above in DBeaver, you can download the ERD fil ## How To Migrate -There are dozens of alternative authentication services to FusionAuth. This guide can offer only general advice to migrate to any of them. Each service allows you to import users (and sometimes other data) in one of three ways: +There are dozens of alternative authentication services to FusionAuth. This guide can offer only general advice to migrate to any of them. Ultimately, migration is the process of copying data about your users from the FA database in its format, to the new system's database in its format, mapping tables and columns appropriately. Each service allows you to import users (and sometimes other data) in one of three ways: - **Direct database import through SQL statements** — These services tend to be free or open source. Examples include Keycloak, the Janssen project from Gluu, Authentic, or Authelia. While free services don't have the funding to pay developers to write user-migration plugins, they allow you complete access to edit your database and insert whatever data you want. In these cases, you would write a SQL script against the FA database that generates a SQL script that is a set of INSERT statements that writes users to the target database. - **API** — Most services will also provide an API, allowing you to manage users from a terminal. Examples include AWS Cognito, Firebase, Keycloak, Gluu/Janssen, Auth0, FrontEgg, Stytch, and Authentik. Instead of writing SQL to generate SQL, write SQL to export a JSON, or YAML file of your users. Then write a script in bash, Python, or Node.js that loops through the file and calls the target service API to create a new user for each existing user. @@ -178,7 +178,7 @@ INSERT INTO user (id, first_name, email, encryption_scheme, password, salt, fact INSERT INTO user (id, first_name, email, encryption_scheme, password, salt, factor) VALUES ('a58ebb5e-a207-4653-824c-b7f41a73c63c', 'Test', 'test@example.com', 'salted-pbkdf2-hmac-sha256', '3JK/aB+MBHtFXvxCvoVqZ4cl5wTkiV843dwA/HKGKBM=', 's1ElWordVlDuCjuy1rhHz5i2GPdGp9NcVCfx+jSFRic=', 24000); ``` -You can now save the output of your this to a SQL file and then run that file on your new database. +You can now save the output to a SQL file and then run that file on your new database. ### API or Plugins @@ -219,7 +219,6 @@ To generate a JSON file to give to user import plugins, or to loop through in yo If you need to map FA columns to columns in the new database, use SQL syntax like: `SELECT u.first_name as name`. Or you could do the mapping in the Python or Node.js code itself. - ## How To Handle Differing Password Hashing Algorithms PasswordHashProvider From 225b43f2c38a1215a04d11421b95232335e05031 Mon Sep 17 00:00:00 2001 From: RichardJECooke Date: Thu, 22 Aug 2024 14:53:48 +0200 Subject: [PATCH 19/32] explain hashing --- .../docs/lifecycle/migrate-users/offboard.mdx | 15 +++++++++++++-- 1 file changed, 13 insertions(+), 2 deletions(-) diff --git a/astro/src/content/docs/lifecycle/migrate-users/offboard.mdx b/astro/src/content/docs/lifecycle/migrate-users/offboard.mdx index 89e6904627..2c70dcbf19 100644 --- a/astro/src/content/docs/lifecycle/migrate-users/offboard.mdx +++ b/astro/src/content/docs/lifecycle/migrate-users/offboard.mdx @@ -117,7 +117,7 @@ If you want to browse the diagram above in DBeaver, you can download the ERD fil ## How To Migrate -There are dozens of alternative authentication services to FusionAuth. This guide can offer only general advice to migrate to any of them. Ultimately, migration is the process of copying data about your users from the FA database in its format, to the new system's database in its format, mapping tables and columns appropriately. Each service allows you to import users (and sometimes other data) in one of three ways: +There are dozens of alternative authentication services to FusionAuth. This guide can offer only general advice to migrate to any of them. Ultimately, migration is the process of copying data about your users from your database in the FA format, to the new system's database in its format, mapping tables and columns appropriately. Each service allows you to import users (and sometimes other data) in different ways: - **Direct database import through SQL statements** — These services tend to be free or open source. Examples include Keycloak, the Janssen project from Gluu, Authentic, or Authelia. While free services don't have the funding to pay developers to write user-migration plugins, they allow you complete access to edit your database and insert whatever data you want. In these cases, you would write a SQL script against the FA database that generates a SQL script that is a set of INSERT statements that writes users to the target database. - **API** — Most services will also provide an API, allowing you to manage users from a terminal. Examples include AWS Cognito, Firebase, Keycloak, Gluu/Janssen, Auth0, FrontEgg, Stytch, and Authentik. Instead of writing SQL to generate SQL, write SQL to export a JSON, or YAML file of your users. Then write a script in bash, Python, or Node.js that loops through the file and calls the target service API to create a new user for each existing user. @@ -221,11 +221,22 @@ If you need to map FA columns to columns in the new database, use SQL syntax lik ## How To Handle Differing Password Hashing Algorithms -PasswordHashProvider +The major difficulty migrating between any authentication services is how each service handles password hashing. + +Passwords are not stored in plaintext in databases. Instead, the password is irreversibly hashed into a sequence of bytes. Sometimes the password is first combined with random bytes, called a salt, before hashing. The salt must be kept with the hash. There are a multitude of different algorithms that hash passwords. Some algorithms include the salt with the hash. + +By default FA uses [Salted PBKDF2 HMAC SHA-256](/docs/reference/password-hashes#salted-pbkdf2-hmac-sha-256). If the new service also uses this hash algorithm, you have no work to do. If the new service uses a different algorithm, you have two choices: +- Do not migrate passwords. Require all your users to "change" their password before logging in, through the forgot password email mechanism. This requires little work from you, but is poor user experience. +- Set the hashing algorithm field for the user in the new service, if the service supports the algorithm. If the service doesn't, it's possible that the service supports hashing extensions. This means that you can write your own code that implements the hash algorithm to check the user's password at login, and add it to the service as an extension. For example, here's how [custom hashing works](https://fusionauth.io/docs/extend/code/password-hashes/custom-password-hashing) in FA. Check if your new service supports something similar. + +A lot of services also have the option to overwrite the hash and algorithm of the migrated user with the service's default algorithm after first login. You should enable this so all users are consistently stored. + +Finally, be aware that FusionAuth users can have different hashing algorithms — don't assume all users use PBKDF2, check the `encryption_scheme` field. This difference will occur only if FusionAuth users were migrated into the database from another system, or if a previous administrator configured FusionAuth to use an algorithm differing from the default. ## Social Logins like Facebook, Google, Or GitHub + ## Online Migration Techniques https://fusionauth.io/docs/apis/ From 323d6f73543a8cf173ad5ead6d4bf18fcec7a120 Mon Sep 17 00:00:00 2001 From: RichardJECooke Date: Thu, 22 Aug 2024 14:56:41 +0200 Subject: [PATCH 20/32] edit --- astro/src/content/docs/lifecycle/migrate-users/offboard.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/astro/src/content/docs/lifecycle/migrate-users/offboard.mdx b/astro/src/content/docs/lifecycle/migrate-users/offboard.mdx index 2c70dcbf19..a983aa1b4d 100644 --- a/astro/src/content/docs/lifecycle/migrate-users/offboard.mdx +++ b/astro/src/content/docs/lifecycle/migrate-users/offboard.mdx @@ -229,7 +229,7 @@ By default FA uses [Salted PBKDF2 HMAC SHA-256](/docs/reference/password-hashes# - Do not migrate passwords. Require all your users to "change" their password before logging in, through the forgot password email mechanism. This requires little work from you, but is poor user experience. - Set the hashing algorithm field for the user in the new service, if the service supports the algorithm. If the service doesn't, it's possible that the service supports hashing extensions. This means that you can write your own code that implements the hash algorithm to check the user's password at login, and add it to the service as an extension. For example, here's how [custom hashing works](https://fusionauth.io/docs/extend/code/password-hashes/custom-password-hashing) in FA. Check if your new service supports something similar. -A lot of services also have the option to overwrite the hash and algorithm of the migrated user with the service's default algorithm after first login. You should enable this so all users are consistently stored. +A lot of services also have the option to overwrite the hash and algorithm of the migrated user with the service's default algorithm after first login. You should enable this, so all users are consistently stored. Finally, be aware that FusionAuth users can have different hashing algorithms — don't assume all users use PBKDF2, check the `encryption_scheme` field. This difference will occur only if FusionAuth users were migrated into the database from another system, or if a previous administrator configured FusionAuth to use an algorithm differing from the default. From bec98d6fc09243d8a08e0e9529e0e16ff3d734bd Mon Sep 17 00:00:00 2001 From: RichardJECooke Date: Thu, 22 Aug 2024 17:32:06 +0200 Subject: [PATCH 21/32] article finished. must proofread and edit --- .../docs/lifecycle/migrate-users/offboard.mdx | 41 +++++++++++-------- 1 file changed, 24 insertions(+), 17 deletions(-) diff --git a/astro/src/content/docs/lifecycle/migrate-users/offboard.mdx b/astro/src/content/docs/lifecycle/migrate-users/offboard.mdx index a983aa1b4d..ec899f44fb 100644 --- a/astro/src/content/docs/lifecycle/migrate-users/offboard.mdx +++ b/astro/src/content/docs/lifecycle/migrate-users/offboard.mdx @@ -24,7 +24,7 @@ import Diagram2 from 'src/components/docs/extend/segmentDiagram2.astro'; - [Direct Database Import Through SQL Statements](#direct-database-import-through-sql-statements) - [API or Plugins](#api-or-plugins) - [How To Handle Differing Password Hashing Algorithms](#how-to-handle-differing-password-hashing-algorithms) -- [Social Logins like Facebook, Google, Or GitHub](#social-logins-like-facebook-google-or-github) +- [Social Logins like Facebook And Google](#social-logins-like-facebook-and-google) - [Online Migration Techniques](#online-migration-techniques) ## Introduction @@ -53,8 +53,8 @@ In this section you will run a new instance of self-hosted FusionAuth with Docke - Install [Docker](https://docs.docker.com/get-docker/) if you don't have it on your machine. - Clone the [FusionAuth example Docker Compose repository](https://github.com/FusionAuth/fusionauth-example-docker-compose) to your computer. - In your terminal, navigate to the `light` directory in the repository. -- Run `docker compose up` to start FA. -- Browse to http://localhost:9011 to check that FA is running. You can log in with `admin@example.com` and `password`. +- Run `docker compose up` to start FusionAuth. +- Browse to http://localhost:9011 to check that FusionAuth is running. You can log in with `admin@example.com` and `password`. - Note the database connection details in the `docker-compose.yml` file and the hidden `.env` file. ## Browse The Database @@ -73,7 +73,7 @@ Now that you know how to access your data, you can write a database export scrip You have three strategy options: - Offline migration: You take your application offline. Migrate your user data to the new system. Switch your application from pointing to FusionAuth for authentication to the new authentication system. Bring your application online again. This process could take from a few minutes to a few hours. -- Online migration: You use webhooks and the FusionAuth API to migrate users individually from FA to the new system. As each user logs in, a webhook event fires, triggering a script you write to migrate the user's data and mark them as using the new system for future logins instead of FA. This is massively more complex than an offline migration. +- Online migration: You use webhooks and the FusionAuth API to migrate users individually from FusionAuth to the new system. As each user logs in, a webhook event fires, triggering a script you write to migrate the user's data and mark them as using the new system for future logins instead of FusionAuth. This is massively more complex than an offline migration. - Hybrid migration: Perform an offline migration without taking your application offline. Use webhooks to send any information that changes during the migration process to update your new system. This is called dual-write — where you update changes to users in both FusionAuth and the new authentication service, until the time you disable FusionAuth completely. An offline migration is the simplest option to perform for most systems, but you need to warn your users your system will be offline for an hour at a convenient time. You will have to use a form of online migration if your database is huge and your application cannot afford to ever be offline. @@ -82,7 +82,7 @@ Whichever option you choose, you will need to test the process thoroughly before ## What To Export? -FA does not have any dedicated documentation that explains the database schema. The schema is discussed briefly in this guide, and hopefully most tables are relatively obvious from their column names. If you need help understanding something in particular, please ask the FA programmers on the [Slack channel](https://fusionauth.io/community). +FusionAuth does not have any dedicated documentation that explains the database schema. The schema is discussed briefly in this guide, and hopefully most tables are relatively obvious from their column names. If you need help understanding something in particular, please ask the FusionAuth programmers on the [Slack channel](https://fusionauth.io/community). Some data types, like users, applications, and roles, are used in most authentication services. But some data is so specific to FusionAuth there is no point in trying to migrate it with a script. This includes settings for webhooks, connectors, lambdas, and user actions. There is also no point in exporting logs like daily login counts and FusionAuth instance settings like themes, because your new authentication service won't use them. You will need to manually reproduce actions and styles like these in whatever format the new service specifies. @@ -108,7 +108,7 @@ Below is the full database diagram for the tables you need to export. Though it ![Database diagram](../../../../../public/img/docs/lifecycle/migrate-users/offboard/databaseDiagram.svg) -Here is the full list of FA database tables you should look at for export: `application_roles`, `applications`, `consents`, `email_templates`, `group_application_roles`, `group_members`, `groups`, `identities`, `identity_provider_links`, `identity_providers`, `identity_providers_applications` +Here is the full list of FusionAuth database tables you should look at for export: `application_roles`, `applications`, `consents`, `email_templates`, `group_application_roles`, `group_members`, `groups`, `identities`, `identity_provider_links`, `identity_providers`, `identity_providers_applications` `identity_providers_tenants`, `tenants`, `user_comments`, `user_consents`, `user_consents_email_plus`, `user_registrations`, `user_registrations_application_roles`, `users`.