Observe allocated deltas in system docs (#808)

* Peak memory observation in the system * Docs improvements (#799) * Update storage info where needed * Update best practics * Add total unreleased amount of deltas to show storage info * Add best practice for delta allocation * Add additional info * Add NuRaft log file flag (#816) * Add health checks docs (#833) * Update HA examples * Add information * Add NuRaft log file env (#848) * Add leader/follower role (#857) Co-authored-by: Antonio Filipovic <61245998+antoniofilipovic@users.noreply.github.com> * Add fix * Add coordinator hostname HA configuration option (#860) * First leader * Update pages/clustering/high-availability.mdx Co-authored-by: Katarina Supe <61758502+katarinasupe@users.noreply.github.com> --------- Co-authored-by: Kruno Golubic <46486712+kgolubic@users.noreply.github.com> Co-authored-by: Katarina Supe <61758502+katarinasupe@users.noreply.github.com> * Add monitoring at runtime docs (#806) * database-less connections update (#853) * Update HA docs * Add durability for coordinator (#859) * add durability docs * Update high-availability.mdx --------- Co-authored-by: Kruno Golubic <46486712+kgolubic@users.noreply.github.com> * Edge type property index (#855) * Add hops limit docs (#881) * add docs for hops limit * revert accidentally changed part of the docs * fix grammar mistakes * implement suggestions * Peak memory observation in the system (#788) * Peak memory observation in the system * Update storage info where needed * Update best practics * Add information from PR review --------- Co-authored-by: Kruno Golubic <46486712+kgolubic@users.noreply.github.com> * removed flags (#854) * Update envs for HA (#878) * fix envs * add 3 missing envs * Update configuration.mdx * Update configuration.mdx --------- Co-authored-by: Kruno Golubic <46486712+kgolubic@users.noreply.github.com> * Add callout for important force reset notice (#877) * add callout for instance to be alive on force reset * Update high-availability.mdx * Enum datatype (v1) (#852) * Add information around enums * Add ALTER to the list of clauses * Add ALTER to the list of Memgraph's Cypher extension * Fix incorrect ALTER command * Update pages/querying/clauses/create.mdx * Update pages/querying/clauses/alter.mdx * Update pages/querying/clauses/alter.mdx * Update graph-modeling.md * Address PR comments --------- Co-authored-by: kgolubic <kgolubic@gmail.com> Co-authored-by: Kruno Golubic <46486712+kgolubic@users.noreply.github.com> * SSO Core - OIDC + OAuth2.0 docs (#876) * wip oidc docs * oidc sso docs * PR changes and update * update package info * add startup info * Update v2.18 experimental feature statuses (#871) * Update v2.18 experimental feature statuses * Join the sentances into paragraphs * Update language and formatting --------- Co-authored-by: Kruno Golubic <46486712+kgolubic@users.noreply.github.com> Co-authored-by: kgolubic <kgolubic@gmail.com> * Fix from PR * Update README.md * Add metrics info docs (#809) --------- Co-authored-by: Andi <andi8647@gmail.com> Co-authored-by: Kruno Golubic <46486712+kgolubic@users.noreply.github.com> Co-authored-by: kgolubic <kgolubic@gmail.com> Co-authored-by: Antonio Filipovic <61245998+antoniofilipovic@users.noreply.github.com> Co-authored-by: Katarina Supe <61758502+katarinasupe@users.noreply.github.com> Co-authored-by: andrejtonev <29177572+andrejtonev@users.noreply.github.com> Co-authored-by: Aidar Samerkhanov <aidar.samerkhanov@memgraph.io> Co-authored-by: David Ivekovic <david.ivekovic@memgraph.io> Co-authored-by: Gareth Andrew Lloyd <gareth@ignition-web.co.uk> Co-authored-by: Ivan Milinović <44698587+imilinovic@users.noreply.github.com> Co-authored-by: Marko Budiselić <marko.budiselic@memgraph.com>
memgraph · Jul 17, 2024 · e458555 · e458555
1 parent 118523c
commit e458555
Show file tree

Hide file tree

Showing 8 changed files with 236 additions and 25 deletions.
diff --git a/README.md b/README.md
@@ -95,14 +95,19 @@ an [issue](https://github.com/memgraph/documentation/issues).
 
 ### Contributing guide
 
-If you want to change the documentation, create a new branch and make
-the appropriate changes. Then, create a pull request to merge these changes into the
+If you want to change the documentation, create a new branch and make the
+appropriate changes. Then, create a pull request to merge these changes into the
 `main` branch.
 
-The pull request should describe the changes it's proposing and all checks must be completed. 
+The pull request should describe the changes it's proposing and all checks must
+be completed. 
 
-Add an appropriate label to the PR, either `status: draft` if you are still working on the PR, or `status: ready` if the PR is ready for review. 
+Add an appropriate label to the PR, either `status: draft` if you are still
+working on the PR, or `status: ready` if the PR is ready for review. 
 
-When the PR is reviewed and approved, the label will be changed to `status: ship it` and merged into the main by the repo admins.
+When the PR is reviewed and approved, the label will be changed to `status: ship
+it` and merged into the main by the repo admins.
 
-If the PR requires changes, the label will be changed to `status: change`. Address the comments and change the documentation appropriately, then re-request a review and change the label to `status: ready` again. 
+If the PR requires changes, the label will be changed to `status: change`.
+Address the comments and change the documentation appropriately, then re-request
+a review and change the label to `status: ready` again. 
diff --git a/pages/data-migration/best-practices.mdx b/pages/data-migration/best-practices.mdx
@@ -559,6 +559,8 @@ It will help you diagnose bottlenecks and high-memory queries which can be optim
 That is most common in import queries because users would want to import the whole dataset with one command only.
 
 For more information, check our [storage memory usage](/fundamentals/storage-memory-usage).
+For more information about `Delta` objects, check the 
+information on the [IN_MEMORY_TRANSACTIONAL storage mode](/fundamentals/storage-memory-usage#in-memory-transactional-storage-mode-default).
 
 ## Do you have more questions?
 

diff --git a/...tabase-management/authentication-and-authorization/auth-system-integrations.mdx b/...tabase-management/authentication-and-authorization/auth-system-integrations.mdx
@@ -602,3 +602,89 @@ allowed to log in because her role (moderator) already exists. Carol and Dave
 won't be allowed to log in because their role (administrator) doesn't exist.
 
 </Steps>
+
+## OpenID Connect (OIDC)
+
+Memgraph also supports authentication and authorization using OIDC (OIDC + OAuth 2.0) protocol with a
+built-in auth module that is packaged with Memgraph Enterprise.
+
+The module currently supports the following SSO providers:
+- Microsoft Entra ID
+- Okta using a custom authorization server
+
+To use the built-in auth module run Memgraph with the `--auth-module-mappings` [flag](/database-management/configuration#auth-module) with the name of each provider and protocol you want to use separater with a semicolon (;).
+For example, to use OIDC with both Entra ID and Okta run Memgraph with `--auth-module-mappings=oidc-okta;oidc-entra-id`.
+
+For SSO provider setup see [this](/data-visualization/user-manual/single-sign-on).
+
+### Module requirements
+
+The module is written in Python 3 and requires the following libraries:
+ - `PyJWT` - used to validate tokens.
+ - `requests` - used to retrieve public keys for validating tokens.
+ - `cryptography` - used to parse tokens.
+
+The required packages are already installed in the docker image. If you are using a native build you to install the packages manually on your machine.
+The list of all required packages is located at `src/auth/reference_modules/requirements.txt`. You can install them by running `pip3 install -r /path/to/requirements.txt`.
+If you are running a native build make sure that you don't have the Python `JWT` library installed (you can check that with `pip freeze`) as `PyJWT` and `JWT` have some conflicts.
+
+### Module configuration
+
+The module is located at `/usr/lib/memgraph/auth_module/oidc.py`.
+The module has no configuration file. You have to run Memgraph with some environmental variables.
+To use multiple SSO providers you have to set the environmental variables associated with that provider.
+
+#### MS Entra ID
+For using OIDC with Microsoft Entra ID you have to set the following environmental variables:
+
+```bash
+MEMGRAPH_SSO_ENTRA_ID_OIDC_TENANT_ID=
+MEMGRAPH_SSO_ENTRA_ID_OIDC_CLIENT_ID=
+MEMGRAPH_SSO_ENTRA_ID_OIDC_ROLE_MAPPING=
+MEMGRAPH_SSO_ENTRA_ID_OIDC_USERNAME=
+```
+
+You can find the tenant and client ID in the app overview in Azure. For a more detailed explanation see [this](/pages/data-visualization/user-manual/single-sign-on.mdx).
+
+##### Role mapping
+Authorization is done by mapping the Entra ID roles from the roles field in the access token to the Memgraph roles.
+The role mapping is defined as a string where each individual mapping is separated by a semicolon (;). Within each mapping, the Entra ID role is separated from the corresponding Memgraph role by a colon (:).
+One example of a role mapping is `entra.admin:memadmin; entra.user:memuser` where the Entra ID role `entra.admin` is mapped to the Memgraph role `memadmin` and the Entra ID role `entra.user` is mapped to the Memgraph role `memuser`.
+Make sure that each user only has one role on Entra ID.
+
+##### Username
+The username variable tells the OIDC module what to use as the username. It has the format `token-type:field`. 
+Token type can be `id` or `access` depending on whether you want to use a field from the access or the ID token for the username. See the following to learn more about [access](https://learn.microsoft.com/en-us/entra/identity-platform/access-tokens) and [id](https://learn.microsoft.com/en-us/entra/identity-platform/id-tokens) tokens.
+By default, it is set to `id:sub` as per the OIDC protocol it is recommended to use the `sub` field from the id token as it is non-mutable and globally unique for each application.
+For MS Entra ID one commonly used field is `id:preferred_username` which is usually the email of the user. For a list of all default claims look at [this](https://learn.microsoft.com/en-us/entra/identity-platform/access-token-claims-reference) and [this](https://learn.microsoft.com/en-us/entra/identity-platform/id-token-claims-reference).
+You can also configure custom claims.
+
+#### Okta
+For using OIDC with Okta you have to set the following environmental variables:
+```bash
+MEMGRAPH_SSO_OKTA_OIDC_ISSUER=
+MEMGRAPH_SSO_OKTA_OIDC_CLIENT_ID=
+MEMGRAPH_SSO_OKTA_OIDC_AUTHORIZATION_SERVER=api://default
+MEMGRAPH_SSO_OKTA_OIDC_ROLE_MAPPING=
+MEMGRAPH_SSO_OKTA_OIDC_USERNAME=
+```
+
+Issuer is `https://{your-okta-domain}.okta.com/oauth2/default/`. You can find the client ID on the Admin panel -> Applications -> General. You can find the authorization server on the Admin panel -> Security -> API -> Authorization Servers -> Audience. By default, it is set to `api://default`.
+
+##### Role mapping
+Authorization is done by mapping the Okta groups from the `groups` field in the access token to the Memgraph roles.
+The role mapping is defined as a string where each individual mapping is separated by a semicolon (;). Within each mapping, the Entra ID role is separated from the corresponding Memgraph role by a colon (:).
+One example of a role mapping is `okta.admin:memadmin; okta.user:memuser` where the Okta group `okta.admin` is mapped to the Memgraph role `memadmin` and the Okta group `okta.user` is mapped to the Memgraph role `memuser`.
+The `groups` field does not exist in Okta by default and you have to set it up manually. More information on how to do that [here](https://developer.okta.com/docs/guides/customize-tokens-returned-from-okta/main/#add-a-custom-claim-to-a-token).
+Make sure that each user is only assigned to one group on Okta.
+
+##### Username 
+The username variable tells the OIDC module what to use as the username. It has the format `token-type:field`. 
+Token type can be `id` or `access` depending on whether you want to use a field from the access or the ID token for the username. See the following to learn more about [access](https://www.okta.com/identity-101/access-token/) and [id](https://developer.okta.com/docs/guides/validate-id-tokens/main/#id-tokens-vs-access-tokens) tokens.
+By default, it is set to `id:sub` as per the OIDC protocol it is recommended to use the `sub` field from the id token as it is non-mutable and globally unique for each application.
+For Okta one commonly used field is `access:sub` which is usually the email of the user. You can also configure [custom claims](https://developer.okta.com/docs/guides/customize-tokens-returned-from-okta/main/).
+
+### Database configuration
+
+OIDC is by default enabled using the Memgraph `oidc.py` module. To use a custom auth module use the `--auth-module-mappings` [flag](/database-management/configuration#auth-module) like the following:
+`--auth-module-mappings=oidc-entra-id:/path/to/oidc-entra-module;oidc-okta:/path/to/oidc-okta-module` depending on the SSO provider you want to use.
diff --git a/pages/database-management/configuration.mdx b/pages/database-management/configuration.mdx
@@ -362,7 +362,7 @@ auth module authentication and authorization mechanisms used by Memgraph.
 | Flag                             | Description
 | ---------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
 | --auth-module-mappings           | Associates auth schemes to external modules. A mapping is structured as follows: `<scheme>:<absolute path to module>` and individual entries are separated with `;`. If the mapping contains whitespace, enclose the flag value with quotation marks.                             |
-| --auth-module-executable         | [DEPRECATED] Path to the executable that should be used for user authentication/authorization. Replaced by `--auth-module-mappings=basic` from Memgraph 2.18.                                                                                                                   |
+| --auth-module-executable         | [DEPRECATED] Path to the executable that should be used for user authentication/authorization. Replaced by `--auth-module-mappings=basic` from Memgraph 2.18.                                                                                                                     |
 | --auth-module-timeout-ms         | Specifies the maximum time that Memgraph will wait for a response from the external auth module.                                                                                                                                                                                  |
 | --auth-password-permit-null      | Can be set to false to disable null passwords.                                                                                                                                                                                                                                    |
 | --auth-password-strength-regex   | The regular expression that should be used to match the entire entered password to ensure its strength. The syntax for regular expressions is derived from a [modified version of the ECMAScript regular expression grammar](https://en.cppreference.com/w/cpp/regex/ecmascript). |

diff --git a/pages/database-management/enabling-memgraph-enterprise.mdx b/pages/database-management/enabling-memgraph-enterprise.mdx
@@ -12,6 +12,7 @@ The following Memgraph features are only available in Enterprise Edition:
 - [SAML integration](/database-management/authentication-and-authorization/auth-system-integrations#saml)
 - [OIDC integration](/database-management/authentication-and-authorization/auth-system-integrations#openid-connect-oidc)
 - [LDAP integration](/database-management/authentication-and-authorization/auth-system-integrations#ldap)
+- [OIDC integration](/database-management/authentication-and-authorization/auth-system-integrations#oidc)
 - [Monitoring via HTTP server](/database-management/monitoring#monitoring-via-http-server-enterprise)
 - [Multi-tenancy](/database-management/multi-tenancy)
 - [Role-based access control](/database-management/authentication-and-authorization/role-based-access-control)

diff --git a/pages/database-management/monitoring.mdx b/pages/database-management/monitoring.mdx
@@ -264,14 +264,21 @@ local Memgraph build will result in a response similar to the one below.
 {
     "General": {
         "average_degree": 0.0,
-        "disk_usage": 1417846,
+        "disk_usage": 238141,
         "edge_count": 0,
-        "memory_usage": 36937728,
+        "memory_usage": 46329856,
+        "peak_memory_usage": 46329856,
+        "unreleased_delta_objects": 0,
         "vertex_count": 0
     },
     "Index": {
         "ActiveLabelIndices": 0,
-        "ActiveLabelPropertyIndices": 0
+        "ActiveLabelPropertyIndices": 0,
+        "ActiveTextIndices": 0
+    },
+    "Memory": {
+        "PeakMemoryRes": 46329856,
+        "UnreleasedDeltaObjects": 0
     },
     "Operator": {
         "AccumulateOperator": 0,
@@ -291,6 +298,8 @@ local Memgraph build will result in a response similar to the one below.
         "ExpandVariableOperator": 0,
         "FilterOperator": 0,
         "ForeachOperator": 0,
+        "HashJoinOperator": 0,
+        "IndexedJoinOperator": 0,
         "LimitOperator": 0,
         "MergeOperator": 0,
         "OnceOperator": 0,
@@ -299,6 +308,9 @@ local Memgraph build will result in a response similar to the one below.
         "ProduceOperator": 0,
         "RemoveLabelsOperator": 0,
         "RemovePropertyOperator": 0,
+        "RollUpApplyOperator": 0,
+        "ScanAllByEdgeIdOperator": 0,
+        "ScanAllByEdgeTypeOperator": 0,
         "ScanAllByIdOperator": 0,
         "ScanAllByLabelOperator": 0,
         "ScanAllByLabelPropertyOperator": 0,
@@ -323,30 +335,33 @@ local Memgraph build will result in a response similar to the one below.
         "WriteQuery": 0
     },
     "Session": {
-        "ActiveBoltSessions": 0,
+        "ActiveBoltSessions": 1,
         "ActiveSSLSessions": 0,
-        "ActiveSessions": 0,
-        "ActiveTCPSessions": 0,
+        "ActiveSessions": 1,
+        "ActiveTCPSessions": 1,
         "ActiveWebSocketSessions": 0,
-        "BoltMessages": 0
+        "BoltMessages": 3
     },
     "Snapshot": {
-        "SnapshotCreationLatency_us_50p": 4860,
-        "SnapshotCreationLatency_us_90p": 4860,
-        "SnapshotCreationLatency_us_99p": 4860,
-        "SnapshotRecoveryLatency_us_50p": 628,
-        "SnapshotRecoveryLatency_us_90p": 628,
-        "SnapshotRecoveryLatency_us_99p": 628
+        "SnapshotCreationLatency_us_50p": 0,
+        "SnapshotCreationLatency_us_90p": 0,
+        "SnapshotCreationLatency_us_99p": 0,
+        "SnapshotRecoveryLatency_us_50p": 0,
+        "SnapshotRecoveryLatency_us_90p": 0,
+        "SnapshotRecoveryLatency_us_99p": 0
     },
     "Stream": {
         "MessagesConsumed": 0,
         "StreamsCreated": 0
     },
     "Transaction": {
         "ActiveTransactions": 0,
-        "CommitedTransactions": 0,
-        "FailedQuery": 0,
-        "RollbackedTransactions": 0
+        "CommitedTransactions": 1,
+        "FailedPrepare": 1,
+        "FailedPull": 0,
+        "FailedQuery": 1,
+        "RollbackedTransactions": 0,
+        "SuccessfulQuery": 1
     },
     "Trigger": {
         "TriggersCreated": 0,