diff --git a/README.md b/README.md index 56086b2ec..95c1cdd96 100644 --- a/README.md +++ b/README.md @@ -95,14 +95,19 @@ an [issue](https://github.com/memgraph/documentation/issues). ### Contributing guide -If you want to change the documentation, create a new branch and make -the appropriate changes. Then, create a pull request to merge these changes into the +If you want to change the documentation, create a new branch and make the +appropriate changes. Then, create a pull request to merge these changes into the `main` branch. -The pull request should describe the changes it's proposing and all checks must be completed. +The pull request should describe the changes it's proposing and all checks must +be completed. -Add an appropriate label to the PR, either `status: draft` if you are still working on the PR, or `status: ready` if the PR is ready for review. +Add an appropriate label to the PR, either `status: draft` if you are still +working on the PR, or `status: ready` if the PR is ready for review. -When the PR is reviewed and approved, the label will be changed to `status: ship it` and merged into the main by the repo admins. +When the PR is reviewed and approved, the label will be changed to `status: ship +it` and merged into the main by the repo admins. -If the PR requires changes, the label will be changed to `status: change`. Address the comments and change the documentation appropriately, then re-request a review and change the label to `status: ready` again. +If the PR requires changes, the label will be changed to `status: change`. +Address the comments and change the documentation appropriately, then re-request +a review and change the label to `status: ready` again. diff --git a/pages/data-migration/best-practices.mdx b/pages/data-migration/best-practices.mdx index 8bd4406e9..d5c7ae5bd 100644 --- a/pages/data-migration/best-practices.mdx +++ b/pages/data-migration/best-practices.mdx @@ -559,6 +559,8 @@ It will help you diagnose bottlenecks and high-memory queries which can be optim That is most common in import queries because users would want to import the whole dataset with one command only. For more information, check our [storage memory usage](/fundamentals/storage-memory-usage). +For more information about `Delta` objects, check the +information on the [IN_MEMORY_TRANSACTIONAL storage mode](/fundamentals/storage-memory-usage#in-memory-transactional-storage-mode-default). ## Do you have more questions? diff --git a/pages/database-management/authentication-and-authorization/auth-system-integrations.mdx b/pages/database-management/authentication-and-authorization/auth-system-integrations.mdx index 627537793..d6db03340 100644 --- a/pages/database-management/authentication-and-authorization/auth-system-integrations.mdx +++ b/pages/database-management/authentication-and-authorization/auth-system-integrations.mdx @@ -602,3 +602,89 @@ allowed to log in because her role (moderator) already exists. Carol and Dave won't be allowed to log in because their role (administrator) doesn't exist. + +## OpenID Connect (OIDC) + +Memgraph also supports authentication and authorization using OIDC (OIDC + OAuth 2.0) protocol with a +built-in auth module that is packaged with Memgraph Enterprise. + +The module currently supports the following SSO providers: +- Microsoft Entra ID +- Okta using a custom authorization server + +To use the built-in auth module run Memgraph with the `--auth-module-mappings` [flag](/database-management/configuration#auth-module) with the name of each provider and protocol you want to use separater with a semicolon (;). +For example, to use OIDC with both Entra ID and Okta run Memgraph with `--auth-module-mappings=oidc-okta;oidc-entra-id`. + +For SSO provider setup see [this](/data-visualization/user-manual/single-sign-on). + +### Module requirements + +The module is written in Python 3 and requires the following libraries: + - `PyJWT` - used to validate tokens. + - `requests` - used to retrieve public keys for validating tokens. + - `cryptography` - used to parse tokens. + +The required packages are already installed in the docker image. If you are using a native build you to install the packages manually on your machine. +The list of all required packages is located at `src/auth/reference_modules/requirements.txt`. You can install them by running `pip3 install -r /path/to/requirements.txt`. +If you are running a native build make sure that you don't have the Python `JWT` library installed (you can check that with `pip freeze`) as `PyJWT` and `JWT` have some conflicts. + +### Module configuration + +The module is located at `/usr/lib/memgraph/auth_module/oidc.py`. +The module has no configuration file. You have to run Memgraph with some environmental variables. +To use multiple SSO providers you have to set the environmental variables associated with that provider. + +#### MS Entra ID +For using OIDC with Microsoft Entra ID you have to set the following environmental variables: + +```bash +MEMGRAPH_SSO_ENTRA_ID_OIDC_TENANT_ID= +MEMGRAPH_SSO_ENTRA_ID_OIDC_CLIENT_ID= +MEMGRAPH_SSO_ENTRA_ID_OIDC_ROLE_MAPPING= +MEMGRAPH_SSO_ENTRA_ID_OIDC_USERNAME= +``` + +You can find the tenant and client ID in the app overview in Azure. For a more detailed explanation see [this](/pages/data-visualization/user-manual/single-sign-on.mdx). + +##### Role mapping +Authorization is done by mapping the Entra ID roles from the roles field in the access token to the Memgraph roles. +The role mapping is defined as a string where each individual mapping is separated by a semicolon (;). Within each mapping, the Entra ID role is separated from the corresponding Memgraph role by a colon (:). +One example of a role mapping is `entra.admin:memadmin; entra.user:memuser` where the Entra ID role `entra.admin` is mapped to the Memgraph role `memadmin` and the Entra ID role `entra.user` is mapped to the Memgraph role `memuser`. +Make sure that each user only has one role on Entra ID. + +##### Username +The username variable tells the OIDC module what to use as the username. It has the format `token-type:field`. +Token type can be `id` or `access` depending on whether you want to use a field from the access or the ID token for the username. See the following to learn more about [access](https://learn.microsoft.com/en-us/entra/identity-platform/access-tokens) and [id](https://learn.microsoft.com/en-us/entra/identity-platform/id-tokens) tokens. +By default, it is set to `id:sub` as per the OIDC protocol it is recommended to use the `sub` field from the id token as it is non-mutable and globally unique for each application. +For MS Entra ID one commonly used field is `id:preferred_username` which is usually the email of the user. For a list of all default claims look at [this](https://learn.microsoft.com/en-us/entra/identity-platform/access-token-claims-reference) and [this](https://learn.microsoft.com/en-us/entra/identity-platform/id-token-claims-reference). +You can also configure custom claims. + +#### Okta +For using OIDC with Okta you have to set the following environmental variables: +```bash +MEMGRAPH_SSO_OKTA_OIDC_ISSUER= +MEMGRAPH_SSO_OKTA_OIDC_CLIENT_ID= +MEMGRAPH_SSO_OKTA_OIDC_AUTHORIZATION_SERVER=api://default +MEMGRAPH_SSO_OKTA_OIDC_ROLE_MAPPING= +MEMGRAPH_SSO_OKTA_OIDC_USERNAME= +``` + +Issuer is `https://{your-okta-domain}.okta.com/oauth2/default/`. You can find the client ID on the Admin panel -> Applications -> General. You can find the authorization server on the Admin panel -> Security -> API -> Authorization Servers -> Audience. By default, it is set to `api://default`. + +##### Role mapping +Authorization is done by mapping the Okta groups from the `groups` field in the access token to the Memgraph roles. +The role mapping is defined as a string where each individual mapping is separated by a semicolon (;). Within each mapping, the Entra ID role is separated from the corresponding Memgraph role by a colon (:). +One example of a role mapping is `okta.admin:memadmin; okta.user:memuser` where the Okta group `okta.admin` is mapped to the Memgraph role `memadmin` and the Okta group `okta.user` is mapped to the Memgraph role `memuser`. +The `groups` field does not exist in Okta by default and you have to set it up manually. More information on how to do that [here](https://developer.okta.com/docs/guides/customize-tokens-returned-from-okta/main/#add-a-custom-claim-to-a-token). +Make sure that each user is only assigned to one group on Okta. + +##### Username +The username variable tells the OIDC module what to use as the username. It has the format `token-type:field`. +Token type can be `id` or `access` depending on whether you want to use a field from the access or the ID token for the username. See the following to learn more about [access](https://www.okta.com/identity-101/access-token/) and [id](https://developer.okta.com/docs/guides/validate-id-tokens/main/#id-tokens-vs-access-tokens) tokens. +By default, it is set to `id:sub` as per the OIDC protocol it is recommended to use the `sub` field from the id token as it is non-mutable and globally unique for each application. +For Okta one commonly used field is `access:sub` which is usually the email of the user. You can also configure [custom claims](https://developer.okta.com/docs/guides/customize-tokens-returned-from-okta/main/). + +### Database configuration + +OIDC is by default enabled using the Memgraph `oidc.py` module. To use a custom auth module use the `--auth-module-mappings` [flag](/database-management/configuration#auth-module) like the following: +`--auth-module-mappings=oidc-entra-id:/path/to/oidc-entra-module;oidc-okta:/path/to/oidc-okta-module` depending on the SSO provider you want to use. diff --git a/pages/database-management/configuration.mdx b/pages/database-management/configuration.mdx index cd2b9a739..752e56f7e 100644 --- a/pages/database-management/configuration.mdx +++ b/pages/database-management/configuration.mdx @@ -362,7 +362,7 @@ auth module authentication and authorization mechanisms used by Memgraph. | Flag | Description | ---------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | --auth-module-mappings | Associates auth schemes to external modules. A mapping is structured as follows: `:` and individual entries are separated with `;`. If the mapping contains whitespace, enclose the flag value with quotation marks. | -| --auth-module-executable | [DEPRECATED] Path to the executable that should be used for user authentication/authorization. Replaced by `--auth-module-mappings=basic` from Memgraph 2.18. | +| --auth-module-executable | [DEPRECATED] Path to the executable that should be used for user authentication/authorization. Replaced by `--auth-module-mappings=basic` from Memgraph 2.18. | | --auth-module-timeout-ms | Specifies the maximum time that Memgraph will wait for a response from the external auth module. | | --auth-password-permit-null | Can be set to false to disable null passwords. | | --auth-password-strength-regex | The regular expression that should be used to match the entire entered password to ensure its strength. The syntax for regular expressions is derived from a [modified version of the ECMAScript regular expression grammar](https://en.cppreference.com/w/cpp/regex/ecmascript). | diff --git a/pages/database-management/enabling-memgraph-enterprise.mdx b/pages/database-management/enabling-memgraph-enterprise.mdx index 9a00e4070..d99191249 100644 --- a/pages/database-management/enabling-memgraph-enterprise.mdx +++ b/pages/database-management/enabling-memgraph-enterprise.mdx @@ -12,6 +12,7 @@ The following Memgraph features are only available in Enterprise Edition: - [SAML integration](/database-management/authentication-and-authorization/auth-system-integrations#saml) - [OIDC integration](/database-management/authentication-and-authorization/auth-system-integrations#openid-connect-oidc) - [LDAP integration](/database-management/authentication-and-authorization/auth-system-integrations#ldap) +- [OIDC integration](/database-management/authentication-and-authorization/auth-system-integrations#oidc) - [Monitoring via HTTP server](/database-management/monitoring#monitoring-via-http-server-enterprise) - [Multi-tenancy](/database-management/multi-tenancy) - [Role-based access control](/database-management/authentication-and-authorization/role-based-access-control) diff --git a/pages/database-management/monitoring.mdx b/pages/database-management/monitoring.mdx index 16350f5f1..b0f14ed11 100644 --- a/pages/database-management/monitoring.mdx +++ b/pages/database-management/monitoring.mdx @@ -264,14 +264,21 @@ local Memgraph build will result in a response similar to the one below. { "General": { "average_degree": 0.0, - "disk_usage": 1417846, + "disk_usage": 238141, "edge_count": 0, - "memory_usage": 36937728, + "memory_usage": 46329856, + "peak_memory_usage": 46329856, + "unreleased_delta_objects": 0, "vertex_count": 0 }, "Index": { "ActiveLabelIndices": 0, - "ActiveLabelPropertyIndices": 0 + "ActiveLabelPropertyIndices": 0, + "ActiveTextIndices": 0 + }, + "Memory": { + "PeakMemoryRes": 46329856, + "UnreleasedDeltaObjects": 0 }, "Operator": { "AccumulateOperator": 0, @@ -291,6 +298,8 @@ local Memgraph build will result in a response similar to the one below. "ExpandVariableOperator": 0, "FilterOperator": 0, "ForeachOperator": 0, + "HashJoinOperator": 0, + "IndexedJoinOperator": 0, "LimitOperator": 0, "MergeOperator": 0, "OnceOperator": 0, @@ -299,6 +308,9 @@ local Memgraph build will result in a response similar to the one below. "ProduceOperator": 0, "RemoveLabelsOperator": 0, "RemovePropertyOperator": 0, + "RollUpApplyOperator": 0, + "ScanAllByEdgeIdOperator": 0, + "ScanAllByEdgeTypeOperator": 0, "ScanAllByIdOperator": 0, "ScanAllByLabelOperator": 0, "ScanAllByLabelPropertyOperator": 0, @@ -323,20 +335,20 @@ local Memgraph build will result in a response similar to the one below. "WriteQuery": 0 }, "Session": { - "ActiveBoltSessions": 0, + "ActiveBoltSessions": 1, "ActiveSSLSessions": 0, - "ActiveSessions": 0, - "ActiveTCPSessions": 0, + "ActiveSessions": 1, + "ActiveTCPSessions": 1, "ActiveWebSocketSessions": 0, - "BoltMessages": 0 + "BoltMessages": 3 }, "Snapshot": { - "SnapshotCreationLatency_us_50p": 4860, - "SnapshotCreationLatency_us_90p": 4860, - "SnapshotCreationLatency_us_99p": 4860, - "SnapshotRecoveryLatency_us_50p": 628, - "SnapshotRecoveryLatency_us_90p": 628, - "SnapshotRecoveryLatency_us_99p": 628 + "SnapshotCreationLatency_us_50p": 0, + "SnapshotCreationLatency_us_90p": 0, + "SnapshotCreationLatency_us_99p": 0, + "SnapshotRecoveryLatency_us_50p": 0, + "SnapshotRecoveryLatency_us_90p": 0, + "SnapshotRecoveryLatency_us_99p": 0 }, "Stream": { "MessagesConsumed": 0, @@ -344,9 +356,12 @@ local Memgraph build will result in a response similar to the one below. }, "Transaction": { "ActiveTransactions": 0, - "CommitedTransactions": 0, - "FailedQuery": 0, - "RollbackedTransactions": 0 + "CommitedTransactions": 1, + "FailedPrepare": 1, + "FailedPull": 0, + "FailedQuery": 1, + "RollbackedTransactions": 0, + "SuccessfulQuery": 1 }, "Trigger": { "TriggersCreated": 0, diff --git a/pages/database-management/server-stats.md b/pages/database-management/server-stats.md index cd11030c4..cbf1aaed2 100644 --- a/pages/database-management/server-stats.md +++ b/pages/database-management/server-stats.md @@ -36,6 +36,7 @@ The result will contain the following fields: | vm_max_map_count | The number of memory-mapped areas that the kernel allows a process to have. If it is unknown, returns -1.
For more info, check out [virtual memory section of the docs](/fundamentals/storage-memory-usage#virtual-memory). | | memory_res | The non-swapped physical RAM memory a task has used, reported by the OS (in B, KiB, MiB, GiB or TiB). | | peak_memory_res | Peak RAM memory usage in the system during the whole run. | +| unreleased_delta_objects | The current number of still allocated objects with the information about the changes that write transactions have made, called Delta objects. Refer to allocation and deallocation of Delta objects [on this page](/fundamentals/storage-memory-usage#in-memory-transactional-storage-mode-default). | | disk_usage | The amount of disk space used by the data directory (in B, KiB, MiB, GiB or TiB). | | memory_tracked | The amount of RAM allocated in the system and tracked by Memgraph (in B, KiB, MiB, GiB or TiB).
For more info, check out [memory control](/fundamentals/storage-memory-usage). | | allocation_limit | The current allocation limit set for this instance (in B, KiB, MiB, GiB or TiB).
For more info, check out the [memory control](/fundamentals/storage-memory-usage#control-memory-usage). | @@ -44,6 +45,106 @@ The result will contain the following fields: | next_session_isolation_level | The current `next` isolation level. | | storage_mode | The current storage mode.
For more info, check out [storage modes](/fundamentals/storage-memory-usage#storage-modes). | +## Metrics information + +Running this query will show the system metrics. Same metrics can be provided from the HTTP endpoint at default +port 9091. This is a query that only works with the Memgraph Enterprise License. For the information about the metrics, +please see the [information about monitoring via HTTP server](/database-management/monitoring#monitoring-via-http-server-enterprise). + +```cypher +SHOW METRICS INFO; +``` + +```console copy=false ++---------------------------------------+---------------+-------------+----------+ +| name | type | metric type | value | ++---------------------------------------+---------------+-------------+----------+ +| "AverageDegree" | "General" | "Gauge" | 0 | +| "EdgeCount" | "General" | "Gauge" | 0 | +| "VertexCount" | "General" | "Gauge" | 0 | +| "ActiveLabelIndices" | "Index" | "Counter" | 0 | +| "ActiveLabelPropertyIndices" | "Index" | "Counter" | 0 | +| "ActiveTextIndices" | "Index" | "Counter" | 0 | +| "UnreleasedDeltaObjects" | "Memory" | "Counter" | 0 | +| "DiskUsage" | "Memory" | "Gauge" | 231838 | +| "MemoryRes" | "Memory" | "Gauge" | 45805568 | +| "PeakMemoryRes" | "Memory" | "Gauge" | 45805568 | +| "AccumulateOperator" | "Operator" | "Counter" | 0 | +| "AggregateOperator" | "Operator" | "Counter" | 0 | +| "ApplyOperator" | "Operator" | "Counter" | 0 | +| "CallProcedureOperator" | "Operator" | "Counter" | 0 | +| "CartesianOperator" | "Operator" | "Counter" | 0 | +| "ConstructNamedPathOperator" | "Operator" | "Counter" | 0 | +| "CreateExpandOperator" | "Operator" | "Counter" | 0 | +| "CreateNodeOperator" | "Operator" | "Counter" | 0 | +| "DeleteOperator" | "Operator" | "Counter" | 0 | +| "DistinctOperator" | "Operator" | "Counter" | 0 | +| "EdgeUniquenessFilterOperator" | "Operator" | "Counter" | 0 | +| "EmptyResultOperator" | "Operator" | "Counter" | 0 | +| "EvaluatePatternFilterOperator" | "Operator" | "Counter" | 0 | +| "ExpandOperator" | "Operator" | "Counter" | 0 | +| "ExpandVariableOperator" | "Operator" | "Counter" | 0 | +| "FilterOperator" | "Operator" | "Counter" | 0 | +| "ForeachOperator" | "Operator" | "Counter" | 0 | +| "HashJoinOperator" | "Operator" | "Counter" | 0 | +| "IndexedJoinOperator" | "Operator" | "Counter" | 0 | +| "LimitOperator" | "Operator" | "Counter" | 0 | +| "MergeOperator" | "Operator" | "Counter" | 0 | +| "OnceOperator" | "Operator" | "Counter" | 0 | +| "OptionalOperator" | "Operator" | "Counter" | 0 | +| "OrderByOperator" | "Operator" | "Counter" | 0 | +| "ProduceOperator" | "Operator" | "Counter" | 0 | +| "RemoveLabelsOperator" | "Operator" | "Counter" | 0 | +| "RemovePropertyOperator" | "Operator" | "Counter" | 0 | +| "RollUpApplyOperator" | "Operator" | "Counter" | 0 | +| "ScanAllByEdgeIdOperator" | "Operator" | "Counter" | 0 | +| "ScanAllByEdgeTypeOperator" | "Operator" | "Counter" | 0 | +| "ScanAllByIdOperator" | "Operator" | "Counter" | 0 | +| "ScanAllByLabelOperator" | "Operator" | "Counter" | 0 | +| "ScanAllByLabelPropertyOperator" | "Operator" | "Counter" | 0 | +| "ScanAllByLabelPropertyRangeOperator" | "Operator" | "Counter" | 0 | +| "ScanAllByLabelPropertyValueOperator" | "Operator" | "Counter" | 0 | +| "ScanAllOperator" | "Operator" | "Counter" | 0 | +| "SetLabelsOperator" | "Operator" | "Counter" | 0 | +| "SetPropertiesOperator" | "Operator" | "Counter" | 0 | +| "SetPropertyOperator" | "Operator" | "Counter" | 0 | +| "SkipOperator" | "Operator" | "Counter" | 0 | +| "UnionOperator" | "Operator" | "Counter" | 0 | +| "UnwindOperator" | "Operator" | "Counter" | 0 | +| "QueryExecutionLatency_us_50p" | "Query" | "Histogram" | 0 | +| "QueryExecutionLatency_us_90p" | "Query" | "Histogram" | 0 | +| "QueryExecutionLatency_us_99p" | "Query" | "Histogram" | 0 | +| "ReadQuery" | "QueryType" | "Counter" | 0 | +| "ReadWriteQuery" | "QueryType" | "Counter" | 0 | +| "WriteQuery" | "QueryType" | "Counter" | 0 | +| "ActiveBoltSessions" | "Session" | "Counter" | 1 | +| "ActiveSSLSessions" | "Session" | "Counter" | 0 | +| "ActiveSessions" | "Session" | "Counter" | 1 | +| "ActiveTCPSessions" | "Session" | "Counter" | 1 | +| "ActiveWebSocketSessions" | "Session" | "Counter" | 0 | +| "BoltMessages" | "Session" | "Counter" | 3 | +| "SnapshotCreationLatency_us_50p" | "Snapshot" | "Histogram" | 0 | +| "SnapshotCreationLatency_us_90p" | "Snapshot" | "Histogram" | 0 | +| "SnapshotCreationLatency_us_99p" | "Snapshot" | "Histogram" | 0 | +| "SnapshotRecoveryLatency_us_50p" | "Snapshot" | "Histogram" | 0 | +| "SnapshotRecoveryLatency_us_90p" | "Snapshot" | "Histogram" | 0 | +| "SnapshotRecoveryLatency_us_99p" | "Snapshot" | "Histogram" | 0 | +| "MessagesConsumed" | "Stream" | "Counter" | 0 | +| "StreamsCreated" | "Stream" | "Counter" | 0 | +| "ActiveTransactions" | "Transaction" | "Counter" | 1 | +| "CommitedTransactions" | "Transaction" | "Counter" | 0 | +| "FailedPrepare" | "Transaction" | "Counter" | 1 | +| "FailedPull" | "Transaction" | "Counter" | 0 | +| "FailedQuery" | "Transaction" | "Counter" | 1 | +| "RollbackedTransactions" | "Transaction" | "Counter" | 0 | +| "SuccessfulQuery" | "Transaction" | "Counter" | 0 | +| "TriggersCreated" | "Trigger" | "Counter" | 0 | +| "TriggersExecuted" | "Trigger" | "Counter" | 0 | ++---------------------------------------+---------------+-------------+----------+ +``` + + + ## Build information Running the following query will return certain information about the build type of @@ -55,4 +156,4 @@ SHOW BUILD INFO; | Field | Description | |------------|-----------------------------------------------------| -| build_type | The optimization level the instance was built with. | \ No newline at end of file +| build_type | The optimization level the instance was built with. | diff --git a/pages/fundamentals/storage-memory-usage.mdx b/pages/fundamentals/storage-memory-usage.mdx index e19fa368e..0bb9e3d63 100644 --- a/pages/fundamentals/storage-memory-usage.mdx +++ b/pages/fundamentals/storage-memory-usage.mdx @@ -736,6 +736,7 @@ SHOW STORAGE INFO; | "vm_max_map_count" | 453125 | | "memory_res" | "43.16MiB" | | "peak_memory_res" | "43.16MiB" | +| "unreleased_delta_objects" | 0 | | "disk_usage" | "104.46KiB" | | "memory_tracked" | "8.52MiB" | | "allocation_limit" | "58.55GiB" | @@ -758,7 +759,7 @@ Here are several tips how you can reduce memory usage and increase scalability: ON :Label(property);` 3. If you don't have properties on relationships, disable them in the [configuration file](/configuration/configuration-settings#storage) by setting the - `-storage-properties-on-edges` flag to `false`. This can significantly reduce + `--storage-properties-on-edges` flag to `false`. This can significantly reduce memory usage because effectively `Edge` objects will not be created, and all information will be inlined under `Vertex` objects. You can disable properties on relationships with a non-empty database, if the relationships