[Security AI Assistant] Adds ability to specify LangSmith config and APM URL for tracing in cloud environments #180227
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
spong
added
enhancement
|
Pinging @elastic/security-solution (Team: SecuritySolution) |
spong
commented
Apr 8, 2024
x-pack/plugins/security_solution/common/experimental_features.ts
Outdated
Show resolved
Hide resolved
💚 Build Succeeded
Metrics [docs]Async chunks
History
To update your PR or re-run it, just comment with: cc @spong |
kibanamachine
added
v8.14.0
backport:skip
spong
added a commit
that referenced
this pull request
Apr 10, 2024
…tion via env vars (#180426) ## Summary With #180227, LangSmith configuration (Project & API Key) could no longer be specified using environment variables when working locally. This fixes that issue, which was caused by sending `''` for `langSmithProject` and `langSmithApiKey` instead of `undefined`. To test, set the below env vars, then start kibana. Be sure to not have the UI trace options set as shown in #180227. ``` # LangChain LangSmith export LANGCHAIN_TRACING_V2=true export LANGCHAIN_ENDPOINT="https://api.smith.langchain.com" export LANGCHAIN_API_KEY="🫣" export LANGCHAIN_PROJECT="Best Project Ever" ```
andrew-goldstein
added a commit
to andrew-goldstein/kibana
that referenced
this pull request
Apr 18, 2024
…d deployments ### Summary This PR enables LangSmith tracing for the [AI Insights](elastic#180611) feature in cloud deployments. LangSmith project names and API keys are specified using the same UI and patterns introduced by @spong in <elastic#180227> ### Details To enable LangSmith tracing in cloud deployments, configure the following `xpack.securitySolution.enableExperimental` feature flags: ``` xpack.securitySolution.enableExperimental: ['assistantModelEvaluation', 'assistantAlertsInsights'] ``` - The `assistantModelEvaluation` feature flag enables the `Evaluation` settings category in the assistant. The LangSmith project name and API key are configured here - The `assistantAlertsInsights` feature flag enables the AI Insights feature, which is off by default at the time of this writing After enabling the feature flags above: 1) Click the `AI Assistant` button anywhere in the Security Solution to launch the assistant 2) Click the settings gear in the assistant 3) Click the `Evaluation` settings category 4) Click `Show Trace Options (for internal use only)` 5) Specify a `LangSmith Project` and `LangSmith API Key` per the screenshot below: 
andrew-goldstein
added a commit
that referenced
this pull request
Apr 18, 2024
…eployments (#181159) ## [Security Solution] [AI Insights] Enable LangSmith tracing in cloud deployments ### Summary This PR enables LangSmith tracing for the [AI Insights](#180611) feature in cloud deployments. LangSmith project names and API keys are specified using the same UI and patterns introduced by @spong in <#180227> ### Details To enable LangSmith tracing in cloud deployments, configure the following `xpack.securitySolution.enableExperimental` feature flags: ``` xpack.securitySolution.enableExperimental: ['assistantModelEvaluation', 'assistantAlertsInsights'] ``` - The `assistantModelEvaluation` feature flag enables the `Evaluation` settings category in the assistant. The LangSmith project name and API key are configured here - The `assistantAlertsInsights` feature flag enables the AI Insights feature, which is off by default at the time of this writing After enabling the feature flags above: 1) Click the `AI Assistant` button anywhere in the Security Solution to launch the assistant 2) Click the settings gear in the assistant 3) Click the `Evaluation` settings category 4) Click `Show Trace Options (for internal use only)` 5) Specify a `LangSmith Project` and `LangSmith API Key` per the screenshot below: 
kibanamachine
pushed a commit
to kibanamachine/kibana
that referenced
this pull request
Apr 18, 2024
…eployments (elastic#181159) ## [Security Solution] [AI Insights] Enable LangSmith tracing in cloud deployments ### Summary This PR enables LangSmith tracing for the [AI Insights](elastic#180611) feature in cloud deployments. LangSmith project names and API keys are specified using the same UI and patterns introduced by @spong in <elastic#180227> ### Details To enable LangSmith tracing in cloud deployments, configure the following `xpack.securitySolution.enableExperimental` feature flags: ``` xpack.securitySolution.enableExperimental: ['assistantModelEvaluation', 'assistantAlertsInsights'] ``` - The `assistantModelEvaluation` feature flag enables the `Evaluation` settings category in the assistant. The LangSmith project name and API key are configured here - The `assistantAlertsInsights` feature flag enables the AI Insights feature, which is off by default at the time of this writing After enabling the feature flags above: 1) Click the `AI Assistant` button anywhere in the Security Solution to launch the assistant 2) Click the settings gear in the assistant 3) Click the `Evaluation` settings category 4) Click `Show Trace Options (for internal use only)` 5) Specify a `LangSmith Project` and `LangSmith API Key` per the screenshot below:  (cherry picked from commit 89609fe)
kibanamachine
referenced
this pull request
Apr 18, 2024
…cloud deployments (#181159) (#181203) # Backport This will backport the following commits from `main` to `8.14`: - [[Security Solution] [AI Insights] Enable LangSmith tracing in cloud deployments (#181159)](#181159) <!--- Backport version: 9.4.3 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport) <!--BACKPORT [{"author":{"name":"Andrew Macri","email":"andrew.macri@elastic.co"},"sourceCommit":{"committedDate":"2024-04-18T19:01:27Z","message":"[Security Solution] [AI Insights] Enable LangSmith tracing in cloud deployments (#181159)\n\n## [Security Solution] [AI Insights] Enable LangSmith tracing in cloud deployments\r\n\r\n### Summary\r\n\r\nThis PR enables LangSmith tracing for the [AI Insights](#180611) feature in cloud deployments.\r\n\r\nLangSmith project names and API keys are specified using the same UI and patterns introduced by @spong in <https://github.com/elastic/kibana/pull/180227>\r\n\r\n### Details\r\n\r\nTo enable LangSmith tracing in cloud deployments, configure the following `xpack.securitySolution.enableExperimental` feature flags:\r\n\r\n```\r\nxpack.securitySolution.enableExperimental: ['assistantModelEvaluation', 'assistantAlertsInsights']\r\n```\r\n\r\n- The `assistantModelEvaluation` feature flag enables the `Evaluation` settings category in the assistant. The LangSmith project name and API key are configured here\r\n- The `assistantAlertsInsights` feature flag enables the AI Insights feature, which is off by default at the time of this writing\r\n\r\nAfter enabling the feature flags above:\r\n\r\n1) Click the `AI Assistant` button anywhere in the Security Solution to launch the assistant\r\n\r\n2) Click the settings gear in the assistant\r\n\r\n3) Click the `Evaluation` settings category\r\n\r\n4) Click `Show Trace Options (for internal use only)`\r\n\r\n5) Specify a `LangSmith Project` and `LangSmith API Key` per the screenshot below:\r\n\r\n","sha":"89609fe596d79b7d2eb4f209c5388824f9b279c1","branchLabelMapping":{"^v8.15.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","Team: SecuritySolution","Team:Security Generative AI","v8.14.0","v8.15.0","Feature:AI Insights"],"title":"[Security Solution] [AI Insights] Enable LangSmith tracing in cloud deployments","number":181159,"url":"https://github.com/elastic/kibana/pull/181159","mergeCommit":{"message":"[Security Solution] [AI Insights] Enable LangSmith tracing in cloud deployments (#181159)\n\n## [Security Solution] [AI Insights] Enable LangSmith tracing in cloud deployments\r\n\r\n### Summary\r\n\r\nThis PR enables LangSmith tracing for the [AI Insights](#180611) feature in cloud deployments.\r\n\r\nLangSmith project names and API keys are specified using the same UI and patterns introduced by @spong in <https://github.com/elastic/kibana/pull/180227>\r\n\r\n### Details\r\n\r\nTo enable LangSmith tracing in cloud deployments, configure the following `xpack.securitySolution.enableExperimental` feature flags:\r\n\r\n```\r\nxpack.securitySolution.enableExperimental: ['assistantModelEvaluation', 'assistantAlertsInsights']\r\n```\r\n\r\n- The `assistantModelEvaluation` feature flag enables the `Evaluation` settings category in the assistant. The LangSmith project name and API key are configured here\r\n- The `assistantAlertsInsights` feature flag enables the AI Insights feature, which is off by default at the time of this writing\r\n\r\nAfter enabling the feature flags above:\r\n\r\n1) Click the `AI Assistant` button anywhere in the Security Solution to launch the assistant\r\n\r\n2) Click the settings gear in the assistant\r\n\r\n3) Click the `Evaluation` settings category\r\n\r\n4) Click `Show Trace Options (for internal use only)`\r\n\r\n5) Specify a `LangSmith Project` and `LangSmith API Key` per the screenshot below:\r\n\r\n","sha":"89609fe596d79b7d2eb4f209c5388824f9b279c1"}},"sourceBranch":"main","suggestedTargetBranches":["8.14"],"targetPullRequestStates":[{"branch":"8.14","label":"v8.14.0","branchLabelMappingKey":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"main","label":"v8.15.0","branchLabelMappingKey":"^v8.15.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/181159","number":181159,"mergeCommit":{"message":"[Security Solution] [AI Insights] Enable LangSmith tracing in cloud deployments (#181159)\n\n## [Security Solution] [AI Insights] Enable LangSmith tracing in cloud deployments\r\n\r\n### Summary\r\n\r\nThis PR enables LangSmith tracing for the [AI Insights](#180611) feature in cloud deployments.\r\n\r\nLangSmith project names and API keys are specified using the same UI and patterns introduced by @spong in <https://github.com/elastic/kibana/pull/180227>\r\n\r\n### Details\r\n\r\nTo enable LangSmith tracing in cloud deployments, configure the following `xpack.securitySolution.enableExperimental` feature flags:\r\n\r\n```\r\nxpack.securitySolution.enableExperimental: ['assistantModelEvaluation', 'assistantAlertsInsights']\r\n```\r\n\r\n- The `assistantModelEvaluation` feature flag enables the `Evaluation` settings category in the assistant. The LangSmith project name and API key are configured here\r\n- The `assistantAlertsInsights` feature flag enables the AI Insights feature, which is off by default at the time of this writing\r\n\r\nAfter enabling the feature flags above:\r\n\r\n1) Click the `AI Assistant` button anywhere in the Security Solution to launch the assistant\r\n\r\n2) Click the settings gear in the assistant\r\n\r\n3) Click the `Evaluation` settings category\r\n\r\n4) Click `Show Trace Options (for internal use only)`\r\n\r\n5) Specify a `LangSmith Project` and `LangSmith API Key` per the screenshot below:\r\n\r\n","sha":"89609fe596d79b7d2eb4f209c5388824f9b279c1"}}]}] BACKPORT--> Co-authored-by: Andrew Macri <andrew.macri@elastic.co>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
While we wait for #178304, this is a PR for allowing users to specify their LangSmith config for tracing in cloud environments by only storing them in session storage. This is also behind an experimental feature flag and must be enabled with the
assistantModelEvaluationflag ala:Note I:xpack.securitySolution.enableExperimentalshould be allowlisted in cloud, but I have manually enabled via source for initial testing.Note II: I have verified the above is configurable on cloud deployments 👍
The new
traceOptionsare stored with theelasticAssistantDefault.traceOptionskey, and the following keys:The
langSmithApiKeyandlangSmithProjectare then sent along with the request to/actions/connector/{connectorId}/_execute, and a newLangChainTraceris created using the values. The tracing infrastructue was already in place for evaluation, so no other changes were necessary.The
apmUrlvalue is now used for theView APM trace for messageaction, so if you have set up a remote APM server, you can now link directly to that instance from the message.A basic UI was added for these fields under the
Runstep of the Evaluation Settings. No need to save or run an evaluation once entering. Fields are immediately stored in session storage upon entry.Test Instructions
Click on the latest Kibana Buildkite build, go to the
ci:cloud-deploycluster (grabbing creds from vault), then set a LangChain Project/API key in the above UI, then make a request to the LLM and verify the trace is collected in the LangSmith UI:Note
Only LangChain codepaths can be traced to LangSmith, so you must ensure LangChain is enabled by either turning on the Knowledge Base or enabling the Alert tools. The former can't be done in default
ci:cloud-deploydeployments as they only have a 1GB ML nodes, so it is easiest to just turn on the Alert tools.Checklist
Delete any items that are not applicable to this PR.