Modernize Node.js + Filebeat ingestion guide for Elastic Stack 9.0 #3422
+321
−306
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
🔍 Preview links for changed docs |
|
@pierrehilbert Hi Pierre, this page contains information that is outdated in version 9.0, and we need to update it. Could you please refer me to someone from your team who can review it? |
|
@belimawr could you please review here? |
belimawr
reviewed
Oct 14, 2025
| ::::: | ||
| In the directory where you created `webserver.js`, you should now find a newly created `log.json` file. Open the file and check the contents. There should be one log entry indicating that Node.js is listening on the localhost port, and another entry for the HTTP request from when you opened `localhost` in your browser. |
There was a problem hiding this comment.
The file log.json is actually created in the logs folder.
Suggested change
| In the directory where you created `webserver.js`, you should now find a newly created `log.json` file. Open the file and check the contents. There should be one log entry indicating that Node.js is listening on the localhost port, and another entry for the HTTP request from when you opened `localhost` in your browser. | |
| In the directory where you created `webserver.js`, you should now find a newly created `logs/log.json` file. Open the file and check the contents. There should be one log entry indicating that Node.js is listening on the localhost port, and another entry for the HTTP request from when you opened `localhost` in your browser. |
| const method = methods[Math.floor(Math.random() * methods.length)] | ||
| const from = addresses[Math.floor(Math.random() * addresses.length)] | ||
| try { | ||
| await got[method]('http://localhost:3000', { headers: { from } }) |
There was a problem hiding this comment.
The original code did not work, but I got it working.
I'm not a Javascript expert, there might be better ways to achieve the same result, but the following works:
Suggested change
| await got[method]('http://localhost:3000', { headers: { from } }) | |
| await got['got'][method]('http://localhost:3000', { headers: { from } }) |
| })() | ||
| ``` | ||
| This Node.js app generates HTTP requests with a random method of type `GET`, `POST`, or `PUT`, and a random `from` request header using various pretend email addresses. The requests are sent at random intervals between 1 and 10 seconds. |
There was a problem hiding this comment.
According to the code, the time between requests is from 2 to 7s.
Suggested change
| This Node.js app generates HTTP requests with a random method of type `GET`, `POST`, or `PUT`, and a random `from` request header using various pretend email addresses. The requests are sent at random intervals between 1 and 10 seconds. | |
| This Node.js app generates HTTP requests with a random method of type `GET`, `POST`, or `PUT`, and a random `from` request header using various pretend email addresses. The requests are sent at random intervals between 2 and 7 seconds. |
| const from = addresses[Math.floor(Math.random() * addresses.length)] | ||
| try { | ||
| await got[method]('http://localhost:3000', { headers: { from } }) | ||
| } catch {} |
There was a problem hiding this comment.
Some error handling is helpful, otherwise the script will just hang there with no output and without making any requests:
Suggested change
| } catch {} | |
| } catch (err){ | |
| console.log("cannot execute request:", err) | |
| } |
Comment on lines
+180
to
+182
| ```json | ||
| {"@timestamp":"2021-09-09T18:42:20.799Z","log.level":"info","message":"handled request","ecs":{"version":"1.6.0"},"http":{"version":"1.1","request":{"method":"POST","headers":{"user-agent":"got (https://github.com/sindresorhus/got)","from":"octopus@the.zoo","accept":"application/json","accept-encoding":"gzip, deflate, br","host":"localhost:3000","connection":"close","content-length":"0"},"body":{"bytes":0}},"response":{"status_code":200,"headers":{"foo":"Bar"}}},"url":{"path":"/","full":"http://localhost:3000/"},"client":{"address":"::ffff:127.0.0.1","ip":"::ffff:127.0.0.1","port":49930},"user_agent":{"original":"got (https://github.com/sindresorhus/got)"}} | ||
| ``` |
There was a problem hiding this comment.
The GH suggestion might not work very well. I believe this example should be updated, following the tutorial today we get logs like this:
{"@timestamp":"2025-10-14T16:11:36.402Z","client":{"address":"::1","ip":"::1","port":42836},"ecs.version":"8.10.0","http":{"request":{"body":{"bytes":0},"headers":{"accept-encoding":"gzip, deflate, br","connection":"keep-alive","content-length":"0","from":"emu@zoo","host":"localhost:3000","user-agent":"got (https://github.com/sindresorhus/got)"},"method":"PUT"},"response":{"headers":{"foo":"Bar"},"status_code":200},"version":"1.1"},"log.level":"info","message":"handled request","url":{"full":"http://localhost:3000/","path":"/"},"user_agent":{"original":"got (https://github.com/sindresorhus/got)"}}One key difference is that the ECS version in the logs is now 8.10.0.
Suggested change
| ```json | |
| {"@timestamp":"2021-09-09T18:42:20.799Z","log.level":"info","message":"handled request","ecs":{"version":"1.6.0"},"http":{"version":"1.1","request":{"method":"POST","headers":{"user-agent":"got (https://github.com/sindresorhus/got)","from":"octopus@the.zoo","accept":"application/json","accept-encoding":"gzip, deflate, br","host":"localhost:3000","connection":"close","content-length":"0"},"body":{"bytes":0}},"response":{"status_code":200,"headers":{"foo":"Bar"}}},"url":{"path":"/","full":"http://localhost:3000/"},"client":{"address":"::ffff:127.0.0.1","ip":"::ffff:127.0.0.1","port":49930},"user_agent":{"original":"got (https://github.com/sindresorhus/got)"}} | |
| ``` | |
| ```json | |
| {"@timestamp":"2025-10-14T16:11:36.402Z","client":{"address":"::1","ip":"::1","port":42836},"ecs.version":"8.10.0","http":{"request":{"body":{"bytes":0},"headers":{"accept-encoding":"gzip, deflate, br","connection":"keep-alive","content-length":"0","from":"emu@zoo","host":"localhost:3000","user-agent":"got (https://github.com/sindresorhus/got)"},"method":"PUT"},"response":{"headers":{"foo":"Bar"},"status_code":200},"version":"1.1"},"log.level":"info","message":"handled request","url":{"full":"http://localhost:3000/","path":"/"},"user_agent":{"original":"got (https://github.com/sindresorhus/got)"}} |
Comment on lines
+366
to
+380
| POST /_security/api_key | ||
| { | ||
| "name": "filebeat-api-key", | ||
| "role_descriptors": { | ||
| "logstash_read_write": { | ||
| "cluster": ["manage_index_templates", "monitor"], | ||
| "index": [ | ||
| { | ||
| "names": ["filebeat-*"], | ||
| "privileges": ["create_index", "write", "read", "manage"] | ||
| } | ||
| } | ||
| } | ||
| ``` | ||
| ] | ||
| } | ||
| } | ||
| } |
There was a problem hiding this comment.
The API Key with those permissions did not work for me with the latest Filebeat. While the ./filebeat test output command works, when Filebeat is running it gets a 403 when it tries to check a ILM policy.
I'll try to fix it after I finish my review.
| ``` | ||
| Let the script run for a few minutes and maybe brew up a quick coffee or tea ☕ . After that, make sure that the *log.json* file is generated as expected and is populated with several log entries. | ||
| Let the script run for a few minutes. After that, make sure that the *log.json* file is generated as expected and is populated with several log entries. |
There was a problem hiding this comment.
Suggested change
| Let the script run for a few minutes. After that, make sure that the *log.json* file is generated as expected and is populated with several log entries. | |
| Let the script run for a few minutes. After that, make sure that the *logs/log.json* file is generated as expected and is populated with several log entries. |
Comment on lines
488
to
493
| 1. [Login to Kibana](../../../deploy-manage/deploy/elastic-cloud/access-kibana.md). | ||
| 2. Open the {{kib}} main menu and select **Management** > **{{kib}}** > **Data views**. | ||
| 3. In the search bar, search for *filebeat*. You should get _filebeat-*_ in the search results. | ||
| 4. Select _filebeat-*_. | ||
| The filebeat data view shows a list of fields and their details. |
There was a problem hiding this comment.
It's the same thing I mentioned before, "Management > Kibana > Data views." cannot be found in the new defaults.
It's better to go: "Menu -> Discover", then select the filebeat-* data view.
| ## Create log visualizations in Kibana [ec-node-logs-view-kibana] | ||
| Now it’s time to create visualizations based off of the application log data. | ||
| ## Create log visualizations in Kibana [ec-node-logs-view-kibana] |
There was a problem hiding this comment.
This whole section is not aligned with the new solutions default view.
I suggest just linking to another docs (maybe from Kibana) that show how to create data visualisations, this way we don't have to keep updating Filebeat docs every time Kibana changes the navigation.
Comment on lines
+392
to
+396
| { | ||
| "api_key": "tV1dnfF-GHI59ykgv4N0U3", | ||
| "id": "2TBR42gBabmINotmvZjv", | ||
| "name": "filebeat-api-key" | ||
| } |
There was a problem hiding this comment.
In 9.1.4 the API also returns the encoded field:
{
"id": "yC9C5JkBk-xuk5nSlGlK",
"name": "filebeat-api-key",
"api_key": "hGDpbhHEeAUbvNaQlStNlg",
"encoded": "eUM5QzVKa0JrLXh1azVuU2xHbEs6aEdEcGJoSEVlQVVidk5hUWxTdE5sZw=="
}
I'm not sure when this was introduced, probably it's ok if we omit it here, but if we want to show the current APIs, then we should also add it.
belimawr
reviewed
Oct 14, 2025
Comment on lines
+366
to
+380
| POST /_security/api_key | ||
| { | ||
| "name": "filebeat-api-key", | ||
| "role_descriptors": { | ||
| "logstash_read_write": { | ||
| "cluster": ["manage_index_templates", "monitor"], | ||
| "index": [ | ||
| { | ||
| "names": ["filebeat-*"], | ||
| "privileges": ["create_index", "write", "read", "manage"] | ||
| } | ||
| } | ||
| } | ||
| ``` | ||
| ] | ||
| } | ||
| } | ||
| } |
There was a problem hiding this comment.
I found a request that works well:
POST /_security/api_key
{
"name": "filebeat-api-key",
"role_descriptors": {
"filebeat_role": {
"cluster": ["manage_index_templates", "monitor", "read_ilm"],
"index": [
{
"names": ["filebeat-*"],
"privileges": ["create_index", "write", "read", "manage"]
}
]
}
}
}
There are two key changes:
- Added the cluster privilege
read_ilm - Renamed the role to
filebeat_role
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR updates the “Ingest logs from a Node.js web application using Filebeat” tutorial to align with Elastic Stack 9.0 GA .
Summary of changes:
Replaced the deprecated
loginput withfilestream.Updated instructions for locating the Cloud ID using the Elastic Cloud console (replacing the outdated Kibana path).
Removed obsolete setup steps for creating deployments and connected securely sections; replaced with current "Before you begin" guidance and links to centralized Cloud setup docs.
Added
::::{stepper}blocks for procedural steps.Closes #1968