Logs Sample Data #22276

alexfrancoeur · 2018-08-22T21:28:18Z

This is one of the data sets we'd like to add for sample data. This is my first take on the data set itself so open to any feedback.

We're using makelogs as the basis here but want to tweak the URL's and requests to be more Elastic-y.

Open to feedback on the dataset, dashboards, etc.

Also note that I did not add any tests. There were a bunch of sample flight data, I'm not sure if they're necessary for other sample data sets. I'll look for @nreese here for guidance and can add some in if necessary.

Screenshots

Sample document

{
  "_index": "kibana_sample_data_logs",
  "_type": "_doc",
  "_id": "hikLbGUBoflyAT6o14Fv",
  "_version": 1,
  "_score": null,
  "_source": {
    "referer": "http://www.elastic-elastic-elastic.com/success/kenneth-ham",
    "clientip": "109.157.83.197",
    "response": "200",
    "tags": [
      "success",
      "info"
    ],
    "message": "109.157.83.197 - - [2018-08-03T13:07:45.046Z] \"GET /apm-server/apm-server-6.3.2-windows-x86.zip HTTP/1.1\" 200 5782 \"-\" \"Mozilla/5.0 (X11; Linux x86_64; rv:6.0a1) Gecko/20110421 Firefox/6.0a1\"",
    "request": "/apm-server/apm-server-6.3.2-windows-x86.zip",
    "geo": {
      "srcdest": "IN:AO",
      "src": "IN",
      "dest": "AO",
      "coordinates": {
        "lat": 39.21837556,
        "lon": -75.59642667
      }
    },
    "phpmemory": null,
    "extension": "zip",
    "index": "kibana_sample_data_logs",
    "url": "https://artifacts.elastic.co/downloads/apm-server/apm-server-6.3.2-windows-x86.zip",
    "memory": null,
    "host": "artifacts.elastic.co",
    "utc_time": "2018-08-03T13:07:45.046Z",
    "machine": {
      "ram": 8589934592,
      "os": "win 7"
    },
    "agent": "Mozilla/5.0 (X11; Linux x86_64; rv:6.0a1) Gecko/20110421 Firefox/6.0a1",
    "ip": "109.157.83.197",
    "bytes": 5782,
    "timestamp": "2018-08-24T13:07:45.046Z"
  },
  "fields": {
    "utc_time": [
      "2018-08-03T13:07:45.046Z"
    ],
    "hour_of_day": [
      13
    ],
    "timestamp": [
      "2018-08-24T13:07:45.046Z"
    ]
  },
  "sort": [
    1535116065046
  ]
}

elasticmachine · 2018-08-22T23:34:42Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request

jamiesmith

Nice job! eCommerce next?

alexfrancoeur · 2018-08-23T12:18:45Z

Updated PR so that it now has a working index pattern. New screenshot and sample doc above.

@jamiesmith @asawariS would love some feedback on the visualization types shown here, metrics to track, etc. Any thoughts on a color schema would be great as well. I'd still like to add annotations to the TSVB chart and better metrics in the table. There needs to be some cleanup, but this is actually a dashboard I use for demo's pretty frequently. More to show off functionality, not necessarily tell a story.

@jamiesmith I know we talked about adding "Elasticy" hosts, requests, etc. Any input here from PMM would be great, I'd really like to make this data fun. I know that you were also thinking of anomalies to add in. Note that I made a few changes in the Sample Data Clean Up doc and removed @ from a few of the field names. I also updated the index to be kibana_sample_data_logs.

@nreese open question here around tests. Do I need to add them in if there are already some for flight sample data? I'm guessing I do, but thought I'd ask before I write any 😄

nreese · 2018-08-23T13:31:04Z

open question here around tests. Do I need to add them in if there are already some for flight sample data? I'm guessing I do, but thought I'd ask before I write any

I think we should add a test that installs the sample data set, opens the dashboard and makes sure the expected number of panels are opened, and then uninstalls the sample data set. I can work on this and give you a PR with the tests.

elasticmachine · 2018-08-23T14:05:21Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request

alexfrancoeur · 2018-08-23T14:55:20Z

@nreese I should be able to add them, will ping with questions. Thanks for the input!

alexfrancoeur · 2018-08-23T14:56:10Z

@nreese I'm assuming I can pretty much copy what you did with flights here

nreese · 2018-08-23T15:03:11Z

@alexfrancoeur yes, just don't copy the skipped tests. Simply opening the dashboard and verifying the panel count is good enough for now.

elasticmachine · 2018-08-23T16:47:04Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request

alexfrancoeur · 2018-08-24T13:12:59Z

Still iterating on the data set a bit with @jamiesmith, updated screenshot and doc in the summary. We're close!

alexfrancoeur · 2018-08-24T13:52:04Z

@nreese how do those tests look?

nreese · 2018-08-24T14:16:24Z

@alexfrancoeur tests look great. Nice work adding another data set

elasticmachine · 2018-08-28T18:31:36Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request

nreese · 2018-08-28T18:47:10Z

test/functional/apps/home/_sample_data.js

@@ -87,15 +106,40 @@ export default function ({ getService, getPageObjects }) {
        const tsvb = await find.existsByCssSelector('.vega-view-container');
        expect(tsvb).to.be(true);
      });
+
+      it.skip('should launch sample logs data set dashboard', async ()=> {


This test should not be skipped. It is failing because PageObjects.home.launchSampleDataSet('logs') needs to be run from from the kibana home page tutorial_directory/sampleData. You had it right by changing after to afterEach.

nreese · 2018-08-28T18:47:31Z

test/functional/apps/home/_sample_data.js

+      it('should uninstall logs sample data set', async ()=> {
+        await PageObjects.home.removeSampleDataSet('logs');
+        await retry.try(async () => {
+          const successToastExists = await PageObjects.home.doesSampleDataSetSuccessfulUninstallToastExist();


Remove the check for the toast notifications. Those are flaky.

++ removed, testing locally (because I just figured out how to do that 😄 )

nreese · 2018-08-28T18:47:39Z

test/functional/apps/home/_sample_data.js

+    it('should install logs sample data set', async ()=> {
+      await PageObjects.home.addSampleDataSet('logs');
+      await retry.try(async () => {
+        const successToastExists = await PageObjects.home.doesSampleDataSetSuccessfulInstallToastExist();


Remove the check for the toast notifications. Those are flaky.

elasticmachine · 2018-08-28T19:05:45Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request

elasticmachine · 2018-08-28T21:10:06Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request

nreese

Looks great. Nice to have another sample data set with a great looking dashboard

lgtm
code review, installed sample data set and viewed dashboard.

alexfrancoeur · 2018-08-28T21:23:13Z

nice! thank you @nreese 😄

AlonaNadler · 2018-08-29T01:19:03Z

@alexfrancoeur @nreese Great job on the sample web logs!!
Few suggestions:

Can you add trend logs to the TSVB table, it's a features users don't know exists, even if it won't make sense it demonstrates a capability we have

Why do you have - in the custom label, it looks like there is an error when it's presented in the dashboard

I prefer to have panels titles, users are dropped to this dashboard, any additional context can help, especially when it comes to logs when your metrics might inspire new ideas in them
Change the prefix you give to all objects from [logs] to [sample logs] since it might get mixed with any future log users will ingest
Change the title of Sample log data to Sample web logs these 2 panels looks identical, it will be easier to differentiate when just quickly scanning them

nreese · 2018-08-29T02:53:00Z

@alexfrancoeur @nreese Great job on the sample web logs!!

It was all @alexfrancoeur.

alexfrancoeur · 2018-08-29T13:33:37Z

@nreese stop being modest - I copied all your code 😄

@AlonaNadler great feedback! Updated and committed. Once green, I'll merge

elasticmachine · 2018-08-29T14:46:04Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request

nreese · 2018-08-29T19:44:41Z

jenkins, test this

elasticmachine · 2018-08-29T21:45:27Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request

Log sample data, check ✅

elasticmachine · 2018-08-30T16:07:35Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request

alexfrancoeur added 2 commits August 22, 2018 16:58

first take at logs sample data

fa05b8a

typo

f8df0e8

alexfrancoeur added Feature:Add Data Add Data and sample data feature on Home Feature:Home Kibana home application labels Aug 22, 2018

alexfrancoeur requested review from jamiesmith, nreese and asawariS August 22, 2018 21:28

smaller and same sized images and function rename

d690cd6

alexfrancoeur mentioned this pull request Aug 22, 2018

Add Additional Sample Data Sets #21604

Closed

fixed index pattern so that it matches dataset

74176da

jamiesmith approved these changes Aug 23, 2018

View reviewed changes

some dashboard tweaks

3213662

alexfrancoeur added 7 commits August 24, 2018 09:41

tests and updated data set

ef08257

first take at logs sample data

06bfaf2

typo

9f497d3

smaller and same sized images and function rename

cd7c407

fixed index pattern so that it matches dataset

1aa2d16

some dashboard tweaks

1f26258

tests and updated data set

19e5c3c

remove flaky test

41ce8f6

nreese reviewed Aug 28, 2018

View reviewed changes

alexfrancoeur added 2 commits August 28, 2018 15:18

really listening to Nathan this time

2fc50b0

really listening to Nathan this time (for real)

b6faa50

nreese approved these changes Aug 28, 2018

View reviewed changes

incorporated Alonas suggestions

b694f12

alexfrancoeur merged commit 3e52847 into elastic:master Aug 29, 2018

alexfrancoeur mentioned this pull request Aug 30, 2018

[6.x] [WIP] Logs Sample Data (#22276) #22530

Merged

alexfrancoeur pushed a commit to alexfrancoeur/kibana that referenced this pull request Aug 30, 2018

[WIP] Logs Sample Data (elastic#22276)

1dcbf4c

Log sample data, check ✅

alexfrancoeur pushed a commit that referenced this pull request Aug 30, 2018

[WIP] Logs Sample Data (#22276) (#22530)

df8b830

Log sample data, check ✅

nreese added release_note:enhancement v7.0.0 v6.5.0 labels Aug 30, 2018

nreese changed the title ~~[WIP] Logs Sample Data~~ Logs Sample Data Aug 30, 2018

nreese added the :Sharing label Aug 30, 2018

This was referenced Sep 10, 2018

[Sample Data] Add Canvas Workpad for Web Logs #22892

Closed

Update "Loading sample data" documentation #22446

Closed

jgowdyelastic mentioned this pull request Feb 28, 2023

[ML] sample data set ends after "now" #152317

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Logs Sample Data #22276

Logs Sample Data #22276

alexfrancoeur commented Aug 22, 2018 •

edited

Loading

elasticmachine commented Aug 22, 2018

jamiesmith left a comment

alexfrancoeur commented Aug 23, 2018

nreese commented Aug 23, 2018

elasticmachine commented Aug 23, 2018

alexfrancoeur commented Aug 23, 2018

alexfrancoeur commented Aug 23, 2018

nreese commented Aug 23, 2018

elasticmachine commented Aug 23, 2018

alexfrancoeur commented Aug 24, 2018

alexfrancoeur commented Aug 24, 2018

nreese commented Aug 24, 2018

elasticmachine commented Aug 28, 2018

nreese Aug 28, 2018

alexfrancoeur Aug 28, 2018

nreese Aug 28, 2018

alexfrancoeur Aug 28, 2018

nreese Aug 28, 2018

elasticmachine commented Aug 28, 2018

elasticmachine commented Aug 28, 2018

nreese left a comment

alexfrancoeur commented Aug 28, 2018

AlonaNadler commented Aug 29, 2018 •

edited

Loading

nreese commented Aug 29, 2018

alexfrancoeur commented Aug 29, 2018

elasticmachine commented Aug 29, 2018

nreese commented Aug 29, 2018

elasticmachine commented Aug 29, 2018

elasticmachine commented Aug 30, 2018

Logs Sample Data #22276

Logs Sample Data #22276

Conversation

alexfrancoeur commented Aug 22, 2018 • edited Loading

Screenshots

Sample document

elasticmachine commented Aug 22, 2018

💚 Build Succeeded

jamiesmith left a comment

Choose a reason for hiding this comment

alexfrancoeur commented Aug 23, 2018

nreese commented Aug 23, 2018

elasticmachine commented Aug 23, 2018

💚 Build Succeeded

alexfrancoeur commented Aug 23, 2018

alexfrancoeur commented Aug 23, 2018

nreese commented Aug 23, 2018

elasticmachine commented Aug 23, 2018

💚 Build Succeeded

alexfrancoeur commented Aug 24, 2018

alexfrancoeur commented Aug 24, 2018

nreese commented Aug 24, 2018

elasticmachine commented Aug 28, 2018

💔 Build Failed

nreese Aug 28, 2018

Choose a reason for hiding this comment

alexfrancoeur Aug 28, 2018

Choose a reason for hiding this comment

nreese Aug 28, 2018

Choose a reason for hiding this comment

alexfrancoeur Aug 28, 2018

Choose a reason for hiding this comment

nreese Aug 28, 2018

Choose a reason for hiding this comment

elasticmachine commented Aug 28, 2018

💔 Build Failed

elasticmachine commented Aug 28, 2018

💚 Build Succeeded

nreese left a comment

Choose a reason for hiding this comment

alexfrancoeur commented Aug 28, 2018

AlonaNadler commented Aug 29, 2018 • edited Loading

nreese commented Aug 29, 2018

alexfrancoeur commented Aug 29, 2018

elasticmachine commented Aug 29, 2018

💔 Build Failed

nreese commented Aug 29, 2018

elasticmachine commented Aug 29, 2018

💚 Build Succeeded

elasticmachine commented Aug 30, 2018

💚 Build Succeeded

alexfrancoeur commented Aug 22, 2018 •

edited

Loading

AlonaNadler commented Aug 29, 2018 •

edited

Loading