Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

For Cloud: include cribl section in configuration output #918

Closed
jrcheli opened this issue May 2, 2022 · 2 comments · Fixed by #966
Closed

For Cloud: include cribl section in configuration output #918

jrcheli opened this issue May 2, 2022 · 2 comments · Fixed by #966
Assignees
@jrcheli
Milestone
1.1.0

Comments

@jrcheli
Copy link
Contributor

jrcheli commented May 2, 2022

For debugging we output the process configuration in our logging output if the logging level is set to do so ("info", I think?). Right now the cribl section is not included in the output which makes it hard to double-check configuration settings when using this cribl mode. Here's what I mean by cribl section (I grabbed this from the input scope.yml file, but we should output the section in json as we do with the rest of our config):

cribl:
  enable: true
  transport:
    type: edge
    host: 127.0.0.1
    port: 10090
    tls:
      enable: false
      validateserver: true
      cacertpath: ''
@ghost ghost added this to the Next Minor (1.1.0) milestone May 4, 2022
@jrcheli
Copy link
Contributor Author

jrcheli commented May 25, 2022

I added the cribl section to json output, and added unit tests for the same.

For confirmation, I ran these commands, which I expected to write the (default) config of ps -ef to /tmp/scope.log.

rm /tmp/scope.log
LD_PRELOAD=lib/linux/x86_64/libscope.so SCOPE_LOG_LEVEL=info ps -ef
cat /tmp/scope.log

From this, I inspected /tmp/scope.log and saw the cribl output I expected:

Scope: ps(pid:30611): [2022-05-25T03:50:07.654+0000] getting DNS info for udp 127.0.0.1:8125
Scope: ps(pid:30611): [2022-05-25T03:50:07.654+0000] fd:3 connect to 127.0.0.1:8125 was successful
Scope: ps(pid:30611): [2022-05-25T03:50:07.654+0000] The following settings have been overridden by a LogStream connection: event, metric and payload transport, Metrics format, 
Scope: ps(pid:30611): [2022-05-25T03:50:07.654+0000] Constructor (Scope Version: v1.1.0-tc0-64-gc94a220c1513+)
Scope: ps(pid:30611): [2022-05-25T03:50:07.654+0000] command w/args: ps -ef
Scope: ps(pid:30611): [2022-05-25T03:50:07.654+0000] {"current":{"metric":{"enable":"true","transport":{"type":"udp","host":"127.0.0.1","port":"8125","tls":{"enable":"false","validateserver":"true","cacertpath":""}},"format":{"type":"ndjson","statsdprefix":"","statsdmaxlen":512,"verbosity":4},"watch":[{"type":"statsd"}]},"libscope":{"log":{"level":"info","transport":{"type":"file","path":"/tmp/scope.log","buffering":"line"}},"configevent":"true","summaryperiod":10,"commanddir":"/tmp"},"event":{"enable":"true","transport":{"type":"edge"},"format":{"type":"ndjson","maxeventpersec":10000,"enhancefs":"true"},"watch":[{"type":"file","name":"(\\/logs?\\/)|(\\.log$)|(\\.log[.\\d])","field":".*","value":".*"},{"type":"console","name":"(stdout)|(stderr)","field":".*","value":".*"},{"type":"http","name":".*","field":".*","value":".*","headers":[]},{"type":"net","name":".*","field":".*","value":".*"},{"type":"fs","name":".*","field":".*","value":".*"},{"type":"dns","name":".*","field":".*","value":".*"}]},"payload":{"enable":"false","dir":"/tmp"},"tags":{},"protocol":[],"cribl":{"enable":"true","transport":{"type":"edge"},"authtoken":""}}}

By piping the the json portion of this last line through jq, I made it easier to confirm that the cribl node and appropriate sub-nodes are there, with default values:

...
    "protocol": [],
    "cribl": {
      "enable": "true",
      "transport": {
        "type": "edge"
      },
      "authtoken": ""
    }
  }
}

jrcheli added a commit that referenced this issue May 25, 2022
@jrcheli
(#918) output the cribl section when outputting config.
dd8f2da
@jrcheli jrcheli mentioned this issue May 25, 2022
Merged
@michalbiesek
Copy link
Contributor

TLDR: I see cribl section in the #966.

I have tested the #966 with a following command:

Terminal 1

SCOPE_LOG_LEVEL=info SCOPE_CRIBL_ENABLE=false LD_PRELOAD=libscope.so curl google.com

Terminal 2

nc -l 9109

Besides seeing the cribl section in /tmp/scope.log In terminal 2 I have retrieved the start info message the one described in #780 and it also contains the cribl section

The start info message

on master branch:

{
   "format":"ndjson",
   "info":{
      "process":{
         "libscopever":"v1.1.0-tc0-64-gc94a220c1513",
         "pid":498186,
         "ppid":378270,
         "gid":1000,
         "groupname":"test_user",
         "uid":1000,
         "username":"test_user",
         "hostname":"test_user",
         "procname":"curl",
         "cmd":"curl google.com",
         "id":"test_user-curl-curl google.com"
      },
      "configuration":{
         "current":{
            "metric":{
               "enable":"true",
               "transport":{
                  "type":"udp",
                  "host":"127.0.0.1",
                  "port":"8125",
                  "tls":{
                     "enable":"false",
                     "validateserver":"true",
                     "cacertpath":""
                  }
               },
               "format":{
                  "type":"statsd",
                  "statsdprefix":"",
                  "statsdmaxlen":512,
                  "verbosity":4
               },
               "watch":[
                  {
                     "type":"statsd"
                  }
               ]
            },
            "libscope":{
               "log":{
                  "level":"warning",
                  "transport":{
                     "type":"file",
                     "path":"/tmp/scope.log",
                     "buffering":"line"
                  }
               },
               "configevent":"true",
               "summaryperiod":10,
               "commanddir":"/tmp"
            },
            "event":{
               "enable":"true",
               "transport":{
                  "type":"tcp",
                  "host":"127.0.0.1",
                  "port":"9109",
                  "tls":{
                     "enable":"false",
                     "validateserver":"true",
                     "cacertpath":""
                  }
               },
               "format":{
                  "type":"ndjson",
                  "maxeventpersec":10000,
                  "enhancefs":"true"
               },
               "watch":[
                  {
                     "type":"file",
                     "name":"(\\/logs?\\/)|(\\.log$)|(\\.log[.\\d])",
                     "field":".*",
                     "value":".*"
                  },
                  {
                     "type":"console",
                     "name":"(stdout)|(stderr)",
                     "field":".*",
                     "value":".*"
                  },
                  {
                     "type":"http",
                     "name":".*",
                     "field":".*",
                     "value":".*",
                     "headers":[
                        
                     ]
                  },
                  {
                     "type":"net",
                     "name":".*",
                     "field":".*",
                     "value":".*"
                  },
                  {
                     "type":"fs",
                     "name":".*",
                     "field":".*",
                     "value":".*"
                  },
                  {
                     "type":"dns",
                     "name":".*",
                     "field":".*",
                     "value":".*"
                  }
               ]
            },
            "payload":{
               "enable":"false",
               "dir":"/tmp"
            },
            "tags":{
               
            },
            "protocol":[
               
            ]
         }
      },
      "environment":{
         
      }
   }
}

on #966:

{
   "format":"ndjson",
   "info":{
      "process":{
         "libscopever":"v1.1.0-tc0-65-gdd8f2daf0088",
         "pid":471150,
         "ppid":378270,
         "gid":1000,
         "groupname":"test_user",
         "uid":1000,
         "username":"test_user",
         "hostname":"test_user",
         "procname":"curl",
         "cmd":"curl google.com",
         "id":"test_user-curl-curl google.com"
      },
      "configuration":{
         "current":{
            "metric":{
               "enable":"true",
               "transport":{
                  "type":"udp",
                  "host":"127.0.0.1",
                  "port":"8125",
                  "tls":{
                     "enable":"false",
                     "validateserver":"true",
                     "cacertpath":""
                  }
               },
               "format":{
                  "type":"statsd",
                  "statsdprefix":"",
                  "statsdmaxlen":512,
                  "verbosity":4
               },
               "watch":[
                  {
                     "type":"statsd"
                  }
               ]
            },
            "libscope":{
               "log":{
                  "level":"warning",
                  "transport":{
                     "type":"file",
                     "path":"/tmp/scope.log",
                     "buffering":"line"
                  }
               },
               "configevent":"true",
               "summaryperiod":10,
               "commanddir":"/tmp"
            },
            "event":{
               "enable":"true",
               "transport":{
                  "type":"tcp",
                  "host":"127.0.0.1",
                  "port":"9109",
                  "tls":{
                     "enable":"false",
                     "validateserver":"true",
                     "cacertpath":""
                  }
               },
               "format":{
                  "type":"ndjson",
                  "maxeventpersec":10000,
                  "enhancefs":"true"
               },
               "watch":[
                  {
                     "type":"file",
                     "name":"(\\/logs?\\/)|(\\.log$)|(\\.log[.\\d])",
                     "field":".*",
                     "value":".*"
                  },
                  {
                     "type":"console",
                     "name":"(stdout)|(stderr)",
                     "field":".*",
                     "value":".*"
                  },
                  {
                     "type":"http",
                     "name":".*",
                     "field":".*",
                     "value":".*",
                     "headers":[
                        
                     ]
                  },
                  {
                     "type":"net",
                     "name":".*",
                     "field":".*",
                     "value":".*"
                  },
                  {
                     "type":"fs",
                     "name":".*",
                     "field":".*",
                     "value":".*"
                  },
                  {
                     "type":"dns",
                     "name":".*",
                     "field":".*",
                     "value":".*"
                  }
               ]
            },
            "payload":{
               "enable":"false",
               "dir":"/tmp"
            },
            "tags":{
               
            },
            "protocol":[
               
            ],
            "cribl":{
               "enable":"false",
               "transport":{
                  "type":"edge"
               },
               "authtoken":""
            }
         }
      },
      "environment":{
         
      }
   }
}

The content of /tmp.scope.log

on master branch:

{
   "current":{
      "metric":{
         "enable":"true",
         "transport":{
            "type":"udp",
            "host":"127.0.0.1",
            "port":"8125",
            "tls":{
               "enable":"false",
               "validateserver":"true",
               "cacertpath":""
            }
         },
         "format":{
            "type":"statsd",
            "statsdprefix":"",
            "statsdmaxlen":512,
            "verbosity":4
         },
         "watch":[
            {
               "type":"statsd"
            }
         ]
      },
      "libscope":{
         "log":{
            "level":"info",
            "transport":{
               "type":"file",
               "path":"/tmp/scope.log",
               "buffering":"line"
            }
         },
         "configevent":"true",
         "summaryperiod":10,
         "commanddir":"/tmp"
      },
      "event":{
         "enable":"true",
         "transport":{
            "type":"tcp",
            "host":"127.0.0.1",
            "port":"9109",
            "tls":{
               "enable":"false",
               "validateserver":"true",
               "cacertpath":""
            }
         },
         "format":{
            "type":"ndjson",
            "maxeventpersec":10000,
            "enhancefs":"true"
         },
         "watch":[
            {
               "type":"file",
               "name":"(\\/logs?\\/)|(\\.log$)|(\\.log[.\\d])",
               "field":".*",
               "value":".*"
            },
            {
               "type":"console",
               "name":"(stdout)|(stderr)",
               "field":".*",
               "value":".*"
            },
            {
               "type":"http",
               "name":".*",
               "field":".*",
               "value":".*",
               "headers":[
                  
               ]
            },
            {
               "type":"net",
               "name":".*",
               "field":".*",
               "value":".*"
            },
            {
               "type":"fs",
               "name":".*",
               "field":".*",
               "value":".*"
            },
            {
               "type":"dns",
               "name":".*",
               "field":".*",
               "value":".*"
            }
         ]
      },
      "payload":{
         "enable":"false",
         "dir":"/tmp"
      },
      "tags":{
         
      },
      "protocol":[
         
      ]
   }
}

on #966:

{
   "current":{
      "metric":{
         "enable":"true",
         "transport":{
            "type":"udp",
            "host":"127.0.0.1",
            "port":"8125",
            "tls":{
               "enable":"false",
               "validateserver":"true",
               "cacertpath":""
            }
         },
         "format":{
            "type":"statsd",
            "statsdprefix":"",
            "statsdmaxlen":512,
            "verbosity":4
         },
         "watch":[
            {
               "type":"statsd"
            }
         ]
      },
      "libscope":{
         "log":{
            "level":"info",
            "transport":{
               "type":"file",
               "path":"/tmp/scope.log",
               "buffering":"line"
            }
         },
         "configevent":"true",
         "summaryperiod":10,
         "commanddir":"/tmp"
      },
      "event":{
         "enable":"true",
         "transport":{
            "type":"tcp",
            "host":"127.0.0.1",
            "port":"9109",
            "tls":{
               "enable":"false",
               "validateserver":"true",
               "cacertpath":""
            }
         },
         "format":{
            "type":"ndjson",
            "maxeventpersec":10000,
            "enhancefs":"true"
         },
         "watch":[
            {
               "type":"file",
               "name":"(\\/logs?\\/)|(\\.log$)|(\\.log[.\\d])",
               "field":".*",
               "value":".*"
            },
            {
               "type":"console",
               "name":"(stdout)|(stderr)",
               "field":".*",
               "value":".*"
            },
            {
               "type":"http",
               "name":".*",
               "field":".*",
               "value":".*",
               "headers":[
                  
               ]
            },
            {
               "type":"net",
               "name":".*",
               "field":".*",
               "value":".*"
            },
            {
               "type":"fs",
               "name":".*",
               "field":".*",
               "value":".*"
            },
            {
               "type":"dns",
               "name":".*",
               "field":".*",
               "value":".*"
            }
         ]
      },
      "payload":{
         "enable":"false",
         "dir":"/tmp"
      },
      "tags":{
         
      },
      "protocol":[
         
      ],
      "cribl":{
         "enable":"false",
         "transport":{
            "type":"edge"
         },
         "authtoken":""
      }
   }
}

michalbiesek pushed a commit to michalbiesek/appscope that referenced this issue May 27, 2022
@jrcheli @michalbiesek
(criblio#918) output the cribl section when outputting config.
1c03ade
@abetones abetones mentioned this issue Jun 10, 2022
Closed
@jrcheli jrcheli mentioned this issue Jun 13, 2022
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@jrcheli jrcheli

Labels
None yet
Projects
None yet
Milestone
1.1.0
Development

Successfully merging a pull request may close this issue.

output the cribl section when outputting config criblio/appscope
2 participants
@michalbiesek @jrcheli