[BUG] Issues detecting custom plugins

[lsnow11]

I have created several custom plugins with a prior version of eventgen that I am trying to port.

I have put my plugins in the lib folder structure that sa-eventgen uses (subdirs for each type of plugin). When running v6.3.0, my generator plugins are found but not raters or output. I added logging to list out all the discovered plugins and I see all my generator plugins but none of the others. Moving or copying them to bin does not change the behavior. For the output plugin, it seems to just be ignoring the outputMode setting entirely - I don't receive any errors, but I'm definitely not hitting the plugin.

To Reproduce
Pull the demo-itsi-2019 repo from Splunk internal git, eventgen6 branch.
On a fresh Splunk install, run /opt/splunk/bin/splunk cmd python configureITSIdemo to install the demo. This will take a while.
Enable the Eventgen modular input.
Search index=itsidemo for at least last 60 minutes. You will see five sourcetypes. There should be more that are created by the output plugin.

Expected: at least eight sourcetypes

Actual: five sourcetypes

Screenshots
list of loaded plugins

Sample files and eventgen.conf file
Please grab from Splunk internal git as noted above

Do you run eventgen with SA-eventgen?
Yes

If you are using SA-Eventgen with Splunk (please complete the following information):

• OS: Ubuntu
• Browser FF
• Eventgen Version 6.3.0
• Splunk Version 7.3.0
• What other apps you have installed in Splunk etc/apps? All ITSI apps, my own apps for the demo

Additional context
Saw the same issue with rater, but I was using an older version and have not tested again with 6.3.0. Not being able to find the rater caused eventgen to throw errors that the plugin could not be found. I do not get errors for the missing output plugin, it is just not used.

[li-wu]

@lsnow11 you should place your custom generator plugins under $SPLUNK_HOME/etc/apps/SA-Eventgen/lib/splunk_eventgen/lib/plugins/generator and rater and output plugin in corresponding folder. Also, could you try the latest version of Eventgen which is 6.5.1? Thanks.

[lsnow11]

Even after moving my plugins to the Eventgen app (v6.5.1) it doesn't look like my plugins are being used, though I do see them being loaded.

[transaction.web.before]
disabled = false
generator = transactiongenerator
outputMode = appsummarize
#rater = alternator2
alternate = 1
interval = 60
count = 12
backfill = -1410m
index = itsidemo
domainName = buttercup.com
isErr = 0
isMonitoring = 0
trending = 1
repeating = 1
duration = 660
productsFile = $SPLUNK_HOME/etc/apps/itsidemo_datagen/samples/products.csv
usersFile = $SPLUNK_HOME/etc/apps/itsidemo_datagen/samples/users.csv
hostsFile = $SPLUNK_HOME/etc/apps/itsidemo_datagen/samples/allhosts.csv
awsFile = $SPLUNK_HOME/etc/apps/itsidemo_datagen/samples/awsinstances.csv
authFile = $SPLUNK_HOME/etc/apps/itsidemo_datagen/samples/external_auth.csv
cdnFile = $SPLUNK_HOME/etc/apps/itsidemo_datagen/samples/cloudfront_edge.csv
trendState = $SPLUNK_HOME/etc/apps/itsidemo_datagen/samples/transaction_web_trend.state
rateState = $SPLUNK_HOME/etc/apps/itsidemo_datagen/samples/transaction_web_trend_rate.state

08-20-2019 20:02:50.417 +0000 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/SA-Eventgen/bin/modinput_eventgen.py" 2019-08-20 20:02:50 eventgen DEBUG MainProcess {'event': "Loading module 'output.appsummarize' from 'appsummarize.py'"}

08-20-2019 20:10:56.935 +0000 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/SA-Eventgen/bin/modinput_eventgen.py" 2019-08-20 20:10:56 eventgen DEBUG MainProcess {'event': "Starting OutputPlugin for sample 'transaction.web.before' with output 'modinput'"}

[li-wu]

@lsnow11 I tried with the demo-itsi-2019 repo with eventgen6 branch. Check with apps under demo-itsi-2019/splunk/etc/apps, there are four apps with eventgen.conf and samples:

• Splunk_TA_akamai
• Splunk_TA_aws
• Splunk_TA_New_Relic
• appd_datagen

I can generate all the data after I copy these four apps folder into a new Splunk instance. One important thing is you need to delete the file Splunk_TA_aws/local/eventgen.conf because it disabled all the data generation stanzas in Splunk_TA_aws.

The other thing is custom plugins. I can not get any information about the custom plugins you have written, so I can not give any advice here.

[lsnow11]

The only app I'm concernd with is itsidemo_datagen. The rest use samples rather than plugins. To the best of my knowledge, samples work fine as you have stated.

[li-wu]

I got your plugins from the code base under SA-Eventgen. I tried the following eventgen stanza in your eventgen.conf:

[transaction.web.before]
disabled = false
generator = transactiongenerator
#outputMode = appsummarize
outputMode = stdout
#rater = alternator2
alternate = 1
interval = 60
count = 12
backfill = -1410m
index = itsidemo
domainName = buttercup.com
isErr = 0
isMonitoring = 0
trending = 1
repeating = 1
duration = 660
productsFile = $SPLUNK_HOME/etc/apps/itsidemo_datagen/samples/products.csv
usersFile = $SPLUNK_HOME/etc/apps/itsidemo_datagen/samples/users.csv
hostsFile = $SPLUNK_HOME/etc/apps/itsidemo_datagen/samples/allhosts.csv
awsFile = $SPLUNK_HOME/etc/apps/itsidemo_datagen/samples/awsinstances.csv
authFile = $SPLUNK_HOME/etc/apps/itsidemo_datagen/samples/external_auth.csv
cdnFile = $SPLUNK_HOME/etc/apps/itsidemo_datagen/samples/cloudfront_edge.csv
trendState = $SPLUNK_HOME/etc/apps/itsidemo_datagen/samples/transaction_web_trend.state
rateState = $SPLUNK_HOME/etc/apps/itsidemo_datagen/samples/transaction_web_trend_rate.state

Things I changes for above stanza:

• Change outputMode to stdout to only verify the generator plugin.
• Replace $SPLUNK_HOME with actual path, because Eventgen will not replace the token for custom configuration.

I can get data from stdout successfully.

Plus: I use eventgen pip module to verify above scenario and I think Eventgen app should do the same since they use the same core code.

[lsnow11]

the generator plugin is working fine, that's not the problem. the problem is that the output plugin is supposed to aggregate some of the data from the generator and generate some new data and this is not happening.

[li-wu]

@lsnow11 I tried with your custom output plugin appsummarize.py and made a few changes to it to make it work. Attached the working version.

Here are the changes I made:

• Add a required parameter output_counter=None to __init__ of SummarizeOutputPlugin class.
• Add a required class attribute useOutputQueue = False.
• Import logger from logging_config and replace self.logger with logger.
• Iterate q using for loop since q is a list.

You can use file compare tool to see the changes. Hope it work for you.
We made some changes to the logging part and did not reflect it in the documentation and we will add those changes to doc later. Thanks.

appsummarize.py.zip

[lsnow11]

I took your appsummarize.py and copied it into the repo, but I'm still not seeing it get used by eventgen:
08-23-2019 18:13:16.679 +0000 ERROR ExecProcessor - message from "python /opt/splunk/etc/apps/SA-Eventgen/bin/modinput_eventgen.py" 2019-08-23 18:13:16 eventgen DEBUG MainProcess {'event': "Starting OutputPlugin for sample 'transaction.web.before' with output 'modinput'"}

[li-wu]

SA-Eventgen uses modular input to generate data into splunk. It prints data to stdout and splunkd read data from stdout and get the data into Splunk.

So if you are using the Eventgen App aka SA-Eventgen, you can not change the outputMode = modinput which is default.

An alternative way is put your output logic in modinput.py.

[li-wu]

Feel free to reopen it if you got further issues.