Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Batch JSONL Output #5705

Merged
merged 2 commits into from
Nov 15, 2024
Merged

Batch JSONL Output #5705

merged 2 commits into from
Nov 15, 2024

Conversation

kchason
Copy link
Contributor

@kchason kchason commented Oct 8, 2024

Proposed changes

Similar to the batch logic in #5688, this adds the ability to batch outputs in the JSONL export to avoid retaining the entire result set in memory.

Checklist

  • Pull request is created against the dev branch
  • All checks passed (lint, unit/integration/regression tests etc.) with my changes
  • I have added tests that prove my fix is effective or that my feature works
  • I have added necessary documentation (if appropriate)

@GeorginaReeder
Copy link

Thanks so much for your contribution @kchason ! :)

@ehsandeep ehsandeep mentioned this pull request Nov 14, 2024
Open
@ehsandeep ehsandeep merged commit ba6a050 into projectdiscovery:dev Nov 15, 2024
11 of 14 checks passed
@BrewTestBot BrewTestBot mentioned this pull request Nov 21, 2024
Merged
@blsaccess blsaccess mentioned this pull request Nov 22, 2024
Merged
Explorer1092 pushed a commit to Explorer1092/nuclei that referenced this pull request Nov 25, 2024
Merge branch 'projectdiscovery-main'
0344a14 
* projectdiscovery-main: (283 commits)
  upgrade
  ci(generate-docs): push w/o pull (projectdiscovery#5843)
  auto gen docs update
  handle env variables in dynamic secret file (projectdiscovery#5835)
  disable self-contained and file protocol templates as default (projectdiscovery#5825)
  chore(deps): bump github.com/projectdiscovery/hmap from 0.0.65 to 0.0.67 (projectdiscovery#5832)
  feat: Added time based delay analyzer to fuzzing implementation (projectdiscovery#5781)
  fix: data race at `protocolstate`, `contextargs` & some outdated test cases (projectdiscovery#5820)
  chore(deps): bump github.com/projectdiscovery/retryabledns
  chore(deps): bump github.com/projectdiscovery/goflags
  chore(deps): bump github.com/projectdiscovery/dsl from 0.3.0 to 0.3.3
  chore(deps): bump github.com/projectdiscovery/rawhttp
  Batch JSONL output and add trailing commas (projectdiscovery#5705)
  ci: refactor workflows (projectdiscovery#5818)
  chore(deps): bump github.com/golang-jwt/jwt/v4 from 4.5.0 to 4.5.1 (projectdiscovery#5795)
  chore(deps): bump github.com/projectdiscovery/rawhttp (projectdiscovery#5809)
  chore(deps): bump github.com/projectdiscovery/wappalyzergo (projectdiscovery#5808)
  update version
  chore(deps): bump github.com/projectdiscovery/gologger
  chore(deps): bump github.com/projectdiscovery/useragent
  ...

# Conflicts:
#	.github/ISSUE_TEMPLATE/feature_request.md
#	.github/ISSUE_TEMPLATE/issue-report.md
#	.run/DSLFunctionsIT.run.xml
#	DESIGN.md
#	README.md
#	README_CN.md
#	README_ID.md
#	README_KR.md
#	cmd/docgen/docgen.go
#	cmd/functional-test/main.go
#	cmd/functional-test/run.sh
#	cmd/integration-test/custom-dir.go
#	cmd/integration-test/dns.go
#	cmd/integration-test/file.go
#	cmd/integration-test/fuzz.go
#	cmd/integration-test/headless.go
#	cmd/integration-test/http.go
#	cmd/integration-test/integration-test.go
#	cmd/integration-test/library.go
#	cmd/integration-test/loader.go
#	cmd/integration-test/offline-http.go
#	cmd/integration-test/ssl.go
#	cmd/integration-test/template-dir.go
#	cmd/integration-test/template-path.go
#	cmd/integration-test/websocket.go
#	cmd/integration-test/whois.go
#	cmd/integration-test/workflow.go
#	cmd/nuclei/main.go
#	examples/advanced/advanced.go
#	examples/simple/simple.go
#	go.mod
#	go.sum
#	internal/colorizer/colorizer.go
#	internal/runner/banner.go
#	internal/runner/healthcheck.go
#	internal/runner/lazy.go
#	internal/runner/options.go
#	internal/runner/proxy.go
#	internal/runner/runner.go
#	internal/runner/runner_test.go
#	internal/runner/templates.go
#	lib/example_test.go
#	lib/sdk.go
#	lib/sdk_private.go
#	pkg/catalog/disk/find.go
#	pkg/catalog/loader/filter/path_filter.go
#	pkg/catalog/loader/loader.go
#	pkg/catalog/loader/loader_test.go
#	pkg/catalog/loader/remote_loader.go
#	pkg/core/engine.go
#	pkg/core/executors.go
#	pkg/core/workflow_execute.go
#	pkg/core/workflow_execute_test.go
#	pkg/external/customtemplates/azure_blob.go
#	pkg/external/customtemplates/github.go
#	pkg/external/customtemplates/github_test.go
#	pkg/external/customtemplates/gitlab.go
#	pkg/external/customtemplates/s3.go
#	pkg/external/customtemplates/templates_provider.go
#	pkg/fuzz/component/path.go
#	pkg/fuzz/fuzz.go
#	pkg/input/formats/swagger/swagger.go
#	pkg/input/provider/list/hmap.go
#	pkg/input/provider/list/hmap_test.go
#	pkg/input/transform.go
#	pkg/installer/template.go
#	pkg/installer/template_test.go
#	pkg/installer/util.go
#	pkg/installer/versioncheck.go
#	pkg/installer/versioncheck_test.go
#	pkg/js/compiler/compiler.go
#	pkg/js/compiler/init.go
#	pkg/js/global/scripts.go
#	pkg/js/libs/mssql/mssql.go
#	pkg/js/libs/postgres/postgres.go
#	pkg/loader/workflow/workflow_loader.go
#	pkg/model/model.go
#	pkg/model/model_test.go
#	pkg/model/types/severity/severities.go
#	pkg/model/types/stringslice/stringslice.go
#	pkg/operators/common/dsl/dsl.go
#	pkg/operators/extractors/compile.go
#	pkg/operators/extractors/extract.go
#	pkg/operators/matchers/compile.go
#	pkg/operators/matchers/match.go
#	pkg/operators/operators.go
#	pkg/output/format_screen.go
#	pkg/output/output.go
#	pkg/output/output_test.go
#	pkg/protocols/common/expressions/expressions.go
#	pkg/protocols/common/expressions/variables.go
#	pkg/protocols/common/generators/generators.go
#	pkg/protocols/common/generators/generators_test.go
#	pkg/protocols/common/generators/options.go
#	pkg/protocols/common/generators/validate.go
#	pkg/protocols/common/helpers/eventcreator/eventcreator.go
#	pkg/protocols/common/helpers/responsehighlighter/response_highlighter.go
#	pkg/protocols/common/helpers/responsehighlighter/response_highlighter_test.go
#	pkg/protocols/common/helpers/writer/writer.go
#	pkg/protocols/common/interactsh/interactsh.go
#	pkg/protocols/common/interactsh/options.go
#	pkg/protocols/common/protocolstate/state.go
#	pkg/protocols/common/replacer/replacer.go
#	pkg/protocols/common/uncover/uncover.go
#	pkg/protocols/common/utils/vardump/dump.go
#	pkg/protocols/common/variables/variables.go
#	pkg/protocols/dns/dns.go
#	pkg/protocols/dns/dns_test.go
#	pkg/protocols/dns/dnsclientpool/clientpool.go
#	pkg/protocols/dns/operators.go
#	pkg/protocols/dns/operators_test.go
#	pkg/protocols/dns/request.go
#	pkg/protocols/dns/request_test.go
#	pkg/protocols/file/file.go
#	pkg/protocols/file/find_test.go
#	pkg/protocols/file/operators.go
#	pkg/protocols/file/operators_test.go
#	pkg/protocols/file/request.go
#	pkg/protocols/file/request_test.go
#	pkg/protocols/headless/engine/engine.go
#	pkg/protocols/headless/engine/http_client.go
#	pkg/protocols/headless/engine/instance.go
#	pkg/protocols/headless/engine/page_actions.go
#	pkg/protocols/headless/engine/page_actions_test.go
#	pkg/protocols/headless/engine/util.go
#	pkg/protocols/headless/headless.go
#	pkg/protocols/headless/operators.go
#	pkg/protocols/headless/request.go
#	pkg/protocols/http/build_request.go
#	pkg/protocols/http/build_request_test.go
#	pkg/protocols/http/cluster.go
#	pkg/protocols/http/http.go
#	pkg/protocols/http/http_test.go
#	pkg/protocols/http/httpclientpool/clientpool.go
#	pkg/protocols/http/operators.go
#	pkg/protocols/http/operators_test.go
#	pkg/protocols/http/request.go
#	pkg/protocols/http/request_annotations.go
#	pkg/protocols/http/request_annotations_test.go
#	pkg/protocols/http/request_fuzz.go
#	pkg/protocols/http/request_generator.go
#	pkg/protocols/http/request_generator_test.go
#	pkg/protocols/http/signature.go
#	pkg/protocols/http/signer/signer.go
#	pkg/protocols/http/signerpool/signerpool.go
#	pkg/protocols/javascript/js.go
#	pkg/protocols/network/network.go
#	pkg/protocols/network/network_test.go
#	pkg/protocols/network/networkclientpool/clientpool.go
#	pkg/protocols/network/operators.go
#	pkg/protocols/network/operators_test.go
#	pkg/protocols/network/request.go
#	pkg/protocols/network/request_test.go
#	pkg/protocols/offlinehttp/find_test.go
#	pkg/protocols/offlinehttp/offlinehttp.go
#	pkg/protocols/offlinehttp/operators.go
#	pkg/protocols/offlinehttp/operators_test.go
#	pkg/protocols/offlinehttp/request.go
#	pkg/protocols/protocols.go
#	pkg/protocols/ssl/ssl.go
#	pkg/protocols/ssl/ssl_test.go
#	pkg/protocols/utils/http/variables.go
#	pkg/protocols/utils/http/variables_test.go
#	pkg/protocols/utils/utils.go
#	pkg/protocols/websocket/websocket.go
#	pkg/protocols/whois/rdapclientpool/clientpool.go
#	pkg/protocols/whois/whois.go
#	pkg/reporting/client.go
#	pkg/reporting/dedupe/dedupe.go
#	pkg/reporting/dedupe/dedupe_test.go
#	pkg/reporting/exporters/es/elasticsearch.go
#	pkg/reporting/exporters/jsonexporter/jsonexporter.go
#	pkg/reporting/exporters/markdown/markdown.go
#	pkg/reporting/exporters/sarif/sarif.go
#	pkg/reporting/exporters/splunk/splunkhec.go
#	pkg/reporting/format/format_utils.go
#	pkg/reporting/format/format_utils_test.go
#	pkg/reporting/options.go
#	pkg/reporting/reporting.go
#	pkg/reporting/trackers/jira/jira.go
#	pkg/scan/scan_context.go
#	pkg/templates/cluster.go
#	pkg/templates/compile.go
#	pkg/templates/compile_test.go
#	pkg/templates/parser_test.go
#	pkg/templates/tag_filter.go
#	pkg/templates/tag_filter_test.go
#	pkg/templates/templates.go
#	pkg/templates/templates_doc_examples.go
#	pkg/templates/types/types.go
#	pkg/templates/workflows.go
#	pkg/testutils/testutils.go
#	pkg/tmplexec/exec.go
#	pkg/tmplexec/flow/flow_executor.go
#	pkg/tmplexec/interface.go
#	pkg/tmplexec/multiproto/multi.go
#	pkg/tmplexec/multiproto/multi_test.go
#	pkg/types/interfaces.go
#	pkg/types/resume.go
#	pkg/types/types.go
#	pkg/utils/template_path.go
#	pkg/utils/utils.go
#	pkg/utils/yaml/preprocess.go
#	pkg/workflows/workflows.go
#	pkg/workflows/workflows_test.go
Explorer1092 pushed a commit to Explorer1092/nuclei that referenced this pull request Nov 25, 2024
Merge branch 'projectdiscovery-main'
74103df 
* projectdiscovery-main: (283 commits)
  upgrade
  ci(generate-docs): push w/o pull (projectdiscovery#5843)
  auto gen docs update
  handle env variables in dynamic secret file (projectdiscovery#5835)
  disable self-contained and file protocol templates as default (projectdiscovery#5825)
  chore(deps): bump github.com/projectdiscovery/hmap from 0.0.65 to 0.0.67 (projectdiscovery#5832)
  feat: Added time based delay analyzer to fuzzing implementation (projectdiscovery#5781)
  fix: data race at `protocolstate`, `contextargs` & some outdated test cases (projectdiscovery#5820)
  chore(deps): bump github.com/projectdiscovery/retryabledns
  chore(deps): bump github.com/projectdiscovery/goflags
  chore(deps): bump github.com/projectdiscovery/dsl from 0.3.0 to 0.3.3
  chore(deps): bump github.com/projectdiscovery/rawhttp
  Batch JSONL output and add trailing commas (projectdiscovery#5705)
  ci: refactor workflows (projectdiscovery#5818)
  chore(deps): bump github.com/golang-jwt/jwt/v4 from 4.5.0 to 4.5.1 (projectdiscovery#5795)
  chore(deps): bump github.com/projectdiscovery/rawhttp (projectdiscovery#5809)
  chore(deps): bump github.com/projectdiscovery/wappalyzergo (projectdiscovery#5808)
  update version
  chore(deps): bump github.com/projectdiscovery/gologger
  chore(deps): bump github.com/projectdiscovery/useragent
  ...

# Conflicts:
#	.github/ISSUE_TEMPLATE/feature_request.md
#	.github/ISSUE_TEMPLATE/issue-report.md
#	.run/DSLFunctionsIT.run.xml
#	DESIGN.md
#	README.md
#	README_CN.md
#	README_ID.md
#	README_KR.md
#	cmd/docgen/docgen.go
#	cmd/functional-test/main.go
#	cmd/functional-test/run.sh
#	cmd/integration-test/custom-dir.go
#	cmd/integration-test/dns.go
#	cmd/integration-test/file.go
#	cmd/integration-test/fuzz.go
#	cmd/integration-test/headless.go
#	cmd/integration-test/http.go
#	cmd/integration-test/integration-test.go
#	cmd/integration-test/library.go
#	cmd/integration-test/loader.go
#	cmd/integration-test/offline-http.go
#	cmd/integration-test/ssl.go
#	cmd/integration-test/template-dir.go
#	cmd/integration-test/template-path.go
#	cmd/integration-test/websocket.go
#	cmd/integration-test/whois.go
#	cmd/integration-test/workflow.go
#	cmd/nuclei/main.go
#	examples/advanced/advanced.go
#	examples/simple/simple.go
#	go.mod
#	go.sum
#	internal/colorizer/colorizer.go
#	internal/runner/banner.go
#	internal/runner/healthcheck.go
#	internal/runner/lazy.go
#	internal/runner/options.go
#	internal/runner/proxy.go
#	internal/runner/runner.go
#	internal/runner/runner_test.go
#	internal/runner/templates.go
#	lib/example_test.go
#	lib/sdk.go
#	lib/sdk_private.go
#	pkg/catalog/disk/find.go
#	pkg/catalog/loader/filter/path_filter.go
#	pkg/catalog/loader/loader.go
#	pkg/catalog/loader/loader_test.go
#	pkg/catalog/loader/remote_loader.go
#	pkg/core/engine.go
#	pkg/core/executors.go
#	pkg/core/workflow_execute.go
#	pkg/core/workflow_execute_test.go
#	pkg/external/customtemplates/azure_blob.go
#	pkg/external/customtemplates/github.go
#	pkg/external/customtemplates/github_test.go
#	pkg/external/customtemplates/gitlab.go
#	pkg/external/customtemplates/s3.go
#	pkg/external/customtemplates/templates_provider.go
#	pkg/fuzz/component/path.go
#	pkg/fuzz/fuzz.go
#	pkg/input/formats/swagger/swagger.go
#	pkg/input/provider/list/hmap.go
#	pkg/input/provider/list/hmap_test.go
#	pkg/input/transform.go
#	pkg/installer/template.go
#	pkg/installer/template_test.go
#	pkg/installer/util.go
#	pkg/installer/versioncheck.go
#	pkg/installer/versioncheck_test.go
#	pkg/js/compiler/compiler.go
#	pkg/js/compiler/init.go
#	pkg/js/global/scripts.go
#	pkg/js/libs/mssql/mssql.go
#	pkg/js/libs/postgres/postgres.go
#	pkg/loader/workflow/workflow_loader.go
#	pkg/model/model.go
#	pkg/model/model_test.go
#	pkg/model/types/severity/severities.go
#	pkg/model/types/stringslice/stringslice.go
#	pkg/operators/common/dsl/dsl.go
#	pkg/operators/extractors/compile.go
#	pkg/operators/extractors/extract.go
#	pkg/operators/matchers/compile.go
#	pkg/operators/matchers/match.go
#	pkg/operators/operators.go
#	pkg/output/format_screen.go
#	pkg/output/output.go
#	pkg/output/output_test.go
#	pkg/protocols/common/expressions/expressions.go
#	pkg/protocols/common/expressions/variables.go
#	pkg/protocols/common/generators/generators.go
#	pkg/protocols/common/generators/generators_test.go
#	pkg/protocols/common/generators/options.go
#	pkg/protocols/common/generators/validate.go
#	pkg/protocols/common/helpers/eventcreator/eventcreator.go
#	pkg/protocols/common/helpers/responsehighlighter/response_highlighter.go
#	pkg/protocols/common/helpers/responsehighlighter/response_highlighter_test.go
#	pkg/protocols/common/helpers/writer/writer.go
#	pkg/protocols/common/interactsh/interactsh.go
#	pkg/protocols/common/interactsh/options.go
#	pkg/protocols/common/protocolstate/state.go
#	pkg/protocols/common/replacer/replacer.go
#	pkg/protocols/common/uncover/uncover.go
#	pkg/protocols/common/utils/vardump/dump.go
#	pkg/protocols/common/variables/variables.go
#	pkg/protocols/dns/dns.go
#	pkg/protocols/dns/dns_test.go
#	pkg/protocols/dns/dnsclientpool/clientpool.go
#	pkg/protocols/dns/operators.go
#	pkg/protocols/dns/operators_test.go
#	pkg/protocols/dns/request.go
#	pkg/protocols/dns/request_test.go
#	pkg/protocols/file/file.go
#	pkg/protocols/file/find_test.go
#	pkg/protocols/file/operators.go
#	pkg/protocols/file/operators_test.go
#	pkg/protocols/file/request.go
#	pkg/protocols/file/request_test.go
#	pkg/protocols/headless/engine/engine.go
#	pkg/protocols/headless/engine/http_client.go
#	pkg/protocols/headless/engine/instance.go
#	pkg/protocols/headless/engine/page_actions.go
#	pkg/protocols/headless/engine/page_actions_test.go
#	pkg/protocols/headless/engine/util.go
#	pkg/protocols/headless/headless.go
#	pkg/protocols/headless/operators.go
#	pkg/protocols/headless/request.go
#	pkg/protocols/http/build_request.go
#	pkg/protocols/http/build_request_test.go
#	pkg/protocols/http/cluster.go
#	pkg/protocols/http/http.go
#	pkg/protocols/http/http_test.go
#	pkg/protocols/http/httpclientpool/clientpool.go
#	pkg/protocols/http/operators.go
#	pkg/protocols/http/operators_test.go
#	pkg/protocols/http/request.go
#	pkg/protocols/http/request_annotations.go
#	pkg/protocols/http/request_annotations_test.go
#	pkg/protocols/http/request_fuzz.go
#	pkg/protocols/http/request_generator.go
#	pkg/protocols/http/request_generator_test.go
#	pkg/protocols/http/signature.go
#	pkg/protocols/http/signer/signer.go
#	pkg/protocols/http/signerpool/signerpool.go
#	pkg/protocols/javascript/js.go
#	pkg/protocols/network/network.go
#	pkg/protocols/network/network_test.go
#	pkg/protocols/network/networkclientpool/clientpool.go
#	pkg/protocols/network/operators.go
#	pkg/protocols/network/operators_test.go
#	pkg/protocols/network/request.go
#	pkg/protocols/network/request_test.go
#	pkg/protocols/offlinehttp/find_test.go
#	pkg/protocols/offlinehttp/offlinehttp.go
#	pkg/protocols/offlinehttp/operators.go
#	pkg/protocols/offlinehttp/operators_test.go
#	pkg/protocols/offlinehttp/request.go
#	pkg/protocols/protocols.go
#	pkg/protocols/ssl/ssl.go
#	pkg/protocols/ssl/ssl_test.go
#	pkg/protocols/utils/http/variables.go
#	pkg/protocols/utils/http/variables_test.go
#	pkg/protocols/utils/utils.go
#	pkg/protocols/websocket/websocket.go
#	pkg/protocols/whois/rdapclientpool/clientpool.go
#	pkg/protocols/whois/whois.go
#	pkg/reporting/client.go
#	pkg/reporting/dedupe/dedupe.go
#	pkg/reporting/dedupe/dedupe_test.go
#	pkg/reporting/exporters/es/elasticsearch.go
#	pkg/reporting/exporters/jsonexporter/jsonexporter.go
#	pkg/reporting/exporters/markdown/markdown.go
#	pkg/reporting/exporters/sarif/sarif.go
#	pkg/reporting/exporters/splunk/splunkhec.go
#	pkg/reporting/format/format_utils.go
#	pkg/reporting/format/format_utils_test.go
#	pkg/reporting/options.go
#	pkg/reporting/reporting.go
#	pkg/reporting/trackers/jira/jira.go
#	pkg/scan/scan_context.go
#	pkg/templates/cluster.go
#	pkg/templates/compile.go
#	pkg/templates/compile_test.go
#	pkg/templates/parser_test.go
#	pkg/templates/tag_filter.go
#	pkg/templates/tag_filter_test.go
#	pkg/templates/templates.go
#	pkg/templates/templates_doc_examples.go
#	pkg/templates/types/types.go
#	pkg/templates/workflows.go
#	pkg/testutils/testutils.go
#	pkg/tmplexec/exec.go
#	pkg/tmplexec/flow/flow_executor.go
#	pkg/tmplexec/interface.go
#	pkg/tmplexec/multiproto/multi.go
#	pkg/tmplexec/multiproto/multi_test.go
#	pkg/types/interfaces.go
#	pkg/types/resume.go
#	pkg/types/types.go
#	pkg/utils/template_path.go
#	pkg/utils/utils.go
#	pkg/utils/yaml/preprocess.go
#	pkg/workflows/workflows.go
#	pkg/workflows/workflows_test.go
@bf-rbrown bf-rbrown mentioned this pull request Nov 27, 2024
Open
1 task
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@dogancanbakir dogancanbakir dogancanbakir approved these changes

@ehsandeep ehsandeep ehsandeep approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@kchason @GeorginaReeder @ehsandeep @dogancanbakir