Skip to content

Commit

Permalink
Merge remote-tracking branch 'upstream/master' into feature/ci-pipeli…
Browse files Browse the repository at this point in the history 
…ne-2.0

* upstream/master: (87 commits)
  [packaging] Normalise GCP bucket folder structure (elastic#20903)
  [Metricbeat] Add billing metricset into googlecloud module (elastic#20812)
  Include python docs in devguide index (elastic#20917)
  Avoid generating incomplete configurations in autodiscover (elastic#20898)
  Improve docs of leaderelection configuration (elastic#20601)
  Document how to set the ES host and Kibana URLs in Ingest Manager (elastic#20874)
  docs: Update beats for APM (elastic#20881)
  Adding cborbeat to community beats (elastic#20884)
  Bump kibana version to 7.9.0 in x-pack/metricbeat (elastic#20899)
  Kubernetes state_daemonset metricset for Metricbeat (elastic#20649)
  [Filebeat][zeek] Add new x509 fields to zeek (elastic#20867)
  [Filebeat][Gsuite] Add note about admin in gsuite docs (elastic#20855)
  Ensure kind cluster has RFC1123 compliant name (elastic#20627)
  Setup python paths in test runner configuration (elastic#20832)
  docs: Add  `processor.event` info to Logstash output (elastic#20721)
  docs: update cipher suites (elastic#20697)
  [ECS] Update ecs to 1.6.0 (elastic#20792)
  Fix path in hits docs (elastic#20447)
  Update filebeat azure module documentation (elastic#20815)
  Remove duplicate ListGroupsForUsers in aws/cloudtrail (elastic#20788)
  ...
  • Loading branch information
@v1v
v1v committed Sep 2, 2020
2 parents d431b02 + 1757cec commit 1faa11d
Show file tree
Hide file tree
Showing 499 changed files with 60,567 additions and 86,725 deletions.
18 changes: 15 additions & 3 deletions .ci/packaging.groovy
View file
Original file line number Diff line number Diff line change
Expand Up @@ -230,11 +230,11 @@ def withMacOSEnv(Closure body){

def publishPackages(baseDir){
def bucketUri = "gs://${JOB_GCS_BUCKET}/snapshots"
if (env.CHANGE_ID?.trim()) {
if (isPR()) {
bucketUri = "gs://${JOB_GCS_BUCKET}/pull-requests/pr-${env.CHANGE_ID}"
}

googleStorageUpload(bucket: "${bucketUri}",
def beatsFolderName = getBeatsName(baseDir)
googleStorageUpload(bucket: "${bucketUri}/${beatsFolderName}",
credentialsId: "${JOB_GCS_CREDENTIALS}",
pathPrefix: "${baseDir}/build/distributions/",
pattern: "${baseDir}/build/distributions/**/*",
Expand All @@ -243,6 +243,18 @@ def publishPackages(baseDir){
)
}

/**
* There is a specific folder structure in https://staging.elastic.co/ and https://artifacts.elastic.co/downloads/
* therefore the storage bucket in GCP should follow the same folder structure.
* This is required by https://github.com/elastic/beats-tester
* e.g.
* baseDir=name -> return name
* baseDir=name1/name2/name3-> return name2
*/
def getBeatsName(baseDir) {
return basedir.replace('x-pack/', '')
}

def withBeatsEnv(Closure body) {
withMageEnv(){
withEnv([
Expand Down
3 changes: 3 additions & 0 deletions CHANGELOG-developer.next.asciidoc
View file
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,7 @@ The list below covers the major changes between 7.0.0-rc2 and master only.
- Remove `common.MapStrPointer` parameter from `cfgfile.Runnerfactory` interface. {pull}19135[19135]
- Replace `ACKCount`, `ACKEvents`, and `ACKLastEvent` callbacks with `ACKHandler` and interface in `beat.ClientConfig`. {pull}19632[19632]
- Remove global ACK handler support via `SetACKHandler` from publisher pipeline. {pull}19632[19632]
- Make implementing `Close` required for `reader.Reader` interfaces. {pull}20455[20455]

==== Bugfixes

Expand Down Expand Up @@ -99,3 +100,5 @@ The list below covers the major changes between 7.0.0-rc2 and master only.
- Added SQL helper that can be used from any Metricbeat module {pull}18955[18955]
- Update Go version to 1.14.4. {pull}19753[19753]
- Update Go version to 1.14.7. {pull}20508[20508]
- Add packaging for docker image based on UBI minimal 8. {pull}20576[20576]
- Make the mage binary used by the build process in the docker container to be statically compiled. {pull}20827[20827]
294 changes: 76 additions & 218 deletions CHANGELOG.asciidoc
View file

Large diffs are not rendered by default.

46 changes: 42 additions & 4 deletions CHANGELOG.next.asciidoc
View file
Original file line number Diff line number Diff line change
Expand Up @@ -19,10 +19,14 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
- Make error message about locked data path actionable. {pull}18667[18667]
- Ensure dynamic template names are unique for the same field. {pull}18849[18849]
- Remove the deprecated `xpack.monitoring.*` settings. Going forward only `monitoring.*` settings may be used. {issue}9424[9424] {pull}18608[18608]
- Added `certificate` TLS verification mode to ignore server name mismatch. {issue}12283[12283] {pull}20293[20293]
- Autodiscover doesn't generate any configuration when a variable is missing. Previously it generated an incomplete configuration. {pull}20898[20898]

*Auditbeat*

- File integrity dataset (macOS): Replace unnecessary `file.origin.raw` (type keyword) with `file.origin.text` (type `text`). {issue}12423[12423] {pull}15630[15630]
- Change event.kind=error to event.kind=event to comply with ECS. {issue}18870[18870] {pull}20685[20685]
- Change network.direction values to ECS recommended values (inbound, outbound). {issue}12445[12445] {pull}20695[20695]

*Filebeat*

Expand Down Expand Up @@ -65,6 +69,7 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
- Add `while_pattern` type to multiline reader. {pull}19662[19662]
- Tracking session end reason in panw module. {pull}18705[18705]
- Fix PANW field spelling "veredict" to "verdict" on event.action {pull}18808[18808]
- Removed experimental modules `citrix`, `kaspersky`, `rapid7` and `tenable`. {pull}20706[20706]

*Heartbeat*

Expand All @@ -85,6 +90,7 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
*Packetbeat*

- Redis: fix incorrectly handle with two-words redis command. {issue}14872[14872] {pull}14873[14873]
- `event.category` no longer contains the value `network_traffic` because this is not a valid ECS event category value. {pull}20556[20556]

*Winlogbeat*

Expand All @@ -96,6 +102,7 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
- Add Powershell module. Support for event ID's: `400`, `403`, `600`, `800`, `4103`, `4014`, `4105`, `4106`. {issue}16262[16262] {pull}18526[18526]
- Fix Powershell processing of downgraded engine events. {pull}18966[18966]
- Fix unprefixed fields in `fields.yml` for Powershell module {issue}18984[18984]
- Remove top level `hash` property from sysmon events {pull}20653[20653]

*Functionbeat*

Expand Down Expand Up @@ -158,6 +165,10 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
- Fix issue in autodiscover that kept inputs stopped after config updates. {pull}20305[20305]
- Log debug message if the Kibana dashboard can not be imported from the archive because of the invalid archive directory structure {issue}12211[12211], {pull}13387[13387]
- Add service resource in k8s cluster role. {pull}20546[20546]
- [Metricbeat][Kubernetes] Change cluster_ip field from ip to keyword. {pull}20571[20571]
- Rename cloud.provider `az` value to `azure` inside the add_cloud_metadata processor. {pull}20689[20689]
- Add missing country_name geo field in `add_host_metadata` and `add_observer_metadata` processors. {issue}20796[20796] {pull}20811[20811]
- Explicitly detect missing variables in autodiscover configuration, log them at the debug level. {issue}20568[20568] {pull}20898[20898]

*Auditbeat*

Expand Down Expand Up @@ -235,18 +246,25 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
- Fix s3 input parsing json file without expand_event_list_from_field. {issue}19902[19902] {pull}19962[19962] {pull}20370[20370]
- Fix millisecond timestamp normalization issues in CrowdStrike module {issue}20035[20035], {pull}20138[20138]
- Fix support for message code 106100 in Cisco ASA and FTD. {issue}19350[19350] {pull}20245[20245]
- Fix event.outcome logic for azure/siginlogs fileset {pull}20254[20254]
- Fix `fortinet` setting `event.timezone` to the system one when no `tz` field present {pull}20273[20273]
- Fix `okta` geoip lookup in pipeline for `destination.ip` {pull}20454[20454]
- Fix mapping exception in the `googlecloud/audit` dataset pipeline. {issue}18465[18465] {pull}20465[20465]
- Fix `cisco` asa and ftd parsing of messages 106102 and 106103. {pull}20469[20469]
- Improve validation checks for Azure configuration {issue}20369[20369] {pull}20389[20389]
- Fix event.kind for system/syslog pipeline {issue}20365[20365] {pull}20390[20390]
- Clone value when copy fields in processors to avoid crash. {issue}19206[19206] {pull}20500[20500]
- Fix event.type for zeek/ssl and duplicate event.category for zeek/connection {pull}20696[20696]
- Fix long registry migration times. {pull}20717[20717] {issue}20705[20705]
- Fix event types and categories in auditd module to comply with ECS {pull}20652[20652]
- Update documentation in the azure module filebeat. {pull}20815[20815]

*Heartbeat*

- Fixed excessive memory usage introduced in 7.5 due to over-allocating memory for HTTP checks. {pull}15639[15639]
- Fixed TCP TLS checks to properly validate hostnames, this broke in 7.x and only worked for IP SANs. {pull}17549[17549]
- Add support for new `service_name` option to all monitors. {pull}19932[19932].
- Stop rescheduling tasks of stopped monitors. {pull}20570[20570]

*Journalbeat*

Expand Down Expand Up @@ -315,6 +333,10 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
- Add required option for `metrics` in app_insights. {pull}20406[20406]
- Groups same timestamp metric values to one event in the app_insights metricset. {pull}20403[20403]
- Updates vm_compute metricset with more info on guest metrics. {pull}20448[20448]
- Add fallback for PdhExpandWildCardPathW failing in perfmon metricset. {issue}20139[20139] {pull}20630[20630]
- Fix resource tags in aws cloudwatch metricset {issue}20326[20326] {pull}20385[20385]
- Fix ec2 disk and network metrics to use Sum statistic method. {pull}20680[20680]
- Fill cloud.account.name with accountID if account alias doesn't exist. {pull}20736[20736]

*Packetbeat*

Expand All @@ -326,6 +348,8 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d

- Fix invalid IP addresses in DNS query results from Sysmon data. {issue}18432[18432] {pull}18436[18436]
- Fields from Winlogbeat modules were not being included in index templates and patterns. {pull}18983[18983]
- Fix `event.outcome` in the security module for non-English languages. {issue}20079[20079] {pull}20564[20564]
- Fix duplicated field error when exporting index-pattern with migration.6_to_7.enabled. {issue}20521[20521] {pull}20540[20540]

*Functionbeat*

Expand Down Expand Up @@ -381,7 +405,8 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
- Set index.max_docvalue_fields_search in index template to increase value to 200 fields. {issue}20215[20215]
- Add leader election for Kubernetes autodiscover. {pull}20281[20281]
- Add capability of enriching process metadata with contianer id also for non-privileged containers in `add_process_metadata` processor. {pull}19767[19767]

- Added support for more message types for Cisco ASA and FTD. {pull}20565[20565]
- Add replace_fields config option in add_host_metadata for replacing host fields. {pull}20490[20490] {issue}20464[20464]

*Auditbeat*

Expand Down Expand Up @@ -474,7 +499,7 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
- Improve ECS categorization field mappings in system module. {issue}16031[16031] {pull}18065[18065]
- Change the `json.*` input settings implementation to merge parsed json objects with existing objects in the event instead of fully replacing them. {pull}17958[17958]
- Improve ECS categorization field mappings in osquery module. {issue}16176[16176] {pull}17881[17881]
- Added http_endpoint input{pull}18298[18298]
- Add http_endpoint input. {pull}18298[18298]
- Add support for array parsing in azure-eventhub input. {pull}18585[18585]
- Added `observer.vendor`, `observer.product`, and `observer.type` to PANW module events. {pull}18223[18223]
- The `logstash` module can now automatically detect the log file format (JSON or plaintext) and process it accordingly. {issue}9964[9964] {pull}18095[18095]
Expand Down Expand Up @@ -523,18 +548,21 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
- Add support for custom header and headersecret for filebeat http_endpoint input {pull}20435[20435]
- Add event.ingested to all Filebeat modules. {pull}20386[20386]
- Return error when log harvester tries to open a named pipe. {issue}18682[18682] {pull}20450[20450]

- Avoid goroutine leaks in Filebeat readers. {issue}19193[19193] {pull}20455[20455]
- Convert httpjson to v2 input {pull}20226[20226]
- Improve Zeek x509 module with `x509` ECS mappings {pull}20867[20867]

*Heartbeat*

- Allow a list of status codes for HTTP checks. {pull}15587[15587]
- Add additional ECS compatible fields for TLS information. {pull}17687[17687]
- Record HTTP response headers. {pull}18327[18327]
- Add index and pipeline settings to monitor configurations. {pull}20610[20610]

*Journalbeat*

- Added an `id` config option to inputs to allow running multiple inputs on the
same journal. {pull}18467{18467}
same journal. {pull}18467[18467]
- Add basic ECS categorization and `log.syslog` fields. {pull}19176[19176]

*Metricbeat*
Expand Down Expand Up @@ -633,7 +661,13 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
- Added cache and connection_errors metrics to status metricset of MySQL module {issue}16955[16955] {pull}19844[19844]
- Update MySQL dashboard with connection errors and cache metrics {pull}19913[19913] {issue}16955[16955]
- Add cloud.instance.name into aws ec2 metricset. {pull}20077[20077]
- Add host inventory metrics into aws ec2 metricset. {pull}20171[20171]
- Add `scope` setting for elasticsearch module, allowing it to monitor an Elasticsearch cluster behind a load-balancing proxy. {issue}18539[18539] {pull}18547[18547]
- Add state_daemonset metricset for Kubernetes Metricbeat module {pull}20649[20649]
- Add host inventory metrics to azure compute_vm metricset. {pull}20641[20641]
- Add host inventory metrics to googlecloud compute metricset. {pull}20391[20391]
- Request prometheus endpoints to be gzipped by default {pull}20766[20766]
- Add billing metricset into googlecloud module. {pull}20812[20812] {issue}20738[20738]

*Packetbeat*

Expand All @@ -646,6 +680,7 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d

*Functionbeat*
- Add basic ECS categorization and `cloud` fields. {pull}19174[19174]
- Add support for parallelization factor for kinesis. {pull}20727[20727]

*Winlogbeat*

Expand All @@ -655,9 +690,11 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
- Add support for event IDs 4673,4674,4697,4698,4699,4700,4701,4702,4768,4769,4770,4771,4776,4778,4779,4964 to the Security module {pull}17517[17517]
- Add registry and code signature information and ECS categorization fields for sysmon module {pull}18058[18058]
- Add new winlogbeat security dashboard {pull}18775[18775]
- Add `event.outcome` to events based on the audit success and audit failure keywords. {pull}20564[20564]

*Elastic Log Driver*
- Add support for `docker logs` command {pull}19531[19531]
- Add support to change beat name, and support for Kibana Logs. {pull}20522[20522]

==== Deprecated

Expand All @@ -671,6 +708,7 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
*Journalbeat*

*Metricbeat*

- Deprecate tags config parameter in cloudwatch metricset. {pull}16733[16733]
- Deprecate tags.resource_type_filter config parameter and replace with resource_type. {pull}19688[19688]

Expand Down
Loading

0 comments on commit 1faa11d

Please sign in to comment.