Skip to content

Commit

Permalink
Merge branch 'main' into processor_regex_replace
Browse files Browse the repository at this point in the history
* main:
  fix: missed import_plugins.sh (alibaba#763)
  feat: move helper to pkg/helper and implement decodeV2 of prometheus (alibaba#742)
  fix: gozstd compatiblity in some environment (alibaba#761)
  • Loading branch information
pj1987111 committed Apr 3, 2023
2 parents ea0e3f5 + fc10d19 commit 0c0de9f
Show file tree
Hide file tree
Showing 183 changed files with 2,964 additions and 376 deletions.
71 changes: 71 additions & 0 deletions .github/workflows/build-core.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,71 @@
# Copyright 2021 iLogtail Authors
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

name: Build Core

on:
pull_request:
push:
branches:
- main
- 1.*
jobs:
CI:
runs-on: ${{ matrix.runner }}
timeout-minutes: 30
strategy:
matrix:
go-version: [1.18]
# https://docs.github.com/en/actions/learn-github-actions/workflow-syntax-for-github-actions#jobsjob_idruns-on
runner: [ubuntu-latest, macos-latest, windows-2019]
fail-fast: true
steps:
# prepare ubuntu environment
- name: prepare ubuntu environment
if: matrix.runner == 'ubuntu-latest'
run: sudo apt-get clean && sudo apt-get update && sudo apt-get install -y libsystemd-dev

# prepare windows environment
# https://github.com/actions/virtual-environments/issues/2549
- name: add mingw32 to path
if: matrix.runner == 'windows-2019'
shell: bash
run: |
echo "C:\msys64\mingw32\bin" >> $GITHUB_PATH
- name: Set up Go ${{ matrix.go-version }}
uses: actions/setup-go@v2
with:
go-version: ${{ matrix.go-version }}

- name: Check out code into the Go module directory
uses: actions/checkout@v2
with:
submodules: true

- name: Build Binary
if: matrix.runner == 'ubuntu-latest'
env:
BUILD_LOGTAIL_UT: ON
ENABLE_COMPATIBLE_MODE: ON
ENABLE_STATIC_LINK_CRT: ON
run: make core

result:
runs-on: ubuntu-latest
timeout-minutes: 30
needs: [CI]
steps:
- name: Build Result
run: echo "Just to make the GitHub merge button green"
6 changes: 3 additions & 3 deletions .github/workflows/build.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -57,14 +57,14 @@ jobs:
- name: Build Binary
if: matrix.runner == 'ubuntu-latest'
env:
BUILD_LOGTAIL_UT: ON
BUILD_LOGTAIL_UT: OFF
ENABLE_COMPATIBLE_MODE: ON
ENABLE_STATIC_LINK_CRT: ON
run: make dist
run: make dist && scripts/check_glibc.sh

- name: Build Docker
if: matrix.runner == 'ubuntu-latest'
run: scripts/check_glibc.sh && make docker
run: make docker

result:
runs-on: ubuntu-latest
Expand Down
4 changes: 3 additions & 1 deletion core/unittest/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -59,4 +59,6 @@ add_subdirectory(reader)
add_subdirectory(sender)
add_subdirectory(profiler)
add_subdirectory(sdk)
add_subdirectory(observer)
if (UNIX)
add_subdirectory(observer)
endif ()
2 changes: 1 addition & 1 deletion docs/cn/data-pipeline/extension/ext-default-decoder.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@
## 配置参数
| 参数 | 类型 | 是否必选 | 说明 |
|-------------------|---------|------|-----------------------------------------------------------------------------------------------------|
| Format | String || 具体的协议,[查看支持的具体协议列表](https://github.com/alibaba/ilogtail/blob/master/helper/decoder/common/comon.go) |
| Format | String || 具体的协议,[查看支持的具体协议列表](https://github.com/alibaba/ilogtail/blob/master/pkg/protocol/decoder/common/comon.go) |
| FieldsExtend | Boolean || 是否启用增强字段功能,默认为false,仅针对Format=influxdb时有效 |
| DisableUncompress | Boolean || 否不解压数据,默认为false,仅针对Format=raw时有效 |

Expand Down
2 changes: 1 addition & 1 deletion docs/cn/data-pipeline/input/service-http-server.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@
| 参数 | 类型 | 是否必选 | 说明 |
|--------------------|-------------------|------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| Type | String || 插件类型,固定为`service_http_server` |
| Format | String || <p>数据格式。</p> <p>支持格式:`sls``prometheus``influxdb``otlp_logv1``otlp_metricv1``otlp_tracev1`, `pyroscope`,statsd`</p> <p>v2版本支持格式:`raw`</p><p>说明:`raw`格式以原始请求字节流传输数据</p> |
| Format | String || <p>数据格式。</p> <p>支持格式:`sls``prometheus``influxdb``otlp_logv1``otlp_metricv1``otlp_tracev1`, `pyroscope`,`statsd`</p> <p>v2版本支持格式:`raw`, `promtheus`, `otlp_metricv1``otlp_tracev1`</p><p>说明:`raw`格式以原始请求字节流传输数据</p> |
| Address | String || <p>监听地址。</p><p></p> |
| Path | String || <p>接收端点, 如Format 为 `otlp_logv1` 时, 默认端点为`/v1/logs` 。</p><p></p> |
| ReadTimeoutSec | String || <p>读取超时时间。</p><p>默认取值为:`10s`。</p> |
Expand Down
54 changes: 29 additions & 25 deletions go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ require (
github.com/apache/pulsar-client-go v0.9.0
github.com/bsm/sarama-cluster v2.1.15+incompatible
github.com/buger/jsonparser v1.1.1
github.com/cespare/xxhash v1.1.0
github.com/cespare/xxhash v1.1.0 // indirect
github.com/cespare/xxhash/v2 v2.2.0
github.com/cihub/seelog v0.0.0-20170130134532-f561c5e57575
github.com/coreos/go-systemd v0.0.0-20190719114852-fd7a80b32e1f
Expand All @@ -22,17 +22,15 @@ require (
github.com/elastic/beats/v7 v7.7.1
github.com/elastic/go-elasticsearch/v8 v8.6.0
github.com/elastic/go-lumber v0.1.0
github.com/go-kit/kit v0.12.0
github.com/go-mysql-org/go-mysql v1.3.0
github.com/go-ping/ping v0.0.0-20211130115550-779d1e919534
github.com/go-sql-driver/mysql v1.5.0
github.com/gofrs/uuid v4.2.0+incompatible
github.com/gofrs/uuid v4.2.0+incompatible // indirect
github.com/gogo/protobuf v1.3.2
github.com/gosnmp/gosnmp v1.31.0
github.com/grafana/loki-client-go v0.0.0-20230116142646-e7494d0ef70c
github.com/hashicorp/golang-lru v0.6.0
github.com/influxdata/go-syslog v1.0.1
github.com/influxdata/influxdb v1.10.0
github.com/jackc/pgx/v4 v4.16.1
github.com/jarcoal/httpmock v1.2.0
github.com/jeromer/syslogparser v0.0.0-20190429161531-5fbaaf06d9e7
Expand All @@ -41,41 +39,43 @@ require (
github.com/knz/strtime v0.0.0-20181018220328-af2256ee352c
github.com/mailru/easyjson v0.7.7
github.com/mindprince/gonvml v0.0.0-20180514031326-b364b296c732
github.com/narqo/go-dogstatsd-parser v0.2.0
github.com/oschwald/geoip2-golang v1.1.0
github.com/paulbellamy/ratecounter v0.2.1-0.20170719102518-a803f0e4f071
github.com/pierrec/lz4 v2.6.0+incompatible
github.com/pingcap/check v0.0.0-20200212061837-5e12011dc712
github.com/pkg/errors v0.9.1
github.com/prometheus/common v0.37.0
github.com/prometheus/common v0.42.0
github.com/prometheus/procfs v0.8.0
github.com/prometheus/prometheus v1.8.2-0.20201119142752-3ad25a6dc3d9
github.com/pyroscope-io/jfr-parser v0.6.0
github.com/pyroscope-io/pyroscope v0.0.0-00010101000000-000000000000
github.com/shirou/gopsutil v3.21.11+incompatible
github.com/siddontang/go-mysql v0.0.0-20180725024449-535abe8f2eba
github.com/sirupsen/logrus v1.8.1
github.com/smartystreets/goconvey v1.7.2
github.com/streadway/handy v0.0.0-20230327021402-6a47ec586270
github.com/stretchr/testify v1.8.1
github.com/stretchr/testify v1.8.2
github.com/syndtr/goleveldb v0.0.0-20170725064836-b89cc31ef797
github.com/xdg-go/scram v1.1.1
go.opentelemetry.io/collector/consumer v0.66.0
go.opentelemetry.io/collector/pdata v0.66.0
go.opentelemetry.io/proto/otlp v0.19.0
go.uber.org/atomic v1.10.0
golang.org/x/sys v0.4.0
google.golang.org/grpc v1.52.1
google.golang.org/protobuf v1.28.1
golang.org/x/sys v0.6.0
google.golang.org/grpc v1.53.0
google.golang.org/protobuf v1.29.0
gotest.tools v2.2.0+incompatible
k8s.io/api v0.26.1
k8s.io/apimachinery v0.26.1
k8s.io/client-go v0.26.1
k8s.io/cri-api v0.23.1
)

require (
cloud.google.com/go/compute v1.14.0 // indirect
github.com/go-kit/kit v0.12.0 // indirect
github.com/pyroscope-io/jfr-parser v0.6.0 // indirect
github.com/pyroscope-io/pyroscope v0.37.2 // indirect
k8s.io/cri-api v0.23.1 // indirect
)

require cloud.google.com/go/compute v1.18.0 // indirect

require (
cloud.google.com/go/compute/metadata v0.2.3 // indirect
github.com/99designs/go-keychain v0.0.0-20191008050251-8e49817e8af4 // indirect
github.com/99designs/keyring v1.2.1 // indirect
Expand Down Expand Up @@ -110,6 +110,7 @@ require (
github.com/grafana/regexp v0.0.0-20220304095617-2e8d9baf4ac2 // indirect
github.com/grpc-ecosystem/grpc-gateway/v2 v2.11.1 // indirect
github.com/gsterjov/go-libsecret v0.0.0-20161001094733-a6f4afe4910c // indirect
github.com/influxdata/influxdb v1.11.0 // indirect
github.com/influxdata/line-protocol/v2 v2.2.1 // indirect
github.com/intel/goresctrl v0.2.0 // indirect
github.com/jpillora/backoff v1.0.0 // indirect
Expand All @@ -122,12 +123,15 @@ require (
github.com/mtibben/percent v0.2.1 // indirect
github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 // indirect
github.com/mwitkow/go-conntrack v0.0.0-20190716064945-2f068394615f // indirect
github.com/narqo/go-dogstatsd-parser v0.2.0 // indirect
github.com/paulmach/orb v0.8.0 // indirect
github.com/pelletier/go-toml v1.9.3 // indirect
github.com/pierrec/lz4 v2.6.1+incompatible // indirect
github.com/pierrec/lz4/v4 v4.1.17 // indirect
github.com/pingcap/errors v0.11.5-0.20210425183316-da1aaba5fb63 // indirect
github.com/pingcap/parser v0.0.0-20210415081931-48e7f467fd74 // indirect
github.com/prometheus/client_golang v1.14.0 // indirect
github.com/prometheus/prometheus v1.8.2-0.20201119142752-3ad25a6dc3d9 // indirect
github.com/segmentio/asm v1.2.0 // indirect
github.com/siddontang/go v0.0.0-20180604090527-bdc77568d726 // indirect
github.com/spaolacci/murmur3 v1.1.0 // indirect
Expand Down Expand Up @@ -158,7 +162,7 @@ require (
github.com/andrewkroh/sys v0.0.0-20151128191922-287798fe3e43 // indirect
github.com/cenkalti/backoff v2.2.1+incompatible // indirect
github.com/containerd/cgroups v1.0.4 // indirect
github.com/containerd/containerd v1.6.8
github.com/containerd/containerd v1.6.8 // indirect
github.com/containerd/continuity v0.2.2 // indirect
github.com/containerd/fifo v1.0.0 // indirect
github.com/containerd/go-cni v1.1.6 // indirect
Expand All @@ -179,7 +183,7 @@ require (
github.com/eapache/go-resiliency v1.2.0 // indirect
github.com/eapache/go-xerial-snappy v0.0.0-20180814174437-776d5712da21 // indirect
github.com/eapache/queue v1.1.0 // indirect
github.com/emicklei/go-restful v2.15.0+incompatible // indirect
github.com/emicklei/go-restful v2.16.0+incompatible // indirect
github.com/fsnotify/fsnotify v1.6.0 // indirect
github.com/go-logfmt/logfmt v0.5.1 // indirect
github.com/go-logr/logr v1.2.3 // indirect
Expand All @@ -189,8 +193,8 @@ require (
github.com/golang-sql/civil v0.0.0-20220223132316-b832511892a9 // indirect
github.com/golang-sql/sqlexp v0.1.0 // indirect
github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da // indirect
github.com/golang/protobuf v1.5.2 // indirect
github.com/golang/snappy v0.0.4
github.com/golang/protobuf v1.5.3 // indirect
github.com/golang/snappy v0.0.4 // indirect
github.com/google/go-cmp v0.5.9 // indirect
github.com/google/gofuzz v1.2.0 // indirect
github.com/google/uuid v1.3.0 // indirect
Expand Down Expand Up @@ -247,14 +251,14 @@ require (
go.uber.org/multierr v1.9.0
go.uber.org/zap v1.24.0 // indirect
golang.org/x/crypto v0.1.0 // indirect
golang.org/x/net v0.5.0 // indirect
golang.org/x/oauth2 v0.4.0 // indirect
golang.org/x/net v0.8.0 // indirect
golang.org/x/oauth2 v0.5.0 // indirect
golang.org/x/sync v0.1.0 // indirect
golang.org/x/term v0.4.0 // indirect
golang.org/x/text v0.6.0 // indirect
golang.org/x/term v0.6.0 // indirect
golang.org/x/text v0.8.0 // indirect
golang.org/x/time v0.3.0 // indirect
google.golang.org/appengine v1.6.7 // indirect
google.golang.org/genproto v0.0.0-20230124163310-31e0e69b6fc2
google.golang.org/genproto v0.0.0-20230306155012-7f2fa6fef1f4 // indirect
gopkg.in/birkirb/loggers.v1 v1.0.3 // indirect
gopkg.in/inf.v0 v0.9.1 // indirect
gopkg.in/natefinch/lumberjack.v2 v2.0.0 // indirect
Expand Down
Loading

0 comments on commit 0c0de9f

Please sign in to comment.