Support spl #251

AVloger · 2024-01-24T08:48:45Z

No description provided.

update readme

CLAassistant · 2024-01-24T08:48:51Z

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
1 out of 2 committers have signed the CLA.

✅ crimson-gao
❌ 泓逸

泓逸 seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

tangkai1719 · 2024-01-24T09:49:53Z

consumer/tasks.go

 	consumer.lastFetchGroupCount = GetLogGroupCount(consumer.lastFetchLogGroupList)
+	if consumer.client.option.Query != "" {
+		consumer.lastFetchRawSize = rawDataSize


这个改法，变量语义上不太清晰，按照java sdk的方法来作

有差异，Java和Python的这个结构是对外暴露的，go sdk这个变量是私有变量，按照JAVA SDK的方式就是又需要在结构体里面塞两个变量，然后又加一个赋值的判断，这样就代码就变冗余了

example/consumer/query_demo/simple_demo_with_query.go

tangkai1719 · 2024-01-24T10:03:03Z

client_store.go

@@ -217,6 +217,11 @@ func (c *Client) PullLogsV2(plr *PullLogRequest) (gl *LogGroupList, nextCursor s
 	return ls.PullLogsV2(plr)
 }

+func (c *Client) PullLogsInner(plr *PullLogRequest) (gl *LogGroupList, nextCursor string, dataSize, rawDataSize, rawDataCount int, err error) {


不要再加函数了，PullLogsV2是上次为了spl加的，扩展它，把一些多的meta参数（nextCursor string, dataSize, rawDataSize, rawDataCount int）合并到一个 PullLogMeta 结构里去

tangkai1719 · 2024-01-24T10:03:54Z

log_store.go

+	return
+}
+
+func (s *LogStore) GetLogsBytesInner(plr *PullLogRequest) (out []byte, nextCursor string, dataSize, rawDataSize, rawDataCount int, err error) {


在 GetLogsBytesV2 上改

所有V2接口都修改成了PullLogMeta的返回值，增加函数convert**V1做V1返回值的转化

log_store.go

tangkai1719 · 2024-01-25T02:33:59Z

log_store.go

 	}

-	out = make([]byte, bodyRawSize)
-	if bodyRawSize != 0 {
+	out = make([]byte, pullLogMeta.DataSize)


L520/L521换一下顺序

tangkai1719 · 2024-01-25T02:35:05Z

log_store.go

+		}
+	}
+	// If it is not in scan mode, exit early
+	if plr.PullMode != "scan_on_stream" {


最好这样改：
if plr.PullMode == "scan_on_stream" {
xxx
}

tangkai1719 · 2024-01-25T02:36:50Z

log_store.go

+	v = r.Header["X-Log-Rawdatasize"]
+	if len(v) > 0 {
+		pullLogMeta.RawDataSize, err = strconv.Atoi(v[0])
+		if err != nil {


header非法或取不到，需要返回用户更容易理解的信息

可以参考以前的：

v, ok = r.Header["X-Log-Bodyrawsize"] if !ok || len(v) == 0 { err = fmt.Errorf("can't find 'x-log-bodyrawsize' header") return }

log_store.go

tangkai1719 · 2024-01-25T13:01:12Z

client_interface.go

@@ -271,7 +271,9 @@ type ClientInterface interface {
 	// @note if you want to pull logs continuous, set endCursor = ""
 	PullLogs(project, logstore string, shardID int, cursor, endCursor string,
 		logGroupMaxCount int) (gl *LogGroupList, nextCursor string, err error)
+	// Deprecated: Use NewFunction instead.


Use PullLogsWithQuery instead

tangkai1719 · 2024-01-25T13:06:11Z

log_store.go

@@ -448,10 +448,15 @@ func (s *LogStore) GetLogsBytes(shardID int, cursor, endCursor string,
 	return s.GetLogsBytesV2(plr)


调用好几次，浪费了

直接 GetLogsBytesWithQuery

log_store.go

model.go

tangkai1719 · 2024-01-25T13:15:10Z

token_auto_update_client.go

@@ -836,7 +836,7 @@ func (c *TokenAutoUpdateClient) PullLogs(project, logstore string, shardID int,
 		EndCursor:        endCursor,
 		LogGroupMaxCount: logGroupMaxCount,
 	}
-	return c.PullLogsV2(plr)
+	return c.logClient.PullLogsV2(plr)


改成 PullLogsWithQuery，用最少的跳数来实现

token_auto_update_client.go

tangkai1719 · 2024-01-25T13:17:39Z

client_interface.go

@@ -271,7 +271,9 @@ type ClientInterface interface {
 	// @note if you want to pull logs continuous, set endCursor = ""


GetLogsBytesV2 也是要废弃的吧？注解

shabicheng · 2024-01-26T02:09:51Z

README.md

@@ -38,6 +38,14 @@ go get -u github.com/aliyun/aliyun-log-go-sdk
   Client = sls.CreateNormalInterfaceV2(Endpoint, credentialsProvider)
   ```

+   为了防止出现配置错误，您可以在创建 Client 之后，测试一下 Client 是否能成功调用 SLS API


一下去掉

shabicheng · 2024-01-26T05:17:29Z

log_store.go

 			return
 		}
 	}
+	// If query is not nil, extract more headers
+	if plr.Query != "" {


重复的代码有点多，可以单独提取一个函数出来，放到util里面，X-Log-Bodyrawsize 也可以复用

此外，Query存在的情况下，要求必须存在X-Log-Rawdatasize等两个字段的吧，如果不存在是否要报错，否则语义就出现问题了

shabicheng · 2024-01-26T07:18:51Z

consumer/shard_worker.go

@@ -145,14 +147,21 @@ func (consumer *ShardConsumerWorker) updateStatus(success bool) {
 }

 func (consumer *ShardConsumerWorker) shouldFetch() bool {
-	if consumer.lastFetchGroupCount >= consumer.client.option.MaxFetchLogGroupCount || consumer.lastFetchRawSize >= 4*1024*1024 {
+	lastFetchRawSize := consumer.lastFetchRawSize


是否可以只使用 lastFetchGroupCount lastFetchRawSize

只需要在Query模式下，把lastFetchRawSizeBeforeQuery 赋值给 lastFetchRawSize

shabicheng · 2024-01-27T01:42:03Z

model.go

 	return urlVal
 }

+type PullLogMeta struct {
+	NextCursor                                           string
+	RawSize, RawSizeBeforeQuery, RawDataCountBeforeQuery int


单行，不美观

myftcode · 2024-02-21T09:35:07Z

现在这个https://github.com/aliyun/aliyun-log-go-sdk/blob/master/example/consumer/query_demo/simple_demo_with_query.go文件运行报错 level=warn time=2024-02-21T09:26:17.609466Z caller=consumer_client.go:158 msg="unknown error when pull log" shardId=1 cursor="MTcwODQ4MDc0MzkzNjAxNjkzMg==" error="can't find 'x-log-rawdatasize' header" tryTimes=3

AVloger · 2024-02-21T09:52:48Z

现在这个https://github.com/aliyun/aliyun-log-go-sdk/blob/master/example/consumer/query_demo/simple_demo_with_query.go文件运行报错 level=warn time=2024-02-21T09:26:17.609466Z caller=consumer_client.go:158 msg="unknown error when pull log" shardId=1 cursor="MTcwODQ4MDc0MzkzNjAxNjkzMg==" error="can't find 'x-log-rawdatasize' header" tryTimes=3

请问是哪个地域

myftcode · 2024-02-22T01:30:36Z

现在这个https://github.com/aliyun/aliyun-log-go-sdk/blob/master/example/consumer/query_demo/simple_demo_with_query.go文件运行报错 level=warn time=2024-02-21T09:26:17.609466Z caller=consumer_client.go:158 msg="unknown error when pull log" shardId=1 cursor="MTcwODQ4MDc0MzkzNjAxNjkzMg==" error="can't find 'x-log-rawdatasize' header" tryTimes=3

请问是哪个地域

杭州 , Endpoint: "cn-hangzhou.log.aliyuncs.com",

crimson-gao and others added 4 commits January 17, 2024 17:56

update readme

7f48d43

Merge pull request aliyun#249 from crimson-gao/client-init-check-ak

cf8f58e

update readme

scan mode fix flow_limit for consumer groups

459259f

README add Query

41c4f15

tangkai1719 reviewed Jan 24, 2024

View reviewed changes

v2 change to pullLogMeta

f87046b

panawala reviewed Jan 25, 2024

View reviewed changes

log_store.go Show resolved Hide resolved

tangkai1719 reviewed Jan 25, 2024

View reviewed changes

泓逸 added 2 commits January 25, 2024 11:53

fix name and the location of flow_limit

974fbee

fix review

2df118b

tangkai1719 approved these changes Jan 25, 2024

View reviewed changes

tangkai1719 requested a review from shabicheng January 25, 2024 06:17

泓逸 added 2 commits January 25, 2024 20:53

revert V2 response & add Deprecated some methods to use

0bfb0a3

name fix

5fb7474

panawala reviewed Jan 25, 2024

View reviewed changes

log_store.go Outdated Show resolved Hide resolved

tangkai1719 reviewed Jan 25, 2024

View reviewed changes

泓逸 added 6 commits January 25, 2024 22:07

reviewer comments repair

6f14df2

deprecated V2

4fae389

deprecated fix

2ab20cc

Merge branch 'master' into support-spl

bb74901

fix put/pull link

9d2f400

[bugfix] return 403 need to flow_limit

bb4079f

shabicheng reviewed Jan 27, 2024

View reviewed changes

add utils.ParseHeaderInt to simply code

b816f96

shabicheng merged commit 4da4cf6 into aliyun:master Jan 29, 2024
1 check was pending

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support spl #251

Support spl #251

AVloger commented Jan 24, 2024

CLAassistant commented Jan 24, 2024

tangkai1719 Jan 24, 2024

AVloger Jan 24, 2024

tangkai1719 Jan 24, 2024

tangkai1719 Jan 24, 2024

AVloger Jan 24, 2024

AVloger Jan 25, 2024

tangkai1719 Jan 25, 2024

tangkai1719 Jan 25, 2024

tangkai1719 Jan 25, 2024

tangkai1719 Jan 25, 2024

tangkai1719 Jan 25, 2024

tangkai1719 Jan 25, 2024

tangkai1719 Jan 25, 2024

shabicheng Jan 26, 2024

shabicheng Jan 26, 2024

shabicheng Jan 26, 2024

shabicheng Jan 27, 2024

myftcode commented Feb 21, 2024

AVloger commented Feb 21, 2024

myftcode commented Feb 22, 2024

		@@ -448,10 +448,15 @@ func (s *LogStore) GetLogsBytes(shardID int, cursor, endCursor string,
		return s.GetLogsBytesV2(plr)

		@@ -271,7 +271,9 @@ type ClientInterface interface {
		// @note if you want to pull logs continuous, set endCursor = ""

Support spl #251

Support spl #251

Conversation

AVloger commented Jan 24, 2024

CLAassistant commented Jan 24, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

myftcode commented Feb 21, 2024

AVloger commented Feb 21, 2024

myftcode commented Feb 22, 2024