Profile base to dev #352

snail007 · 2023-12-06T09:21:06Z

No description provided.

* Profile base bytesbuffer (#326) * add WriteString to BytesBuffer * Encode WriteString * Profile base object reuse (#327) * reuse bytesbuffer & stream * Optimise StringMap Range (#328) * Profile base object reuse (#332) * reuse bytesbuffer & stream

* Profile base add metric (#338) * Optimise filter addMetric * Profile base findcluster (#340) * Profile base findcluster (#337) * Profile base do acccesslog (#339) * bugfix: BytesBuffer.Reset (#334) * Access log (#336) * Profile base do accesslog (#343) Accesslogv2 (#342) * Profile base preview escape (#345) * optimise Escape * Profile base timenow (#344) (#347) * Profile base do accesslog (#343) * update: log/bytesbuffer (#341) * bytesBuffer release * Accesslogv2 (#346) * update: log/bytesbuffer

* code beautify

motan2 endpoint

# Conflicts: # protocol/motanProtocol.go

* code beautify

* memory copy & bytesBuffer resue

* code beautify

rayzhang0603 · 2023-12-11T04:08:12Z

agent.go

@@ -437,7 +442,7 @@ func (a *Agent) initHTTPClusters() {
 		}
 		httpCluster := cluster.NewHTTPCluster(url, true, a.Context, a.extFactory)
 		if httpCluster == nil {
-			vlog.Errorf("Create http cluster %s failed", id)
+			vlog.Errorf("—Create http cluster %s failed", id)


rayzhang0603 · 2023-12-19T03:50:31Z

agent.go

@@ -490,7 +490,7 @@ func (a *Agent) reloadClusters(ctx *motan.Context) {
 		}

 		service := url.Path
-		mapKey := getClusterKey(url.Group, url.GetStringParamsWithDefault(motan.VersionKey, "0.1"), url.Protocol, url.Path)
+		mapKey := getClusterKey(url.Group, url.GetStringParamsWithDefault(motan.VersionKey, "1.0"), url.Protocol, url.Path)


默认版本号改为全局常量（包含其他相关处）

rayzhang0603 · 2023-12-19T05:51:36Z

server/motanserver.go

+	if motanReq, ok := mreq.(*motan.MotanRequest); ok {
+		motan.ReleaseMotanRequest(motanReq)
+	}
+	if motanResp, ok := mres.(*motan.MotanResponse); ok {


对mres的HttpMotanResponse类型判断放在else中，不需要重复判断两次

rayzhang0603 · 2023-12-19T06:10:28Z

agent_test.go

 		{"test", "g1", "http", "1.3", "No refers for request"},
-		{"test", "g1", "http", "1.2", "less condition to select cluster"},
+		{"test", "b", "c", "d", "no cluster matches the request"},
+		// 同一个service有多个cluster可以匹配，但是group没有传


改成英文注释吧

rayzhang0603 · 2023-12-19T06:37:44Z

core/bytes.go

+	if bb.buf == nil {
+		bb.buf = make([]byte, initSize)
+	}
+	if bb.temp == nil {


默认属性（temp）的初始化都放到池构造func中，不用对每个buf都做判空了。buf是因为涉及到初始大小的问题，与此方法的入参相关，可以放在这里

rayzhang0603 · 2023-12-19T10:40:15Z

http/httpProxy.go

@@ -433,23 +448,27 @@ func MotanRequestToFasthttpRequest(motanRequest core.Request, fasthttpRequest *f
 	motanRequest.GetAttachments().Range(func(k, v string) bool {
 		// ignore some specified key
 		for _, attachmentKey := range httpProxySpecifiedAttachments {
-			if strings.EqualFold(k, attachmentKey) {
+			if k == attachmentKey {


优化匹配逻辑，命中概率大的放在前面快速终止流程。比如InnerAttachmentsConvertMap的匹配

rayzhang0603 · 2023-12-19T11:07:48Z

http/httpProxy.go

@@ -501,12 +520,13 @@ func FasthttpResponseToMotanResponse(motanResponse core.Response, fasthttpRespon
 	fasthttpResponse.Header.VisitAll(func(k, v []byte) {
 		motanResponse.SetAttachment(string(k), string(v))
 	})
-	if resp, ok := motanResponse.(*core.MotanResponse); ok {
+	if resp, ok := motanResponse.(*core.HttpMotanResponse); ok {
 		httpResponseBody := fasthttpResponse.Body()
 		if httpResponseBody != nil {


没有必要判空

rayzhang0603 · 2023-12-19T11:15:28Z

log/bytes.go

+
+func acquireBytesBuffer() *innerBytesBuffer {
+	b := innerBytesBufferPool.Get()
+	if b == nil {


不会为nil

rayzhang0603 · 2023-12-19T11:16:27Z

log/log.go

@@ -390,12 +391,13 @@ func (d *defaultLogger) doAccessLog(logObject *AccessLogEntity) {
 			zap.String("exception", logObject.Exception),
 			zap.String("upstreamCode", logObject.UpstreamCode))
 	} else {
-		var buffer bytes.Buffer
+		buffer := newInnerBytesBuffer()


改用acquireBytesBuffer，显示使用池化buffer

rayzhang0603 · 2023-12-19T11:19:08Z

log/log.go

+
+func AcquireAccessLogEntity() *AccessLogEntity {
+	v := accessLogEntityPool.Get()
+	if v == nil {


无效分支

rayzhang0603 · 2023-12-20T03:32:25Z

endpoint/motanCommonEndpoint.go

 			s.channel.heartbeatLock.Unlock()
 		} else {
 			s.channel.streamLock.Lock()
-			delete(s.channel.streams, s.streamId)
+			if _, exist = s.channel.heartbeats[s.streamId]; exist {


remove的不对

rayzhang0603 · 2023-12-20T03:35:48Z

endpoint/motanCommonEndpoint.go

+	default:
+	}
+	//use atomic.Swap avoid data race between goroutines(Channel.Call and Stream.notify) when ReleaseStream
+	atomic.SwapPointer((*unsafe.Pointer)(unsafe.Pointer(&s.channel)), nil)


channel重置了多次；
res需要重置，避免长时间持有大对象；

rayzhang0603 · 2023-12-20T03:51:44Z

endpoint/motanCommonEndpoint.go

 	select {
 	case <-s.recvNotifyCh:
 		msg := s.res
 		if msg == nil {
 			return nil, errors.New("recv err: recvMsg is nil")
 		}
 		return msg, nil
-	case <-timer.C:
+	case <-s.timer.C:
+		s.release = false


需要在注释解释stream的release机制，几个关键点：stream重置后初值默认进行回收；Channel.Call是唯一回收点；当超时等特殊情况，回收状态需要由remove重新决定

rayzhang0603 · 2023-12-20T03:59:58Z

protocol/motanProtocol.go

+// DecodeWithTime the parameter rs is a slice pointer, so the
+// extension of *rs will affect the slice outside in order to
+// reuse the readSlice which is used by reading header info from buf
+func DecodeWithTime(buf *bufio.Reader, rs *[]byte, maxContentLength int) (msg *Message, start time.Time, err error) {


rs参数命名调整。tempBuf之类的

rayzhang0603 · 2023-12-20T04:01:50Z

protocol/motanProtocol.go

-	metabuf := motan.NewBytesBuffer(256)
+// EncodeWithoutBody encode message expect Message.Body
+// unexpected if call Message.GetEncodedBytes after rewrite result(*motan.BytesBuffer)
+func (msg *Message) EncodeWithoutBody() *motan.BytesBuffer {


不用和Encode方法进行区分，不用显示表明body处理方式，由Encode方法不再返回结果，只能使用GetEncodedBytes方法强制限定使用姿势

rayzhang0603 · 2023-12-20T04:44:35Z

protocol/motanProtocol.go

 	msg.Metadata.Range(func(k, v string) bool {
-		if k == "" || v == "" {
-			return true
+		if VerifyMeta(k, v) {


为啥拆分开？

rayzhang0603 · 2023-12-20T04:49:35Z

protocol/motanProtocol.go

-func DecodeWithTime(buf *bufio.Reader, maxContentLength int) (msg *Message, start time.Time, err error) {
-	temp := make([]byte, HeaderLength, HeaderLength)
-
+func DecodeWithoutBody(buf *bufio.Reader, rs *[]byte, maxContentLength int) (msg *Message, bodySize int, err error) {


不需要单独拆分方法，这个方法没有复用的场景，也不会进行内联

rayzhang0603 · 2023-12-20T04:53:43Z

protocol/motanProtocol.go

@@ -660,11 +740,15 @@ func ConvertToResMessage(response motan.Response, serialize motan.Serialization)
 				res.Body = b
 			} else {
 				vlog.Warningf("convert response value fail! serialized value not []byte. res:%+v", response)
+				res.SetAlreadySend()


alreadySend改为canRelease

rayzhang0603 · 2023-12-20T05:20:17Z

provider/httpProvider.go

@@ -140,6 +140,20 @@ func (h *HTTPProvider) SetContext(context *motan.Context) {
 	h.gctx = context
 }

+// Rewrite do rewrite
+func (h *HTTPProvider) Rewrite(httpReq *fasthttp.Request, request motan.Request) (string, error) {


改为私有方法

rayzhang0603 · 2023-12-20T05:29:01Z

provider/httpProvider.go

 		return resp
 	}
-	resp.RequestID = request.GetRequestID()


给response赋值requestid的语句呢？

rayzhang0603 · 2023-12-20T05:56:12Z

provider/httpProvider.go

+	//do rewrite
+	rewritePath := request.GetMethod()
+	var err error
+	if h.enableRewrite {


f h.enableRewrite 的判定可以放到Rewrite方法中

rayzhang0603 · 2023-12-20T05:58:44Z

provider/httpProvider.go

-		err := mhttp.MotanRequestToFasthttpRequest(request, httpReq, h.defaultHTTPMethod)
-		if err != nil {
-			fillExceptionWithCode(resp, http.StatusBadRequest, t, err)
+	if h.enableHttpException {


判定条件合并。下同

* memory copy & bytesBuffer resue

* use ants pool

Profile base code beautify

snail007 and others added 30 commits November 28, 2023 15:02

preview to base (#333)

4705fe3

* Profile base bytesbuffer (#326) * add WriteString to BytesBuffer * Encode WriteString * Profile base object reuse (#327) * reuse bytesbuffer & stream * Optimise StringMap Range (#328) * Profile base object reuse (#332) * reuse bytesbuffer & stream

fix:stream.close (#353)

d1ac994

optimise code format and comments

f82ae9f

optimise code format and comments

b562874

optimise code format and comments

14dd5be

optimise code format and comments

34736c3

code beautify

2faf7b4

optimise code format and comments

4738ce5

Merge branch 'weibocom:profile_base' into profile_base

805f1ab

beautify code

ff79f24

liangwei beautify code (#354)

86bd259

* code beautify

update testing on go1.20 go1.21

9f7fe4f

Merge remote-tracking branch 'origin/profile_base' into profile_base

cae883c

update testing

5ab5e31

Merge branch 'weibocom:profile_base' into profile_base

d6cf525

beautify code, add unit test

b90bac9

Profile base liangwei (#355)

5d9a877

* code beautify

pretty code

131f4c9

Merge remote-tracking branch 'origin/profile_base' into profile_base

0456692

fix message pool bug

0796175

Merge branch 'weibocom:profile_base' into profile_base

41f84c1

fix message pool bug (#356)

cd54cbf

* code beautify

memory copy & bytesBuffer resue (#357)

2f43dd1

motan2 endpoint

Beautify code (#358)

c8105a6

bugfix: stream reuse (#359)

cd04a8e

Merge remote-tracking branch 'origin/profile_base' into profile_base

c31a18f

# Conflicts: # protocol/motanProtocol.go

optimize http provider

7bb6f72

http provider optimize (#360)

9e2ecf5

* code beautify

Beautify code (#361)

08df2ff

* memory copy & bytesBuffer resue

Hoofffman and others added 12 commits December 18, 2023 13:59

Merge branch 'weibocom:profile_base' into profile_base

9d1101e

fix motanv1 and url.ismatch

50c9ac2

fix motanv1 and url.ismatch

4576f30

Profile base (#362)

b1745cd

* code beautify

Merge branch 'weibocom:profile_base' into profile_base

1facc58

fix motanv1 and url.ismatch

0941da5

Profile base (#363)

fd18d2e

* code beautify

Merge branch 'weibocom:profile_base' into profile_base

24a5322

beautify code

1f17cb3

beautify code (#364)

49e52e4

* code beautify

Beautify code (#365)

20b0d67

Beautify code (#366)

971cd52

rayzhang0603 reviewed Dec 19, 2023

View reviewed changes

TestBytesBuffer_WriteString_Grow write empty

d9f31c5

rayzhang0603 reviewed Dec 20, 2023

View reviewed changes

cocowh and others added 7 commits December 21, 2023 11:34

Beautify code (#367)

dbbacbf

* memory copy & bytesBuffer resue

Profile base ant pool (#368)

d4cafa9

* use ants pool

Merge branch 'weibocom:profile_base' into profile_base

60128a9

beautify code

11b5f14

Merge pull request #369 from Hoofffman/profile_base

7ae4d07

Profile base code beautify

add url int param cache (#370)

5902191

fix init with method params

c669f3c

snail007 merged commit ecfa6a0 into dev Dec 26, 2023

rayzhang0603 deleted the profile_base branch April 7, 2024 03:08

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Profile base to dev #352

Profile base to dev #352

snail007 commented Dec 6, 2023

rayzhang0603 Dec 11, 2023

rayzhang0603 Dec 19, 2023

rayzhang0603 Dec 19, 2023

rayzhang0603 Dec 19, 2023

rayzhang0603 Dec 19, 2023

rayzhang0603 Dec 19, 2023

rayzhang0603 Dec 19, 2023

rayzhang0603 Dec 19, 2023

rayzhang0603 Dec 19, 2023

rayzhang0603 Dec 19, 2023

rayzhang0603 Dec 20, 2023

rayzhang0603 Dec 20, 2023

rayzhang0603 Dec 20, 2023

rayzhang0603 Dec 20, 2023

rayzhang0603 Dec 20, 2023

rayzhang0603 Dec 20, 2023

rayzhang0603 Dec 20, 2023

rayzhang0603 Dec 20, 2023

rayzhang0603 Dec 20, 2023

rayzhang0603 Dec 20, 2023

rayzhang0603 Dec 20, 2023

rayzhang0603 Dec 20, 2023

Profile base to dev #352

Profile base to dev #352

Conversation

snail007 commented Dec 6, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment