EOS GRPC interface #1471

ffurano · 2021-02-11T14:37:58Z

No description provided.

ffurano · 2021-02-11T15:23:37Z

Why these things in the build fail, as they are not testing eosgrpc ? Maybe I have to wait some time and rebase ?

go.mod

go.sum

pkg/eosclient/eosgrpc/eosgrpc.go

ffurano · 2021-03-04T15:29:15Z

Hi Ishank, thank for the review. I'll go through all these points in the next days

lgtm-com · 2021-03-08T18:41:50Z

This pull request introduces 2 alerts when merging 2f5a2e1 into 2011cb5 - view on LGTM.com

new alerts:

2 for Redundant check for negative value

lgtm-com · 2021-03-09T13:14:00Z

This pull request introduces 2 alerts when merging 7724aea into 491455c - view on LGTM.com

new alerts:

2 for Redundant check for negative value

lgtm-com · 2021-03-12T11:55:15Z

This pull request introduces 2 alerts when merging b6646ac into 1808b1c - view on LGTM.com

new alerts:

2 for Redundant check for negative value

lgtm-com · 2021-03-12T14:16:10Z

This pull request introduces 2 alerts when merging df43e99 into 1808b1c - view on LGTM.com

new alerts:

2 for Redundant check for negative value

lgtm-com · 2021-03-15T13:40:16Z

This pull request introduces 2 alerts when merging c0231a7 into 3b7e1f0 - view on LGTM.com

new alerts:

2 for Redundant check for negative value

lgtm-com · 2021-03-19T11:23:04Z

This pull request introduces 2 alerts when merging 9423a36 into 03346dd - view on LGTM.com

new alerts:

2 for Redundant check for negative value

lgtm-com · 2021-03-19T12:10:23Z

This pull request introduces 2 alerts when merging 91cb5c4 into 03346dd - view on LGTM.com

new alerts:

2 for Redundant check for negative value

lgtm-com · 2021-03-19T12:46:26Z

This pull request introduces 2 alerts when merging f2e244e into 03346dd - view on LGTM.com

new alerts:

2 for Redundant check for negative value

lgtm-com · 2021-03-29T08:41:56Z

This pull request introduces 1 alert when merging e838b6e into e8a00d9 - view on LGTM.com

new alerts:

1 for Useless assignment to local variable

ishank011 · 2021-05-20T11:26:22Z

@ffurano can you rebase instead of merging? It's impossible to review

ffurano · 2021-05-21T07:05:41Z

@ffurano can you rebase instead of merging? It's impossible to review

Uhm... I only did rebase so far

ffurano · 2021-05-21T07:43:18Z

@ffurano can you rebase instead of merging? It's impossible to review

Uhm... I only did rebase so far

Actually I have the impression that you have put into master some previous stage of my work. I am reviewing all the new conflicts, yet now it's really a mess

ffurano · 2021-05-21T08:14:45Z

@ffurano can you rebase instead of merging? It's impossible to review

Rebase done

lgtm-com · 2021-05-27T09:54:27Z

This pull request introduces 1 alert when merging 328a1da into 1c0c24a - view on LGTM.com

new alerts:

1 for Missing error check

labkode · 2021-06-04T09:49:13Z

@ffurano I see a dependency to a private repo of yours:

github.com/ffurano/grpc-proto

Could you replace it with an upstream version?

labkode

Remove private repos

ffurano · 2021-06-05T07:35:56Z

@ffurano I see a dependency to a private repo of yours:
github.com/ffurano/grpc-proto
Could you replace it with an upstream version?

There's no such a thing by now...

…true

ffurano · 2021-06-11T09:35:26Z

If I keep it in the Client class then it will leak. It has to be unique in the whole app

Don't we have just one instance of the client struct (not the HTTP client, the one defined in eosgrpc) in the whole app?

Uhm, nothing enforces that in the Client, and it seems pretty awkward to assume that there is just one instance.

The creator can create as many clients it needs, as it goes through the config file:
func New(opt *Options) *Client

ffurano · 2021-06-11T09:36:53Z

If you have a better way to implement a singleton in golang please let me know... so far I have done more or less like this: https://medium.com/@ishagirdhar/singleton-pattern-in-golang-9f60d7fdab23

ishank011

I'm not sure this should be a singleton in the first place. We'll have one instance of the eos driver to connect to each of the EOS instances and I don't see any advantage of sharing the transport among them. I'd prefer it to be a part of the Client struct. We also need to increase the defaults and make transport configurable.

ishank011 · 2021-06-14T13:21:00Z

pkg/eosclient/eosgrpc/eos_http/eoshttp.go

+// From the basepath and the file path... build an url
+func (c *Client) buildFullURL(urlpath, uid, gid string) (string, error) {
+	s := c.opt.BaseURL
+	if len(urlpath) > 0 && urlpath[0] != '/' {


Can you do something like https://github.com/cs3org/reva/blob/master/pkg/ocm/share/manager/json/json.go#L321-L326?

ishank011 · 2021-06-14T13:21:57Z

pkg/eosclient/eosgrpc/eos_http/eoshttp.go

+	// I feel safer putting here a check, to prohibit malicious users to
+	// inject a false uid/gid into the url
+	// Who knows, maybe it's redundant? Better more than nothing.
+	p1 := strings.Index(urlpath, "eos.ruid")


https://github.com/cs3org/reva/blob/master/internal/grpc/services/gateway/webdavstorageprovider.go#L281

ishank011 · 2021-06-14T13:23:43Z

pkg/eosclient/eosgrpc/eos_http/eoshttp.go

+				Certificates: []tls.Certificate{cert},
+			},
+			MaxIdleConns:        10,
+			MaxConnsPerHost:     8,


Can you make these configurable?

Yes, of course

ishank011 · 2021-06-14T13:25:50Z

pkg/eosclient/eosgrpc/eos_http/eoshttp.go

+		resp, err := c.cl.Do(req)
+
+		// Let's support redirections... and if we retry we have to retry at the same FST, avoid going back to the MGM
+		if resp != nil && (resp.StatusCode == 307 || resp.StatusCode == 302) {


(resp.StatusCode == http.StatusFound || resp.StatusCode == http.StatusTemporaryRedirect)

ffurano · 2021-06-14T13:56:28Z

I'm not sure this should be a singleton in the first place. We'll have one instance of the eos driver to connect to each of the EOS instances and I don't see any advantage of sharing the transport among them. I'd prefer it to be a part of the Client struct. We also need to increase the defaults and make transport configurable.

Oh, I think you make a good point here about sharing the transport, I'll have a look if it an eos instance can keep the same transport to cache the conns

…ll of its parameters configurable

ishank011 · 2021-06-14T15:12:26Z

@ffurano GetHTTPCl is called on every PUT and GET call, which creates an instance of *ehttp.Client.
I think here we can store the *ehttp.Client

reva/pkg/eosclient/eosgrpc/eosgrpc.go

Lines 197 to 201 in 338598b

    
           t, err := htopts.Init() 
        
           if err != nil { 
        
           	panic("Cant't init the EOS http client options") 
        
           } 
        
           c.httptransport = t

and GetHTTPCl can just return that client.

ffurano · 2021-06-17T08:05:29Z

@ffurano GetHTTPCl is called on every PUT and GET call, which creates an instance of *ehttp.Client.
I think here we can store the *ehttp.Client

reva/pkg/eosclient/eosgrpc/eosgrpc.go

Lines 197 to 201 in 338598b

t, err := htopts.Init()

if err != nil {

panic("Cant't init the EOS http client options")

}

c.httptransport = t

and GetHTTPCl can just return that client.

This is how it's meant to work, the creation of a new http client is based on the common instance of the transport class, which caches the real connections behind (plus keepalive, max numbers etc). Instances of the client class are meant to be just local and short lived

ffurano · 2021-06-17T08:09:14Z

@ffurano GetHTTPCl is called on every PUT and GET call, which creates an instance of *ehttp.Client.
I think here we can store the *ehttp.Client

reva/pkg/eosclient/eosgrpc/eosgrpc.go

Lines 197 to 201 in 338598b

t, err := htopts.Init()

if err != nil {

panic("Cant't init the EOS http client options")

}

c.httptransport = t

and GetHTTPCl can just return that client.

Ah, and now the transport class is not a singleton anymore, there is one instance per EOS endpoint configured

…ent. Use the golang url class to parse urls

ffurano requested a review from labkode as a code owner February 11, 2021 14:37

ffurano force-pushed the master branch from e97ecf9 to 58fb8dc Compare February 15, 2021 13:12

ffurano force-pushed the master branch from a52cfc8 to e9b2fd1 Compare February 24, 2021 08:10

ishank011 reviewed Mar 4, 2021

View reviewed changes

hound bot deleted a comment from ffurano Mar 12, 2021

ffurano force-pushed the master branch from cf6da42 to 07a5e51 Compare March 22, 2021 15:07

ffurano force-pushed the master branch from d90665e to 3618837 Compare April 20, 2021 07:39

ffurano force-pushed the master branch from fc70237 to 4f62e04 Compare May 21, 2021 08:10

ffurano force-pushed the master branch from 1bca395 to 62c7b07 Compare June 2, 2021 07:50

glpatcern changed the title ~~[WIP] EOS Grpc interface~~ EOS GRPC interface Jun 4, 2021

labkode requested changes Jun 4, 2021

View reviewed changes

root and others added 11 commits June 11, 2021 11:15

Pass the upload file size to the http layer if WriteUsesLocalTemp is …

cc06cef

…true

Fix minor warning

8ae8330

Use the embedded eos grpc proto

fb41a53

Don't exit, just panic

29df48d

Fix comment

658f2b0

Cosmetics

691a91c

EOS negative resp logged as error

1d72cf2

Missing license header

a38337b

Set MaxConnsPerHost, MaxIdleConnsPerHost defaults in the http transport

af1b261

mod tidy

ea83b38

make the http options local to Client

007005c

ffurano force-pushed the master branch from f98b551 to 007005c Compare June 11, 2021 09:16

labkode requested review from labkode and ishank011 June 11, 2021 09:43

Don't set twice SSL_CERT_DIR

eba9d4d

ishank011 reviewed Jun 14, 2021

View reviewed changes

Keep the http transport struct on a per EOS instance base, and make a…

338598b

…ll of its parameters configurable

Don't reinitialize the http options and transport when creating a cli…

b6ff7a4

…ent. Use the golang url class to parse urls

ishank011 previously approved these changes Jun 17, 2021

View reviewed changes

Http EOS client: more robust url parsing

1baa5a7

ffurano dismissed ishank011’s stale review via 1baa5a7 June 17, 2021 11:28

ishank011 approved these changes Jun 17, 2021

View reviewed changes

ishank011 merged commit 692f38f into cs3org:master Jun 17, 2021

C0rby mentioned this pull request Jun 23, 2021

[full-ci] update reva to v1.9 owncloud/ocis#2210

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

EOS GRPC interface #1471

EOS GRPC interface #1471

ffurano commented Feb 11, 2021

ffurano commented Feb 11, 2021

ffurano commented Mar 4, 2021

lgtm-com bot commented Mar 8, 2021

lgtm-com bot commented Mar 9, 2021

lgtm-com bot commented Mar 12, 2021

lgtm-com bot commented Mar 12, 2021

lgtm-com bot commented Mar 15, 2021

lgtm-com bot commented Mar 19, 2021

lgtm-com bot commented Mar 19, 2021

lgtm-com bot commented Mar 19, 2021

lgtm-com bot commented Mar 29, 2021

ishank011 commented May 20, 2021

ffurano commented May 21, 2021

ffurano commented May 21, 2021

ffurano commented May 21, 2021

lgtm-com bot commented May 27, 2021

labkode commented Jun 4, 2021 •

edited

Loading

labkode left a comment

ffurano commented Jun 5, 2021

ffurano commented Jun 11, 2021

ffurano commented Jun 11, 2021

ishank011 left a comment •

edited

Loading

ishank011 Jun 14, 2021

ishank011 Jun 14, 2021

ishank011 Jun 14, 2021

ffurano Jun 14, 2021

ishank011 Jun 14, 2021 •

edited

Loading

ffurano commented Jun 14, 2021

ishank011 commented Jun 14, 2021

ffurano commented Jun 17, 2021

ffurano commented Jun 17, 2021

EOS GRPC interface #1471

EOS GRPC interface #1471

Conversation

ffurano commented Feb 11, 2021

ffurano commented Feb 11, 2021

ffurano commented Mar 4, 2021

lgtm-com bot commented Mar 8, 2021

lgtm-com bot commented Mar 9, 2021

lgtm-com bot commented Mar 12, 2021

lgtm-com bot commented Mar 12, 2021

lgtm-com bot commented Mar 15, 2021

lgtm-com bot commented Mar 19, 2021

lgtm-com bot commented Mar 19, 2021

lgtm-com bot commented Mar 19, 2021

lgtm-com bot commented Mar 29, 2021

ishank011 commented May 20, 2021

ffurano commented May 21, 2021

ffurano commented May 21, 2021

ffurano commented May 21, 2021

lgtm-com bot commented May 27, 2021

labkode commented Jun 4, 2021 • edited Loading

labkode left a comment

Choose a reason for hiding this comment

ffurano commented Jun 5, 2021

ffurano commented Jun 11, 2021

ffurano commented Jun 11, 2021

ishank011 left a comment • edited Loading

Choose a reason for hiding this comment

ishank011 Jun 14, 2021

Choose a reason for hiding this comment

ishank011 Jun 14, 2021

Choose a reason for hiding this comment

ishank011 Jun 14, 2021

Choose a reason for hiding this comment

ffurano Jun 14, 2021

Choose a reason for hiding this comment

ishank011 Jun 14, 2021 • edited Loading

Choose a reason for hiding this comment

ffurano commented Jun 14, 2021

ishank011 commented Jun 14, 2021

ffurano commented Jun 17, 2021

ffurano commented Jun 17, 2021

labkode commented Jun 4, 2021 •

edited

Loading

ishank011 left a comment •

edited

Loading

ishank011 Jun 14, 2021 •

edited

Loading