finagle-protobuf #91

george-vacariuc · 2012-06-27T16:16:57Z

We made a couple of improvements based on our experience using the protobuf based RPC in production.

…hutdown

PatrickOsborne · 2014-03-27T17:41:43Z

Hi All,

I have built a basic protobuf implementation on the mux transport and it turned out to be fairly easy using thriftmux for some inspiration.

I'm now working on getting the tracing to work with the new implementation. I have added Tracers to the DefaultClient and DefaultServer created with the mux transporter but the service name and method name aren't filled in. So it looks like I need to call something like:

  Trace.recordRpcname(serviceName, methodName)

after the TracingFilter executes and pushes the Tracer and sets up the trace id. But I'm not sure how best to insert this call into client and server instances.

I've read through much of the thriftmux implementation and I have been unable to find how it is done there. Is thriftmux recording the service name and method name somehow that I am missing?

It does look like in the DefaultServer the 'prepare' field (ServiceFactory) could be used to record the rpc information. Is that correct? Or is there a better place to do that?

On the client side, I haven't found some sort of hook to use yet. Is there something in DefaultClient I can use? or maybe the ClientDispatcher?

Thanks for the help,
Patrick.

bmdhacks · 2014-03-27T17:48:50Z

I'm willing to bet that recordRpcname is not called in mux and that's a bug. Lemme look into it some more.

PatrickOsborne · 2014-03-27T18:02:49Z

Thanks for the quick response.

My thinking is that the mux transport has no way of knowing what the service name and method name should be. So that is why I was looking for some sort of hook in the client and server to provide the information.

Also on a side note, I've been looking at thriftmux and the fact that the client and server builders aren't compatible with mux and I'm starting to wonder if anyone has used the mux transport in production.

Is mux just experimental at this point or is it production ready?

Thanks again,
Patrick

mariusae · 2014-03-27T22:05:09Z

Also on a side note, I've been looking at thriftmux and the fact that the client and server builders aren't compatible with mux and I'm starting to wonder if anyone has used the mux transport in production.

The intention is to use the new-style APIs; ClientBuilders and ServerBuilders are not going to be provided for new protocols. @roanta has a change coming—it should hit github very soon—which allows the same kind of customization with the new APIs.

PatrickOsborne · 2014-03-27T23:01:47Z

Thanks Marius. I understand not using the builders with the new protocols. In the meantime I've passed the tracer into the default client and server.

The driver behind us moving the protobuf implementation to mux is so that we can hook up Zipkin for distributed tracing in production. So I have two primary concerns:

getting the tracing working with the service and method names
and knowing that the mux transport is production ready

Do you have some thoughts on that?

Thanks,
Patrick

bmdhacks · 2014-03-28T00:49:26Z

Both of these issues are huge focusses for the finagle team at twitter, and we're actively responding to any issues that arise with them. I'm not promising it's all bug-free, but we're extra-responsive on this stuff this quarter. You're on the cutting edge, but I don't think you're too experimental to be in production.

PatrickOsborne · 2014-03-28T02:46:32Z

Great. Thanks for the info. If I can help with the mux code let me know.

I'm working on integrating the protobuf mux implementation with our existing services to see how the implementation shakes out. I'll let you know if I see any other issues.

PatrickOsborne · 2014-03-31T17:12:36Z

Hi @bmdhacks,

Do you have an update on the rpc service name and method name issue with the mux transport?

Should I open an issue?

Thanks,
Patrick

PatrickOsborne · 2014-04-01T22:49:56Z

Is there an update on the mux tracing issue? Should I open an issue?

Thanks,
Patrick

bmdhacks · 2014-04-01T23:39:34Z

It's fixed internally, we're just working on getting a release out with the code.

PatrickOsborne · 2014-04-09T16:53:04Z

what is the timeframe for the tracing fix?

-- Patrick

bmdhacks · 2014-04-09T17:29:33Z

We'll have something up within a week

stuhood · 2014-04-09T18:44:12Z

@PatrickOsborne: Would you mind updating this pull request, or opening a new one with your implementation? We might as well begin the review process while we wait for that fix. Also, I don't think the fix actually blocks landing the patch?

bmdhacks · 2014-04-10T23:23:54Z

Ok, the mux tracing fix is now pushed

mosesn · 2014-04-18T23:59:21Z

And now it has been merged back into github.

mosesn · 2014-08-13T18:00:27Z

@george-vacariuc we've been thinking about how it's a huge pain that finagle keeps changing underneath your feet so it's hard to merge in finagle-protobuf. We've also been putting similar projects under the finagle [organization][0]. What do you think about the idea of making finagle-protobuf its own project under the finagle org, so that it doesn't have to go through the same stringent code review process? We can also make you an owner of the project, so you'll be able to merge pull requests etc yourself, which makes the most sense since you're the expert on finagle-protobuf.

We'll still be happy to consult on finagle-protobuf, but we won't have to end up in these sinkholes where nothing gets done.

Thoughts?

george-vacariuc · 2014-08-15T02:38:21Z

Moses, that's a good idea. I will check with our compliance people if
they're ok with me working on this and I will follow up with you in the
next week or so.

On Wed, Aug 13, 2014 at 12:00 PM, Moses Nakamura notifications@github.com
wrote:

@george-vacariuc https://github.com/george-vacariuc we've been thinking
about how it's a huge pain that finagle keeps changing underneath your feet
so it's hard to merge in finagle-protobuf. We've also been putting similar
projects under the finagle [organization][0]. What do you think about the
idea of making finagle-protobuf its own project under the finagle org, so
that it doesn't have to go through the same stringent code review process?
We can also make you an owner of the project, so you'll be able to merge
pull requests etc yourself, which makes the most sense since you're the
expert on finagle-protobuf.

We'll still be happy to consult on finagle-protobuf, but we won't have to
end up in these sinkholes where nothing gets done.

Thoughts?

—
Reply to this email directly or view it on GitHub
#91 (comment).

mosesn · 2014-12-29T17:48:27Z

@george-vacariuc any update from your compliance folks?

george-vacariuc · 2014-12-30T03:24:07Z

Hey Moses,

Please accept my apologies for having not followed up; unfortunately I will
not be able to contribute to the project.

--George

On Mon, Dec 29, 2014 at 10:48 AM, Moses Nakamura notifications@github.com
wrote:

@george-vacariuc https://github.com/george-vacariuc any update from
your compliance folks?

—
Reply to this email directly or view it on GitHub
#91 (comment).

travisbrown · 2015-03-13T14:41:33Z

For the record, after a conversation with @chrisphelps yesterday I rebased this PR against develop and split finagle-protobuf out into its own project in the Finagle org. It's likely that we'll remove the subproject from this repository, and once that happens I'll close this PR.

chrisphelps · 2015-03-13T14:52:34Z

Thanks, @travisbrown

Problem finagle-protobuf is not being compiled, tested, or used at Twitter and is badly out of date. Solution I've created a new finagle-protobuf project in the Finagle organization on GitHub from the subproject here (after rebasing #91). This commit completes the move by deleting the unused subproject and removing the commented-out build config. Result finagle-protobuf lives in the Finagle organization. RB_ID=611556

travisbrown · 2015-03-24T13:11:15Z

finagle-protobuf is now a stand-alone project in the Finagle organization and has been removed as a subproject here.

Problem finagle-protobuf is not being compiled, tested, or used at Twitter and is badly out of date. Solution I've created a new finagle-protobuf project in the Finagle organization on GitHub from the subproject here (after rebasing twitter#91). This commit completes the move by deleting the unused subproject and removing the commented-out build config. Result finagle-protobuf lives in the Finagle organization. RB_ID=611556

George Vacariuc and others added 30 commits April 11, 2012 10:55

added load test, refactored

a844e92

moved method lookup tests to scala

4a7e0eb

Merge branch 'master' of git://github.com/twitter/finagle

d8d8ba0

added finagle-protobuf pom

5502ea4

added release() for the stub

397a250

sbt eclipse support

e619a53

eclipsify finagle-serversets

7e4780d

temporary workaround for ChannelClosedExceptions raise when servers s…

443040f

…hutdown

move snapshot

b218582

add scm pointer to finagle-protobuf pom

be6b50d

[maven-release-plugin] prepare release finagle-protobuf-3.0.2

afda314

[maven-release-plugin] prepare for next development iteration

562b63b

release 3.0.2

46ef82f

merged

48a9b19

preconditions check

4f7e2eb

handle runtime exceptions and return empty messages

563b64e

[maven-release-plugin] prepare release finagle-protobuf-3.0.4-TENDRIL

ff85203

[maven-release-plugin] prepare for next development iteration

85a3b4e

using a promise for async computations

47d1437

Merge branch 'master' of https://fisheye.tendrilinc.com/git/finagle

572cbcc

fixed def to val

997b0a4

release 3.0.5-TENDRIL

69e9136

add .gitignore

9b40464

bump finagle-protobuf to next snapshot version

40dfc44

added loggin

0963750

added loggin

8ccd3d4

3.0.7 snapshot

02477fb

3.0.7

07a3375

3.0.8 SNAPSHOT

0498fd5

use toString()

9da0f7d

bmdhacks closed this Mar 27, 2014

bmdhacks reopened this Mar 27, 2014

bmdhacks closed this Apr 9, 2014

bmdhacks reopened this Apr 9, 2014

groestl mentioned this pull request Sep 19, 2014

LoadService gets stuck recursively descending into /sys #309

Closed

travisbrown closed this Mar 24, 2015

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

finagle-protobuf #91

finagle-protobuf #91

george-vacariuc commented Jun 27, 2012

PatrickOsborne commented Mar 27, 2014

bmdhacks commented Mar 27, 2014

PatrickOsborne commented Mar 27, 2014

mariusae commented Mar 27, 2014

PatrickOsborne commented Mar 27, 2014

bmdhacks commented Mar 28, 2014

PatrickOsborne commented Mar 28, 2014

PatrickOsborne commented Mar 31, 2014

PatrickOsborne commented Apr 1, 2014

bmdhacks commented Apr 1, 2014

PatrickOsborne commented Apr 9, 2014

bmdhacks commented Apr 9, 2014

stuhood commented Apr 9, 2014

bmdhacks commented Apr 10, 2014

mosesn commented Apr 18, 2014

mosesn commented Aug 13, 2014

george-vacariuc commented Aug 15, 2014

mosesn commented Dec 29, 2014

george-vacariuc commented Dec 30, 2014

travisbrown commented Mar 13, 2015

chrisphelps commented Mar 13, 2015

travisbrown commented Mar 24, 2015

finagle-protobuf #91

finagle-protobuf #91

Conversation

george-vacariuc commented Jun 27, 2012

PatrickOsborne commented Mar 27, 2014

bmdhacks commented Mar 27, 2014

PatrickOsborne commented Mar 27, 2014

mariusae commented Mar 27, 2014

PatrickOsborne commented Mar 27, 2014

bmdhacks commented Mar 28, 2014

PatrickOsborne commented Mar 28, 2014

PatrickOsborne commented Mar 31, 2014

PatrickOsborne commented Apr 1, 2014

bmdhacks commented Apr 1, 2014

PatrickOsborne commented Apr 9, 2014

bmdhacks commented Apr 9, 2014

stuhood commented Apr 9, 2014

bmdhacks commented Apr 10, 2014

mosesn commented Apr 18, 2014

mosesn commented Aug 13, 2014

george-vacariuc commented Aug 15, 2014

mosesn commented Dec 29, 2014

george-vacariuc commented Dec 30, 2014

travisbrown commented Mar 13, 2015

chrisphelps commented Mar 13, 2015

travisbrown commented Mar 24, 2015