-
-
Notifications
You must be signed in to change notification settings - Fork 636
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[V2-UI] Move logging to Rust #6817
[V2-UI] Move logging to Rust #6817
Conversation
@stuhood @illicitonion This is not ready for merging yet, I post it in case you want to take a look. |
@blorente : So, I would definitely recommend splitting #6004 into two separate PRs:
I think that maybe this PR tries to dive into 2 without first tackling 1, so I'd recommend inverting that. If this makes sense, I can update the steps in #6004 (comment) to incorporate that split? |
No need, upon reading it again I understand what you mean. The only thing I
have doubts about is how the instances of the UX subscribe to logging. I'm
imagining something like `<Global logger>.subscribe_ux(UI instance)>`, but
I'm not sure how to do that in Rust, like what kind of reference I need to
pass.
…On Tue, 27 Nov 2018, 19:45 Stu Hood ***@***.*** wrote:
@blorente <https://github.com/blorente> : So, I would definitely
recommend splitting #6004
<#6004> into two separate PRs:
1. Moving logging from python to rust
2. Integrating the v2-ui with the rust-side logging
I think that maybe this PR tries to dive into 2 without first tackling 1,
so I'd recommend inverting that. If this makes sense, I can update the
steps in #6004 (comment)
<#6004 (comment)>
to incorporate that split?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#6817 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AFlvPloInuPVoonoAsDyLsdhbX7U2AEFks5uzZZ3gaJpZM4Y1zI2>
.
|
The global logging instance would be a |
Yeah, that I already fought with for this PR, finally understood it. My
doubt is what kind of reference to register, like how does ownership of the
UXs look. It makes sense that the lifetime of the UX is the same as the
call to `execute`, so I have to find out what kind of reference is actually
registered in the logger.
Once I have that tomorrow, the rest is actually cake.
…On Tue, 27 Nov 2018, 20:48 Stu Hood ***@***.*** wrote:
I'm imagining something like <Global logger>.subscribe_ux(UI instance)>,
but I'm not sure how to do that in Rust, like what kind of reference I need
to pass.
The global logging instance would be a static (possibly initialized via
lazy_static!).
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#6817 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AFlvPtQDn-vlhChiYHyypVGA4WpqoOzZks5uzaU3gaJpZM4Y1zI2>
.
|
Looks like wrapping the UIs in an `Arc` is the way to go, I'll learn about
those tomorrow.
On Tue, 27 Nov 2018, 20:56 Borja Lorente Escobar <bescobar@twitter.com
wrote:
… Yeah, that I already fought with for this PR, finally understood it. My
doubt is what kind of reference to register, like how does ownership of the
UXs look. It makes sense that the lifetime of the UX is the same as the
call to `execute`, so I have to find out what kind of reference is actually
registered in the logger.
Once I have that tomorrow, the rest is actually cake.
On Tue, 27 Nov 2018, 20:48 Stu Hood ***@***.*** wrote:
> I'm imagining something like <Global logger>.subscribe_ux(UI instance)>,
> but I'm not sure how to do that in Rust, like what kind of reference I need
> to pass.
>
> The global logging instance would be a static (possibly initialized via
> lazy_static!).
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <#6817 (comment)>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AFlvPtQDn-vlhChiYHyypVGA4WpqoOzZks5uzaU3gaJpZM4Y1zI2>
> .
>
|
Either that, or just directly in the |
f832238
to
2922ed0
Compare
a6651a3
to
8cf5ddd
Compare
So I think we now log everything we need to log (sample: https://asciinema.org/a/4uE48nHD5XyLohLxeMlcrjDjG). Now it remains to:
However, it's close to reviewable. |
Also, it might be a good time to see if we want to keep the old format. |
I left some TODOs here and there, but I think this is mostly reviewable (cc @illicitonion @stuhood ) |
Also, it doesn't have any tests yet. I'll try to add them on the plane on Wednesday. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Very exciting! Looking forward to it landing!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks @blorente ! Overall, this looks really good. Doing a bit more to clean up the python side of things will be important before landing. As mentioned, I'll open a review to try and clean up Native
initialization, which might help here.
src/rust/engine/logging/src/lib.rs
Outdated
|
||
pub type Logger = logger::Logger; | ||
|
||
// TODO My intuition is that these belog here, because putting them outside of the `logging` |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Stale comment?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
No, this was in response to a comment by @illicitonion with the question of whether this should go on the FFI definition because it's tied to Rust/Python interaction, or whether it should go in the logging lib because it's exclusively relevant to logging. I decided for the latter, and that comment is my reasoning. Can delete it, though.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Relevant comment: #6817 (comment)
### Problem Previously the binary blob loaded by the `Native` instance was fetched using `BinaryUtils`, which meant that fetching it required bootstrap options, which put constraints on where in the process you could use the native instance. But a while back that binary blob was moved to being loaded with python resources, so instantiating the `Native` instance no longer required any configuration... although its API continued to suggest that it did. ### Solution Move the last remaining argument for `Native.create(..)` to `Scheduler.__init__`, and replace `Native.create(..)` with `Native.instance()`. While it should also be possible now to remove the passing of the `native` instance in a bunch of places, this PR does not do that, in order to avoid conflicts with other in flight PRs. ### Result The true requirements of loading a `Native` instance are more apparent, and the initialization code in #6817 should be simplified.
6a1fb11
to
e1f7c66
Compare
After Stu's fantastic work on #6979, I have fixed most of the comments and TODOs. I need help with the following item: fd99e5d#diff-42b494f83e5fd1826f7b469b00c79cc3R97 Other than that, I think the PR is mostly ready, except for the blatant lack of tests, which will be my next step. |
Is this the reason why we are missing logs at the |
That is not a direct result of that bug, but rather a result of me try to fix that bug and installing two handlers, here and here. I think that may create a race condition of logs, to the pantsd logfile. (@cosmicexplorer ) |
e1f7c66
to
263bdf7
Compare
@cosmicexplorer Re: the race condition. This also means that solving the redirection issue (for which I asked help, I will tackle it tomorrow) will very likely solve this race condition as well. |
263bdf7
to
6447a02
Compare
So there's currently two big separate issues blocking this:
|
c3fbfbe
to
9d84495
Compare
Woohoo!
…On Wed, 10 Apr 2019, 20:42 Daniel Wagner-Hall, ***@***.***> wrote:
Merged #6817 <#6817> into master.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#6817 (comment)>, or mute
the thread
<https://github.com/notifications/unsubscribe-auth/AFlvPpc9Sl_Aq4sJD1v2ENgYKNntrhvQks5vfj6QgaJpZM4Y1zI2>
.
|
This appears to have broken CI: Specifically, the new regexps expect the leading |
As discussed in #slack, this was likely an inconsistency between the shards in your CI run, as master has been green a few times since that landed. |
Problem
Currently, Rust calls Python through FFI to handle logging. As we move to the V2 UI, we would like to make the reverse happen, where the V2 engine logs to the UI.
Solution
The Rust code owns logging (which the Python code can forward to) instead of the other way around.
Note that this PR changes some logging formats (adding timestamps, adding PIDs, and possibly a few other minor changes).