Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

close #351. in storm.py: turn off stdout to avoid messing with JSON payload sent back to JAVA layer #698

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

francis-shuoch
Copy link

I write a python decorator to silence prints' output into stdout. Meanwhile sendMsgToParent() function doesn't get affected, so the communication to Java Layer is not affected, too. Use this wrapper on any custom python scripts containing prints without worrying those prints would mess up the tuple emits. I tested with storm-starter's WordCountTopology.

stdout. Meanwhile sendMsgToParent() function doesn't get affected, so the
communication to Java Layer is not affected, too. Use this wrapper on
any custom python scripts containing prints without worrying those
prints would mess up the tuple emits. I tested with storm-starter's WordCountTopology.
@xumingming
Copy link
Collaborator

So developers need to call this nullify_stdout method manually?

@francis-shuoch
Copy link
Author

Yes, as lbustelo pointed out in issue #351, custom python function containing prints can use this decorator to wrapper it, so user defined prints won't mess up the tuple emits from the bolts. Like this:

@nullify_stdout
def custom_fun:
    print "this won't come into python process's stdout to mess up emits."

sorry I didn't add an example like this in my PR.

@francis-shuoch
Copy link
Author

Another way is to assign sys.stdout = NullDevice() globally in the module, while sendMsgToParent() use original_stdout so it can write to stdout normally. But this way I don't know when to reassign sys.stdout back. Any thoughts here? I would like to follow it.

@xumingming
Copy link
Collaborator

Would it be better if storm.py handles all the silence prints details, so it will be transparent to the developers?

@francis-shuoch
Copy link
Author

Then I think assigning sys.stdout = NullDevice() globally in the module while only sendMsgToParent() using original_stdout will suffice. What do you think?

@xumingming
Copy link
Collaborator

About reassigning sys.stdout back, one way I can think of is wrapping all calls to self.process(tuple), self.initialize(stormconf, context) etc with your nullify_stdout.

@francis-shuoch
Copy link
Author

I gave it some fresh thoughts today, now I think, since all prints will be read by storm's JVM-component, we must forbid any prints outside of sendMsgToParent() function. So, there is no need to reassign sys.stdout back. Does this make sense to you?

Or, alternatively, we can reassign sys.stdout back at the end of run() function, since this should be the last call of custom python script (see splitsentence.py example for reference).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants