Skip to content
This repository has been archived by the owner on Mar 3, 2023. It is now read-only.

Remove usage of NullFoo objects #1469

Merged
merged 3 commits into from
Oct 5, 2016

Conversation

billonahill
Copy link
Contributor

@billonahill billonahill commented Oct 4, 2016

For all Foo interfaces that use reflection, we implement a NullFoo object with an empty implementation so we can test classes that rely on reflection to load Foos. Refactoring to use a more idiomatic approach using mock objects instead. This has the following benefits:

  1. We don't need to maintain dummy classes. This is tedious when adding methods and refactoring and it clutters the code base
  2. We don't need to maintain build targets and dependency chains for the dummy classes.
  3. We can customize the expected method call behavior of the mock objects based on the context of their use, instead of relying on hard-coded implementations.

This pr only removes usage of the NullFoo objects. Removing the actual classes can happen in a follow-up.

@billonahill billonahill added this to the 0.14.4 milestone Oct 4, 2016
@billonahill billonahill self-assigned this Oct 4, 2016
Copy link
Contributor

@kramasamy kramasamy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good!

@@ -1,5 +1,5 @@
# uploader class for transferring the topology jar/tar files to storage
heron.class.uploader: com.twitter.heron.uploader.NullUploader
heron.class.uploader: com.twitter.heron.uploader.localfs.LocalFileSystemUploader
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The YARN scheduler does not need to upload heron core, A use case for a no-op uploader. Replacing it with this uploader will result in unnecessary copies of the binaries. Will it be ok to let the NullUploader exist?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I changed this because all other uploaders use the local fs uploader, which just moves it to another directory, so I thought that made sense for yarn as well. Is the yarn workflow different than all the others with respect to it's uploader needs?

$ grep -r heron.class.uploader heron/config/.
heron/config/./src/yaml/conf/aurora/uploader.yaml:heron.class.uploader:                            com.twitter.heron.uploader.localfs.LocalFileSystemUploader
heron/config/./src/yaml/conf/local/uploader.yaml:heron.class.uploader:                            com.twitter.heron.uploader.localfs.LocalFileSystemUploader
heron/config/./src/yaml/conf/localzk/uploader.yaml:heron.class.uploader:                            com.twitter.heron.uploader.localfs.LocalFileSystemUploader
heron/config/./src/yaml/conf/marathon/uploader.yaml:heron.class.uploader:                            com.twitter.heron.uploader.localfs.LocalFileSystemUploader
heron/config/./src/yaml/conf/mesos/uploader.yaml:heron.class.uploader:                            com.twitter.heron.uploader.localfs.LocalFileSystemUploader
heron/config/./src/yaml/conf/slurm/uploader.yaml:heron.class.uploader:                            com.twitter.heron.uploader.localfs.LocalFileSystemUploader
heron/config/./src/yaml/conf/yarn/uploader.yaml:heron.class.uploader:                            com.twitter.heron.uploader.NullUploader

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

which means it will copy to another directory - which incurs overhead and also a configuration of directory where the file needs to be moved as well. Probably, it might be a good idea to have NullUploader.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@billonahill Yes, the YARN scheduler skips the uploader step. The heron binaries are added to classpath by the launcher in this case.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've changed back to NullUploader for now, but the proper fix is really to change the yarn impl (or the shared classes it relies on) to not invoke an uploader at all. This is a cleaner approach than calling a class written to do nothing.

@billonahill billonahill merged commit 85ab6cd into apache:master Oct 5, 2016
@billonahill billonahill deleted the billg/remove_null_objects branch October 5, 2016 22:22
nicknezis pushed a commit that referenced this pull request Sep 14, 2020
* Remove usage of NullFoo objects

* Remove usage of NullFoo objects
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants