-
Notifications
You must be signed in to change notification settings - Fork 226
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
ANDROIDX_TEST_ORCHESTRATOR support #264
ANDROIDX_TEST_ORCHESTRATOR support #264
Conversation
sample/src/androidTest/java/com/facebook/testing/screenshot/sample/MainActivityTest.kt
Outdated
Show resolved
Hide resolved
core/src/main/java/com/facebook/testing/screenshot/internal/AlbumImpl.java
Outdated
Show resolved
Hide resolved
core/src/main/java/com/facebook/testing/screenshot/internal/AlbumImpl.java
Outdated
Show resolved
Hide resolved
@xiphirx , please check the approach in the PR. If you're happy with it, I will prettify code and open non draft pull request. |
@xiphirx , fixed everything that I didn't like in the code. Now it's ready for review from my point of view. Please take a look when you get a chance. |
Hey thanks for taking this on! I'm currently busy with other things and will be out for about a week. I will try to get back to you asap. A few thoughts:
|
@xiphirx , thanks for the reply. My following actions then:
|
Performance measurements of adb pull tar vs directory I created a tar bundle, by running sample app. To increase screenshots count, one test generated 1001 screenshots for (i in 0..1000) {
Screenshot.snapActivity(activity).setName("test$i").record()
} screenshot_bundle.tar was 111 Mb. I extracted all content to test directory. Then I pulled archive and directory. adb pull /sdcard/screenshots/com.facebook.testing.screenshot.example.test/screenshots-default/screenshot_bundle.tar test.tar
/sdcard/screenshots/com.facebook.testi...203.3 MB/s (111338496 bytes in 0.522s) adb pull /sdcard/screenshots/com.facebook.testing.screenshot.example.test/screenshots-default/test test3
/sdcard/screenshots/com.facebook.testing.screenshot.example.test/screenshots-default/test/: 14148 files pulled. 21.6 MB/s (99918186 bytes in 4.412s) To make competition fair we should also count tar bundle extracting time, for this I copied bundle extracting related python code to extract.py
Results: tar: 1.6 seconds, directory 4.4 seconds. So tar bundle 2.75 times faster in my tests. @xiphirx , what do you think about it? does the time of adb pull for directory seem acceptable for you? |
@xiphirx ,
I haven't found any doc that answers this question. But I did a quick research in source code. Basically orchestrator runs tests one by one using So the answer is only one process is alive, so no sync of writes/read is required. |
@xiphirx , if we want to write metadata to one file, |
@xiphirx |
Got rid of tar bundle, pulling whole directory. works fine for sample project |
core/src/main/java/com/facebook/testing/screenshot/internal/AlbumImpl.java
Outdated
Show resolved
Hide resolved
return; | ||
} | ||
|
||
private String readPreviousTestRunId() { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why are readPreviousTestRunId
and writePreviousTestRunId
required? We should be able to just read the test run id off the arguments passed into the Instrumentation
instance: https://developer.android.com/reference/androidx/test/platform/app/InstrumentationRegistry#getArguments()
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Still wondering this
core/src/main/java/com/facebook/testing/screenshot/internal/SingleTestRunArtifactsManager.java
Outdated
Show resolved
Hide resolved
core/src/main/java/com/facebook/testing/screenshot/internal/SingleTestRunArtifactsManager.java
Outdated
Show resolved
Hide resolved
core/src/main/java/com/facebook/testing/screenshot/internal/SingleTestRunArtifactsManager.java
Outdated
Show resolved
Hide resolved
core/src/main/java/com/facebook/testing/screenshot/internal/SingleTestRunArtifactsManager.java
Outdated
Show resolved
Hide resolved
private static final String ROOT_TAG_OPEN = "<screenshots>"; | ||
private static final String ROOT_TAG_CLOSE = "</screenshots>"; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
How about switching to JSON + reading in the existing file + appending in memory + re-writing the entire file?
If you're worried about that taking too long or the file growing too large, then we can just save a metadata file per test.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
switched.
@xiphirx , I was struggling a lot with python plugin, not sure that I've changed everything that should be changed, especially tests 😢 but it woks for test example
I will appreciate your help with it, advices or commits 😉
return tileName; | ||
} | ||
|
||
/** Delete all screenshots associated with this album */ | ||
@Override | ||
public void cleanup() { | ||
if (mCurrentTestRunId.equals(mPreviousTestRunId)) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The cleanup function can disappear now because we write to a unique directory each time.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Who will remove images after test? should it be plugin?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
That already doesn't really happen, it just "cleans up" because a new run will overwrite images in the same directory.
We can make the plugin clear the directory on the device after pulling, yeah.
plugin/src/main/kotlin/com/facebook/testing/screenshot/build/ScreenshotsPlugin.kt
Outdated
Show resolved
Hide resolved
Co-authored-by: Hilal Alsibai <993832+xiphirx@users.noreply.github.com>
Co-authored-by: Hilal Alsibai <993832+xiphirx@users.noreply.github.com>
Co-authored-by: Hilal Alsibai <993832+xiphirx@users.noreply.github.com>
Co-authored-by: Hilal Alsibai <993832+xiphirx@users.noreply.github.com>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Generally approving the changes, still had a few comments. Thanks for persisting and continuing to work on this, I really appreciate it!
I will try to import this code at some point this week, likely to have it merged next week.
return; | ||
} | ||
|
||
private String readPreviousTestRunId() { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Still wondering this
|
||
move_all_files_to_different_directory(bundle_name_local_file, dir) | ||
# clean up | ||
shutil.rmtree(bundle_name_local_file) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is where you could also issue a cleanup command to the device
@@ -503,16 +496,21 @@ def pull_all(package, dir, adb_puller): | |||
pull_images(dir, device_dir, adb_puller=adb_puller) | |||
|
|||
|
|||
def pull_filtered(package, dir, adb_puller, filter_name_regex=None): | |||
def pull_filtered(package, dir, adb_puller, testRunId, filter_name_regex=None): | |||
device_dir = pull_metadata(package, dir, adb_puller=adb_puller) | |||
_validate_metadata(dir) | |||
metadata.filter_screenshots(join(dir, 'metadata.xml'), name_regex=filter_name_regex) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Seems like this needs to be metadata.json
, likely should be a constant value that is reused everywhere
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This needs to be updated.
@xiphirx , please wait a bit, don't merge, I've just realized that I've tested only screenshots recording. I see that there's still some python code that uses |
@xiphirx , I was trying to implement proper cleaning up feature, but I realized that I can't do it without having a big picture of how python plugin works. Not sure that I have enough time to explore it. So I just leave current implementation. I tested @xiphirx , please tell me if I can do anything else in the scope of this PR. Thanks |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Just a few small things, then I'll import and test internally + merge!
import java.io.OutputStream; | ||
import java.io.RandomAccessFile; | ||
|
||
class RandomAccessFileOutputStream extends OutputStream { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This class is now not used, please remove.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
removed, thanks
@@ -503,16 +496,21 @@ def pull_all(package, dir, adb_puller): | |||
pull_images(dir, device_dir, adb_puller=adb_puller) | |||
|
|||
|
|||
def pull_filtered(package, dir, adb_puller, filter_name_regex=None): | |||
def pull_filtered(package, dir, adb_puller, testRunId, filter_name_regex=None): | |||
device_dir = pull_metadata(package, dir, adb_puller=adb_puller) | |||
_validate_metadata(dir) | |||
metadata.filter_screenshots(join(dir, 'metadata.xml'), name_regex=filter_name_regex) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This needs to be updated.
if (metadataFile.exists()) { | ||
Gson gson = new Gson(); | ||
JsonReader jsonReader = new JsonReader(new FileReader(getMetadataFile())); | ||
mMetadata = gson.fromJson(jsonReader, new TypeToken<List<Map<String, Object>>>() { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why isnt this type token for the type List<ScreenshotMetadata>
so that it matches the type of mMetadata
?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
fixed, thanks
@xiphirx , for some reason github doesn't let me reply for the comment #264 (comment) I checked |
Nope thats ok. I'll hopefully merge this in this week, thank you! |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@xiphirx has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.
Explanation
Feature is implemented according to discussion in #173.
Of course some unexpected difficulties revealed during implementation. All of them related to appending to files, because when we run tests using orchestrator, new process is created for each test. So that each test should append data to files that left from the previous test. Standard Java tooling doesn't support this kind of scenarios.
Append to XML file
XmlSerializer
can't add node to the middle of the file. For example:To append nodes to existing XML file I use
RandomAccessFile
, that points to position before the root closing tag. And after appending I have to add new root closing tag manually.Write bitmap
Despite the fact that ZIP file format designed to support appending of new file to existing archive without full rewrite, this operation isn't supported. It isn't supported for TAR format as well, but because of format structure it's much easier to create workaround for TAR format. It works similar to XML workaround. I use
RandomAccessFile
to seek for end of the last entry, and start append new entities after the last one. I use Apache Commons Comperes to work with TAR, I'm happy to switch to any other lib/tool if you advice me.@xiphirx , please tell me what do you think about approach in general. I will also appreciate it if you give me some advises regarding code style and structuring.