Android: SAF & FileDescriptor

[PaulWoitaschek]

I need to use this library on android using an uri from SAF.
Therefore it would be great if you implemented an api for using a file descriptor.

An example for how to implement this so the uir can be passed to the NDK can be found here:
https://github.com/wseemann/FFmpegMediaMetadataRetriever/blob/b42986fb4044c54131379407598d8ac0ff1a3326/gradle/fmmr-library/core/src/main/java/wseemann/media/FFmpegMediaMetadataRetriever.java#L214

[tanersener]

Can you please give some more info like what is the issue you're facing right now and why that feature needs to be implemented in the library rather than the application?

[PaulWoitaschek]

I need to parse metadata of a lot of media files from the users phone storage.

On Android 10 you don't have access to the users sd card using the file api:
https://developer.android.com/about/versions/10/privacy/changes#scoped-storage

Instead you need to use the SAF framework:
https://developer.android.com/guide/topics/providers/document-provider

You fire an intent:
https://developer.android.com/reference/android/content/Intent.html#ACTION_OPEN_DOCUMENT

And get an uri in onActivityResult. This uri is no file uri but a content provider uri.
Therefore you can't use it with mobile-ffmpeg at all (unless you make a deep temporary copy of the file, which is no option here as I can't duplicate gigabytes of media.)

You can use the content provider to obtain a file descriptor:
contentResolver.openFileDescriptor(uri, "r"). and work with that.
But this need integration with mobile-ffmpeg.

[tanersener]

Thanks for explaining the issue in detail.

MobileFFmpeg for Android is made of two modules; one native module (written in C/C++) and one Java module. There is a JNI layer between these two and Java module sends API requests to native module through this JNI layer.

Argument parsing and opening input/output files are all implemented in native module. Supported protocols are also defined there. As far as I know, SAF is not available in NDK. That means that I can't use SAF inside the native module. So, I don't think that SAF can be integrated with MobileFFmpeg. Do you have any recommendations?

[PaulWoitaschek]

@wseemann

Can you help out here?

[HBiSoft]

@tanersener I'm not that familiar with NDK and I do not want to highjack this issue, I just thought I would give my 2 cents.

As @PaulWoitaschek mentioned, on Android we can not access the file (or its path) directly, unless the file is located in our application's directory. Instead, we have to make use of the ContentResolver to open a FileDescriptor. As shown below:

FileDescriptor fd = getContentResolver().openFileDescriptor(Uri, "r").getFileDescriptor();

---

In the project that was provided as an example above, the owner of that library checks to see if the Uri that was passed is a content:// Uri. If it was then he uses a FileDescriptor instead of passing the path of the file, please have a look here:

https://github.com/wseemann/FFmpegMediaMetadataRetriever/blob/b42986fb4044c54131379407598d8ac0ff1a3326/gradle/fmmr-library/core/src/main/java/wseemann/media/FFmpegMediaMetadataRetriever.java#L214

---

So you can create 2 native methods, one for handling file paths(as you have it currently):

mobile-ffmpeg/android/app/src/main/java/com/arthenica/mobileffmpeg/Config.java

Line 642 in 917c22e

native static int nativeFFmpegExecute(final String[] arguments);

and one for passing the FileDescriptor, something like this:

native static int nativeFFmpegExecute((final String[] arguments, final FileDescriptor fd) {

On the C/C++ side you can then handle this depending on which one of the above was called.
Of course, there will also have to be another method added in FFmpeg.java:

mobile-ffmpeg/android/app/src/main/java/com/arthenica/mobileffmpeg/FFmpeg.java

Line 55 in 917c22e

public static int execute(final String[] arguments) {

Like this:

public static int execute(final String[] arguments, final FileDescriptor fd) {

---

Developers using your library could then have the option to pass a FileDescriptor, something like this:

FileDescriptor fd = getContentResolver().openFileDescriptor(Uri, "r").getFileDescriptor();
//Note that, instead of passing the path at the end, FileDiscriptor is passed instead.
FFmpeg.execute("some_commands_by_developer"+outputPath, fd);

---

Edit:

This is how he handles it from C:

https://github.com/wseemann/FFmpegMediaMetadataRetriever/blob/b42986fb4044c54131379407598d8ac0ff1a3326/gradle/fmmr-library/library/src/main/jni/metadata/ffmpeg_mediametadataretriever.c#L304

int set_data_source_fd(State **ps, int fd, int64_t offset, int64_t length) {
    char path[256] = "";

	State *state = *ps;
	
	ANativeWindow *native_window = NULL;

	if (state && state->native_window) {
		native_window = state->native_window;
	}

	init(&state);

	state->native_window = native_window;
    	
    int myfd = dup(fd);

    char str[20];
    sprintf(str, "pipe:%d", myfd);
    strcat(path, str);
    
    state->fd = myfd;
    state->offset = offset;
    
	*ps = state;
    
    return set_data_source_l(ps, path);
}

[alexcohn]

There is no need to change the C++ side at all. A neat trick lets you work with a file path instead of the FileDescriptor. It's based on the specs of the Linux /proc/self/fd files.

As service to the community, the integer descriptor may be extracted on the Java side of the library (if it sniffs the content:// prefix), but may be easily done by the app developer who uses the library.

[kingvhit]

Hi, another reason that must support FileDescriptor is defined here
#185

Above trick, I am testing with the other project with same purpose, but this behaviour can be changed by Google in the future.

[tanersener]

I know you expect the library to handle URIs internally but I think using the following method is the most elegant solution for SAF.

File descriptors received from SAF can be used using the pipe protocol. Both input and output files can be defined with pipe protocol. But some file formats require the output to be seekable, so pipe protocol will fail with them. Be careful about them.

String safUriToFFmpegPath(final Uri uri) {
    try {
        ParcelFileDescriptor parcelFileDescriptor = getContentResolver().openFileDescriptor(uri, "r");
        return String.format(Locale.getDefault(), "pipe:%d", parcelFileDescriptor.getFd());
    } catch (FileNotFoundException e) {
        return "";
    }
}

[RowlandOti]

All this is unnecessary, I pondered upon this myself and hated this elegant library for it. I did it using a utility function like:

val filePath = RealPathUtil.getRealPath(
                           context,
                            currentUri
                        )

so, contentUri to filePath is what you need. It is just as fast.

[HBiSoft]

@RowlandOti
The RealPathUtil you are referring to is probably querying the _data column to get the path of the file. I have released a library myself that converts a Uri to a "real" path. The problem is that the _data column can no longer be accessed when you are targeting 29.

[RowlandOti]

You are right, this will only work for API < 29. This was the only caveat. @HBiSoft

[HBiSoft]

@tanersener
I've tested the pipe protocol and it is working.

But some file formats require the output to be seekable

Can you please explain what is meant with the above?

[tanersener]

@HBiSoft

Muxers of some file formats like mp4 and mov requires the output to be seekable to write file metadata/header information correctly. Unfortunately pipe protocol does not support seeking. Therefore you may receive muxer does not support non seekable output error if you try to use pipe protocol in an output file.

There are some options that can be used to overcome this limitation, like -movflags frag_keyframe or -movflags empty_moov. But using these options has consequences. For example, you may not have a seekable video file at the end. So, you need to be careful about the options you use. FFmpeg's mov, mp4, ismv guide includes the list of options and explains what they are used for. It may help you to understand the those consequences better.

Update: Unfortunately input files are affected from this limitation too. You can't use mp4 and mov files with pipe protocol as input unless they are created using -movflags +faststart. There is not way to overcome this limitation.

[HBiSoft]

@tanersener
Thank you for your informative reply.
As I understand it, this will only be an issue if I use the pipe protocol with the output and not when I use it with the input? If this is the case, I will then be able to set the output path to my application directory and all will be good.

[kingvhit]

There is no need to change the C++ side at all. A neat trick lets you work with a file path instead of the FileDescriptor. It's based on the specs of the Linux /proc/self/fd files.

As service to the community, the integer descriptor may be extracted on the Java side of the library (if it sniffs the content:// prefix), but may be easily done by the app developer who uses the library.

About this solution?, why does not any one interest this one?

[tanersener]

@HBiSoft I missed some details about input files. Updated my previous post with the following information. It looks like safUriToFFmpegPath can be used as a temporary workaround but a protocol like saf: or fd: has to be implemented.

Unfortunately input files are affected from this limitation too. You can't use mp4 and mov files with pipe protocol as input unless they are created using -movflags +faststart. There is not way to overcome this limitation.

@kingvhit That solution is a trick that works for some devices and doesn't work on some others. I tested it and it didn't work for me. I receive Permission denied errors when I try. You can try and see if it works for you.

[kingvhit]

@tanersener : Hi, I have tested on Xiaomi Mi 8 Pro (Adroid 9), and Google Pixel XL 3 (android 10) and all of them working well.

I have concern when you told that it's is not really work for all devices :'(. Could you please tell me which device are you testing on before.

Just I want to re-check it on my app.

[HBiSoft]

@tanersener I agree with:

a protocol like saf: or fd: has to be implemented.

I think the best is to use fd: because we can open a FileDescriptor with any(almost) Uri. Then we, using the library, can pass the FileDescriptor instead of the path. This will work on all API's and no "hacks" are involved. I know my first comment means more work, but I believe it is the best/correct way - https://developer.android.com/training/data-storage/shared/media#native

[tanersener]

@kingvhit
My old Galaxy A5 (2016) (Android 7.0) receives /proc/self/fd/66: Permission denied.
Also Pixel 2 Simulator with API 24 also receives permission denied error for me.

[alexcohn]

@tanersener I am afraid this only means that there is no universal way of handling this. I didn't expect Permission denied on Android 7, but anyways you only need this workaround for API 29 and higher. BTW, my Android 7.0 device handles this perfectly. I only bothered to handle input in my simple exercise, but it works even cross-process: see alexcohn/FFmpeg-Android@686c4b5

[tanersener]

@alexcohn Tried to test your pull request to see whether I'm missing something but I still receive /proc/11193/fd/52: Permission denied on my device. Attached the logs here if you want to take a look.

[alexcohn]

Which file did you choose? For me, uri content://com.android.providers.downloads.documents/document/raw%3A%2Fstorage%2Femulated%2F0%2FDownload%2Fqq.mov points to /sdcard/Download/qq.mov (a.k.a. /storage/emulated/0/Download/qq.mov).

[HBiSoft]

I tested with a file on my SD Card, the content uri is content://com.android.externalstorage.documents/document/6330-6333%3ATest.mp4. It is pointing to /storage/6330-6333/Test.mp4 and I also get permissions denied -
/proc/23074/fd/70: Permission denied

I tested the above by calling:

protected void onActivityResult(int requestCode, int resultCode, Intent data) {

        if (requestCode == SELECT_VIDEO_REQUEST && resultCode == RESULT_OK) {
                ParcelFileDescriptor parcelFileDescriptor = null;
                try {
                    parcelFileDescriptor = getContentResolver().openFileDescriptor(data.getData(), "r");
                } catch (FileNotFoundException e) {
                    e.printStackTrace();
                }
                int fd = parcelFileDescriptor.getFd();
                int pid = Process.myPid();
                String mediaFile = "/proc/" + pid + "/fd/" + fd;

                int rc = FFmpeg.execute("-i "+mediaFile+" -crf 18 -c:v libx264 -c:a copy "+outputPath");
                if (rc == RETURN_CODE_SUCCESS) {
                    Log.i(Config.TAG, "Command execution completed successfully.");
                } else if (rc == RETURN_CODE_CANCEL) {
                    Log.i(Config.TAG, "Command execution cancelled by user.");
                } else {
                    Log.i(Config.TAG, String.format("Command execution failed with rc=%d and the output below.", rc));
                    Config.printLastCommandOutput(Log.INFO);
                }
        }
    }

[RowlandOti]

By default, by experimentation, I noticed that drives mounted to /storage/XXXX-XXX will by default have write permissions denied.

I don't think we should face similar problems with /sdcard/Download/ aka /storage/emulated/0/Download/ as both have read/write access.

[tanersener]

@alexcohn

If I select /storage/emulated/0/Download/video1.mp4 file, it gets content://com.android.externalstorage.documents/document/primary%3ADownload%2Fvideo1.mp4 uri. I don't know why it has a different format then yours or from @HBiSoft's file.

opened content://com.android.externalstorage.documents/document/primary%3ADownload%2Fvideo1.mp4 as /proc/14517/fd/52
/proc/14517/fd/52: Permission denied

[HBiSoft]

@alexcohn
I'm not sure, I think MANAGE_EXTERNAL_STORAGE is intented for app that requires broad access like file managing applications.

Some apps have a core use case that requires broad file access, such as file management or backup & restore operations.

The permissions to access it also looks ugly Allow access to manage all files where I only want the user to select one file.

Also from the docs:

Starting in Android 11, apps that have the READ_EXTERNAL_STORAGE permission can read a device's media files using direct file paths and native libraries. This new capability allows your app to work more smoothly with third-party media libraries.

But this doesn't help because how are we supposed to retrieve the path from MediaStore.. We can use the file path but be can't retrieve it (unless there is a way I don't know of, without using the _data column).

[pawaom]

Android has put up some additional documentation regarding Storage access in android 11

here are the links
https://medium.com/androiddevelopers/modern-user-storage-on-android-e9469e8624f9

https://developer.android.com/preview/privacy/storage

and This is from commonsware

https://commonsware.com/blog/2020/03/22/r-raw-paths-all-files-access.html

@tanersener , @HBiSoft we can even file feedback and get specific help for use cases impacted by this feature

here

https://google.qualtrics.com/jfe/form/SV_9HOzzyeCIEw0ij3?Source=scoped-storage

[HBiSoft]

@pawaom
I've opened an issue on issuetracker

It seems that we can continue using the data column to access the file path. I have tested this and I was able to pass a file to mobile-ffmpeg on Android R. This includes the SD Card.

The only issue I have now is that on Android Q, this doesn't work. I'm waiting for Google to reply. I feel they should give us the option to enable legacy storage programmatically. We will then be able to access files on Android Q as well.

I will let you know when they reply.

@tanersener
I think you can close this issue as there is nothing you have to do on the library side. Thank you for your support and great library.

[pawaom]

@HBiSoft , according to this

https://medium.com/androiddevelopers/modern-user-storage-on-android-e9469e8624f9

they have written

However, we understand that many apps depend heavily on APIs that use file paths, including third-party libraries, that cannot easily switch to using file descriptors. So in Android 11, APIs and libraries that use file paths will be enabled again. Your app can use the requestLegacyExternalStorage manifest attribute to ensure compatibility for users running Android 10

also if you read commonware's blog, it seems we need to use old way for upto android p , then for android Q requestLegacyExternalStorage and again for android R use the new access Raw file path thing, all this is really confusing

I have also filed an issue with them from the link they provided

https://google.qualtrics.com/jfe/form/SV_9HOzzyeCIEw0ij3?Source=scoped-storage

If few others do the same they might consider it to be urgent

[HBiSoft]

@pawaom

Your app can use the requestLegacyExternalStorage manifest attribute to ensure compatibility for users running Android 10

If that's the case, then I'm all sorted. I've tested and everything seems to be working fine.

[tanersener]

Thanks everyone who contributed to sort this out 👍

[pawaom]

finally they have understood it

https://issuetracker.google.com/issues/151407044#comment10

Do keep requestLegacyExternalStorage in the manifest if you need legacy storage on Q devices. It will be ignored on R devices once you target R. We are improving the documentation on this.

Re DATA column: will publish more guidance on this soon

I am adding this comment just for others who might refer to it in the future

[pawaom]

this can be useful, @alexcohn , @HBiSoft @tanersener
https://youtu.be/RjyYCUW-9tY?t=300

and

android/uamp#325 (comment)

[HBiSoft]

@pawaom
I'm not sure why you are sharing this?

We've had access to files/paths from the MediaStore since API 1, the only issue (why this question was started in the first place) was that it was going to be removed(as it is in Android 10 without requesting legacy storage), but Google decided to not remove access in Android R (Instead they introduced FUSE, which is basically the same as it always was).

I clarified everything in my answer here as well - https://stackoverflow.com/a/60917774/5550161

[pawaom]

https://developer.android.com/preview/privacy/storage#media-direct-file-native

@HBiSoft , according to the new explanation , this is just an updated info for any one who might check it in the future, I shared tagging you and a few others, because the changes in Android 10 had caused confusion, I was unaware of your answer in Stackoverflow, sorry for the trouble

[alexcohn]

There are two little problems with the fopen() approach. For one, it does not work on Android 10 and there doesn't seem to be an intention to fix that for the ~20% of devices currently active. Second, this does not work smoothly for Google Drive and other document providers. I have addressed both issues in #440.

[pawaom]

There are two little problems with the fopen() approach. For one, it does not work on Android 10 and there doesn't seem to be an intention to fix that for the ~20% of devices currently active. Second, this does not work smoothly for Google Drive and other document providers. I have addressed both issues in #440.

they have mentioned

If you access media files using the File API or native libraries, it's recommended that you opt out of scoped storage by setting requestLegacyExternalStorage to true in your app's manifest file. That way, your app behaves as expected on devices that run Android 10.

[HBiSoft]

@alexcohn
I had a quick look at your pull request, it looks very promising. Could you perhaps push it to a different branch so we can test it?

[alexcohn]

Which different branch could it be? Feel free to test directly from https://github.com/alexcohn/mobile-ffmpeg/tree/experiment/scoped-storage

[SoluLabLive]

@alexcohn @tanersener any plan to give updated Gradle version for these changes?

[xiaos]

But soon Android 11 will come, and we cannot opt out of scoped storage, how to solve this issue?

[Naresh95375]

any progress in support on Android 11?

[MenilV]

After a couple of days of struggle with this one, here's a workaround that supports Android 11 and will probably be good enough until this one gets released:

• since the FFmpeg.execute(...) requires a file path you can always create a temp File in your cache dir with val copy = File.createTempFile("file-name", ".extension", context.cacheDir)
• use the contentResolver to open the inputstream from the Uri and copy the input stream to the newly created file:

context.contentResolver.openInputStream(uri)?.apply{
	copy.outputStream().use { fileOut ->
            this.copyTo(fileOut)
        }
}

• now you can send the file path to FFmpeg copy.path and be Android 11 compatible.
• cleanup the temp file afterward copy.delete()

Tested on Samsung s8+ (Android 10 rooted) and Pixel 4a (Android 11).

Drawbacks of this approach might include memory issues if the file itself is larger.

[HBiSoft]

I'm not sure why people keep asking for Android 11 support. You can pass an Uri thanks to this pull request - #440

There's no need to copy the file first, just pass the Uri to the library.

There's only one issue left, I could find on Android 11 - #634 (it looks like this will be fixed soon)

[aGreatDay]

@HBiSoft

There's no need to copy the file first, just pass the Uri to the library.

Is this available in release 4.4?

First, I create a new MediaStore entry with contentResolver.insert which returns a Uri, and pass the uri.toString() as the outputfile for FFmpeg.execute(...).
This results in:

mobile-ffmpeg: [NULL @ 0xf465a000] Unable to find a suitable output format for 'content://media/external_primary/video/media/2908'
mobile-ffmpeg: content://media/external_primary/video/media/2908: Invalid argument

[RowlandOti]

Also just to chip into those who will make the mistake of trying to copy the file first - it is going to very slow for some devices and users on the other end may notice lagging especially if the file is large. If reading the file is a repetitive and on-demand thing, then that approach is highly inefficient.

On the other hand, reading is as a Uri resource directly is pretty smooth. I build an app around other android users consuming media content hosted on android media hubs and the experience was very smooth.

[blue-peacock]

@aGreatDay

I'm facing the same error using release 4.4. Has anybody a solution?

This results in:

mobile-ffmpeg: [NULL @ 0xf465a000] Unable to find a suitable output format for 'content://media/external_primary/video/media/2908'
mobile-ffmpeg: content://media/external_primary/video/media/2908: Invalid argument

[alexcohn]

@aGreatDay, @blue-peacock you can not simply use a content Uri on "command line". You must wrap it as shown in the test app example SafTabFragment.encodeVideo():

String videoPath = Config.getSafParameterForWrite(requireContext(), outUri);

[blue-peacock]

@alexcohn thank you, but when I pass the Uri content://media/external/images/media/2854 to Config.getSafParameterForWrite() I'm getting saf:173/image1617797380694.jpg as output for videoPath. The ffmpeg command then fails with the error:

[image2 @ 0x71493a6600] Could not open file : saf:173/image1617797380694.jpg av_interleaved_write_frame(): I/O error

The content: Uri is created by the following code:

 val filePrefix = "image"
        val fileExtn = ".jpg"
        val contentValues = ContentValues()
        contentValues.put(MediaStore.Images.Media.RELATIVE_PATH, "Pictures/" + "Test")
        contentValues.put(MediaStore.Images.Media.TITLE, filePrefix + System.currentTimeMillis())
        contentValues.put(MediaStore.Images.Media.DISPLAY_NAME, filePrefix + System.currentTimeMillis() + fileExtn)
        contentValues.put(MediaStore.Images.Media.MIME_TYPE, "image/jpeg")
        val uri = context.contentResolver.insert(MediaStore.Images.Media.EXTERNAL_CONTENT_URI, contentValues)