x/mobile: Android Storage Access Framework support

[Nutomic]

On Android Kitkat and higher, the Storage Access Framework is the only way to read/write files on an external SD card (eg /storage/sdcard1/).

However, access to that is handled through URIs like this:

content://com.android.providers.media.documents/document/image:62

These URIs can't easily be converted to paths [1]. We need this for syncthing-android, and have discussed this already [2] [3].

Go Version: 1.4.2
OS: Android, any architecture

[1] http://stackoverflow.com/q/20067508/1837158
[2] syncthing/syncthing-android#29
[3] https://forum.syncthing.net/t/writing-to-external-sd-card-in-android-5-0-and-above/2492

[lfarkas]

is there any reason why it's unplanned? currently there is not any other way on android to use external storage!?

[crawshaw]

Unplanned means we want to do it, but have not found the programmer hours to do it. We are not a large team.

[bassmaster187]

i think, syncthing android is useless, without access to sdcard.

bittorrent sync is able to sync the pictures of my phone (sony xperia z3 on lolipop) to all my windows clients and i really want to get rid of it.

[fcastilloec]

Any progress? It's been almost a year since this was reported. Access to the SD card is an extremely important feature.

[Eddy2909]

bump

Are there any news on that?

[fti7]

Guys, any new on that? Its an Important Feature :-(

[fti7]

@crawshaw do you accept bounties for that Issue? Maybe we can sponsor the "Programmer Hours" somehow?

[urza]

This is really a very important feature! I will also donate few dollars if needed..

[AudriusButkevicius]

How could this be implemented though?
As far as I understand, it's impossible without tapping into the NDK, hence adding a mandatory cgo dependency?

[lkwg82]

For me I solved it with marshmallow release. I extend the space to the sdcard. So I have 70 sth gig on my S5.
But it does notsolve it for Lollipop releases.

[Cysioland]

@lkwg82 Not to mention, that not all Marshmallow phones allow Adoptable Storage
(vendors ruin all the fun, unfortunately).

[fti7]

Thats the problem... Adoptable Storage is disabled by some Vendors (I guess for Performance-Reasons) :-(

[Cysioland]

@fti7 Or for we-fucked-up-the-rom-badly reasons

[frispete]

@lkwg82: you're paying a price for this: hope, you never uninstall syncthing, because all your synced data with go with it... Good luck!

I'm syncing about 100G with syncthing. Even using root, it creates major headaches to use arbitrary folders on an external SD card. Some are documented here:
syncthing/syncthing-android#719
Swallowing my sediments on this topic in order to not offend anybody.
The real causer don't see this anyway.

[leoluk]

The Syncthing guys have a $600 bounty over there:

https://forum.syncthing.net/t/600-bounty-for-writing-to-external-sd-card-in-android-5-0/2550

I'll contribute to a bounty to make this happen in Go.

[frispete]

Can't #16876 help to further advance on this issue?

[AudriusButkevicius]

So is the problem only path conversion?
Or is the problem that the writes have to go through some API that's only available in Java?
I know reads work via Go (as we have a workaround for read only access)

[Kaned1as]

Or is the problem that the writes have to go through some API that's only available in Java?

Yes, this is the problem. Android renders external SD card as Read-Only and only passes specific file descriptors with write-bit set if requested with ACTION_OPEN_DOCUMENT.

[LeonardoGentile]

Still unplanned after 2 years?

[Kuurusch]

Hey, any progress here?

[LeonardoGentile]

Could please someone from the team give an honest answer here?
Something unplanned for 2 year is very unlikely to be implemented.
Since the first answer from @crawshaw nobody else from the team bothered to answer to anybody, also ignoring a bounty and requests to further contribute to implement this.

This is blocking an essential feature for syncthing and apparently there is nothing the syncthing team can do about it.

If you don't have the time or you are not willing to do it just please tell us and we will stop to complain and look for alternatives.

Thanks 👍

[AudriusButkevicius]

So I am pretty sure nobody in the core team is interested to solve this, and I am fine with that as they have bigger fish to fry before Go becomes even mildly popular on Android. The thing I am not fine with however is lack of guidance. Even if I had the hours to do it, I haven't been provided any guidance on how this could be done even on a high level.

[griesemer]

Unfortunately there's no one on the Go team actively supporting x/mobile at the moment. We are a small team and we are spread thin, and currently have to focus on different priorities. My apologies.

cc: @crawshaw in case he wants to add anything to this.

[dolanor]

cc: @eliasnaur do you have any ideas on what is the problem here, and what can be done about it?
(I'm pinging you because I know you're quite knowledgeable on the mobile part, sorry if I'm mistaken)

[eliasnaur]

It's not clear to me what this issue is about. Is it enough to use

https://developer.android.com/reference/android/content/ContentProvider.html#openFile(android.net.Uri, java.lang.String)

to access the file with a specific URL and then use

https://developer.android.com/reference/android/os/ParcelFileDescriptor.html#getFd()

to extract a file descriptor to use with https://godoc.org/os#NewFile?

If not, please clarify.

[AudriusButkevicius]

I don't see a way to handle exceptions in case of reverse bindings?
I guess I am expected to write a Java wrapper that handles that gracefully?

[Kaned1as]

@eliasnaur , it's read-only fd that you can get this way. Getting rw fd is a bit more painful task involving showing activity to the user with access request, e.g. see SAF-wrapper here

[AudriusButkevicius]

Read FDs you can already get via os.Open...

[frispete]

The SAF-wrapper, that @Adonai points to seems exactly the part, that is missing so badly in syncthing...
Syncing is omnidirectional, hence RW file operations are required here.

[eliasnaur]

I still don't think this issue is specific enough. It seems to me, syncthing can create a Java wrapper for its file operations and expose that wrapper to Go through gomobile bind. If not, what specifically is blocking them?

[AudriusButkevicius]

Probably you are right, that should be sufficient. I need to workout how to do the android part, as the go part seems trivial.

[Nutomic]

From what I understand, it should be possible to use gobind and call the Storage Access Framework APIs directly from Go. Main blocker for that is making Syncthing work with gomobile. So I guess this issue could be closed.

[AudriusButkevicius]

That's not a blocker, as we don't need to make syncthing work with storage access framework, it's the other way around, we need to make a library that uses Java reverse bindings, and then use that in syncthing.

[AudriusButkevicius]

Lets not close it until we work up a solution though.

[eliasnaur]

If you create a Java wrapper, you don't need the reverse bindings. Creating and binding a Go interface and implementing it on the Java side should suffice. Note that gomobile bind doesn't copy []byte arguments, so read/writes across the language barrier should be efficient.

I'm going to close this as out of scope; I don't think direct support for Android SAF is something that belongs in gomobile so it's not clear how this issue can be resolved. A re-opens or a new issue has to specify (for example with small example programs) specific features wanted from gomobile bind that either doesn't work (a bug) or isn't yet done (a feature request) that blocks the writing of SAF support elsewhere.

[AudriusButkevicius]

Well you definately need a wrapper, because as it stands now, there is no way to handle JVM exceptions in Go, so you must have a exception free wrapper which you then reverse bind to.

[dolanor]

I don't know gomobile enough, but wouldn't a Java exception bubble up as a Go panic, and therefore can be recover'ed to make it safe?

[eliasnaur]

I believe both bindings and reverse bindings support Java Exceptions. If not, please raise an issue about what doesn't work.

If you need examples, take a look in bind/testpkg.

[Kaned1as]

Hi, I see this is finally closed, how soon Syncthing will pick up the changes?

[AudriusButkevicius]

It's closed as it's not something Go developers are going to be addressing in the standard library, not that it has been resolved.

Either the docs are thin or I am bad at searching, but looking at #21033 it seemed to me that exception handling is not handled?

[AudriusButkevicius]

Also #18210 @eliasnaur

[eliasnaur]

#21033 is about unexpected Java exceptions bubbling up to the Go side. It's the equivalent of a Go panic that isn't recovered. If you make sure to return error from your interface methods, any Java exception will be converted to a Go error for you.

#18210 is about a build failure. I can't reproduce it and I'm not sure syncthing even needs reverse bindings.

[AudriusButkevicius]

Sorry, but how does it access java APIs if we don't reverse bind? (Forward bind I assume is Java calls go, reverse is go calls java).

[eliasnaur]

No, reverse bind is importing and accessing Java classes directly. It's a (very) experimental feature to explore not having to write Java code at all.

Regular bindings is automaticall generating Java wrapper classes and interfaces from Go exported structs, interfaces, functions, variables and constants.

With (regular) gomobile bind you can (for example) define in Go

package saf

type Writer interface {
     Write(b []byte) (int, error)
}

And then, from Java

import go.saf.Writer;

public class SAFWriter implements Writer {
     @Override public int write(byte[] b) throws Exception {
          ....
      }
}

And then pass instances of SAFWriter to a Go function or method expecting a Writer.

If you need more help, please write to golang-nuts where your questions are exposed a much larger audience.

[tomachinz]

I want this so I can remotely delete images, photos, audio from my phone. Fairly sure I got it to work once with Syncthing maybe on a different phone (not a Samsung). Definitely was able to get BTSync to sync the whole SD card such that I can even delete images from DCIM etc.

[TRSx80]

And this is why I don't want to use anything that touches Google in any way.

I have been hearing about Syncthing all over lately, finally had some time to dig into it, was starting to get very excited, everything sounds great, but now this.

Every time. So sick of it. For years now.

And this is why I will give no help, mind share, bug reports, free discussion/promotion of, etc. anything that has anything to do with Google. There are lots of other actually Free software projects that deserve our support and attention instead. In fact, whenever Syncthing is brought up in LUG, IRC, or anywhere else, I will be sure to let people know you can't even use it on your Android SD card.

Every. Freaking. Time. When will people learn?

EDIT: Purism Librem 5 can't get here soon enough. Once we can get actual GNU/Linux on a phone, Android's days will be numbered.

[AudriusButkevicius]

So just to quantify, this is possible to solve, but essentially your go application has to become a java application calling out to Go, rather than the other way around.

For any vanilla Go application that is run as a simple binary has no way to get a handle on/initialize the JVM to call through to any Java API's. In syncthings case we have a java ui that pokes a go binary via a rest api. In this case we'd have to turn our binary into a library and load/link it in the java process, and then call Go API's from Java, which is a large effort.