Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enabling TLS1.1/1.2 on Android 4.4 devices (API 19/KitKat) #2792

Merged
merged 4 commits into from
Dec 14, 2019
Merged

Enabling TLS1.1/1.2 on Android 4.4 devices (API 19/KitKat) #2792

merged 4 commits into from
Dec 14, 2019

Conversation

mqus
Copy link
Contributor

@mqus mqus commented Nov 8, 2019

This is an attempt to fix #2777 . I already got the Downloader and the Extractor parts working. What I may need some help on is the player. I simply don't know how to approach this and where to start.

Please give me some feedback on the "finished" parts, too.

@theScrabi
Copy link
Member

Player may be complicated as it is a Wohle framework.

@mqus
Copy link
Contributor Author

mqus commented Nov 10, 2019

I found google/ExoPlayer#1819 which says I should set HttpsURLConnection.setDefaultSSLSocketFactory() "prior to using exoplayer" or implementing a new DataSource. Could you comment on what you would find more feasible/desirable?

Where would be the best place to call that static method and where should I call it if I also want to use it for the other two connections?

As I wrote, I'm clueless, but motivated. I can do some research myself but I wanted to try the easy way (asking), first ;-).

@mqus
Copy link
Contributor Author

mqus commented Nov 11, 2019

I tried the static-function approach and put HttpsURLConnection.setDefaultSSLSocketFactory() at the start of MainActivity. I tried it and everything works now. It should not impact other devices which don't run Android 4.4.2 because everywhere I set the new SslFactory I checked for the android version first, but I'm not completely sure.

In any case, please check the given commits. Should I roll them up into one?

@mqus
Copy link
Contributor Author

mqus commented Nov 18, 2019

I'm sorry for my impatience, but Is there a way to speed this process up?

@TobiGr TobiGr added device/software specific Issues that only happen on some devices or with some specific hardware/software bug Issue is related to a bug labels Dec 3, 2019
@TobiGr TobiGr mentioned this pull request Dec 3, 2019
1 task
Copy link
Contributor

@TobiGr TobiGr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am sorry for the silence. There are just too many open PRs and KitKat has only a small number of active users so other PRs had a higher priority. There was a second PR which changed some important aspects of the downloader. Therefore, this one needed to wait, too.

However, we have time to review your changes now. can you please rebase your commits? The new downloader was moved to https://github.com/TeamNewPipe/NewPipe/blob/dev/app/src/main/java/org/schabi/newpipe/DownloaderImpl.java

@Stypox, @yausername, @theScrabi or @mauriciocolli Can someone review this PR, too? This is security relevant and I am not an expert on this topic. I'll try to give a hint regarding the player within the next week, but I'd be glad if someone else could take a look at it.

@TobiGr TobiGr added peertube Service, https://joinpeertube.org/ soundcloud Service, https://soundcloud.com/ labels Dec 3, 2019
@mqus
Copy link
Contributor Author

mqus commented Dec 4, 2019

Thanks for reviewing so far!

I rebased and added most of your suggestions. It should look much cleaner now.

I tested it on my device and everything still works.

@mqus mqus closed this Dec 8, 2019
@mqus
Copy link
Contributor Author

mqus commented Dec 8, 2019

I now rebased on the current dev
I didn't mean to close the PR, force-pushed the wrong stuff accidently.

@mqus mqus reopened this Dec 8, 2019
@mqus
Copy link
Contributor Author

mqus commented Dec 11, 2019

So I rebased and tried out PeerTube.

It didn't work. Here is why:
TLS operates with a variable set of cipher suites which have to be matched up when starting a TLS connection to choose one which both client and server support.

The following are enabled by default on an Android 4.4.2 Emulator (All Lists are sorted alphabetically to make comparison easier.)

SSL_DHE_DSS_EXPORT_WITH_DES40_CBC_SHA
SSL_DHE_DSS_WITH_DES_CBC_SHA
SSL_DHE_DSS_WITH_3DES_EDE_CBC_SHA
SSL_DHE_RSA_EXPORT_WITH_DES40_CBC_SHA
SSL_DHE_RSA_WITH_DES_CBC_SHA
SSL_DHE_RSA_WITH_3DES_EDE_CBC_SHA
SSL_RSA_EXPORT_WITH_DES40_CBC_SHA
SSL_RSA_EXPORT_WITH_RC4_40_MD5
SSL_RSA_WITH_3DES_EDE_CBC_SHA
SSL_RSA_WITH_DES_CBC_SHA
SSL_RSA_WITH_RC4_128_MD5
SSL_RSA_WITH_RC4_128_SHA
TLS_DHE_DSS_WITH_AES_128_CBC_SHA
TLS_DHE_DSS_WITH_AES_256_CBC_SHA
TLS_DHE_RSA_WITH_AES_128_CBC_SHA
TLS_DHE_RSA_WITH_AES_256_CBC_SHA
TLS_ECDH_ECDSA_WITH_3DES_EDE_CBC_SHA
TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA
TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA
TLS_ECDH_ECDSA_WITH_RC4_128_SHA
TLS_ECDH_RSA_WITH_3DES_EDE_CBC_SHA
TLS_ECDH_RSA_WITH_AES_128_CBC_SHA
TLS_ECDH_RSA_WITH_AES_256_CBC_SHA
TLS_ECDH_RSA_WITH_RC4_128_SHA
TLS_ECDHE_ECDSA_WITH_3DES_EDE_CBC_SHA
TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA
TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA
TLS_ECDHE_ECDSA_WITH_RC4_128_SHA
TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA
TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA
TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA
TLS_ECDHE_RSA_WITH_RC4_128_SHA
TLS_RSA_WITH_AES_128_CBC_SHA
TLS_RSA_WITH_AES_256_CBC_SHA

The following are supported on an Android 4.4.2 Emulator, but not enabled:

SSL_DH_anon_EXPORT_WITH_DES40_CBC_SHA
SSL_DH_anon_EXPORT_WITH_RC4_40_MD5
SSL_DH_anon_WITH_3DES_EDE_CBC_SHA
SSL_DH_anon_WITH_DES_CBC_SHA
SSL_DH_anon_WITH_RC4_128_MD5
SSL_RSA_WITH_NULL_MD5
SSL_RSA_WITH_NULL_SHA
TLS_DH_anon_WITH_AES_128_CBC_SHA
TLS_DH_anon_WITH_AES_256_CBC_SHA
TLS_ECDH_anon_WITH_RC4_128_SHA
TLS_ECDH_anon_WITH_AES_128_CBC_SHA
TLS_ECDH_anon_WITH_AES_256_CBC_SHA
TLS_ECDH_anon_WITH_3DES_EDE_CBC_SHA
TLS_ECDH_anon_WITH_NULL_SHA
TLS_ECDH_ECDSA_WITH_NULL_SHA
TLS_ECDH_RSA_WITH_NULL_SHA
TLS_ECDHE_ECDSA_WITH_NULL_SHA
TLS_ECDHE_RSA_WITH_NULL_SHA

(these are either anon, which means they are unsuitable for TLS or NULL which means unencrypted, so none are desirable for us)

Those will be enabled by the last commit, if possible: (OkHttp moderntls + TLS_ECDHE_ECDSA_WITH_AES_{128,256}_CBC_SHA), adapted from https://github.com/AntennaPod/AntennaPod/pull/3439/files#diff-f0eab34b36f9ab46c533bdd783fddc46:

SSL_RSA_WITH_3DES_EDE_CBC_SHA
TLS_AES_128_GCM_SHA256
TLS_AES_256_GCM_SHA384
TLS_AES_128_CCM_SHA256
TLS_AES_256_CCM_8_SHA256
TLS_CHACHA20_POLY1305_SHA256
TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA
TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA
TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384
TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256
TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA
TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA
TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256
TLS_RSA_WITH_AES_128_GCM_SHA256
TLS_RSA_WITH_AES_256_GCM_SHA384
TLS_RSA_WITH_AES_128_CBC_SHA
TLS_RSA_WITH_AES_256_CBC_SHA

On an Android 4.4.2 device without vendor changes to the suites, this means that the following cipher suites are enabled with tls1.2:

TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA
TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA
TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA
TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA
TLS_RSA_WITH_AES_128_CBC_SHA
TLS_RSA_WITH_AES_256_CBC_SHA

Which doesn't overlap with the Framatube.org cipher suites, so no connection can be made. If you notice, it does say that kitkat devices can connect below but I simply can't find the source for that statement; I also couldn't enable the given Ciphersuite in any way.

If we want to fix that on the server (we probably have to), we could start by changing the default nginx config recommended by Peertube introduced here: (Chocobozzz/PeerTube@e883399)

@mqus
Copy link
Contributor Author

mqus commented Dec 11, 2019

There was another thing I noticed, while trying out mediaCCC some more:

Sometimes,when I download a video, it will work at first but then stall, while the console throws the following error(with context):

D/DownloadMission: 0:[request]  Range=bytes=22544384-23068671
    0:[response] Code=206
    0:[response] Content-Length=524288
    0:[response] Content-Range=bytes 22544384-23068671/61431254
D/DownloadMission: 2:[request]  Range=bytes=21495808-22020095
    2:[response] Code=206
    2:[response] Content-Length=524288
    2:[response] Content-Range=bytes 21495808-22020095/61431254
D/DownloadRunnable: 0:position 43 stopped 23068672/23068671
D/dalvikvm: GC_FOR_ALLOC freed 6166K, 35% free 34766K/53176K, paused 40ms, total 40ms
D/DownloadMission: 1:[request]  Range=bytes=22020096-22544383
    1:[response] Code=206
    1:[response] Content-Length=524288
    1:[response] Content-Range=bytes 22020096-22544383/61431254
D/DownloadRunnable: 0:acquired block at position=44 done=0
D/DownloadMission: 0:[request]  Range=bytes=23068672-23592959
    0:[response] Code=206
    0:[response] Content-Length=524288
    0:[response] Content-Range=bytes 23068672-23592959/61431254
D/DownloadRunnable: 0:position 44 stopped 23592960/23592959
    0:acquired block at position=45 done=0
D/DownloadRunnable: 2:position 41 stopped 22020096/22020095
    2:acquired block at position=46 done=0
E/DownloadMission: notifyError()
    javax.net.ssl.SSLHandshakeException: javax.net.ssl.SSLProtocolException: SSL handshake aborted: ssl=0x65da54e8: Failure in SSL library, usually a protocol error
    error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure (external/openssl/ssl/s23_clnt.c:741 0x5fb14d74:0x00000000)
        at com.android.org.conscrypt.OpenSSLSocketImpl.startHandshake(OpenSSLSocketImpl.java:448)
        at com.android.okhttp.Connection.upgradeToTls(Connection.java:146)
        at com.android.okhttp.Connection.connect(Connection.java:107)
        at com.android.okhttp.internal.http.HttpEngine.connect(HttpEngine.java:294)
        at com.android.okhttp.internal.http.HttpEngine.sendSocketRequest(HttpEngine.java:255)
        at com.android.okhttp.internal.http.HttpEngine.sendRequest(HttpEngine.java:206)
        at com.android.okhttp.internal.http.HttpURLConnectionImpl.execute(HttpURLConnectionImpl.java:345)
        at com.android.okhttp.internal.http.HttpURLConnectionImpl.getResponse(HttpURLConnectionImpl.java:296)
        at com.android.okhttp.internal.http.HttpURLConnectionImpl.getResponseCode(HttpURLConnectionImpl.java:503)
        at com.android.okhttp.internal.http.HttpsURLConnectionImpl.getResponseCode(HttpsURLConnectionImpl.java:136)
        at us.shandian.giga.get.DownloadMission.establishConnection(DownloadMission.java:249)
        at us.shandian.giga.get.DownloadRunnable.run(DownloadRunnable.java:84)
     Caused by: javax.net.ssl.SSLProtocolException: SSL handshake aborted: ssl=0x65da54e8: Failure in SSL library, usually a protocol error
    error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure (external/openssl/ssl/s23_clnt.c:741 0x5fb14d74:0x00000000)
        at com.android.org.conscrypt.NativeCrypto.SSL_do_handshake(Native Method)
        at com.android.org.conscrypt.OpenSSLSocketImpl.startHandshake(OpenSSLSocketImpl.java:405)
        	... 11 more
    notifyError() code = 1004
I/DownloadManagerService: Active network [connected=true metered=false] NetworkInfo: type: WIFI[], state: CONNECTED/CONNECTED, reason: (unspecified), extra: "TomateMozarella v4", roaming: false, failover: false, isAvailable: true, smCause: 0, isConnectedToProvisioningNetwork: false
W/DownloadMission: thread alive: DownloadMission[1] Immer noch Nichts zu verbergen_.mp4
D/DownloadMission: 2:[request]  Range=bytes=24117248-24641535
    2:[response] Code=206
    2:[response] Content-Length=524288
    2:[response] Content-Range=bytes 24117248-24641535/61431254
D/DownloadRunnable: thread 2 exited from main download loop
    The mission has been paused. Passing.
W/DownloadMission: thread alive: DownloadMission[2] Immer noch Nichts zu verbergen_.mp4
D/MissionAdapter: checkMasterButtonsVisibility() running=true paused=false
I/ViewRootImpl: ViewRoot's Touch Event : Touch Down
D/DownloadRunnable: 0:acquired block at position=42 done=0
D/DownloadRunnable: 2:acquired block at position=45 done=0
D/DownloadRunnable: 1:acquired block at position=46 done=0
D/MissionAdapter: checkMasterButtonsVisibility() running=true paused=false
I/ViewRootImpl: ViewRoot's Touch Event : Touch UP
D/DownloadMission: 2:[request]  Range=bytes=23592960-24117247
    2:[response] Code=206
    2:[response] Content-Length=524288
    2:[response] Content-Range=bytes 23592960-24117247/61431254
D/DownloadMission: 1:[request]  Range=bytes=24117248-24641535
    1:[response] Code=206
    1:[response] Content-Length=524288
    1:[response] Content-Range=bytes 24117248-24641535/61431254
D/DownloadMission: 0:[request]  Range=bytes=22020096-22544383
    0:[response] Code=206
    0:[response] Content-Length=524288
    0:[response] Content-Range=bytes 22020096-22544383/61431254
V/AudioSystem: ioConfigChanged() event 0, ioHandle 3
    ioConfigChanged() opening already existing output! 3
V/AudioSystem: ioConfigChanged() event 0, ioHandle 2
    ioConfigChanged() opening already existing output! 2
I/ViewRootImpl: ViewRoot's Touch Event : Touch Down
I/ViewRootImpl: ViewRoot's Touch Event : Touch UP
I/ActivityManager: Timeline: Activity_launch_request id:org.schabi.newpipe.debug time:141260754

I can pause and resume the download and it will continue where it stopped, so It's not a huge issue but it 'll be annoying. I'll investigate that some more.

@mqus
Copy link
Contributor Author

mqus commented Dec 11, 2019

I tried watching with wireshark and only found one mirror (selfhost.de) having an unspecified TLS error while connecting... I don't know how to fix this so i'll let it be for now.

@yausername
Copy link
Contributor

After a reread I understand that peertube won't work on kitkat unless the instances update their ssl_ciphers.

@mqus
Copy link
Contributor Author

mqus commented Dec 11, 2019

I tried your delegate suggestion (I read about it myself and then later forgot ;) ) and it doesn't seem to do anything for the second problem either. I guess that it fixed a problem in a previous version of okhttp but isn't neccessary now that you have to supply a TrustManager when setting a SSLSocketFactory... . I think the second problem arises from Kitkat having a pretty old root cert store while a few sites depend on a new one... but I'm not entirely sure.

@mqus
Copy link
Contributor Author

mqus commented Dec 11, 2019

Interestingly, the peertube docker config contains a traefik instance which would work with kitkat: https://github.com/Chocobozzz/PeerTube/blob/develop/support/docker/production/config/traefik.toml#L27
It enables TLS_RSA_WITH_AES_256_CBC_SHA(0x35) which ssllabs considers weak, but not insecure. It's also in the above list of Ciphers which okhttp will try.

@mqus
Copy link
Contributor Author

mqus commented Dec 11, 2019

For the meantime, I tried another instance (tube.tchncs.de) and tested downloading and playing.

While browsing works, downloading obviously only works for videos from instances with similar TLS settings.
But: playing videos doesn't work at all, it throws a wall of the following repeating logstream with errors (https://pastebin.com/c901HP13) while constantly showing the toaster message "Recovering from player error". This is similar when viewing videos of other instance. I think this isn't a case of TLS issues (?)

For instances with "wrong" tls settings, that doesn't happen and it will instead repeat the message W/SimpleExoPlayer: Player is accessed on the wrong thread. See https://exoplayer.dev/issues/player-accessed-on-wrong-thread slowly. Downloading from those instances will repeatedly throw SSLHandShakeErrors (or something similar) as expected.

@TobiGr TobiGr added this to the 0.18.1 milestone Dec 12, 2019
@mqus
Copy link
Contributor Author

mqus commented Dec 13, 2019

With help from @Chocobozzz, framatube.org is working flawlessly on Android 4.4 and the upstream nginx template was altered (Chocobozzz/PeerTube#2330)!

I checked video playback again and found that this is mostly an emulator issue. On the x86 Kitkat emulator without play services, video playback was not possible. On my personal phone it works just fine though.

As this thread has gotten a bit long and unwieldy, I'll summarize the past and present issues with newpipe on Kitkat:

MediaCCC:

  • Browse: fixed
  • Playback: fixed
    • Doesn't work in my Kitkat emulator(x86, no play services), but on my device.
  • Download: mostly fixed
    • still minor issues with some mirrors, I don't know how to fix them or if they appear on other Android versions, works otherwise

Peertube instances with matching ciphersuites (set server-side):

  • Browse: fixed
  • Playback: fixed
    • Doesn't work in my Kitkat emulator(x86, no play services), but on my device.
  • Download: fixed
  • This now includes framatube.org (Thanks, @framasoft !)

Peertube instances without matching ciphersuites (set server-side):

  • Browse: doesn't work
    • works via other instances, but it's not easy to figure out from which instance a video/channel is hosted
  • Playback: doesn't work at all
  • Download: doesn't work at all

This is now in a state I would consider finished. The last big group has to be solved by writing kind emails to the admins of affected instances and maybe tuning the Newpipe UI for Peertube a bit by displaying instences for remote videos. If there are no other opinions for the commented-out code in the reviewed section, I'll clean it up and this is ready to get merged.

@yausername
Copy link
Contributor

Great work @mqus!
@TobiGr Can we include this in the latest release? I believe you would have to merge this into release_0.18.0 branch.

Copy link
Contributor

@TobiGr TobiGr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice, this is on my list for 0.18.0.
@mqus Can you please rebase on the current dev. @yausername I'll rebase the release branch after merging.

@TobiGr TobiGr removed this from the 0.18.1 milestone Dec 13, 2019
mqus added 3 commits December 13, 2019 21:42
This enables modern TLS versions in the collection browser, the Downloader and the Player.
This is neccessary because media.ccc.de rejects all older TLS connection attempts, see issue #2777.
@mqus
Copy link
Contributor Author

mqus commented Dec 13, 2019

Rebase&fixing is done. Github still wants me to adress your concerns but wont let me resolve them...

EDIT: just to be clear though, I've applied every change you requested. I don't think that an error screen on that position is good (when starting the app and when initializing the DownloaderImpl). The Connection errors shown currently don't say much but I think that is good enough for now and would require more extensive changes.

Copy link
Contributor

@TobiGr TobiGr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mqus Thanks.

I checked video playback again and found that this is mostly an emulator issue. On the x86 Kitkat emulator without play services, video playback was not possible. On my personal phone it works just fine though.

I've tested this with two Android KitKat emulators and could not watch PeerTube videos for some reason, too (tested with framatube and video.blender.org). But accessing the video info and performing searches works fine. I have the strong feeling, that the emulators are far from being perfect. They are way too buggy. Unfortunately, I do not own a phone running KitKat. I'll trust you and we'll see, whether playback works fine on other devices, too.

I don't think that an error screen on that position is good

I agree.

Github still wants me to address your concerns but wont let me resolve them...

Yep, that's my job :)

@TobiGr TobiGr merged commit 755dff3 into TeamNewPipe:dev Dec 14, 2019
This was referenced Dec 15, 2019
@mqus
Copy link
Contributor Author

mqus commented Dec 15, 2019

Thank you @ all! If there are any connection errors with kitkat devices in the future or any other stuff I should reproduce with my phone, just ping me and I'll see what I can do. I'll also write some more E-Mails to instance admins but I probably wont cover them all.

@TobiGr
Copy link
Contributor

TobiGr commented Dec 15, 2019

@mqus thank you for the effort! I'll gladly come back to your offer ❤️

@danielzgtg danielzgtg mentioned this pull request Dec 28, 2019
3 tasks
@mqus mqus mentioned this pull request Jun 26, 2020
5 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Issue is related to a bug device/software specific Issues that only happen on some devices or with some specific hardware/software peertube Service, https://joinpeertube.org/ soundcloud Service, https://soundcloud.com/
Projects
None yet
Development

Successfully merging this pull request may close these issues.

mediaCCC network error
5 participants