Rotation issue

[emakovsky]

1. record a video in portrait using native android camcorder (nexus 4)
2. play it using native player. Rotation is correct
3. play it using demo app (SimplePlayerActivity). Video is rotated.

Example of the test video:
https://drive.google.com/file/d/0B9_5dr6in8cWcmlGZlJrSlhHS1U/view?usp=sharing

Metadata of the mp4 contains a property:
Rotation : 90°
but it's ignored by the exo player.

[ojw28]

Hm. This is pretty easy to fix on Lollypop. It actually already rotates the video on Lollypop, the only thing that's still wrong is that the video dimensions aren't also swapped (for 90 and 270 degree).

Pre-Lollypop any fix is going to be more difficult, because MediaExtractor doesn't parse out the video rotation.

[emakovsky]

So let's mark the issue as a bug. Because the behavior is not as expected and it is possible to fix it somehow.

[ryango]

in the mean time you can use mediametadataretriever to get the rotation and a textureview and apply a transform to the texture pretty easily (thanks ojw for the hint)

[abhishekgupta92]

@ojw28 Is it possible to do so by parsing rotation using MediaMetadataRetriever? Would love to stick to SurfaceView as the only advantage that we seem to get from using TextureView would be the right rotation.

[IanDBird]

We're also having issues with this lack of functionality.

[andrejvanderzee]

Same problem here.

[andrejvanderzee]

Any plans on fixing this?

[ryango]

I would argue this is a MediaCodec/libstagefright bug not an ExoPlayer bug as MediaCodec is what's doing the rendering. ExoPlayer is just managing the samples and timing. Also this is evident in the Lollipop libstagefright fix in which it respects the rotation (which screws things up for my use and I had to add this to counteract it)

            MediaFormat format = extractor.getTrackFormat(trackIndex);
            if (format.containsKey("rotation-degrees")) {
                // Decoded video is rotated automatically in Android 5.0 lollipop.
                // refer: https://android.googlesource.com/platform/frameworks/av/+blame/lollipop-release/media/libstagefright/Utils.cpp
                format.setInteger("rotation-degrees", 0);
            }

[IanDBird]

Has anyone seen whether it's still an issue if you're using the MP4 support currently in development? It's currently available on the dev branch here (https://github.com/google/ExoPlayer/blob/dev/demo/src/main/java/com/google/android/exoplayer/demo/player/Mp4RendererBuilder.java#L35). Arguably, that's one of the benefits of adding MP4 support into ExoPlayer rather than relying on the MediaCodec/libstagefright support.

[ojw28]

Pulling the extractor up to ExoPlayer wont fix the issue, because the issue is in the decoder (not being pulled up) not the extractor (being pulled up).

[andrejvanderzee]

So what would be the recommended way to solve this? The TextureView instead of SurfaceView approach like Inikkila kindly posted?

[Ood-Tsen]

Hi @IanDBird @ryango
You may check my quick update parse mp4 rotation degree

"rotation-degrees" is defined in the ACodec, but it's hidden.
we cannot guarantee it still available in the future.

"rotation-degrees" is available in Android 5.0, not for Android KK.

[Ood-Tsen]

This is another update for rotate by demo app

[ojw28]

The behavior has been fixed as far as is possible in the change above.

Background: The underlying platform changed in Lollypop to start automatically applying rotation when decoding video into a surface. Prior to Lollypop the rotation was not applied. Since it's not possible to perform the rotation at the application level when using SurfaceView, the Lollypop+ behavior is the correct long term solution.

Solution: After the change above, on Lollypop+ the rotation will always be applied correctly. On earlier versions of Android the rotation will not be applied, however the player will now report the unapplied rotation through the MediaCodecVideoTrackRenderer listener:

ExoPlayer/library/src/main/java/com/google/android/exoplayer/MediaCodecVideoTrackRenderer.java

Line 74 in d3995ea

void onVideoSizeChanged(int width, int height, int unappliedRotationDegrees,

This allows an application using TextureView to easily apply the rotation by making the appropriate call to TextureView.setTransform. Note that on Lollypop+ unappliedRotationDegrees will always be equal to 0.

[Ood-Tsen]

Thanks @ojw28 , you complete the solution.

[Shusshu]

Is there a snapshot/dev repo to test this quickly in our app? or when will it get released ?

[ojw28]

You can get the latest source from the dev branch and build it yourself (or include the source directly in your project). We don't provide jar/aar binaries prior to release.

Note that the APIs (in general - nothing specific to rotation) are a little unstable on the dev brach due to the arrival of multi-track support, so if you go down this route please be aware that they may change again before the next release.

[Shusshu]

From dev I had other issues so finally I cherry-picked the changes + had to do extra changes that I took from dev to make it work:
If anyone needs it: https://github.com/Shusshu/ExoPlayer/tree/142-rotation

[dnutcracker]

@ojw28 Just a small remark - the automatic applying of video orientation while decoding to a surface is true only for the default hardware decoder - when using software decoding for example (using OMX.google.h264.decoder) it does not apply rotation even on Lollipop+.
So people using custom decoders (software or other) might still need the orientation parameter even on Lollipop+ (So maybe the orientation param should always be included and ignored on Lollipop+?)

[ojw28]

Urgh, this is a bit of a mess at the platform level. Application of rotation is inconsistent between both API levels (pre-L v.s. L+) and (in L at least) between decoders at a fixed API level. It appears that on M+ OMX.google.h264.decoder does actually apply the rotation, but you're correct in noting that this does not happen when running L.

Unfortunately, as of now, there's no easy way to determine whether a decoder has actually applied the rotation. I'm not sure how consistent the application of rotation is between decoders in M. I'll be pushing for more consistent application of rotation (if this hasn't happened already) and/or the ability to determine whether the rotation was applied in the next platform release.

For ExoPlayer, always propagating the rotation value doesn't seem like the right thing to do. That just moves the problem of working out what to do from the library to every individual application developer, who will then each have to figure out what different decoders do on different API levels. Instead, I think we probably want to work out what the current behaviors are and hardcode these inside the library, so that the rotation delivered to the application always ends up being correct (i.e. 0 when using a decoder that applied the rotation, non-zero otherwise). Hopefully we can also get consistent behavior across all video decoders from some version of the platform and above, since this makes the developer's life significantly easier.

Re-opening for tracking. First step is to try and figure out exactly what behavior we should be expecting across the various API levels / decoder implementations.

[dnutcracker]

Sounds good. Good luck :)

On Tue, Sep 1, 2015 at 3:46 PM, ojw28 notifications@github.com wrote:

Urgh, this is a bit of a mess at the platform level. Application of
rotation is inconsistent between both API levels (pre-L v.s. L+) and (in L
at least) between decoders at a fixed API level. It appears that on M+
OMX.google.h264.decoder does actually apply the rotation, but you're
correct in noting that this does not happen when running L.

Unfortunately, as of now, there's no easy way to determine whether a
decoder has actually applied the rotation. I'm not sure how consistent the
application of rotation is between decoders in M. I'll be pushing for more
consistent application of rotation (if this hasn't happened already) and/or
the ability to determine whether the rotation was applied in the next
platform release.

For ExoPlayer, always propagating the rotation value doesn't seem like the
right thing to do. That just moves the problem of working out what to do
from the library to every individual application developer, who will then
each have to figure out what different decoders do on different API levels.
Instead, I think we probably want to work out what the current behaviors
are and hardcode these inside the library, so that the rotation delivered
to the application always ends up being correct (i.e. 0 when using a
decoder that applied the rotation, non-zero otherwise). Hopefully we can
also get consistent behavior across all video decoders from some version of
the platform and above, since this makes the developer's life significantly
easier.

Re-opening for tracking. First step is to try and figure out exactly what
behavior we should be expecting across the various API levels / decoder
implementations.

—
Reply to this email directly or view it on GitHub
#91 (comment).

[jacekTymicki]

Do we have any timeline for fixing this issue?

[ojw28]

For most cases it is fixed already (the rotation will be applied, or the rotation that needs to be applied will be propagated as unappliedRotationDegrees). So it's unclear exactly what you mean by this. Unless you've modified the player to use a software decoder then I think behavior should be correct already.

[BackPackerDz]

I have the same issue, please any solution ?

[fasteque]

I see the same issue with the latest ExoPlayer version, r2.0.0: any solution while using the SimpleExoPlayerView?

[langrenfengzi]

So there was no ways to rotate a SurfaceView?

[rakshith-ravi]

Any update on this? I'm working on a commercial app and we kinda need this fixed ASAP. Would we see a fix anytime soon or would it be better to switch to a different solution for video playback?

[gabbimark]

same issue here. my videos playback correcly using any web or android video player that i try. But when playing back using exoplayer the videos are flipped 90 degrees on some phones. Will this be fixed soon?

[Ranjith121]

Same issue. Is any update on pre-L rotation?

[rakshith-ravi]

Considering I didn't see a fix, I moved to media player....The default solution for video playback. Would like to see a fix though

[Ranjith121]

ya am to moved to media player. thanks

[Satsrag]

        int viewWidth = textureView.getWidth();
        int viewHeight = textureView.getHeight();
        Scale bestScale = getBestScale(new Size(width, height), new Size(viewWidth, viewHeight));
        float pivotX = viewWidth / 2f;
        float pivotY = viewHeight / 2f;
        Matrix transform = new Matrix();
        transform.postScale(bestScale.isW ? 1 : bestScale.scale, bestScale.isW ? bestScale.scale : 1, pivotX, pivotY);
        transform.postRotate(unAppliedRotationDegrees, pivotX, pivotY);
        if (unAppliedRotationDegrees == 90 || unAppliedRotationDegrees == 270) {
            final float scaledVideoWidth = bestScale.isW ? viewWidth : (viewWidth * bestScale.scale);
            final float scaledVideoHeight = bestScale.isW ? viewWidth * bestScale.scale : viewWidth;
            Scale scale = getBestScale(new Size((int) scaledVideoHeight, (int) scaledVideoWidth), new Size(viewWidth, viewHeight));
            transform.postScale(scale.scale, scale.scale, pivotX, pivotY);
        }
        if (textureView instanceof TextureView) {
            ((TextureView) textureView).setTransform(transform);
        }

[Kovartthan]

@Saqrag your code makes correct rotation for the video in pre- lollipop versions ?

[crossle]

plz support display matrix rotation, like

    for (i = 0; i < 3; i++) {
        display_matrix[i][0] = avio_rb32(pb);   // 16.16 fixed point
        display_matrix[i][1] = avio_rb32(pb);   // 16.16 fixed point
        display_matrix[i][2] = avio_rb32(pb);   //  2.30 fixed point
    }

[ArikYa]

Why doesn't SimpleExoPlayerView handle the rotation on older device? It already listens to the onVideoSizeChanged() and can handle it. As it stands, because it tries to handle the aspect ratio by itself, it prevents the application from doing it correctly

[ojw28]

We could (and probably should) do this if the SimpleExoPlayerView instance is configured to use TextureView. For SurfaceView, which is the default it's somewhat less clear what to do. We could just log a warning, or we could dynamically switch to TextureView for that specific case.

[ojw28]

@botaydotcom - Might be fun to take a look at.

[alexandreprado]

Hi guys!

Any update on this?

Thank you!

[botaydotcom]

We have updated SimpleExoPlayerView so that if it uses TextureView, it will apply the rotation automatically. This change will be pushed to dev-v2 branch soon.

[ojw28]

The change @botaydotcom mentions has now been pushed to dev-v2.

[shelarsuhas]

I have an application where I am using MediaRecorder to record a video and play it back using ExoPlayer. The video is recorded only in Portrait mode. I am setting the camera setRotation and MediaRecorder setOrientationHint appropriately, yet in some phones the video gets played in landscape mode.

I have checked the meta data of both the files where there is a rotation issue and where there is not and I can see that the rotation is indeed getting set to 270 degree as expected and I am still not getting the desired output.

The one on the left doesn't get played in correct orientation and the one on the right gets played. This is not specific to any particular API level.

[zagzy]

@botaydotcom great, that you have updated textureView.
Guys, what about rotation on SurfaceView? Any ideas?

[botaydotcom]

It's unfortunately not possible to do from application level (including from within ExoPlayer library) with SurfaceView. Please refer to @ojw28's comments in this thread.

If you use SurfaceView on L+, it should be applied automatically at platform level (note that there are a few issues with software decoders on L).
If you need to support SurfaceView on pre-L, one option is to detect if unAppliedRotationDegrees is not 0, and switch your app to use a TextureView dynamically, in which case ExoPlayer will handle the rotation for you automatically.