Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

VideoFeedUpdate hook not working #18

Open
jeryini opened this issue Sep 22, 2020 · 20 comments · Fixed by #26
Open

VideoFeedUpdate hook not working #18

jeryini opened this issue Sep 22, 2020 · 20 comments · Fixed by #26
Labels
bug Something isn't working enhancement New feature or request
Milestone

Comments

@jeryini
Copy link

jeryini commented Sep 22, 2020

Hello!

I'm trying to use hooks on FPVWidgetState, specifically data class VideoFeedUpdate(val videoBuffer: ByteArray?, val size: Int) : FPVWidgetState() to get video frame each time it's updated. I'm using the following example in Java to subscribe to updates:

fpvWidget.getWidgetStateUpdate()
            .observeOn(AndroidSchedulers.mainThread())
            .subscribe(fpvWidgetState -> {
                if (fpvWidgetState instanceof FPVWidget.FPVWidgetState.ProductConnected) {
                    FPVWidget.FPVWidgetState.ProductConnected productConnected = (FPVWidget.FPVWidgetState.ProductConnected) fpvWidgetState;
                    Log.d(TAG, "Product connected: " + productConnected.isConnected());
                }
                if (fpvWidgetState instanceof FPVWidget.FPVWidgetState.VideoFeedUpdate) {
                    FPVWidget.FPVWidgetState.VideoFeedUpdate videoFeedUpdate = (FPVWidget.FPVWidgetState.VideoFeedUpdate) fpvWidgetState;
                    Log.d(TAG, "Feed size: " + videoFeedUpdate.getSize());
                    Log.d(TAG, "Buffer: " + Arrays.toString(videoFeedUpdate.getVideoBuffer()));
                }
            });

Unfortunately I don't see anything in the logs regarding outputted feed size and buffer. I've also tried ProductConnected state and that actually works fine, I can see the output in the log. Do I have to set up something else on the codecmanager or FPVWidget level to get the back the update for the video feed?

@alainpimentel alainpimentel added the bug Something isn't working label Sep 22, 2020
@alainpimentel alainpimentel added this to the v0.5.0 milestone Sep 22, 2020
@dji-lisafedane
Copy link

@jeryini - thanks for reporting on this bug and the good news is that there'll be a fix for this in the next beta release

@jeryini
Copy link
Author

jeryini commented Sep 23, 2020

Hello @dji-lisafedane !

Nice to hear that! Can't wait for it.

@alainpimentel
Copy link
Contributor

@jeryini btw thanks for trying and reporting this!

@jeryini
Copy link
Author

jeryini commented Oct 12, 2020

Hello @alainpimentel !

Do you maybe have any ETA when this will be fixed. Many thanks!

Cheers, Jernej

@alainpimentel
Copy link
Contributor

Hi @jeryini! We have a fix that will come out with the new MSDK 4.14. Keep an eye on https://developer.dji.com/user/roadmap/#msdk for more updates on the release date. Our next beta release will be shortly after 4.14 is out.

@alainpimentel alainpimentel linked a pull request Dec 22, 2020 that will close this issue
@jeryini
Copy link
Author

jeryini commented Dec 22, 2020

Hello @alainpimentel @siddutgikar !

First congratulations on the new awesome release with so many new features. Though I'm still having issues with getting VideoFeedUpdated model state updates.

My configuration:

  • Android Mobile SDK 4.14-trial1
  • Mobile-UXSDK-Beta-Android v0.5.0
  • DJI Mavic 2 Zoom, firmware version: 01.00.0670

I'm using the following code:

fpvWidget.getWidgetStateUpdate()
                        .observeOn(AndroidSchedulers.mainThread())
                        .subscribe(fpvWidgetState -> {
                            Log.d(TAG, fpvWidgetState.toString());
                            if (fpvWidgetState instanceof FPVWidget.ModelState.VideoFeedUpdated) {
                                FPVWidget.ModelState.VideoFeedUpdated videoFeedUpdated = (FPVWidget.ModelState.VideoFeedUpdated) fpvWidgetState;
                                Log.d(TAG, "Feed size: " + videoFeedUpdated.getSize());
                                Log.d(TAG, "Buffer: " + Arrays.toString(videoFeedUpdated.getVideoBuffer()));
                            }
                        });

My logs show the following:

2020-12-22 13:19:16.039 26881-26881/co.dronecontrol.app D/GroundControlStationActivity: VideoFeedSourceUpdated(videoFeed=UNKNOWN)
2020-12-22 13:19:16.039 26881-26881/co.dronecontrol.app D/GroundControlStationActivity: VideoFeedSourceUpdated(videoFeed=CAMERA)
2020-12-22 13:19:16.039 26881-26881/co.dronecontrol.app D/GroundControlStationActivity: OrientationUpdated(orientation=UNKNOWN)
2020-12-22 13:19:16.039 26881-26881/co.dronecontrol.app D/GroundControlStationActivity: CameraNameUpdated(cameraName=)
2020-12-22 13:19:16.040 26881-26881/co.dronecontrol.app I/chatty: uid=10376(co.dronecontrol.app) identical 2 lines
2020-12-22 13:19:16.040 26881-26881/co.dronecontrol.app D/GroundControlStationActivity: CameraNameUpdated(cameraName=)
2020-12-22 13:19:16.040 26881-26881/co.dronecontrol.app D/GroundControlStationActivity: CameraNameUpdated(cameraName=Mavic 2 Zoom Camera)
2020-12-22 13:19:16.040 26881-26881/co.dronecontrol.app I/chatty: uid=10376(co.dronecontrol.app) identical 1 line
2020-12-22 13:19:16.040 26881-26881/co.dronecontrol.app D/GroundControlStationActivity: CameraNameUpdated(cameraName=Mavic 2 Zoom Camera)
2020-12-22 13:19:16.040 26881-26881/co.dronecontrol.app D/GroundControlStationActivity: CameraSideUpdated(cameraSide=Unknown)
2020-12-22 13:19:16.040 26881-26881/co.dronecontrol.app I/chatty: uid=10376(co.dronecontrol.app) identical 5 lines
2020-12-22 13:19:16.040 26881-26881/co.dronecontrol.app D/GroundControlStationActivity: CameraSideUpdated(cameraSide=Unknown)
2020-12-22 13:19:16.040 26881-26881/co.dronecontrol.app D/GroundControlStationActivity: ProductConnected(isConnected=false)
2020-12-22 13:19:16.040 26881-26881/co.dronecontrol.app D/GroundControlStationActivity: ProductConnected(isConnected=true)

As you can see almost all model state events do show up, except for the VideoFeedUpdated state. Is there perhaps some additional configuration needed?

@alainpimentel alainpimentel reopened this Dec 22, 2020
@alainpimentel
Copy link
Contributor

@jeryini great to see you are already testing the new release! Good find. Something is still broken so we'll be taking a look.

@jeryini
Copy link
Author

jeryini commented Dec 30, 2020

Hello @alainpimentel !

Have you maybe pinpointed the issue? Is this something that can be fixed in this library, or does it need another update at the MSDK level?

@snmillar
Copy link
Contributor

Hello @jeryini

I've been looking into the issue and I found that the raw video feed is not available for Mavic 2 Zoom because it needs lens distortion calibration, so the transcoded video feed should be used instead.

Would the data from the transcoded video feed work for your purposes? I can add another hook that gets this data.

Or if you would like to get the transcoded video feed yourself without waiting for an update to this library, you can use the following method:

VideoFeeder.getInstance().provideTranscodedVideoFeed().addVideoDataListener(VideoDataListener listener)

@jeryini
Copy link
Author

jeryini commented Dec 31, 2020

Hello @snmillar !

This might work. Currently I'm using YuvData callback:

codecManager.enabledYuvData(true);
codecManager.setYuvDataCallback

I'm curious, what is actually the difference between subscribing to video via this transcoded video feed vs. using YUV data callback? Is there a difference in video quality? Also I searched inside this library and noticed you don't use YUV data callback to get the video feed, right?

In the end I want to use subscribe to your video feed mechanism because I'm currently using my own version to parse the generated YUV data inside onYuvDataReceived method to get the raw video feed. And that implementation is working fine, but I think it has a bit worse performance then what I see in the official DJI GO app.

@alainpimentel
Copy link
Contributor

@jeryini we are looking into the difference "between subscribing to video via this transcoded video feed vs. using YUV data callback". Will keep you posted when we have a good answer.
In the meantime, hope the MSDK API helps you move forward. We will be adding this as a hook in the next release.

@alainpimentel alainpimentel modified the milestones: v0.5.0, 0.6.0 Jan 5, 2021
@alainpimentel alainpimentel added the enhancement New feature or request label Jan 5, 2021
@hummelslei
Copy link

As I know, the onYuvDataReceived will give the YUV format data and the provideTranscodedVideoFeed provide the H264 data. Also, the provideTranscodedVideoFeed channel has method setTranscodingDataRate to change the rate.

@jeryini
Copy link
Author

jeryini commented Jan 6, 2021

Hello @alainpimentel !

Actually having YUV data available via subscription would be nice! Though based on my experience, if you enable YUV data, for some reason main FPV TextureView stops displaying camera image. Not sure if you can get around this.

@alainpimentel
Copy link
Contributor

We will need to explore why enabling YUV data causes issues tho, do you see that with other drones other than Mavic 2 Zoom? If you have any available.

@jeryini
Copy link
Author

jeryini commented Jan 11, 2021

Yes I tried with Mavic Mini and it's the same. If I remember correctly from one of the Github issue threads in Mobile SDK project, it's due to DJICodecManager unable to render to SurfaceView at the same time while processing YUV data. I actually tried to create two instances of DJICodecManager, enabling YUV on only one, but the result was the same.

@alainpimentel alainpimentel modified the milestones: 0.6.0, v0.5.1 Jan 14, 2021
@natxopedreira
Copy link

Hello

I want to get video frames from a fpvwidget, im trying to suscribe using getWidgetStateUpdate() but is not showing as a method of fpvwidget..... what im missing?????

Also is not clear to me if is possible or not to get that video frame as according to this thread is this hook not working? is there any workarround¿

Thanks and sorry for the newbie question

@jeryini
Copy link
Author

jeryini commented Jan 25, 2021

Hello @alainpimentel, @snmillar !

Do you have any progress with exposing YUV data? It would help me a lot and probably also other users. I'm having a hard time getting YUV data while DJICodecManager would still be able to render video to SurfaceView, I think it's mutually exclusive, i.e. you either get raw YUV data or you render it to TextureView. I even checked and tried this example (DJI-Mobile-SDK-Tutorials/Android-VideoStreamDecodingSample#29 (comment)), but came across another issue. Basically I tried to use that example to pass the raw byte buffer that I get via your VideoFeedUpdate hook and then pass it to DJIVideoStreamDecoder:

DJIVideoStreamDecoder.getInstance().changeSurface(null);
DJIVideoStreamDecoder.getInstance().setYuvDataListener(djiDroneData);
fpvWidget.getWidgetStateUpdate()
    .observeOn(AndroidSchedulers.mainThread())
    .subscribe(fpvWidgetState -> {
        if (fpvWidgetState instanceof FPVWidget.ModelState.VideoFeedUpdated) {
            FPVWidget.ModelState.VideoFeedUpdated videoFeedUpdated = (FPVWidget.ModelState.VideoFeedUpdated) fpvWidgetState;
            Log.d(TAG, "Feed size: " + videoFeedUpdated.getSize());
            DJIVideoStreamDecoder.getInstance().parse(videoFeedUpdated.getVideoBuffer(), videoFeedUpdated.getSize());
        }
    });

But as mentioned in that issue comment, I just get blank screen now, no video being rendered to textureview_fpv.

@jeryini
Copy link
Author

jeryini commented Feb 26, 2021

Hello @alainpimentel @snmillar !

I've just seen new release for Beta 5.1, great job! I've quickly scrolled over the codebase for FPVWidget.kt but don't see any option to subscribe to YUV data. I guess that feature didn't make it to the latest release? Are you maybe still planning to support it in the later versions? This would help us third party developers a lot to be able to just seamlessly subscribe to the stream like I can do now for VideoFeedUpdated.

@mayurpatil307
Copy link

mayurpatil307 commented Apr 12, 2021

Hi I am facing same issue,
I'm trying to use hooks on FPVWidgetState,

fpvWidget.getWidgetStateUpdate()
            .observeOn(AndroidSchedulers.mainThread())
            .subscribe(fpvWidgetState -> {
                if (fpvWidgetState instanceof FPVWidget.FPVWidgetState.ProductConnected) {
                    FPVWidget.FPVWidgetState.ProductConnected productConnected = (FPVWidget.FPVWidgetState.ProductConnected) fpvWidgetState;
                    Log.d(TAG, "Product connected: " + productConnected.isConnected());
                }
                if (fpvWidgetState instanceof FPVWidget.FPVWidgetState.VideoFeedUpdate) {
                    FPVWidget.FPVWidgetState.VideoFeedUpdate videoFeedUpdate = (FPVWidget.FPVWidgetState.VideoFeedUpdate) fpvWidgetState;
                    Log.d(TAG, "Feed size: " + videoFeedUpdate.getSize());
                    Log.d(TAG, "Buffer: " + Arrays.toString(videoFeedUpdate.getVideoBuffer()));
                }
            });

But, the logs are not getting executed.

I tried:

 fpvWidget.setCodecManagerCallback(new FPVWidget.CodecManagerCallback() {
            @Override
            public void onCodecManagerChanged(@Nullable DJICodecManager codecManager) {
                Log.d(TAG, "Codec manager changed");

                codecManager = new DJICodecManager(activity, null, 1280, 720, UsbAccessoryService.VideoStreamSource.Camera);
                codecManager.enabledYuvData(true);

                Log.d(TAG, "YUV data enabled");

                codecManager.setYuvDataCallback(new DJICodecManager.YuvDataCallback() {
                    @Override
                    public void onYuvDataReceived(MediaFormat mediaFormat, ByteBuffer yuvFrame, int dataSize, int width, int height) {
                        Log.d(TAG, "New YUV data received");
                    }
                });
            }
        });

I observed following:
The setYuvDataCallback is not called continuously while we are in FPVWidget activity.
Its called continuously when we go back to previous activity or any other activity from FPVWidget's activity.

@jeryini
Copy link
Author

jeryini commented May 1, 2021

Hello @mayurpatil307 !

Are you maybe using Mavic 2 (Pro/Zoom)? There's an issue with those drones, you need to enable some other stuff via SDK calls, see above posts for solution.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working enhancement New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

7 participants