Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add JPEG hint to app video streams #1635

Merged
merged 2 commits into from
Dec 2, 2022

Conversation

bhaney
Copy link
Member

@bhaney bhaney commented Dec 2, 2022

Currently, app gets back RawRGBALazy images by default from remote cameras.

This switches all camera streams to request JPEGLazy frames from the cameras if they are using a gRPC GetImage() method.

The alternative could be to change the camera/client.go default request to be empty ( rather than the current default to request RawRGBALazy) such that the server can provide its own default MIME type in the response.

@bhaney bhaney added the safe to test This pull request is marked safe to test from a trusted zone label Dec 2, 2022
@viambot viambot removed the safe to test This pull request is marked safe to test from a trusted zone label Dec 2, 2022
@bhaney bhaney added the safe to test This pull request is marked safe to test from a trusted zone label Dec 2, 2022
@bhaney bhaney requested a review from edaniels December 2, 2022 22:09
@bhaney
Copy link
Member Author

bhaney commented Dec 2, 2022

Going to test this now on my own computer using a remote camera on my desk

@github-actions
Copy link
Contributor

github-actions bot commented Dec 2, 2022

Code Coverage

Package Line Rate Delta Health
go.viam.com/rdk/components/arm 61% 0.00%
go.viam.com/rdk/components/arm/universalrobots 13% 0.00%
go.viam.com/rdk/components/arm/xarm 2% 0.00%
go.viam.com/rdk/components/arm/yahboom 7% 0.00%
go.viam.com/rdk/components/audioinput 55% 0.00%
go.viam.com/rdk/components/base 67% 0.00%
go.viam.com/rdk/components/base/agilex 62% 0.00%
go.viam.com/rdk/components/base/boat 41% 0.00%
go.viam.com/rdk/components/base/wheeled 76% 0.00%
go.viam.com/rdk/components/board 68% 0.00%
go.viam.com/rdk/components/board/arduino 10% 0.00%
go.viam.com/rdk/components/board/commonsysfs 47% 0.00%
go.viam.com/rdk/components/board/fake 38% 0.00%
go.viam.com/rdk/components/board/numato 19% 0.00%
go.viam.com/rdk/components/board/pi 50% 0.00%
go.viam.com/rdk/components/camera 65% -0.27%
go.viam.com/rdk/components/camera/fake 67% 0.00%
go.viam.com/rdk/components/camera/ffmpeg 76% -1.43%
go.viam.com/rdk/components/camera/transformpipeline 77% 0.00%
go.viam.com/rdk/components/camera/videosource 56% +0.51%
go.viam.com/rdk/components/encoder/fake 77% 0.00%
go.viam.com/rdk/components/gantry 68% 0.00%
go.viam.com/rdk/components/gantry/multiaxis 84% 0.00%
go.viam.com/rdk/components/gantry/oneaxis 86% 0.00%
go.viam.com/rdk/components/generic 83% 0.00%
go.viam.com/rdk/components/gripper 80% 0.00%
go.viam.com/rdk/components/input 87% 0.00%
go.viam.com/rdk/components/input/gpio 83% 0.00%
go.viam.com/rdk/components/motor 81% 0.00%
go.viam.com/rdk/components/motor/dimensionengineering 63% 0.00%
go.viam.com/rdk/components/motor/dmc4000 69% 0.00%
go.viam.com/rdk/components/motor/fake 57% 0.00%
go.viam.com/rdk/components/motor/gpio 64% +2.16%
go.viam.com/rdk/components/motor/gpiostepper 56% -0.59%
go.viam.com/rdk/components/motor/tmcstepper 62% 0.00%
go.viam.com/rdk/components/movementsensor 75% 0.00%
go.viam.com/rdk/components/movementsensor/cameramono 40% 0.00%
go.viam.com/rdk/components/movementsensor/gpsnmea 37% 0.00%
go.viam.com/rdk/components/movementsensor/gpsrtk 28% 0.00%
go.viam.com/rdk/components/posetracker 86% 0.00%
go.viam.com/rdk/components/sensor 86% 0.00%
go.viam.com/rdk/components/sensor/ultrasonic 34% 0.00%
go.viam.com/rdk/components/servo 78% 0.00%
go.viam.com/rdk/components/servo/gpio 71% 0.00%
go.viam.com/rdk/config 76% 0.00%
go.viam.com/rdk/control 57% 0.00%
go.viam.com/rdk/data 77% 0.00%
go.viam.com/rdk/grpc 25% 0.00%
go.viam.com/rdk/ml 67% 0.00%
go.viam.com/rdk/ml/inference 70% 0.00%
go.viam.com/rdk/motionplan 68% -0.10%
go.viam.com/rdk/octree 93% 0.00%
go.viam.com/rdk/operation 82% 0.00%
go.viam.com/rdk/pointcloud 71% 0.00%
go.viam.com/rdk/protoutils 62% 0.00%
go.viam.com/rdk/referenceframe 69% 0.00%
go.viam.com/rdk/registry 88% 0.00%
go.viam.com/rdk/resource 83% 0.00%
go.viam.com/rdk/rimage 78% 0.00%
go.viam.com/rdk/rimage/depthadapter 94% 0.00%
go.viam.com/rdk/rimage/transform 73% 0.00%
go.viam.com/rdk/rimage/transform/cmd/extrinsic_calibration 67% 0.00%
go.viam.com/rdk/robot 93% 0.00%
go.viam.com/rdk/robot/client 79% 0.00%
go.viam.com/rdk/robot/framesystem 68% 0.00%
go.viam.com/rdk/robot/impl 81% 0.00%
go.viam.com/rdk/robot/server 59% +0.93%
go.viam.com/rdk/robot/web 60% +0.07%
go.viam.com/rdk/robot/web/stream 87% 0.00%
go.viam.com/rdk/services/armremotecontrol 71% 0.00%
go.viam.com/rdk/services/armremotecontrol/builtin 25% 0.00%
go.viam.com/rdk/services/baseremotecontrol 71% 0.00%
go.viam.com/rdk/services/baseremotecontrol/builtin 66% 0.00%
go.viam.com/rdk/services/datamanager 65% 0.00%
go.viam.com/rdk/services/datamanager/builtin 80% 0.00%
go.viam.com/rdk/services/datamanager/datacapture 21% 0.00%
go.viam.com/rdk/services/datamanager/datasync 72% 0.00%
go.viam.com/rdk/services/motion 63% 0.00%
go.viam.com/rdk/services/motion/builtin 88% 0.00%
go.viam.com/rdk/services/navigation 54% 0.00%
go.viam.com/rdk/services/sensors 77% 0.00%
go.viam.com/rdk/services/sensors/builtin 97% 0.00%
go.viam.com/rdk/services/shell 14% 0.00%
go.viam.com/rdk/services/slam 84% 0.00%
go.viam.com/rdk/services/slam/builtin 72% 0.00%
go.viam.com/rdk/services/vision 80% 0.00%
go.viam.com/rdk/services/vision/builtin 74% 0.00%
go.viam.com/rdk/spatialmath 85% 0.00%
go.viam.com/rdk/subtype 96% 0.00%
go.viam.com/rdk/utils 71% 0.00%
go.viam.com/rdk/vision 26% 0.00%
go.viam.com/rdk/vision/chess 80% 0.00%
go.viam.com/rdk/vision/delaunay 87% 0.00%
go.viam.com/rdk/vision/keypoints 92% 0.00%
go.viam.com/rdk/vision/objectdetection 82% 0.00%
go.viam.com/rdk/vision/odometry 60% 0.00%
go.viam.com/rdk/vision/odometry/cmd 0% 0.00%
go.viam.com/rdk/vision/segmentation 49% 0.00%
go.viam.com/rdk/web/server 26% 0.00%
Summary 66% (19738 / 29961) +0.03%

@bhaney
Copy link
Member Author

bhaney commented Dec 2, 2022

Yea, it looks like it is JPEG now!

2022/12/02 17:20:22 camera::client::Read 85 ms
2022/12/02 17:20:22    rimage::DecodeImage::image/jpeg+lazy 0 ms
2022/12/02 17:20:22 rimage::DecodeImage::image/jpeg 8 ms
2022-12-02T17:20:22.103-0500    DEBUG   robot_server.webrtc     zap/options.go:212      finished client unary call      {"ch": 0, "system": "grpc", "span.kind": "client", "grpc.service": "viam.component.camera.v1.CameraService", "grpc.method": "GetImage", "grpc.code": "OK
", "grpc.time_ms": 98.962}

and it looks like it's up to around 10 fps too

Copy link
Contributor

@edaniels edaniels left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM if it works

@bhaney bhaney merged commit 3625762 into viamrobotics:main Dec 2, 2022
@bhaney bhaney deleted the jpeg-context-hint branch April 21, 2023 15:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
safe to test This pull request is marked safe to test from a trusted zone
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants