Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[DATA-851] Data CLI: retry query requests, logging, file naming #1642

Merged
merged 10 commits into from
Dec 6, 2022
Merged

[DATA-851] Data CLI: retry query requests, logging, file naming #1642

merged 10 commits into from
Dec 6, 2022

Conversation

agreenb
Copy link
Contributor

@agreenb agreenb commented Dec 6, 2022 •
edited
Loading

  • Add retry to data requests to handle expected intermittent "connection reset by peer" errors when the server connection is resetting
  • Prepend binary data/metadata filenames with a timestamp since more useful and can sort in browser
    • Note: plan to switch BE to support nanoseconds, but still possible for multiple collectors of the same type to capture at the same ns, so ensure uniqueness with file ID.
  • Add progress logging

The above addresses errors and feedback received on CLI data export usage.
Tested locally for both binary and tabular export.

@viambot viambot added the safe to test This pull request is marked safe to test from a trusted zone label Dec 6, 2022
@agreenb agreenb changed the title Data CLI improvements [DATA-851] Data CLI improvements Dec 6, 2022
@agreenb agreenb changed the title [DATA-851] Data CLI improvements [DATA-851] Data CLI: retry query requests, logging, file naming Dec 6, 2022
AaronCasas
AaronCasas approved these changes Dec 6, 2022
View reviewed changes
Copy link
Contributor

@AaronCasas AaronCasas left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me! Just one nitty suggestion on the time stuff + a question about the connection reset thing

cli/data.go Outdated
metadataDir = "metadata"
maxRetryCount = 5
logEveryN = 100
timeFormat = "2006-01-02T150405.0000Z"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: If this is a standard a standard time format (e.g. rfc3339) it would be helpful to include a comment indicating that or use a const exposed by aa time package to indicate that (e.g. time.RFC3339)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I originally did this to get nanoseconds (but we have file IDs to enforce uniqueness) and not use colons (since the mac browser shows them as "/"), but better to use the same format we're using when capturing data. Updated to RFC3339.

@AaronCasas
Copy link
Contributor

when the server connection is resetting

When/why does this happen? This gives me the impression we might be leaving orphaned connections

@viambot viambot added safe to test This pull request is marked safe to test from a trusted zone and removed safe to test This pull request is marked safe to test from a trusted zone labels Dec 6, 2022
@agreenb
Copy link
Contributor Author

agreenb commented Dec 6, 2022

when the server connection is resetting

When/why does this happen? This gives me the impression we might be leaving orphaned connections

This happened after running for ~20 mins when attempting to export 20k images, and @edaniels mentioned that it's highly likely to hit this error after so many consecutive requests.

Follow-up PR will be batching/parallelizing requests to make this faster.

@viambot viambot added safe to test This pull request is marked safe to test from a trusted zone and removed safe to test This pull request is marked safe to test from a trusted zone labels Dec 6, 2022
@viambot viambot added safe to test This pull request is marked safe to test from a trusted zone and removed safe to test This pull request is marked safe to test from a trusted zone labels Dec 6, 2022
@viambot viambot added safe to test This pull request is marked safe to test from a trusted zone and removed safe to test This pull request is marked safe to test from a trusted zone labels Dec 6, 2022
@github-actions
Copy link
Contributor

github-actions bot commented Dec 6, 2022

Code Coverage

Package Line Rate Delta Health
go.viam.com/rdk/components/arm 61% 0.00% ➖
go.viam.com/rdk/components/arm/universalrobots 13% 0.00% ❌
go.viam.com/rdk/components/arm/xarm 2% 0.00% ❌
go.viam.com/rdk/components/arm/yahboom 7% 0.00% ❌
go.viam.com/rdk/components/audioinput 55% -0.34% ➖
go.viam.com/rdk/components/base 67% 0.00% ➖
go.viam.com/rdk/components/base/agilex 62% 0.00% ➖
go.viam.com/rdk/components/base/boat 41% 0.00% ❌
go.viam.com/rdk/components/base/wheeled 76% 0.00% ✅
go.viam.com/rdk/components/board 68% 0.00% ➖
go.viam.com/rdk/components/board/arduino 10% 0.00% ❌
go.viam.com/rdk/components/board/commonsysfs 47% 0.00% ❌
go.viam.com/rdk/components/board/fake 38% 0.00% ❌
go.viam.com/rdk/components/board/numato 19% 0.00% ❌
go.viam.com/rdk/components/board/pi 50% 0.00% ➖
go.viam.com/rdk/components/camera 65% 0.00% ➖
go.viam.com/rdk/components/camera/fake 67% 0.00% ➖
go.viam.com/rdk/components/camera/ffmpeg 76% -1.43% ✅
go.viam.com/rdk/components/camera/transformpipeline 78% 0.00% ✅
go.viam.com/rdk/components/camera/videosource 55% 0.00% ➖
go.viam.com/rdk/components/encoder/fake 77% 0.00% ✅
go.viam.com/rdk/components/gantry 68% 0.00% ➖
go.viam.com/rdk/components/gantry/multiaxis 84% 0.00% ✅
go.viam.com/rdk/components/gantry/oneaxis 86% 0.00% ✅
go.viam.com/rdk/components/generic 83% 0.00% ✅
go.viam.com/rdk/components/gripper 80% 0.00% ✅
go.viam.com/rdk/components/input 87% 0.00% ✅
go.viam.com/rdk/components/input/gpio 83% 0.00% ✅
go.viam.com/rdk/components/motor 81% 0.00% ✅
go.viam.com/rdk/components/motor/dimensionengineering 63% 0.00% ➖
go.viam.com/rdk/components/motor/dmc4000 69% 0.00% ➖
go.viam.com/rdk/components/motor/fake 57% 0.00% ➖
go.viam.com/rdk/components/motor/gpio 64% 0.00% ➖
go.viam.com/rdk/components/motor/gpiostepper 57% 0.00% ➖
go.viam.com/rdk/components/motor/tmcstepper 62% 0.00% ➖
go.viam.com/rdk/components/movementsensor 75% 0.00% ✅
go.viam.com/rdk/components/movementsensor/cameramono 40% 0.00% ❌
go.viam.com/rdk/components/movementsensor/gpsnmea 37% 0.00% ❌
go.viam.com/rdk/components/movementsensor/gpsrtk 28% 0.00% ❌
go.viam.com/rdk/components/posetracker 86% 0.00% ✅
go.viam.com/rdk/components/sensor 86% 0.00% ✅
go.viam.com/rdk/components/sensor/ultrasonic 34% 0.00% ❌
go.viam.com/rdk/components/servo 78% 0.00% ✅
go.viam.com/rdk/components/servo/gpio 71% 0.00% ✅
go.viam.com/rdk/config 76% 0.00% ✅
go.viam.com/rdk/control 57% 0.00% ➖
go.viam.com/rdk/data 77% 0.00% ✅
go.viam.com/rdk/grpc 25% 0.00% ❌
go.viam.com/rdk/ml 67% 0.00% ➖
go.viam.com/rdk/ml/inference 70% 0.00% ✅
go.viam.com/rdk/motionplan 59% -11.51% ➖
go.viam.com/rdk/octree 93% 0.00% ✅
go.viam.com/rdk/operation 82% 0.00% ✅
go.viam.com/rdk/pointcloud 72% +0.52% ✅
go.viam.com/rdk/protoutils 62% 0.00% ➖
go.viam.com/rdk/referenceframe 69% 0.00% ➖
go.viam.com/rdk/registry 88% 0.00% ✅
go.viam.com/rdk/resource 83% 0.00% ✅
go.viam.com/rdk/rimage 78% 0.00% ✅
go.viam.com/rdk/rimage/depthadapter 94% 0.00% ✅
go.viam.com/rdk/rimage/transform 73% 0.00% ✅
go.viam.com/rdk/rimage/transform/cmd/extrinsic_calibration 67% 0.00% ➖
go.viam.com/rdk/robot 93% 0.00% ✅
go.viam.com/rdk/robot/client 79% 0.00% ✅
go.viam.com/rdk/robot/framesystem 68% 0.00% ➖
go.viam.com/rdk/robot/impl 81% 0.00% ✅
go.viam.com/rdk/robot/server 59% +0.93% ➖
go.viam.com/rdk/robot/web 60% 0.00% ➖
go.viam.com/rdk/robot/web/stream 87% 0.00% ✅
go.viam.com/rdk/services/armremotecontrol 71% 0.00% ✅
go.viam.com/rdk/services/armremotecontrol/builtin 25% 0.00% ❌
go.viam.com/rdk/services/baseremotecontrol 71% 0.00% ✅
go.viam.com/rdk/services/baseremotecontrol/builtin 66% 0.00% ➖
go.viam.com/rdk/services/datamanager 65% 0.00% ➖
go.viam.com/rdk/services/datamanager/builtin 80% 0.00% ✅
go.viam.com/rdk/services/datamanager/datacapture 21% 0.00% ❌
go.viam.com/rdk/services/datamanager/datasync 72% 0.00% ✅
go.viam.com/rdk/services/motion 63% 0.00% ➖
go.viam.com/rdk/services/motion/builtin 88% 0.00% ✅
go.viam.com/rdk/services/navigation 54% 0.00% ➖
go.viam.com/rdk/services/sensors 77% 0.00% ✅
go.viam.com/rdk/services/sensors/builtin 97% 0.00% ✅
go.viam.com/rdk/services/shell 14% 0.00% ❌
go.viam.com/rdk/services/slam 84% 0.00% ✅
go.viam.com/rdk/services/slam/builtin 72% 0.00% ✅
go.viam.com/rdk/services/vision 80% 0.00% ✅
go.viam.com/rdk/services/vision/builtin 74% 0.00% ✅
go.viam.com/rdk/spatialmath 85% 0.00% ✅
go.viam.com/rdk/subtype 96% 0.00% ✅
go.viam.com/rdk/utils 71% -0.19% ✅
go.viam.com/rdk/vision 26% 0.00% ❌
go.viam.com/rdk/vision/chess 80% 0.00% ✅
go.viam.com/rdk/vision/delaunay 87% 0.00% ✅
go.viam.com/rdk/vision/keypoints 92% 0.00% ✅
go.viam.com/rdk/vision/objectdetection 82% 0.00% ✅
go.viam.com/rdk/vision/odometry 60% 0.00% ➖
go.viam.com/rdk/vision/odometry/cmd 0% 0.00% ❌
go.viam.com/rdk/vision/segmentation 49% 0.00% ❌
go.viam.com/rdk/web/server 26% 0.00% ❌
Summary 65% (19580 / 29973) -0.74% ➖

@agreenb agreenb merged commit ec3ee49 into viamrobotics:main Dec 6, 2022
@bazile-clyde bazile-clyde mentioned this pull request Feb 7, 2023
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@AaronCasas AaronCasas AaronCasas approved these changes

Assignees
No one assigned
Labels
safe to test This pull request is marked safe to test from a trusted zone
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@agreenb @AaronCasas @viambot