Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[RSDK-515] Modular Resources #1363

Merged
merged 75 commits into from
Dec 17, 2022
Merged

Conversation

Otterverse
Copy link
Member

@Otterverse Otterverse commented Sep 21, 2022

Lots of changes involved, so I'm not expecting any speedy reviews here. Also, I'd like at least one solid pass of reviews (especially about the higher level architecture) before starting to create proper tests for the new module methods as well.

As for where to look, two main places to start with: There is a new "module" section in the top level, which includes a doc.go file with a fair bit of explanation for how this is expected to work. https://github.com/Otterverse/rdk/blob/modular-resources/module/doc.go

The most obvious entrypoint though, is a through set of examples collected in https://github.com/Otterverse/rdk/tree/modular-resources/examples/customresources There is a README there with instructions for how to run things, and it's all tied up in a bow with makefile targets, so it should "just run."

Other points of interest include:

  • Modules section as top-level of config
  • Changing the concept of "model" from a single string, to a namespaced triplet. Ex: acme:demo:mygizmo
  • IsMoving() moved from Local[COMPONENT] to just [COMPONENT] since this must be checkable via grpc for modular stuff

Lots of other changes (obviously) as well. Please don't hesitate to ask me questions via Slack! This needs to all make sense to everyone, and be the right way forward.

Also, note that this depends on patches to both api and goutils. PRs are in place for those as well.

@CLAassistant
Copy link

CLAassistant commented Oct 3, 2022

CLA assistant check
All committers have signed the CLA.

Copy link
Contributor

@edaniels edaniels left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This LGTM. There are plenty of nits though in this review and one minor thing that I'd like addressed prior to merging. That being said, the overall production code is LGTM. Great work James.

components/servo/server.go Outdated Show resolved Hide resolved
@@ -21,7 +21,7 @@
{
"name": "base1",
"type": "base",
"model": "fake",
"model": "acme:test:model",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

assuming making these changes doesn't have us lose test coverage in other places.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not as far as I know. If find a case where you think it does, please let me know.

config/proto_conversions.go Outdated Show resolved Hide resolved
config/proto_conversions.go Outdated Show resolved Hide resolved
config/proto_conversions.go Outdated Show resolved Hide resolved
module/module_interceptors_test.go Outdated Show resolved Hide resolved
return err
web, err := r.webService()
if err == nil {
web.Stop()
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

so we Stop and Close web?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, Stop() stops just the network facing web service, preventing any further client connections, while leaving the module grpc running, in case teardown needs to communicate with modules. (E.g. a close on a base might want to send a Stop() to its motors.) The final close then shuts down everything.

robot/impl/local_robot.go Outdated Show resolved Hide resolved
}

switch req["command"] {
case "sleep":
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

neat idea

return err
}
if m.Name == "" {
return errors.New("model name field for resource missing")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

isn't it just name missing?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Until one of your previous comments today, I didn't realize we tried to use the json field names in errors. I was just using plain english and saying "model name". Reworded it to say "name field for model missing"

Copy link
Member

@cheukt cheukt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great!


// IsMoving queries of a component is in motion.
func (s *subtypeServer) IsMoving(ctx context.Context, req *pb.IsMovingRequest) (*pb.IsMovingResponse, error) {
base, err := s.getArm(req.GetName())
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: base -> arm


// IsMoving queries of a component is in motion.
func (s *subtypeServer) IsMoving(ctx context.Context, req *pb.IsMovingRequest) (*pb.IsMovingResponse, error) {
base, err := s.getGripper(req.GetName())
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: base -> gripper

@@ -63,6 +64,12 @@ func (c *Config) Ensure(fromCloud bool) error {
}
}

for idx := 0; idx < len(c.Modules); idx++ {
if err := c.Modules[idx].Validate(fmt.Sprintf("%s.%d", "modules", idx)); err != nil {
return err
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

since we allow partial start up now, should we error out just because a module config is wrong?

resource/resource.go Outdated Show resolved Hide resolved
@viambot viambot added safe to test This pull request is marked safe to test from a trusted zone and removed safe to test This pull request is marked safe to test from a trusted zone labels Dec 16, 2022
Copy link
Member Author

@Otterverse Otterverse left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe I've addressed everything from this round of reviews, except for two pending questions. One to @npmenard about subtype, as I want confirmation the tests I added there cover the case he's concerned about (and therefore it wasn't a problem.) Second, @edaniels I have questions on how you want me to approach the web_test.go comment about opID testing on streams.

Otherwise, hopefully all good. I'll get to the api/goutils comments in the morning.

@@ -21,7 +21,7 @@
{
"name": "base1",
"type": "base",
"model": "fake",
"model": "acme:test:model",
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not as far as I know. If find a case where you think it does, please let me know.

config/proto_conversions.go Outdated Show resolved Hide resolved
config/proto_conversions.go Outdated Show resolved Hide resolved
config/proto_conversions.go Outdated Show resolved Hide resolved

if config.Model.Namespace == "" {
config.Model.Namespace = resource.ResourceNamespaceRDK
config.Model.ModelFamily = resource.DefaultModelFamily
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wrote a couple of short paragraphs at the top of resource/resource.go so please look over those there.

return err
}
if m.Name == "" {
return errors.New("model name field for resource missing")
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Until one of your previous comments today, I didn't realize we tried to use the json field names in errors. I was just using plain english and saying "model name". Reworded it to say "name field for model missing"

examples/customresources/demos/complexmodule/module.go Outdated Show resolved Hide resolved
subtype/subtype.go Show resolved Hide resolved
subtype/subtype.go Show resolved Hide resolved
Name: "simple-module",
ExePath: modExe,
}
err := mgr.AddModule(ctx, modCfg)
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I didn't realize subtests fired off in goroutines, I thought it was JUST a labeling thing. Switched to t.Log() for that, as I think this is better to test in sequence. (Rather than setup/teardown for every function.)

and useful mostly for organization/grouping. Note that each "tier" contains the tier to the left it. Such that ModelFamily contains
Namespace, and Model itself contains ModelFamily.

An example resource (say, a motor) may use the motor API, and thus have the Subtype "rdk:component:motor" and have a model such as
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nice

@edaniels
Copy link
Contributor

left remaining comments! thanks for that quick turnaround time

@viambot viambot added safe to test This pull request is marked safe to test from a trusted zone and removed safe to test This pull request is marked safe to test from a trusted zone labels Dec 16, 2022
@viambot viambot added safe to test This pull request is marked safe to test from a trusted zone and removed safe to test This pull request is marked safe to test from a trusted zone labels Dec 16, 2022
@viambot viambot added safe to test This pull request is marked safe to test from a trusted zone and removed safe to test This pull request is marked safe to test from a trusted zone labels Dec 16, 2022
@github-actions
Copy link
Contributor

Code Coverage

Package Line Rate Delta Health
go.viam.com/rdk/components/arm 56% -5.26%
go.viam.com/rdk/components/arm/universalrobots 13% 0.00%
go.viam.com/rdk/components/arm/xarm 2% 0.00%
go.viam.com/rdk/components/arm/yahboom 7% 0.00%
go.viam.com/rdk/components/audioinput 55% 0.00%
go.viam.com/rdk/components/base 63% -4.44%
go.viam.com/rdk/components/base/agilex 62% 0.00%
go.viam.com/rdk/components/base/boat 41% 0.00%
go.viam.com/rdk/components/base/wheeled 76% 0.00%
go.viam.com/rdk/components/board 68% 0.00%
go.viam.com/rdk/components/board/arduino 10% 0.00%
go.viam.com/rdk/components/board/commonsysfs 47% 0.00%
go.viam.com/rdk/components/board/fake 38% 0.00%
go.viam.com/rdk/components/board/numato 19% 0.00%
go.viam.com/rdk/components/board/pi 50% 0.00%
go.viam.com/rdk/components/camera 65% 0.00%
go.viam.com/rdk/components/camera/align 64% +0.48%
go.viam.com/rdk/components/camera/fake 67% 0.00%
go.viam.com/rdk/components/camera/ffmpeg 77% +1.43%
go.viam.com/rdk/components/camera/transformpipeline 78% -0.33%
go.viam.com/rdk/components/camera/videosource 50% +0.64%
go.viam.com/rdk/components/encoder/fake 77% 0.00%
go.viam.com/rdk/components/gantry 60% -7.22%
go.viam.com/rdk/components/gantry/multiaxis 84% 0.00%
go.viam.com/rdk/components/gantry/oneaxis 86% 0.00%
go.viam.com/rdk/components/generic 83% 0.00%
go.viam.com/rdk/components/gripper 68% -12.09%
go.viam.com/rdk/components/input 87% 0.00%
go.viam.com/rdk/components/input/gpio 84% 0.00%
go.viam.com/rdk/components/motor 77% -4.06%
go.viam.com/rdk/components/motor/dimensionengineering 63% 0.00%
go.viam.com/rdk/components/motor/dmc4000 69% 0.00%
go.viam.com/rdk/components/motor/fake 57% 0.00%
go.viam.com/rdk/components/motor/gpio 64% 0.00%
go.viam.com/rdk/components/motor/gpiostepper 56% 0.00%
go.viam.com/rdk/components/motor/tmcstepper 62% 0.00%
go.viam.com/rdk/components/movementsensor 75% 0.00%
go.viam.com/rdk/components/movementsensor/cameramono 40% 0.00%
go.viam.com/rdk/components/movementsensor/gpsnmea 37% 0.00%
go.viam.com/rdk/components/movementsensor/gpsrtk 28% 0.00%
go.viam.com/rdk/components/posetracker 86% 0.00%
go.viam.com/rdk/components/sensor 86% 0.00%
go.viam.com/rdk/components/sensor/ultrasonic 34% 0.00%
go.viam.com/rdk/components/servo 68% -10.81%
go.viam.com/rdk/components/servo/gpio 71% 0.00%
go.viam.com/rdk/config 75% -0.80%
go.viam.com/rdk/control 57% 0.00%
go.viam.com/rdk/data 77% 0.00%
go.viam.com/rdk/examples/customresources/demos/complexmodule 0% N/A
go.viam.com/rdk/examples/customresources/demos/remoteserver 0% N/A
go.viam.com/rdk/grpc 25% 0.00%
go.viam.com/rdk/ml 67% 0.00%
go.viam.com/rdk/ml/inference 70% 0.00%
go.viam.com/rdk/module 64% N/A
go.viam.com/rdk/module/modmanager 79% N/A
go.viam.com/rdk/motionplan 71% +0.20%
go.viam.com/rdk/octree 98% 0.00%
go.viam.com/rdk/operation 82% -0.33%
go.viam.com/rdk/pointcloud 72% 0.00%
go.viam.com/rdk/protoutils 59% 0.00%
go.viam.com/rdk/referenceframe 70% 0.00%
go.viam.com/rdk/registry 89% -0.90%
go.viam.com/rdk/resource 86% +2.79%
go.viam.com/rdk/rimage 78% 0.00%
go.viam.com/rdk/rimage/depthadapter 94% 0.00%
go.viam.com/rdk/rimage/transform 73% 0.00%
go.viam.com/rdk/rimage/transform/cmd/extrinsic_calibration 67% 0.00%
go.viam.com/rdk/robot 85% 0.00%
go.viam.com/rdk/robot/client 82% -0.18%
go.viam.com/rdk/robot/framesystem 68% 0.00%
go.viam.com/rdk/robot/impl 81% -0.69%
go.viam.com/rdk/robot/server 59% +2.94%
go.viam.com/rdk/robot/web 62% +2.75%
go.viam.com/rdk/robot/web/stream 87% 0.00%
go.viam.com/rdk/services/armremotecontrol 71% 0.00%
go.viam.com/rdk/services/armremotecontrol/builtin 52% -0.25%
go.viam.com/rdk/services/baseremotecontrol 71% 0.00%
go.viam.com/rdk/services/baseremotecontrol/builtin 79% -0.12%
go.viam.com/rdk/services/datamanager 65% 0.00%
go.viam.com/rdk/services/datamanager/builtin 80% -0.14%
go.viam.com/rdk/services/datamanager/datacapture 21% 0.00%
go.viam.com/rdk/services/datamanager/datasync 72% 0.00%
go.viam.com/rdk/services/motion 63% 0.00%
go.viam.com/rdk/services/motion/builtin 88% 0.00%
go.viam.com/rdk/services/navigation 53% -1.42%
go.viam.com/rdk/services/sensors 77% 0.00%
go.viam.com/rdk/services/sensors/builtin 97% 0.00%
go.viam.com/rdk/services/shell 14% 0.00%
go.viam.com/rdk/services/slam 84% 0.00%
go.viam.com/rdk/services/slam/builtin 72% +0.05%
go.viam.com/rdk/services/vision 80% 0.00%
go.viam.com/rdk/services/vision/builtin 74% -0.05%
go.viam.com/rdk/session 97% 0.00%
go.viam.com/rdk/spatialmath 83% 0.00%
go.viam.com/rdk/subtype 92% -4.49%
go.viam.com/rdk/utils 72% -0.18%
go.viam.com/rdk/vision 26% 0.00%
go.viam.com/rdk/vision/chess 80% 0.00%
go.viam.com/rdk/vision/delaunay 87% 0.00%
go.viam.com/rdk/vision/keypoints 92% 0.00%
go.viam.com/rdk/vision/objectdetection 82% 0.00%
go.viam.com/rdk/vision/odometry 60% 0.00%
go.viam.com/rdk/vision/odometry/cmd 0% 0.00%
go.viam.com/rdk/vision/segmentation 49% 0.00%
go.viam.com/rdk/web/server 26% 0.00%
Summary 66% (21055 / 31683) -0.11%

@Otterverse Otterverse merged commit c720c2e into viamrobotics:main Dec 17, 2022
@Otterverse Otterverse deleted the modular-resources branch December 17, 2022 00:05
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
safe to test This pull request is marked safe to test from a trusted zone
Projects
None yet
Development

Successfully merging this pull request may close these issues.