feat: initial implemention of openziti to go-mod-bootstrap #659

dovholuknf · 2024-02-01T03:19:20Z

closes #650

This is the initial PR to adopt OpenZiti into EdgexFoundry go-mod-bootstrap. Much of the functionality will be used in other repos. I've tried to be as thorough testing this as I could but I'm sure there will need to be work to fix/doc/etc. I wanted to get the code PR up as soon as possible to start getting feedback.

Loggging - OpenZiti uses pfxlog which uses logrus. A hook is added to callback into EdgexFoundry's logger along with setting the default logger output to discard.
auth_middleware is updated to allow connections from OpenZiti identities, defering authorization to OpenZiti
SecretProvider now allows setting the HttpTransport to use. This is replaced by an OpenZiti implementation if zero trust is enabled, otherwise it's the default http transport
any 'server' is now able to be removed from the IP-based underlay network entirely and able to be made available only via OpenZiti. This makes classic IP-based attacks impossible.
SecurityOptions has been added to ServiceInfo and ClientInfo and is configured similarly to something like:

Service:
    SecurityOptions:
        Mode: "zerotrust"
        OpenZitiController: "openziti:1280"
        OpenZitiServiceName: "edgex.ui"
		
		
    core-command:
        Protocol: http
        Host: "core-command.edgex.ziti"
        Port: 80
        SecurityOptions:
            Mode: "zerotrust"
            OpenZitiController: "openziti:1280"
            OpenZitiServiceName: "edgex.core-command"

PR Checklist

Please check if your PR fulfills the following requirements:

[x ] I am not introducing a breaking change (if you are, flag in conventional commit message with BREAKING CHANGE: describing the break)
I am not introducing a new dependency (add notes below if you are)
I have added unit tests for the new feature or bug fix (if not, why?)
I have fully tested (add details below) this the new feature or bug fix (if not, why?)
I have opened a PR for the related docs change (if not, why?)

Testing Instructions

This is a sweeping change that crosses numerous repos. I don't know how to effectively answer this particular question and this one change is insufficient for testing. A larger, cross-cutting PR/effort is needed to put all the pieces together. I could use help with knowing how to satisfy this

New Dependency Instructions (If applicable)

github.com/openziti/sdk-golang (and transitives)

Using https://wiki.edgexfoundry.org/display/FA/Vetting+Process+for+3rd+Party+Dependencies - new dependency review:
Total Increase in new imports: 2

github.com/openziti/sdk-golang
github.com/sirupsen/logrus

github.com/openziti/sdk-golang

Releases/Tags Count: 705+ (frequent updates)
Contributor Count: 18
License: Apache v2
Stars/Forks/Watchers: on this repo - 86/12/15 but the main /ziti repo has 1768/113/34
godoc.org imported by count: 2
Subjective opinion of the reviewers: necessary to implement zero trust connectivity. the project is very well-established and lots of engineering effort has already been put into the project. recreating the same level of complexity and completeness would not be a good investment of time.

github.com/sirupsen/logrus

Releases/Tags Count: 18 (it's not the kind of project that needs many updates, still actively maintained)
Contributor Count: 262
License: MIT license
Stars/Forks/Watchers: on this repo - 23.8k/2.3k/315
godoc.org imported by count: 145,083
Subjective opinion of the reviewers: necessary only to quiet logging from OpenZiti. OpenZiti include logrus transitively, but to quiet the logging and provide an adapter to edgex's logging, it is needed.

lenny-goodell

Initial comments. Still more to review.

bootstrap/container/zerotrust.go

bootstrap/container/logging.go

go.mod

bootstrap/container/zerotrust.go

bootstrap/handlers/httpserver.go

bootstrap/handlers/clients.go

bootstrap/utils/logging.go

bootstrap/handlers/auth_middleware.go

bootstrap/container/secret.go

bootstrap/bootstrap.go

bootstrap/handlers/httpserver.go

bootstrap/secret/secure.go

bootstrap/secret/insecure.go

bootstrap/interfaces/secret.go

bootstrap/handlers/httpserver.go

lenny-goodell

LGTM, Nice work!

lenny-goodell · 2024-02-26T20:23:06Z

@jumpingliu , @cloudxxx8 , @JamesKButcher , there are two new dependencies added by this PR. @clint has done the vetting and put the information in the PR description per our process. Though it is not explicit in the documented process, we have always had a TSC vote to add new 3rd party dependencies,

codecov-commenter · 2024-02-26T20:40:10Z

Codecov Report

Attention: Patch coverage is 2.40000% with 122 lines in your changes are missing coverage. Please review.

Project coverage is 56.01%. Comparing base (8bd6798) to head (4c525f1).

Files	Patch %	Lines
bootstrap/handlers/httpserver.go	0.00%	55 Missing ⚠️
bootstrap/utils/logging.go	0.00%	29 Missing ⚠️
bootstrap/secret/secure.go	0.00%	11 Missing ⚠️
bootstrap/secret/jwtprovider.go	0.00%	9 Missing ⚠️
bootstrap/handlers/auth_middleware.go	0.00%	8 Missing ⚠️
bootstrap/secret/insecure.go	0.00%	6 Missing ⚠️
bootstrap/handlers/clients.go	42.85%	2 Missing and 2 partials ⚠️

❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #659      +/-   ##
==========================================
- Coverage   58.61%   56.01%   -2.60%     
==========================================
  Files          31       32       +1     
  Lines        2537     2658     +121     
==========================================
+ Hits         1487     1489       +2     
- Misses        953     1070     +117     
- Partials       97       99       +2

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

bootstrap/handlers/httpserver.go

bootstrap/secret/secure.go

Signed-off-by: dovholuknf <46322585+dovholuknf@users.noreply.github.com>

…is used in edgex-go but not in this project Signed-off-by: dovholuknf <46322585+dovholuknf@users.noreply.github.com>

Signed-off-by: dovholuknf <46322585+dovholuknf@users.noreply.github.com>

…e mock regen issue Signed-off-by: dovholuknf <46322585+dovholuknf@users.noreply.github.com>

Signed-off-by: dovholuknf <46322585+dovholuknf@users.noreply.github.com>

…on as it doesn't seem used Signed-off-by: dovholuknf <46322585+dovholuknf@users.noreply.github.com>

Signed-off-by: dovholuknf <46322585+dovholuknf@users.noreply.github.com>

fix: make and use constants fix: add a tiny bit of additional debug logging just in case fix: move the ConnContext before the .Serve calls fix: make the `SetHttpTransport` function one-time only (guard from being overwritten) Signed-off-by: dovholuknf <46322585+dovholuknf@users.noreply.github.com>

dovholuknf · 2024-03-16T14:23:45Z

i've amended the initial commit and signed it to fix the DCO and addressed the PR comments.

cloudxxx8 · 2024-03-18T03:21:19Z

DCO may be failed by one of your commits, so you can squash all into one commit and make sure it can pass DCT

go.mod

Signed-off-by: dovholuknf <46322585+dovholuknf@users.noreply.github.com>

dovholuknf · 2024-03-18T11:52:32Z

thanks. fixed. i really do need to setup 'auto signoff'...

cloudxxx8

LGTM

judehung · 2024-11-08T12:45:31Z

bootstrap/handlers/httpserver.go

+				break
+			}
+
+			zc.c = &zitiCtx


@dovholuknf I'm refactoring your code for #789. I'm confused about the purpose of local zc variable. It looks to me that the code only assigns &zitiCtx to zc.c here without any further processing. Do we really need to keep zc?

from what I see, it appears to be totally vestigal. I don't find any references to it whatsoever other than this one line. I expect at one point a function was defined on the struct, the function removed, leaving this behind.

Thanks for reply. I'll remove both this variable and ZitiContext struct in my PR.

The usage of OpenZiti packages to support zero trust feature significantly increases the build size. For example, core-metadata increases from 14MB to 21MB, core-command increases from 9.8MB to 17MB, device-virtual increases from 18MB to 31MB, and app-service-configurable increases from 22Mb to 34MB. As many edge user scenarios require to deploy EdgeX services on resource-constrained devices without security, allowing that the services can be built without OpenZiti packages and ZeroTrust features can be helpful to user cases which don't need zero trust feature. This commit refactors the codes importing openziti packages with //go:build !no_openziti directive and creates para codes that don't use openziti packages with //go:build no_openziti directive, so users can simply build services by specifying no_openziti tag. Also remove vestigal zc variables and ZitiContext struct per discussion with https://github.com/dovholuknf in edgexfoundry#659 (comment) Signed-off-by: Jude Hung <jude@iotechsys.com>

closes edgexfoundry#789 The usage of OpenZiti packages to support zero trust feature significantly increases the build size. For example, core-metadata increases from 14MB to 21MB, core-command increases from 9.8MB to 17MB, device-virtual increases from 18MB to 31MB, and app-service-configurable increases from 22Mb to 34MB. As many edge user scenarios require to deploy EdgeX services on resource-constrained devices without security, allowing that the services can be built without OpenZiti packages and ZeroTrust features can be helpful to user cases which don't need zero trust feature. This commit refactors the codes importing openziti packages with //go:build !no_openziti directive and creates para codes that don't use openziti packages with //go:build no_openziti directive, so users can simply build services by specifying no_openziti tag. Also remove vestigal zc variables and ZitiContext struct per discussion with https://github.com/dovholuknf in edgexfoundry#659 (comment) Signed-off-by: Jude Hung <jude@iotechsys.com>

* feat: Add new go build tag no_openziti to reduce build size closes #789 The usage of OpenZiti packages to support zero trust feature significantly increases the build size. For example, core-metadata increases from 14MB to 21MB, core-command increases from 9.8MB to 17MB, device-virtual increases from 18MB to 31MB, and app-service-configurable increases from 22Mb to 34MB. As many edge user scenarios require to deploy EdgeX services on resource-constrained devices without security, allowing that the services can be built without OpenZiti packages and ZeroTrust features can be helpful to user cases which don't need zero trust feature. This commit refactors the codes importing openziti packages with //go:build !no_openziti directive and creates para codes that don't use openziti packages with //go:build no_openziti directive, so users can simply build services by specifying no_openziti tag. Also remove vestigal zc variables and ZitiContext struct per discussion with https://github.com/dovholuknf in #659 (comment) Signed-off-by: Jude Hung <jude@iotechsys.com> * feat: rename the zerotrust_no_ziti.go to no_ziti.go Signed-off-by: Jude Hung <jude@iotechsys.com> --------- Signed-off-by: Jude Hung <jude@iotechsys.com>

lenny-goodell requested changes Feb 5, 2024

View reviewed changes

bootstrap/container/zerotrust.go Outdated Show resolved Hide resolved

bootstrap/container/logging.go Outdated Show resolved Hide resolved

go.mod Outdated Show resolved Hide resolved

lenny-goodell requested changes Feb 6, 2024

View reviewed changes

dovholuknf force-pushed the add-openziti branch 5 times, most recently from 9e3ce1e to 285799d Compare February 12, 2024 15:28

lenny-goodell requested changes Feb 12, 2024

View reviewed changes

bootstrap/handlers/clients.go Outdated Show resolved Hide resolved

lenny-goodell reviewed Feb 12, 2024

View reviewed changes

bootstrap/utils/logging.go Show resolved Hide resolved

lenny-goodell requested changes Feb 12, 2024

View reviewed changes

bootstrap/handlers/auth_middleware.go Outdated Show resolved Hide resolved

lenny-goodell requested changes Feb 12, 2024

View reviewed changes

dovholuknf force-pushed the add-openziti branch from 74087bf to e65d472 Compare February 12, 2024 23:58

lenny-goodell requested changes Feb 13, 2024

View reviewed changes

bootstrap/handlers/httpserver.go Outdated Show resolved Hide resolved

lenny-goodell approved these changes Feb 13, 2024

View reviewed changes

lenny-goodell added dependencies Pull requests that update a dependency file security_audit Track issues that are related to CVE/CVSS/CWE auditing etc security-review Pull requests that require review by a member of the Security WG labels Feb 26, 2024

dovholuknf force-pushed the add-openziti branch from 5427a8f to 81f355e Compare February 26, 2024 20:22

dovholuknf force-pushed the add-openziti branch from 81f355e to 24d1ce0 Compare February 26, 2024 20:29

lenny-goodell approved these changes Feb 26, 2024

View reviewed changes

cloudxxx8 requested changes Mar 4, 2024

View reviewed changes

bootstrap/handlers/httpserver.go Outdated Show resolved Hide resolved

bootstrap/handlers/httpserver.go Outdated Show resolved Hide resolved

bootstrap/handlers/httpserver.go Outdated Show resolved Hide resolved

bootstrap/secret/secure.go Show resolved Hide resolved

dovholuknf force-pushed the add-openziti branch from c9f8607 to 3729b23 Compare March 16, 2024 13:35

dovholuknf added 6 commits March 16, 2024 10:20

feat: add openziti/zero trust to go-mod-bootstrap. resolve conflicts

a7d006c

Signed-off-by: dovholuknf <46322585+dovholuknf@users.noreply.github.com>

fix: address test failure

abf5491

Signed-off-by: dovholuknf <46322585+dovholuknf@users.noreply.github.com>

fix: address PR comment. also restore HttpTransportFromService which …

96f5c72

…is used in edgex-go but not in this project Signed-off-by: dovholuknf <46322585+dovholuknf@users.noreply.github.com>

fix: address cyclical compilaiton issue from edgex-go

75426c4

Signed-off-by: dovholuknf <46322585+dovholuknf@users.noreply.github.com>

fix: address failing test. remove unneeded nil check after solving th…

a098b41

…e mock regen issue Signed-off-by: dovholuknf <46322585+dovholuknf@users.noreply.github.com>

fix: no need for the debug variable after figuring out the issue

929ef1f

Signed-off-by: dovholuknf <46322585+dovholuknf@users.noreply.github.com>

dovholuknf added 7 commits March 16, 2024 10:20

fix: add zero trust flag to secret provider. regen mock. adapt code

25dd8a7

Signed-off-by: dovholuknf <46322585+dovholuknf@users.noreply.github.com>

fix: continued PR feedback/refinement

36daeb5

Signed-off-by: dovholuknf <46322585+dovholuknf@users.noreply.github.com>

fix: rearrange imports to minimize changes

af1cb36

Signed-off-by: dovholuknf <46322585+dovholuknf@users.noreply.github.com>

fix: didn't push the rename changes. remove the HandleOpenZiti functi…

4b92d54

…on as it doesn't seem used Signed-off-by: dovholuknf <46322585+dovholuknf@users.noreply.github.com>

fix: go test ./... succeeds but make test fails?

42049ab

Signed-off-by: dovholuknf <46322585+dovholuknf@users.noreply.github.com>

fix: use struct as key for context

d73bf1c

Signed-off-by: dovholuknf <46322585+dovholuknf@users.noreply.github.com>

dovholuknf force-pushed the add-openziti branch from 3729b23 to 9f9a216 Compare March 16, 2024 14:21

cloudxxx8 reviewed Mar 18, 2024

View reviewed changes

go.mod Show resolved Hide resolved

fix: use constants and tweak log message

96197b9

Signed-off-by: dovholuknf <46322585+dovholuknf@users.noreply.github.com>

dovholuknf force-pushed the add-openziti branch from 4c525f1 to 96197b9 Compare March 18, 2024 11:48

fix: use 3.2.0-dev.13

173b050

Signed-off-by: dovholuknf <46322585+dovholuknf@users.noreply.github.com>

cloudxxx8 approved these changes Mar 18, 2024

View reviewed changes

cloudxxx8 merged commit 4121a9d into edgexfoundry:main Mar 18, 2024
3 checks passed

dovholuknf mentioned this pull request Mar 18, 2024

feat: allow clients to be zero trust #678

Merged

5 tasks

judehung reviewed Nov 8, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: initial implemention of openziti to go-mod-bootstrap #659

feat: initial implemention of openziti to go-mod-bootstrap #659

dovholuknf commented Feb 1, 2024 •

edited

Loading

lenny-goodell left a comment

lenny-goodell left a comment

lenny-goodell commented Feb 26, 2024

codecov-commenter commented Feb 26, 2024 •

edited

Loading

dovholuknf commented Mar 16, 2024

cloudxxx8 commented Mar 18, 2024

dovholuknf commented Mar 18, 2024

cloudxxx8 left a comment

judehung Nov 8, 2024

dovholuknf Nov 8, 2024

judehung Nov 8, 2024

feat: initial implemention of openziti to go-mod-bootstrap #659

feat: initial implemention of openziti to go-mod-bootstrap #659

Conversation

dovholuknf commented Feb 1, 2024 • edited Loading

PR Checklist

Testing Instructions

New Dependency Instructions (If applicable)

github.com/openziti/sdk-golang

github.com/sirupsen/logrus

lenny-goodell left a comment

Choose a reason for hiding this comment

lenny-goodell left a comment

Choose a reason for hiding this comment

lenny-goodell commented Feb 26, 2024

codecov-commenter commented Feb 26, 2024 • edited Loading

Codecov Report

dovholuknf commented Mar 16, 2024

cloudxxx8 commented Mar 18, 2024

dovholuknf commented Mar 18, 2024

cloudxxx8 left a comment

Choose a reason for hiding this comment

judehung Nov 8, 2024

Choose a reason for hiding this comment

dovholuknf Nov 8, 2024

Choose a reason for hiding this comment

judehung Nov 8, 2024

Choose a reason for hiding this comment

dovholuknf commented Feb 1, 2024 •

edited

Loading

codecov-commenter commented Feb 26, 2024 •

edited

Loading