Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add UDS Operator and consolidate UDS Policies #66

Merged
merged 111 commits into from
Jan 16, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
111 commits
Select commit Hold shift + click to select a range
1f0e828
wip: UDS Operator
jeff-mccoy Dec 15, 2023
2ffb847
continue working istio migration
jeff-mccoy Dec 17, 2023
b74742e
wip: consolidate pepr things
jeff-mccoy Dec 18, 2023
bd43cd8
I hate you prettier
jeff-mccoy Dec 18, 2023
479eec5
fix neuvector service name
jeff-mccoy Dec 18, 2023
2e621d1
add `name` prop for expose to avoid creating extra VS on service name…
jeff-mccoy Dec 18, 2023
c14adc1
add initial CRD validator
jeff-mccoy Dec 18, 2023
f50f31c
update pkg for new required name field
jeff-mccoy Dec 18, 2023
821e8db
update test package to use CRD
jeff-mccoy Dec 18, 2023
a1e08e4
allow simpler operator config + add real API server netpol
jeff-mccoy Dec 21, 2023
4cf40d3
uds/zarf version bump
jeff-mccoy Dec 31, 2023
842f464
pepr watcher rc update
jeff-mccoy Dec 31, 2023
9c8a1f4
dont rely on name for resource pruning/updating
jeff-mccoy Dec 31, 2023
df8f1ec
match name convention of standard netpols
jeff-mccoy Dec 31, 2023
0e00743
update test pkg uds CRs
jeff-mccoy Dec 31, 2023
a03974f
forec:true & owenerref common func
jeff-mccoy Jan 1, 2024
c714e2f
ouch fail
jeff-mccoy Jan 1, 2024
6783fce
make netpol labels consistent
jeff-mccoy Jan 1, 2024
2a4b11e
pepr 0.21
jeff-mccoy Jan 1, 2024
1ab7f47
cleanup zarf/uds cli version updates
jeff-mccoy Jan 1, 2024
2c7fd9a
back to zarf v0.31.4 until UDS CLI is updated
jeff-mccoy Jan 1, 2024
389bd8e
auto-wire ingress allow NetPol for generated Istio VS
jeff-mccoy Jan 1, 2024
205dfe4
Merge branch 'main' into uds-operator
jeff-mccoy Jan 1, 2024
4b73dc8
update test app expose entries
jeff-mccoy Jan 3, 2024
2cab4fc
pepr 0.21.1 (PatchStatus)
jeff-mccoy Jan 3, 2024
acb4803
update app uds pkg crds
jeff-mccoy Jan 3, 2024
18f3676
move controller logic to controllers folder
jeff-mccoy Jan 3, 2024
f98ecb9
istio ns label brittleness
jeff-mccoy Jan 3, 2024
3e492e3
create separate reconciler file + use PathStatus() for status updates
jeff-mccoy Jan 3, 2024
a4b79e3
add enqueue to control reconcile flow
jeff-mccoy Jan 3, 2024
87f38e4
register the CRD for watch or dev mode
jeff-mccoy Jan 3, 2024
f4e5fc5
update CRD def
jeff-mccoy Jan 3, 2024
774976e
leave bully.ts there for now until we decided if its worth adding a s…
jeff-mccoy Jan 3, 2024
f7d9804
Merge branch 'uds-operator' of https://github.com/defenseunicorns/uds…
jeff-mccoy Jan 3, 2024
9016659
update renovate tag for zarf init pkg
jeff-mccoy Jan 3, 2024
d29fca8
Update src/pepr/istio/index.ts
jeff-mccoy Jan 3, 2024
1061ad1
get apiserver address for netpols from endpointslices vs `/api`
jeff-mccoy Jan 4, 2024
ab68092
update name pattern for netpols
jeff-mccoy Jan 4, 2024
ea332b8
do not retry same failed generation
jeff-mccoy Jan 4, 2024
484db63
add todo
jeff-mccoy Jan 4, 2024
22007fc
add uds to netpol name
jeff-mccoy Jan 4, 2024
cf3213e
fix var name
jeff-mccoy Jan 4, 2024
e23d72c
update metrics server UDS Package
jeff-mccoy Jan 4, 2024
03f212e
clenaup remoteGenerator things
jeff-mccoy Jan 4, 2024
8feb914
more networkpolicy migration pain
jeff-mccoy Jan 4, 2024
9d376ae
remove protocol, add ports options for allow rules
jeff-mccoy Jan 4, 2024
6e31197
so those are case sensitive...
jeff-mccoy Jan 5, 2024
7a74fce
use generator for default policies + remove Promise
jeff-mccoy Jan 5, 2024
c40f4b7
add uds to default deny policy name
jeff-mccoy Jan 5, 2024
22bf462
simplify remoteNamespaceLabels to remoteNamespace
jeff-mccoy Jan 5, 2024
d0bbb7f
make better netpol names
jeff-mccoy Jan 5, 2024
db38f10
add netpol descriptions
jeff-mccoy Jan 5, 2024
45f97bb
add targetPort to accomdate port difference between svc/pod (VS and N…
jeff-mccoy Jan 8, 2024
b0b52bd
Merge branch 'main' into uds-operator
jeff-mccoy Jan 8, 2024
aa8e94b
only deal with ns label if istio is enabled
jeff-mccoy Jan 8, 2024
1e649d8
cleanup crd imports
jeff-mccoy Jan 8, 2024
b374143
drop the stress test file
jeff-mccoy Jan 8, 2024
1e40554
convert promtail to udspackage
jeff-mccoy Jan 8, 2024
76e80a3
Readme, unit test updates
jeff-mccoy Jan 8, 2024
f38366f
add another note about uds operator
jeff-mccoy Jan 8, 2024
6149efd
add diagram
jeff-mccoy Jan 8, 2024
edf11c8
Merge branch 'main' into uds-operator
jeff-mccoy Jan 8, 2024
99b0035
shame
jeff-mccoy Jan 8, 2024
9bac1d3
Merge branch 'main' into uds-operator
jeff-mccoy Jan 8, 2024
2d6919c
Update src/pepr/operator/crd/validator.ts
jeff-mccoy Jan 8, 2024
eac1d6f
I cant spell
jeff-mccoy Jan 8, 2024
4ea5290
Update src/pepr/istio/README.md
jeff-mccoy Jan 9, 2024
2e1d355
Update src/pepr/operator/README.md
jeff-mccoy Jan 9, 2024
719ffbe
ensure default-deny is unique per pkg
jeff-mccoy Jan 9, 2024
3f47f82
Merge branch 'main' into uds-operator
jeff-mccoy Jan 9, 2024
73704fe
Canine Infectious Respiratory Disease
jeff-mccoy Jan 9, 2024
90a5709
add policy description to annotations
jeff-mccoy Jan 9, 2024
c67502e
include uds pkg name in netpol name
jeff-mccoy Jan 9, 2024
29ca359
Merge branch 'main' into uds-operator
jeff-mccoy Jan 9, 2024
5411421
move truncate netpol name function
jeff-mccoy Jan 10, 2024
31c36d6
Update src/pepr/operator/controllers/network/policies.ts
jeff-mccoy Jan 10, 2024
ba02db7
more DRY dequeue
jeff-mccoy Jan 10, 2024
b8c0714
remove test file
jeff-mccoy Jan 10, 2024
04b6a1d
consolidate resource name sanitization
jeff-mccoy Jan 10, 2024
3c3a698
chore(deps): update uds to v0.5.3
renovate[bot] Jan 11, 2024
35e3eef
properly wire passthrough tls for VirtualServices
jeff-mccoy Jan 11, 2024
3d37395
update UDS CLI min version in READMEs
jeff-mccoy Jan 11, 2024
fee2c55
chore(deps): update uds-k3d to v0.3.0
renovate[bot] Jan 10, 2024
8fb20af
chore(deps): update zarf to v0.32.1
renovate[bot] Jan 4, 2024
826b28a
fix zarf & uds-k3d versions
jeff-mccoy Jan 11, 2024
fa00eb0
package.json cleanup
jeff-mccoy Jan 11, 2024
70b21e9
Merge branch 'renovate/uds' into uds-operator
jeff-mccoy Jan 11, 2024
78a7fb4
add `uds run dev` task
jeff-mccoy Jan 11, 2024
edf68fa
allow login host for passtrhough gw
jeff-mccoy Jan 12, 2024
b9dbd20
fix passthrough gw config
jeff-mccoy Jan 12, 2024
792c785
cleanup ns labels on pkg delete
jeff-mccoy Jan 12, 2024
f5b5621
Merge branch 'main' into uds-operator
jeff-mccoy Jan 12, 2024
85ca673
readme update
jeff-mccoy Jan 12, 2024
912ba68
Merge branch 'main' into uds-operator
jeff-mccoy Jan 12, 2024
e19e50a
handle pod restart with istio injection
jeff-mccoy Jan 12, 2024
4486c6a
prevent netpol name colissions
jeff-mccoy Jan 12, 2024
95e5115
run npx pepr format
jeff-mccoy Jan 12, 2024
88951c0
Update src/pepr/operator/controllers/network/policies.ts
mjnagel Jan 12, 2024
16765b4
chore: add descriptions to nps
mjnagel Jan 12, 2024
1bcd80a
rename ns cleanup function
jeff-mccoy Jan 12, 2024
2f422c0
pepr 0.22.0
jeff-mccoy Jan 13, 2024
05e9655
make netpols run before istio things
jeff-mccoy Jan 13, 2024
3f0829b
bind to kube-dns for dns egress
jeff-mccoy Jan 13, 2024
c0b4520
properly handle apiserver endpointslice updates
jeff-mccoy Jan 13, 2024
0f71619
allow try/catch on `pending` statud bump
jeff-mccoy Jan 13, 2024
e240591
Merge branch 'main' into uds-operator
jeff-mccoy Jan 13, 2024
e0590e3
Merge branch 'main' into uds-operator
jeff-mccoy Jan 13, 2024
91551e8
fix: intra namespace for loki gateway comms
mjnagel Jan 16, 2024
bf68e87
add uds package for prometheus stack
mjnagel Jan 16, 2024
f13cf99
fix: missing grafana np for prom
mjnagel Jan 16, 2024
f60a093
Merge branch 'main' into uds-operator
jeff-mccoy Jan 16, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions .vscode/settings.json
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@
"enableTurboSourcemaps": true,
"resolveSourceMapLocations": [
"${workspaceFolder}/**",
"node_modules/kubernetes-fluent-client/**",
"node_modules/pepr/**"
]
},
Expand Down
11 changes: 6 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,10 +12,10 @@ The core applications are:
- [Istio](https://istio.io/) - Service Mesh
- [KeyCloak](https://www.keycloak.org/) - Identity & Access Management
- [Kiali](https://kiali.io/) - Service Mesh Observability
- [Kyverno](https://kyverno.io/) - Policy Engine
- [Loki](https://grafana.com/oss/loki/) - Log Aggregation
- [Metrics Server](https://github.com/kubernetes-sigs/metrics-server) - Metrics
- [Neuvector](https://open-docs.neuvector.com/) - Container Security
- [Pepr](https://pepr.dev) - UDS policy engine & operator
- [Prometheus Stack](https://github.com/prometheus-operator/kube-prometheus) - Monitoring
- [Promtail](https://grafana.com/docs/loki/latest/send-data/promtail/) - Log Aggregation
- [Tempo](https://grafana.com/docs/tempo/latest/getting-started/) - Tracing
Expand Down Expand Up @@ -47,17 +47,18 @@ uds deploy oci://ghcr.io/defenseunicorns/packages/uds/bundles/k3d-core:arm64
uds deploy oci://ghcr.io/defenseunicorns/packages/uds/bundles/k3d-core:amd64
```

The bundle includes the uds.dev certs by default. You can use the UDS environment variables to override the default values. E.g.
The bundle includes the uds.dev certs by default. You can use the UDS environment variables to override the default values. E.g.

```bash
# Set environment variables with the contents of your certificate and key files
# Set environment variables with the contents of your certificate and key files (must be base64 encoded)
UDS_ADMIN_TLS_CERT=$(cat admin.crt)
UDS_ADMIN_TLS_KEY=$(cat admin.key)
UDS_TENANT_TLS_CERT=$(cat tenant.crt)
UDS_TENANT_TLS_KEY=$(cat tenant.key)

# AMD version
uds deploy ocs://ghcr.io/defenseunicorns/package/uds/bundles/k3d-core:amd64
UDS_DOMAIN=example.com

uds deploy oci://ghcr.io/defenseunicorns/package/uds/bundles/k3d-core:amd64
```

### UDS Core Packages
Expand Down
2 changes: 1 addition & 1 deletion bundles/k3d-istio/uds-bundle.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ packages:
- name: policies
description: "Minio policies"
path: policies

- name: init
repository: ghcr.io/defenseunicorns/packages/init
# renovate: datasource=github-tags depName=defenseunicorns/zarf versioning=semver
Expand Down
2 changes: 1 addition & 1 deletion bundles/k3d-standard/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,10 +12,10 @@ The core applications are:
- [Istio](https://istio.io/) - Service Mesh
- [KeyCloak](https://www.keycloak.org/) - Identity & Access Management
- [Kiali](https://kiali.io/) - Service Mesh Observability
- [Kyverno](https://kyverno.io/) - Policy Engine
- [Loki](https://grafana.com/oss/loki/) - Log Aggregation
- [Metrics Server](https://github.com/kubernetes-sigs/metrics-server) - Metrics
- [Neuvector](https://open-docs.neuvector.com/) - Container Security
- [Pepr](https://pepr.dev) - UDS policy engine & operator
- [Prometheus Stack](https://github.com/prometheus-operator/kube-prometheus) - Monitoring
- [Promtail](https://grafana.com/docs/loki/latest/send-data/promtail/) - Log Aggregation
- [Tempo](https://grafana.com/docs/tempo/latest/getting-started/) - Tracing
Expand Down
31 changes: 0 additions & 31 deletions bundles/tls-certs.ts

This file was deleted.

12 changes: 6 additions & 6 deletions package-lock.json

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

4 changes: 4 additions & 0 deletions package.json
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,10 @@
"zarf"
],
"labels": []
},
"env": {
"UDS_DOMAIN": "###ZARF_VAR_DOMAIN###",
"UDS_WITH_ISTIO": "###ZARF_CONST_WITH_ISTIO###"
}
},
"scripts": {
Expand Down
52 changes: 11 additions & 41 deletions pepr.ts
Original file line number Diff line number Diff line change
@@ -1,48 +1,18 @@
import { Capability, PeprModule } from "pepr";
import { PeprModule } from "pepr";

import cfg from "./package.json";

import { policies } from "./src/policies";
import { istio } from "./src/istio/pepr";
import { istio } from "./src/pepr/istio";
import { operator } from "./src/pepr/operator";
import { policies } from "./src/pepr/policies";

new PeprModule(cfg, [
// UDS Core Operator
operator,

/**
* This the root of the UDS Core Pepr Module. To operate on a specific source package, you can
* set the `UDS_PKG` environment variable to the name of the package.
*
* Example:
* UDS_PKG=istio npx pepr build
*/
const sortedCapabilities: Record<string, Capability>[] = [
// UDS Core Policies
{ policies },
policies,

// Istio service mesh
{ istio },
];

// Otherwise, use all capabilities
const allCapabilities = sortedCapabilities.flatMap(data => {
return Object.values(data).flat();
});

const pkg = process.env.UDS_PKG;

if (!pkg || pkg === "all") {
// Start the Pepr module
new PeprModule(cfg, allCapabilities);
} else {
console.log(
`\n\n************** Pepr capabilities limited to only ${pkg} source package **************n\n`,
);

// If the UDS_PKG environment variable is set, then only use that source package
const activeCapability = sortedCapabilities.find(data => data[pkg])?.[pkg];

if (!activeCapability) {
console.error(`Source package ${pkg} not found. Exiting...`);
process.exit(1);
}

// Start the Pepr module
new PeprModule(cfg, [activeCapability]);
}
istio,
]);

This file was deleted.

18 changes: 0 additions & 18 deletions src/grafana/chart/templates/networkpolicies/egress-grafana.yaml

This file was deleted.

19 changes: 0 additions & 19 deletions src/grafana/chart/templates/networkpolicies/egress-istiod.yaml

This file was deleted.

20 changes: 0 additions & 20 deletions src/grafana/chart/templates/networkpolicies/egress-tempo.yaml

This file was deleted.

24 changes: 0 additions & 24 deletions src/grafana/chart/templates/networkpolicies/ingress-istio.yaml

This file was deleted.

23 changes: 0 additions & 23 deletions src/grafana/chart/templates/networkpolicies/ingress-tempo.yaml

This file was deleted.

This file was deleted.

37 changes: 37 additions & 0 deletions src/grafana/chart/templates/uds-package.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
apiVersion: uds.dev/v1alpha1
kind: Package
metadata:
name: grafana
namespace: {{ .Release.Namespace }}
spec:
network:
expose:
- service: grafana
podLabels:
app.kubernetes.io/name: grafana
host: grafana
gateway: admin
port: 80
targetPort: 3000

allow:
- direction: Ingress
podLabels:
app.kubernetes.io/name: grafana
remoteNamespace: tempo
remotePodLabels:
app.kubernetes.io/name: tempo
port: 9090
description: "Tempo Datasource"

- direction: Egress
podLabels:
app.kubernetes.io/name: grafana
remoteGenerated: Anywhere

- direction: Egress
remoteNamespace: tempo
remotePodLabels:
app.kubernetes.io/name: tempo
port: 9411
description: "Tempo"
6 changes: 0 additions & 6 deletions src/grafana/values/values.yaml
Original file line number Diff line number Diff line change
@@ -1,9 +1,3 @@
service:
labels:
uds/istio-gateway: admin
uds/istio-host: grafana
uds/istio-port: "80"

sidecar:
image:
# -- The Docker registry
Expand Down
Loading