docs: Draft docs for self-hosted (WIP) #30491

kay-kim · 2024-11-15T00:49:47Z

Draft. Just placing here to facilitate some specific questions as I work on this.
Added patch for general information architecture changes + updated self-hosted -> self-managed:
Since not part of the left-hand nav:

self-managed/
- installation/
  - install-on-local-kind
  - install-on-local-minikube
  - install-on-aws (just a placeholder for now)
  - install-on-gcp (just a placeholder for now)
- configuration
- operational-guideline
- troubleshooting
- upgrading

kay-kim · 2024-11-15T00:51:48Z

Every single time I want to mark as Draft ... I fat finger to PR. drat.

doc/user/content/self-hosted/_index.md

kay-kim · 2024-11-15T00:53:49Z

doc/user/content/self-hosted/_index.md

+1. Set up the Materialize operator Helm repository.
+
+   a. <red>TBD whether this is needed</red>. Add Helm to install charts that are
+      hosted in the Materialize operator Helm repository:


I wasn't sure if we'd be having people add materialize or we expect people to just have the repo so that they can just install.

right now the helm chart is just in the materialize repo... at some point soon we'll host it on a separate endpoint. for now I think we just tell people to download the materialize repo, check out a specific tag, and add the local path

kay-kim · 2024-11-15T00:54:38Z

doc/user/content/self-hosted/_index.md

+This command removes all the Kubernetes components associated with the chart and
+deletes the release.
+
+## Deploying Materialize Environments


Have no idea what is meant by "deploy a Materialize environment"

good point -- we should scrub "environment" from the docs. that's what we use internally to describe an individual customer region within our cloud product, but is a pretty overloaded term

with the Helm chart / in public facing docs, I think we can just talk about "Deploying Materialize" and the "Materialize CR" (custom resource)

kay-kim · 2024-11-15T00:56:00Z

doc/user/data/materialize_operator_chart_parameter.yml

+parameters:
+  - parameter: clusterd.nodeSelector
+    description: |
+      <red>Replace with description content here.</red>


Will need devs to:

Specify which of these parameters should be user-facing.

For those that are user facing, should update the descriptions.

kay-kim · 2024-11-15T01:04:04Z

doc/user/content/self-hosted/_index.md

+title: "Materialize Kubernetes Operator"
+description: ""
+
+---


We could also make this an overview page.
And move the content to an Install on AWS page (and possibly an Install locally on kind -- since that's how I'm running through the steps anyhow).

But, put the PR up quickly, so that at least you can get an idea about the content.

kay-kim · 2024-11-15T01:08:10Z

doc/user/content/self-hosted/configuration.md

+
+You can configure the Materialize operator chart. For example:
+
+- **RBAC**


Below where I have the ## Parameters section, I could section off the parameters ... so that we don't need to "For example" here.

That is, in the parameters section, I could create subsections

## Parameters ### Network policies ### Observability ### RBAC

kay-kim · 2024-11-15T01:16:25Z

doc/user/content/self-hosted/_index.md

@@ -0,0 +1,242 @@
+---
+title: "Materialize Kubernetes Operator"


https://preview.materialize.com/materialize/30491/self-hosted/

kay-kim · 2024-11-15T01:16:39Z

doc/user/content/self-hosted/configuration.md

@@ -0,0 +1,61 @@
+---
+title: "Materialize Operator Configuration"


https://preview.materialize.com/materialize/30491/self-hosted/configuration/

kay-kim · 2024-11-15T01:16:44Z

doc/user/content/self-hosted/troubleshooting.md

@@ -0,0 +1,21 @@
+---
+title: "Troubleshooting"


https://preview.materialize.com/materialize/30491/self-hosted/troubleshooting/

pH14 · 2024-11-15T15:14:16Z

doc/user/content/self-hosted/_index.md

+
+This tutorial uses `kubectl`. To install, refer to the [`kubectl` documentationq](https://kubernetes.io/docs/tasks/tools/).
+
+### Kubernetes Storage Configuration


just noting that this is lacking context. we should have an explainer on why local storage is valuable (spilling to disk, operating on datasets larger than main memory, more graceful degradation rather than OOMing), and also note that this is optional (though highly recommended)

pH14 · 2024-11-15T15:16:46Z

doc/user/content/self-hosted/_index.md

+  requestRollout: 22222222-2222-2222-2222-222222222222
+  forceRollout: 33333333-3333-3333-3333-333333333333
+  inPlaceRollout: false
+  backendSecretName: materialize-backend


this is how Materialize gets the connection string to talk to its metadata database (postgres or crdb)... which makes me realize we have no mention of that whole set up here :)

noting that we'll need sections about setting up blob storage + metadata database

pH14 · 2024-11-15T15:17:20Z

Thanks for getting this off the ground @kay-kim !

kay-kim · 2024-11-20T01:48:55Z

doc/user/content/self-hosted/install-on-local-kind.md

+        kubectl apply -f misc/helm-charts/testing/minio.yaml   
+        ```
+
+1. Install the following metrics service:


I realize that the testing/readme.md says we need this, but this errors for me on my mac.

metrics-server should be optional... it's needed for some system table metrics to work, but not blocking to testing materialize

kay-kim · 2024-11-20T01:49:22Z

doc/user/content/self-hosted/install-on-local-kind.md

+      service/mzfhj38ptdjs-console   NodePort    10.96.97.5      <none>        9000:30847/TCP
+      ```
+
+   1. Forward the Materialize console service to your local machine:


I had to forward on my mac.

We discussed this some more on the Cloud team, and likely will keep forwarding as the solution here, and generally let users decide what ingress strategy they want

kay-kim · 2024-11-20T02:02:32Z

doc/user/content/self-hosted/_index.md

+     name: materialize-backend
+     namespace: materialize-environment
+   stringData:
+     metadata_backend_url: "postgres:// materialize_user:materialize_pass@postgres.materialize.svc.cluster. local:5432/materialize_db?sslmode=disable"


Wasn't sure if we think we'd have people use postgres as the metadata db (at least in the beginning).
If not ... then, we can make this more general.

(ditto for the blob storage)

👍 this seems reasonable. we could add a comment that these params match if using the sample yamls

kay-kim · 2024-11-20T02:08:25Z

doc/user/content/self-hosted/_index.md

+## Deploying Materialize
+
+### Set up the metadata database
+


Let me know if you want me to stub this with our testing yaml file.

Will add generic content here tomorrow. But wanted to push up quickly the information architecture changes along with the self-hosted -> self-managed nomenclature changes.

kay-kim · 2024-11-20T02:08:30Z

doc/user/content/self-hosted/_index.md

+
+
+### Set up blob storage
+


Let me know if you want me to stub this with our testing yaml file.

seems fine to point to our yaml files for setting up basic metadata db/blob storage

ohoh, now I understand this is on the generic page. hm... maybe we can add an explainer about what's needed for each of these (postgres or cockroach for metadata + s3-compatible blob storage)? and have the more specific install in kind, install on aws pages fill in with more detailed recommendations

kay-kim · 2024-11-20T02:09:15Z

doc/user/content/self-hosted/_index.md

+   apiVersion: v1
+   kind: Namespace
+   metadata:
+     name: materialize-environment


I know we had the discussion about scrubbing "Environment" -- not sure if we actually want to do that with our namespace and such.

I vote we do... IMO it's worth removing the term entirely. From the discussion earlier this week, we can just reference "Materialize" or the Materialize custom resource when possible (and omit "environment"), and "Materialize instance" when we must refer to the resources associated with a specific Materialize CR.

Since we already have a materialize namespace for the operator, is there a preference for this?

kay-kim · 2024-11-20T02:16:20Z

doc/user/content/self-hosted/install-on-local-kind.md

+---
+title: "Appendix: Install locally on kind"
+description: ""
+---


https://preview.materialize.com/materialize/30491/self-hosted/install-on-local-kind/

pH14 · 2024-11-20T18:22:01Z

doc/user/content/self-hosted/_index.md

+   apiVersion: v1
+   kind: Namespace
+   metadata:
+     name: materialize-environment


I vote we do... IMO it's worth removing the term entirely. From the discussion earlier this week, we can just reference "Materialize" or the Materialize custom resource when possible (and omit "environment"), and "Materialize instance" when we must refer to the resources associated with a specific Materialize CR.

pH14 · 2024-11-20T18:23:02Z

doc/user/content/self-hosted/_index.md

+
+
+### Set up blob storage
+


seems fine to point to our yaml files for setting up basic metadata db/blob storage

pH14 · 2024-11-20T18:23:49Z

doc/user/content/self-hosted/_index.md

+     name: materialize-backend
+     namespace: materialize-environment
+   stringData:
+     metadata_backend_url: "postgres:// materialize_user:materialize_pass@postgres.materialize.svc.cluster. local:5432/materialize_db?sslmode=disable"


👍 this seems reasonable. we could add a comment that these params match if using the sample yamls

pH14 · 2024-11-20T18:25:14Z

doc/user/content/self-hosted/install-on-local-kind.md

+              kind: InitConfiguration
+              nodeRegistration:
+                  kubeletExtraArgs:
+                    node-labels: "materialize.cloud/disk=true,workload=materialize-instance"


these shouldn't be needed now

pH14 · 2024-11-20T18:26:23Z

doc/user/content/self-hosted/install-on-local-kind.md

+        kubectl apply -f misc/helm-charts/testing/minio.yaml   
+        ```
+
+1. Install the following metrics service:


metrics-server should be optional... it's needed for some system table metrics to work, but not blocking to testing materialize

pH14 · 2024-11-20T18:27:14Z

doc/user/content/self-hosted/install-on-local-kind.md

+      service/mzfhj38ptdjs-console   NodePort    10.96.97.5      <none>        9000:30847/TCP
+      ```
+
+   1. Forward the Materialize console service to your local machine:


We discussed this some more on the Cloud team, and likely will keep forwarding as the solution here, and generally let users decide what ingress strategy they want

kay-kim requested a review from a team as a code owner November 15, 2024 00:49

kay-kim marked this pull request as draft November 15, 2024 00:51

kay-kim commented Nov 15, 2024

View reviewed changes

doc/user/content/self-hosted/_index.md Outdated Show resolved Hide resolved

kay-kim commented Nov 15, 2024

View reviewed changes

kay-kim force-pushed the docs-self-hosted-draft-for-meeting branch from feea0d6 to d82a1c0 Compare November 15, 2024 01:01

kay-kim commented Nov 15, 2024

View reviewed changes

pH14 reviewed Nov 15, 2024

View reviewed changes

docs: Draft docs for self-hosted (WIP)

3495b49

kay-kim force-pushed the docs-self-hosted-draft-for-meeting branch from d82a1c0 to fa0f19e Compare November 20, 2024 01:47

kay-kim commented Nov 20, 2024

View reviewed changes

docs: Add install locally on kind

10a65be

kay-kim force-pushed the docs-self-hosted-draft-for-meeting branch from fa0f19e to 10a65be Compare November 20, 2024 01:56

kay-kim commented Nov 20, 2024

View reviewed changes

pH14 reviewed Nov 20, 2024

View reviewed changes

Add install on minikube + rename to self managed + reorg

1571ceb

kay-kim force-pushed the docs-self-hosted-draft-for-meeting branch from 929ef2a to 1571ceb Compare November 22, 2024 03:01

kay-kim added 3 commits November 21, 2024 22:18

fix lint errors

dba82c1

rm required column from config table

e49bd3e

add robots=noindex,nofollow to self-managed docs

74171d8


		You can configure the Materialize operator chart. For example:

		- RBAC

		@@ -0,0 +1,242 @@
		---
		title: "Materialize Kubernetes Operator"

		@@ -0,0 +1,61 @@
		---
		title: "Materialize Operator Configuration"


		This tutorial uses `kubectl`. To install, refer to the [`kubectl` documentationq](https://kubernetes.io/docs/tasks/tools/).

		### Kubernetes Storage Configuration

docs: Draft docs for self-hosted (WIP) #30491

Are you sure you want to change the base?

docs: Draft docs for self-hosted (WIP) #30491

Conversation

kay-kim commented Nov 15, 2024 • edited Loading

kay-kim commented Nov 15, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kay-kim Nov 15, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pH14 commented Nov 15, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kay-kim Nov 20, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kay-kim Nov 20, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pH14 Nov 20, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kay-kim commented Nov 15, 2024 •

edited

Loading

kay-kim Nov 15, 2024 •

edited

Loading

kay-kim Nov 20, 2024 •

edited

Loading

kay-kim Nov 20, 2024 •

edited

Loading

pH14 Nov 20, 2024 •

edited

Loading