Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add Nomadic's and BCD tezos indexers #166

Closed
wants to merge 37 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
37 commits
Select commit Hold shift + click to select a range
42a174d
add Nomadic's and BCD tezos indexers
nicolasochem Mar 24, 2021
0e5dbc8
Merge branch 'master' into indexer_new
nicolasochem Mar 25, 2021
9ff649e
restore to upstream version of bcd in anticipation for the release
nicolasochem Mar 25, 2021
7ea5e98
Merge branch 'master' into indexer_new
nicolasochem Mar 25, 2021
c1b17c5
remove dep for now
nicolasochem Mar 25, 2021
3910c59
bump bcdhub version
nicolasochem Mar 25, 2021
8181980
make storage size parametrizable
nicolasochem Mar 25, 2021
c8d25de
Merge branch 'size_param' into indexer_new
nicolasochem Mar 25, 2021
c60057d
hardcode space requirements for indexing mainnet for now
nicolasochem Mar 25, 2021
2e8697e
postgres does not want data folder to be an entire mount point:
nicolasochem Mar 26, 2021
eee2ac1
after last bcd release, it's no longer needed to rebuild
nicolasochem Apr 1, 2021
d243424
Merge branch 'master' into indexer_new
nicolasochem Apr 6, 2021
9ce5ebc
templatize and remove hardcoding
nicolasochem Apr 6, 2021
14e0491
hardcode v9.0-rc1 for edo2
nicolasochem Apr 6, 2021
15d255b
name rpc port
nicolasochem Apr 8, 2021
4053ca5
target node for rpc service
nicolasochem Apr 8, 2021
29a2013
set uid/gid to 1000 to make elastic work on eks
nicolasochem Apr 9, 2021
56989c5
fix rabbit perm issue on eks
nicolasochem Apr 9, 2021
02a4bb3
bump bcd to latest version 3.5.2
nicolasochem Apr 9, 2021
f2723a6
Revert "bump bcd to latest version 3.5.2"
nicolasochem Apr 9, 2021
bd1c8cf
Make serviceName of indexer yamls empty str
harryttd Apr 9, 2021
5643fe3
Don't create statefulsets and headless svc if replicas = 0
harryttd Apr 9, 2021
a0b8984
Merge pull request #2 from tqtezos/indexer-updates
nicolasochem Apr 13, 2021
1dda418
Expose bcd api port
harryttd Apr 20, 2021
d85ebf9
Add bcd metrics container to indexer pod
harryttd Apr 20, 2021
3a7a7b2
Create a shared volume used by some bcd indexer containers
harryttd Apr 20, 2021
fe9b4f8
Remove unused configmap
harryttd Apr 20, 2021
6994b50
Change naming of indexer props in values.yaml
harryttd Apr 20, 2021
cca2e0b
Make default indexer bcd
harryttd Apr 22, 2021
6ef7354
Remove appType label from indexer statefulsets
harryttd Apr 20, 2021
c8420ac
Make rpc service select on appType=tezos-node
harryttd Apr 20, 2021
c38bc06
Add imagePullPolicy=IfNotPresent to some containers
harryttd Apr 22, 2021
02c674e
Update template functions that determine which indexer to deploy
harryttd Apr 22, 2021
2fc5ff8
Add bcd service
harryttd Apr 22, 2021
9d58717
Merge pull request #3 from tqtezos/bcd-indexer-updates
nicolasochem Apr 22, 2021
79019bb
Templatize storageClassName for bcd indexer
harryttd Apr 23, 2021
03f587e
Merge pull request #4 from tqtezos/bcd-storage-class
nicolasochem Apr 23, 2021
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 6 additions & 6 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -160,7 +160,7 @@ Running either of these commands results in:
You can find your node in the tqtezos namespace with some status information using `kubectl`.

```shell
kubectl -n tqtezos get pods -l appType=tezos
kubectl -n tqtezos get pods -l appType=tezos-node
```

You can monitor (and follow using the `-f` flag) the logs of the snapshot downloader/import container:
Expand All @@ -172,7 +172,7 @@ kubectl logs -n tqtezos statefulset/tezos-node -c snapshot-downloader -f
You can view logs for your node using the following command:

```shell
kubectl -n tqtezos logs -l appType=tezos -c tezos-node -f --prefix
kubectl -n tqtezos logs -l appType=tezos-node -c tezos-node -f --prefix
```

IMPORTANT:
Expand Down Expand Up @@ -281,13 +281,13 @@ perform the following tasks:
You can find your node in the tqtezos namespace with some status information using kubectl.

```shell
kubectl -n tqtezos get pods -l appType=tezos
kubectl -n tqtezos get pods -l appType=tezos-node
```

You can view (and follow using the `-f` flag) logs for your node using the following command:

```shell
kubectl -n tqtezos logs -l appType=tezos -c tezos-node -f --prefix
kubectl -n tqtezos logs -l appType=tezos-node -c tezos-node -f --prefix
```

Congratulations! You now have an operational Tezos based permissioned
Expand Down Expand Up @@ -325,7 +325,7 @@ helm upgrade $CHAIN_NAME tqtezos/tezos-chain \

The nodes will start up and establish peer-to-peer connections in a full mesh topology.

List all of your running nodes: `kubectl -n tqtezos get pods -l appType=tezos`
List all of your running nodes: `kubectl -n tqtezos get pods -l appType=tezos-node`

## Adding external nodes to the cluster

Expand Down Expand Up @@ -361,7 +361,7 @@ Congratulations! You now have a multi-node Tezos based permissioned chain.
On each computer, run this command to check that the nodes have matching heads by comparing their hashes (it may take a minute for the nodes to sync up):

```shell
kubectl get pod -n tqtezos -l appType=tezos -o name |
kubectl get pod -n tqtezos -l appType=tezos-node -o name |
while read line;
do kubectl -n tqtezos exec $line -c tezos-node -- /usr/local/bin/tezos-client rpc get /chains/main/blocks/head/hash;
done
Expand Down
2 changes: 1 addition & 1 deletion baker-endorser/Dockerfile
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
FROM tezos/tezos:v8-release
FROM tezos/tezos:v9.0-rc1
RUN sudo apk add jq bash
COPY entrypoint.sh /
ENTRYPOINT ["/entrypoint.sh"]
Expand Down
2 changes: 1 addition & 1 deletion chain-initiator/Dockerfile
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
FROM tezos/tezos:v8-release
FROM tezos/tezos:v9.0-rc1
RUN sudo apk add jq
COPY entrypoint.sh /
ENTRYPOINT ["/entrypoint.sh"]
Expand Down
6 changes: 0 additions & 6 deletions charts/tezos/Chart.lock

This file was deleted.

4 changes: 0 additions & 4 deletions charts/tezos/Chart.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,3 @@ keywords:
- tezos
- blockchain
home: https://github.com/tqtezos/tezos-k8s
dependencies:
- name: authenticated-rpc-tezos-chain
repository: file://../rpc-auth
condition: rpc_auth
59 changes: 59 additions & 0 deletions charts/tezos/templates/_helpers.tpl
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,34 @@
{{- end }}
{{- end }}

{{/*
Don't deploy the baker statefulset and its headless service if
there are no bakers specified.
Returns a string "true" or empty string which is falsey.
*/}}
{{- define "tezos.shouldDeployBakerStatefulset" -}}
{{- $baking_nodes := .Values.nodes.baking | default dict }}
{{- if and (not .Values.is_invitation) ($baking_nodes | len) }}
{{- "true" }}
{{- else }}
{{- "" }}
{{- end }}
{{- end }}

{{/*
Don't deploy the regular node statefulset and its headless service if
there are no regular nodes specified.
Returns a string "true" or empty string which is falsey.
*/}}
{{- define "tezos.shouldDeployRegularNodeStatefulset" -}}
{{- $regular_nodes := .Values.nodes.regular | default dict }}
{{- if ($regular_nodes | len) }}
{{- "true" }}
{{- else }}
{{- "" }}
{{- end }}
{{- end }}

{{/*
Checks if a protocol should be activated. There needs to be a protocol_hash
and protocol_parameters.
Expand All @@ -52,3 +80,34 @@
{{- "" }}
{{- end }}
{{- end }}

{{/*
BCD indexer
Checks if indexer config has an indexer and rpc_url set.
Then checks if indexer name is "bcd".
Returns the true type or empty string which is falsey.
*/}}
{{- define "tezos.shouldDeployBcdIndexer" -}}
{{- $index_config := .Values.indexer | default dict }}
{{- if and $index_config.name $index_config.rpc_url }}
{{- if eq $index_config.name "bcd" }}
{{- "true" }}
{{- else }}
{{- "" }}
{{- end }}
{{- end }}
{{- end }}

{{/*
Nomadic indexer
*/}}
{{- define "tezos.shouldDeployNomadicIndexer" -}}
{{- $index_config := .Values.indexer | default dict }}
{{- if and $index_config.name $index_config.rpc_url }}
{{- if eq $index_config.name "nomadic" }}
{{- "true" }}
{{- else }}
{{- "" }}
{{- end }}
{{- end }}
{{- end }}
4 changes: 2 additions & 2 deletions charts/tezos/templates/baker.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -57,7 +57,7 @@ spec:
name: var-volume
{{ end }}
---
{{- if not .Values.is_invitation }}
{{- if (include "tezos.shouldDeployBakerStatefulset" .) }}
apiVersion: apps/v1
kind: StatefulSet
metadata:
Expand All @@ -74,7 +74,7 @@ spec:
metadata:
labels:
app: {{ .Values.baker_statefulset.name }}
appType: tezos
appType: tezos-node
spec:
containers:
{{- include "tezos.container.node" . | indent 8 }}
Expand Down
145 changes: 145 additions & 0 deletions charts/tezos/templates/bcd_indexer.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,145 @@
{{- if (include "tezos.shouldDeployBcdIndexer" .) }}
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: {{ .Values.bcd_indexer_statefulset.name }}
namespace: {{ .Release.Namespace }}
spec:
podManagementPolicy: Parallel
selector:
matchLabels:
app: {{ .Values.bcd_indexer_statefulset.name }}
serviceName: ""
template:
metadata:
labels:
app: {{ .Values.bcd_indexer_statefulset.name }}
spec:
securityContext:
fsGroup: 1000
containers:
- image: bakingbad/bcdhub-api:3.5
name: api
ports:
- containerPort: 14000
name: bcd
volumeMounts:
- mountPath: /etc/bcd
name: shared-volume
envFrom:
- configMapRef:
name: bcd-env
- configMapRef:
name: db-creds
- image: bakingbad/bcdhub-indexer:3.5
name: indexer
volumeMounts:
- mountPath: /etc/bcd
name: shared-volume
envFrom:
- configMapRef:
name: bcd-env
- image: bakingbad/bcdhub-metrics:3.5
name: metrics
volumeMounts:
- mountPath: /etc/bcd
name: shared-volume
envFrom:
- configMapRef:
name: bcd-env
- configMapRef:
name: db-creds
- image: postgres:alpine
name: db
imagePullPolicy: IfNotPresent
volumeMounts:
- mountPath: /var/lib/postgresql/
name: indexer-db
envFrom:
- configMapRef:
name: db-creds
- image: rabbitmq:latest
name: mq
imagePullPolicy: IfNotPresent
volumeMounts:
- mountPath: /var/lib/rabbitmq/mnesia
name: mq-data
env:
- name: RABBITMQ_DEFAULT_USER
value: guest
- name: RABBITMQ_DEFAULT_PASS
value: guest
- image: bakingbad/bcdhub-elastic:3.5
name: elastic
volumeMounts:
- mountPath: /usr/share/elasticsearch/data
name: esdata
env:
- name: "bootstrap.memory_lock"
value: "true"
- name: "discovery.type"
value: single-node
- name: ES_JAVA_OPTS
value: "-Xms256m -Xmx256m"
volumeClaimTemplates:
- metadata:
name: esdata
spec:
accessModes:
- ReadWriteOnce
storageClassName: {{ .Values.bcd_indexer_statefulset.storageClassName }}
resources:
requests:
storage: "300Gi"
- metadata:
name: mq-data
spec:
accessModes:
- ReadWriteOnce
storageClassName: {{ .Values.bcd_indexer_statefulset.storageClassName }}
resources:
requests:
storage: "5Gi"
- metadata:
name: indexer-db
spec:
accessModes:
- ReadWriteOnce
storageClassName: {{ .Values.bcd_indexer_statefulset.storageClassName }}
resources:
requests:
storage: "300Gi"
- metadata:
name: shared-volume
spec:
accessModes:
- ReadWriteOnce
storageClassName: {{ .Values.bcd_indexer_statefulset.storageClassName }}
resources:
requests:
storage: "100Gi"

---
apiVersion: v1
kind: ConfigMap
metadata:
name: bcd-env
data:
BCD_ENV: sandbox
GIN_MODE: debug
ELASTIC_HOSTNAME: localhost
RABBITMQ_HOSTNAME: localhost
DB_HOSTNAME: 127.0.0.1
RABBITMQ_DEFAULT_USER: guest
RABBITMQ_DEFAULT_PASS: guest
SANDBOX_NODE_URI: {{ .Values.indexer.rpc_url }}
---
apiVersion: v1
kind: ConfigMap
metadata:
name: db-creds
data:
POSTGRES_USER: root
POSTGRES_PASSWORD: root
POSTGRES_DB: bcd
{{- end }}
4 changes: 3 additions & 1 deletion charts/tezos/templates/node.yaml
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
{{- if (include "tezos.shouldDeployRegularNodeStatefulset" .) }}
apiVersion: apps/v1
kind: StatefulSet
metadata:
Expand All @@ -15,7 +16,7 @@ spec:
metadata:
labels:
app: {{ .Values.regular_node_statefulset.name }}
appType: tezos
appType: tezos-node
spec:
containers:
{{- include "tezos.container.node" . | indent 8 }}
Expand Down Expand Up @@ -43,3 +44,4 @@ spec:
resources:
requests:
storage: {{ .Values.regular_node_statefulset.storage_size }}
{{- end }}
Loading