error starting with out of the box configs #21076

tsadigovAgmail · 2022-01-26T15:09:33Z

Summary

trying to start che on ubuntu minikube

authorization server does not start

and I get https result 503 when trying to access che

Relevant information

kubectl logs pod/che-gateway -n eclipse-che -c auth-proxy

[main.go:54] invalid configuration : idc provider requires an oidc issuer url

l0rd · 2022-01-26T23:30:32Z

@tsadigovAgmail please provide more informations:

How are you installing Che (provide the exact commands)?
What version of minikube are you using?
Please share installation logs

tsadigovAgmail · 2022-01-27T03:11:28Z

@l0rd here are the relevant information

What version of minikube are you using?
minikube version: v1.25.1
commit: 3e64b11ed75e56e4898ea85f96b2e4af0301f43d

How are you installing Che (provide the exact commands)?
minikube start --addons=ingress --disk-size=80G --memory=15192 --cpus=5
kubectl apply -f https://github.com/jetstack/cert-manager/releases/download/v1.6.1/cert-manager.yaml
chectl server:deploy --k8spoddownloadimagetimeout=600000 --k8spoderrorrechecktimeout=600000 --k8spodreadytimeout=6000000 --k8spodwaittimeout=6000000 --debug --multiuser --platform minikube

tsadigovAgmail · 2022-01-27T03:23:00Z

out put of the chectl
❯ Eclipse Che Server pod bootstrap
✔ Scheduling...done
✔ Downloading images...done
✖ Starting...failed
→ Failed to start a pod, reason: Error, exitCode: 137

`
$ cat /home/tural/.cache/chectl/error.log

2022-01-26T22:01:16.203Z Error: Command server:deploy failed. Error log: /home/tural/.cache/chectl/error.log.
2022-01-26T22:01:16.203Z at newError (/usr/local/lib/chectl/lib/util.js:199:19)
2022-01-26T22:01:16.203Z at Object.wrapCommandError (/usr/local/lib/chectl/lib/util.js:195:12)
2022-01-26T22:01:16.203Z at Deploy. (/usr/local/lib/chectl/lib/commands/server/deploy.js:226:35)
2022-01-26T22:01:16.203Z at Generator.throw ()
2022-01-26T22:01:16.203Z at rejected (/usr/local/lib/chectl/node_modules/tslib/tslib.js:115:69)
2022-01-26T22:01:16.203Z at runMicrotasks ()
2022-01-26T22:01:16.203Z Cause: Error: Failed to start a pod, reason: Error, exitCode: 137
2022-01-26T22:01:16.203Z at KubeTasks. (/usr/local/lib/chectl/lib/tasks/kube.js:134:35)
2022-01-26T22:01:16.203Z at Generator.next ()
2022-01-26T22:01:16.203Z at fulfilled (/usr/local/lib/chectl/node_modules/tslib/tslib.js:114:62)
2022-01-26T22:01:16.203Z at runMicrotasks ()
`

`
$ kubectl get pods -n eclipse-che

NAME READY STATUS RESTARTS AGE
che-84f54f777d-lmfs8 0/1 Running 42 (3m59s ago) 5h27m
che-dashboard-6db7fddfd-wcvhb 1/1 Running 0 5h27m
che-gateway-7cc8f594c8-2gmrs 3/4 CrashLoopBackOff 67 (80s ago) 5h27m
che-operator-79f57554b5-jk9ff 1/1 Running 1 (5h30m ago) 5h31m
che-tls-job-2rftl 0/1 Completed 0 6h15m
che-tls-job-7rnzg 0/1 Completed 0 5h36m
che-tls-job-7xcc4 0/1 Completed 0 5h31m
che-tls-job-qvzff 0/1 Completed 0 6h7m
devfile-registry-7bf49dbf4b-s7kbk 1/1 Running 0 5h28m
plugin-registry-67c8c7d8f-f466c 1/1 Running 0 5h28m
postgres-588b8fb5bf-ghpcb 1/1 Running 0 5h28m`

`
$kubectl logs che-gateway-7cc8f594c8-2gmrs -n eclipse-che -c oauth-proxy

[2022/01/27 03:19:18] [main.go:54] invalid configuration:
oidc provider requires an oidc issuer URL`

tolusha · 2022-01-27T08:02:30Z

Pls, provide a full chectl installation log

tsadigovAgmail · 2022-01-28T01:42:22Z

@tolusha started from ground up with new machine

here is the output

chectl server:deploy --k8spoddownloadimagetimeout=60000000 --k8spoderrorrechecktimeout=60000000 --k8spodreadytimeout=60000000 --k8spodwaittimeout=60000000 --platform=minikube
› Installer type is set to: 'operator'
› Current Kubernetes context: 'minikube'
✔ Verify Kubernetes API...OK
✔ 👀 Looking for an already existing Eclipse Che instance
✔ Verify if Eclipse Che is deployed into namespace "eclipse-che"...it is not
✔ ✈️ Minikube preflight checklist
✔ Verify if kubectl is installed
✔ Verify if minikube is installed
✔ Verify if minikube is running
↓ Start minikube [skipped]
→ Minikube is already running.
✔ Check Kubernetes version: Found v1.23.1.
✔ Verify if minikube ingress addon is enabled
↓ Enable minikube ingress addon [skipped]
→ Ingress addon is already enabled.
✔ Retrieving minikube IP and domain for ingress URLs...192.168.49.2.nip.io.
✔ Checking minikube version... 1.25.1
↓ Check if cluster accessible [skipped]
✔ Following Eclipse Che logs
✔ Start following Operator logs...done
✔ Start following Eclipse Che Server logs...done
✔ Start following PostgreSQL logs...done
✔ Start following Keycloak logs...done
✔ Start following Plug-in Registry logs...done
✔ Start following Devfile Registry logs...done
✔ Start following Eclipse Che Dashboard logs...done
✔ Start following namespace events...done
⠙ Create Namespace eclipse-che
✔ Create Namespace eclipse-che...[Exists]
✔ 🏃‍ Running the Eclipse Che operator
✔ Create ServiceAccount che-operator in namespace eclipse-che...done.
✔ Read Roles and Bindings...done.
✔ Creating Roles and Bindings...done.
✔ Create CRD checlusters.org.eclipse.che...done.
✔ Create backup and restore CRDs...done.
✔ Waiting 5 seconds for the new Kubernetes resources to get flushed...done.
✔ Create deployment che-operator in namespace eclipse-che...done.
✔ Operator pod bootstrap
✔ Scheduling...done
✔ Downloading images...done
✔ Starting...done
✔ Prepare Eclipse Che cluster CR...Done.
✔ Create the Custom Resource of type checlusters.org.eclipse.che in the namespace eclipse-che...done.
❯ ✅ Post installation checklist
✔ PostgreSQL pod bootstrap
✔ Scheduling...done
✔ Downloading images...done
✔ Starting...done
✔ Keycloak pod bootstrap...skipped
✔ Devfile Registry pod bootstrap
✔ Scheduling...done
✔ Downloading images...done
✔ Starting...done
✔ Plug-in Registry pod bootstrap
✔ Scheduling...done
✔ Downloading images...done
✔ Starting...done
✔ Eclipse Che Dashboard pod bootstrap
✔ Scheduling...done
✔ Downloading images...done
✔ Starting...done
❯ Eclipse Che Server pod bootstrap
✔ Scheduling...done
✔ Downloading images...done
✖ Starting...failed
→ Failed to start a pod, reason: Error, exitCode: 137
Eclipse Che status check
Retrieving Che self-signed CA certificate
Prepare post installation output
Error: Command server:deploy failed. Error log: /home/tural/.cache/chectl/error.log.

tolusha · 2022-01-28T11:14:13Z

Dex is supposed to be deployed on Minibube but I don't see that in the output.
I can't find the chectl version, so try chectl update next or chectl update stable (depending on versions you prefer)
then:

chectl server:delete -y --delete-namespace
chectl server:deploy --platform minikube

tsadigovAgmail · 2022-01-28T12:56:07Z

chectl version is up to date stable version 7.42.0. I will try with chectl update next

tsadigovAgmail · 2022-01-28T14:58:57Z

chectl version
chectl/0.0.20220128-next.0a7174a

› Installer type is set to: 'operator'
› Current Kubernetes context: 'minikube'
✔ Verify Kubernetes API...OK
✔ 👀 Looking for an already existing Eclipse Che instance
✔ Verify if Eclipse Che is deployed into namespace "eclipse-che"...it is not
↓ Check if OIDC Provider installed [skipped]
→ Dex will be automatically installed as OIDC Identity Provider
✔ 🧪 DevWorkspace engine
✔ Verify cert-manager installation
✔ Check Cert Manager deployment...already deployed
✔ Wait for Cert Manager...ready
✔ ✈️ Minikube preflight checklist
✔ Verify if kubectl is installed
✔ Verify if minikube is installed
✔ Verify if minikube is running
↓ Start minikube [skipped]
→ Minikube is already running.
✔ Check Kubernetes version: Found v1.23.1.
✔ Verify if minikube ingress addon is enabled
↓ Enable minikube ingress addon [skipped]
→ Ingress addon is already enabled.
✔ Retrieving minikube IP and domain for ingress URLs...192.168.49.2.nip.io.
✔ Checking minikube version... 1.25.1
↓ Check if cluster accessible [skipped]
✔ Following Eclipse Che logs
✔ Start following Operator logs...done
✔ Start following Eclipse Che Server logs...done
✔ Start following PostgreSQL logs...done
✔ Start following Keycloak logs...done
✔ Start following Plug-in Registry logs...done
✔ Start following Devfile Registry logs...done
✔ Start following Eclipse Che Dashboard logs...done
✔ Start following namespace events...done
✔ Create Namespace eclipse-che...[Exists]
❯ Deploy Dex
✔ Create namespace: dex...[Exists]
✔ Provide Dex certificate
✔ Check Cert Manager deployment...already deployed
✔ Wait for Cert Manager...ready
✔ Check Cert Manager CA certificate...already exists
✔ Set up Eclipse Che certificates issuer...done
✔ Request self-signed certificate...done
✔ Wait for self-signed certificate...ready
✔ Read Dex certificate...[OK]
✔ Save Dex certificate...[OK: /tmp/dex-ca.crt]
✔ Add Dex certificate to Eclipse Che certificates bundle...[OK]
✔ Create Dex service account...[OK]
✔ Create Dex cluster role...[OK]
✔ Create Dex cluster role binding...[OK]
✔ Create Dex service...[OK]
✔ Create Dex ingress...[OK]
✔ Generate Dex username and password...[OK: admin:admin]
✔ Create Dex configmap...[OK]
✔ Create Dex deployment...[OK]
✔ Wait for Dex is ready...[OK]
❯ Configure API server
✔ Create /etc/ca-certificates directory...[OK]
✔ Copy Dex certificate into Minikube...[OK]
✖ Configure Minikube API server
→ - kubelet.housekeeping-interval=5m
Wait for Minikube API server
🏃‍ Running the Eclipse Che operator
Error: Command server:deploy failed. Error log: /home/tural/.cache/chectl/error.log.

tolusha · 2022-01-28T15:08:41Z

Pls. re run the same command. It was timeouted to configure minikube API server

tsadigovAgmail · 2022-01-28T15:30:14Z

I have already tried it 3 times
how can I increase timeout duration?

tsadigovAgmail · 2022-01-28T17:01:13Z

I delete and start minikube
And I get this

chectl server:deploy --platform=minikube --k8spoddownloadimagetimeout=6000000 --k8spoderrorrechecktimeout=600000 --k8spodreadytimeout=6000000 --k8spodwaittimeout=6000000
› Installer type is set to: 'operator'
› Current Kubernetes context: 'minikube'
✔ Verify Kubernetes API...OK
✔ 👀 Looking for an already existing Eclipse Che instance
✔ Verify if Eclipse Che is deployed into namespace "eclipse-che"...it is not
↓ Check if OIDC Provider installed [skipped]
→ Dex will be automatically installed as OIDC Identity Provider
✔ 🧪 DevWorkspace engine
✔ Verify cert-manager installation
✔ Check Cert Manager deployment...already deployed
✔ Wait for Cert Manager...ready
✔ ✈️ Minikube preflight checklist
✔ Verify if kubectl is installed
✔ Verify if minikube is installed
✔ Verify if minikube is running
↓ Start minikube [skipped]
→ Minikube is already running.
✔ Check Kubernetes version: Found v1.23.1.
✔ Verify if minikube ingress addon is enabled
↓ Enable minikube ingress addon [skipped]
→ Ingress addon is already enabled.
✔ Retrieving minikube IP and domain for ingress URLs...192.168.49.2.nip.io.
✔ Checking minikube version... 1.25.1
↓ Check if cluster accessible [skipped]
✔ Following Eclipse Che logs
✔ Start following Operator logs...done
✔ Start following Eclipse Che Server logs...done
✔ Start following PostgreSQL logs...done
✔ Start following Keycloak logs...done
✔ Start following Plug-in Registry logs...done
✔ Start following Devfile Registry logs...done
✔ Start following Eclipse Che Dashboard logs...done
✔ Start following namespace events...done
✔ Create Namespace eclipse-che...[Exists]
❯ Deploy Dex
✔ Create namespace: dex...[Exists]
✔ Provide Dex certificate
✔ Check Cert Manager deployment...already deployed
✔ Wait for Cert Manager...ready
✔ Check Cert Manager CA certificate...already exists
✔ Set up Eclipse Che certificates issuer...done
✔ Request self-signed certificate...done
✔ Wait for self-signed certificate...ready
✔ Read Dex certificate...[OK]
✔ Save Dex certificate...[OK: /tmp/dex-ca.crt]
✔ Add Dex certificate to Eclipse Che certificates bundle...[OK]
✔ Create Dex service account...[OK]
✔ Create Dex cluster role...[OK]
✔ Create Dex cluster role binding...[OK]
✔ Create Dex service...[OK]
✔ Create Dex ingress...[OK]
✔ Generate Dex username and password...[OK: admin:admin]
✔ Create Dex configmap...[OK]
✔ Create Dex deployment...[OK]
✔ Wait for Dex is ready...[OK]
❯ Configure API server
✔ Create /etc/ca-certificates directory...[OK]
✔ Copy Dex certificate into Minikube...[OK]
✖ Configure Minikube API server
→ - kubelet.housekeeping-interval=5m
Wait for Minikube API server
🏃‍ Running the Eclipse Che operator
Error: Command server:deploy failed. Error log: /home/tural/.cache/chectl/error.log.

tural@Ubuntu20Desktop:~/Desktop$ chectl server:deploy --platform=minikube --k8spoddownloadimagetimeout=6000000 --k8spoderrorrechecktimeout=600000 --k8spodreadytimeout=6000000 --k8spodwaittimeout=6000000
› Installer type is set to: 'operator'
› Current Kubernetes context: 'minikube'
✔ Verify Kubernetes API
❯ 👀 Looking for an already existing Eclipse Che instance
✖ Verify if Eclipse Che is deployed into namespace "eclipse-che"
→ connect ECONNREFUSED 192.168.49.2:8443
Check if OIDC Provider installed
🧪 DevWorkspace engine
Error: Command server:deploy failed. Error log: /home/tural/.cache/chectl/error.log. Eclipse Che logs: /tmp/chectl-logs/1643388927386.

tsadigovAgmail · 2022-01-28T17:08:51Z

If I try running the commands you provided at this step I get error

tural@Ubuntu20Desktop:~$ chectl server:delete -y --delete-namespace
› Current Kubernetes context: 'minikube'
✔ Verify Kubernetes API
✔ Delete oauthClientAuthorizations...OK
✖ Delete the Custom Resource of type checlusters.org.eclipse.che
→ connect ECONNREFUSED 192.168.49.2:8443
Delete CRDs
Delete Roles and Bindings
Delete service accounts che-operator
Delete PVC che-operator
Check if OLM is pre-installed on the platform
Check if operator is installed
Delete(OLM) custom catalog source eclipse-che-custom-catalog-source
Delete(OLM) nigthly catalog source eclipse-che-preview
Delete role prometheus-k8s
Delete role binding prometheus-k8s
Delete all deployments
Delete all services
Delete all ingresses
Delete configmaps for Eclipse Che server and operator
Delete rolebindings che, che-workspace-exec and che-workspace-view
Delete service accounts che, che-workspace
Delete PVCs
Delete consoleLink che
Wait until Eclipse Che Server pod is deleted
Wait until Eclipse Che Dashboard pod is deleted
Wait until Keycloak pod is deleted
Wait until PostgreSQL pod is deleted
Wait until Devfile Registry pod is deleted
Wait until Plug-in Registry pod is deleted
Uninstall Dev Workspace Operator
Delete namespace eclipse-che
Error: Command server:delete failed. Error log: /home/tural/.cache/chectl/error.log. Eclipse Che logs: /tmp/chectl-logs/1643389696121.

tsadigovAgmail · 2022-01-28T17:23:45Z

Is it ok that stable and next versions flow differ that much for chectl?
I just want to have a working setup
If there is a known working combination of chectl + OS + minikube version I am happy to use it

tsadigovAgmail · 2022-01-29T23:44:32Z

I decided to try all options without going into details, to see if there is a painless option to start

Ubuntu+minikube nope
Ubuntu+minishift nope
Ubuntu+CRC nope
Fedora +minishift no luck
With so many failing options I start to suspect there is something wrong with me :)
Centos+CRC seems to be installed. I hope it works. Both che dashboard and openshift get stuck in loading screen.

tolusha · 2022-01-31T10:28:50Z

I will try look into (minikube + Ubuntu)
It seems the latest version of the minikube requires more time to reconfigure its API sever

tsadigovAgmail · 2022-01-31T21:27:34Z

thanks @tolusha
what do you think is it also somewhat related to the oauth-proxy container in gateway pod not starting?

che-gateway-7cc8f594c8-2gmrs 3/4 CrashLoopBackOff 67 (80s ago) 5h27m

$kubectl logs che-gateway-7cc8f594c8-2gmrs -n eclipse-che -c oauth-proxy
[2022/01/27 03:19:18] [main.go:54] invalid configuration:
oidc provider requires an oidc issuer URL

tsadigovAgmail · 2022-02-01T01:57:26Z

I am still trying variations and I have an chectl serverv:deploy run without error
Chectl:stable + Ubuntu + minikube:driver=none

No error messages on deploy. BUT The output still has nothing about Dex, the oauth-proxy container under che-gateway pod is not running. And when checking the url https://192.168.178.150.nlp.io in the browser I get nginx 503 temporary unavailable error.

Not sure if I am missing some additional step or installation is problematic

tolusha · 2022-02-01T08:49:43Z

Ubuntu + minikube:drive does not work. That's a know issue. #20969

what do you think is it also somewhat related to the oauth-proxy container in gateway pod not starting?

We did not configure something properly. I will look deeply to find the cause.

tsadigovAgmail · 2022-02-02T13:18:03Z

@tolusha for my scenario I would prefer to have a suggestion on working versions I should use, rather than disabling some functionality. As far as I understand is was working fine in che version 7.40. Should I also use specific version of minikube?

I am going to demo this to the team. So I need to have authentication.

tolusha · 2022-02-02T14:34:51Z

@tsadigovAgmail

Pls try working configuation:

minikube v1.24.0
chectl 7.42.0 (or next)

tsadigovAgmail · 2022-02-05T13:35:53Z

@tolusha no luck
I am trying to set up a k8s cluster to try there

tsadigovAgmail · 2022-02-07T21:21:26Z

@tolusha the documents have different page for each --platform. But I can not find any documentation for --platform=k8s. Where can I look for it?

tolusha · 2022-02-10T11:42:58Z

There is no difference between --platform k8s and --platform minkube in term of depoying Eclipse Che.
The only difference are preflight checks:

tolusha · 2022-02-10T11:48:01Z

Which issue did you have with this config #21076 (comment)?

tsadigovAgmail · 2022-03-07T16:27:37Z

I tried with next 2 versions of chectl and with different driver=docker for minikube, it works like a charm. I will not go over trying other parameter options.
Great job.

hellomaybe · 2022-06-07T10:53:54Z

too much bug on this eclipese-che ,just dont use it

thenets · 2022-11-08T16:45:31Z

I got exactly the same bug.

# Version
[user@domain ~]# chectl --version
chectl/7.56.0 linux-x64 node-v16.13.2


# Installation
[user@domain ~]# chectl server:deploy -p=k8s --skip-oidc-provider-check --domain=my.domain.com


# Waiting the installation
[...]
    ❯ Eclipse Che Server pod bootstrap
      ✔ Scheduling...[OK]
      ✔ Downloading images...[OK]
      ⠧ Starting <----- froze here
      Eclipse Che status check


# Retrieving the logs
[user@domain ~]# kubectl -n eclipse-che logs pod/che-gateway-595f746df9-jhlnh oauth-proxy
[2022/11/08 16:36:48] [main.go:54] invalid configuration:
  provider missing setting: client-id
  missing setting: client-secret or client-secret-file
  oidc provider requires an oidc issuer URL

@tolusha @tsadigovAgmail I believe the --skip-oidc-provider-check flag doesn't work, despite existing.

tolusha · 2022-11-09T13:10:19Z

--skip-oidc-provider-check simply does not check if OIDC provider is configured on the cluster.
It does nothing else. It is admin responsibility to configure OIDC provider and ask chectl to use it:

Prepare patch file

cat >>cr-patch.yaml <<EOF
apiVersion: org.eclipse.che/v2
spec:
  networking:
    auth:
      identityProviderURL: <IDENTITY_PROVIDER_URL>
      oAuthClientName: <CLIENT_ID>
      oAuthSecret: <CLIENT_SECRET>
EOF

Deploy Eclipse Che

chectl server:deploy --platform k8s --che-operator-cr-patch-yaml cr-patch.yaml --skip-oidc-provider-check --domain <DOMAIN>

Related issue #21759

tsadigovAgmail added the kind/question Questions that haven't been identified as being feature requests or bugs. label Jan 26, 2022

che-bot added the status/need-triage An issue that needs to be prioritized by the curator responsible for the triage. See https://github. label Jan 26, 2022

l0rd added status/info-needed More information is needed before the issue can move into the “analyzing” state for engineering. and removed status/need-triage An issue that needs to be prioritized by the curator responsible for the triage. See https://github. labels Jan 26, 2022

tolusha added the area/install Issues related to installation, including offline/air gap and initial setup label Jan 27, 2022

tolusha added sprint/current team/deploy and removed status/info-needed More information is needed before the issue can move into the “analyzing” state for engineering. labels Jan 31, 2022

tolusha mentioned this issue Jan 31, 2022

Che Deploy Sprint #214 #21094

Closed

18 tasks

tolusha added this to the 7.44 milestone Feb 1, 2022

tolusha mentioned this issue Feb 2, 2022

Disable authentication when Eclipse Che is deployed on minikube or docker-desktop platforms #21107

Closed

tolusha mentioned this issue Feb 21, 2022

Che Deploy Sprint #215 #21190

Closed

4 tasks

tolusha removed sprint/current team/deploy labels Feb 22, 2022

tolusha removed this from the 7.44 milestone Feb 22, 2022

tsadigovAgmail closed this as completed Mar 7, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

error starting with out of the box configs #21076

error starting with out of the box configs #21076

tsadigovAgmail commented Jan 26, 2022

l0rd commented Jan 26, 2022

tsadigovAgmail commented Jan 27, 2022 •

edited

Loading

tsadigovAgmail commented Jan 27, 2022 •

edited

Loading

tolusha commented Jan 27, 2022

tsadigovAgmail commented Jan 28, 2022

tolusha commented Jan 28, 2022 •

edited

Loading

tsadigovAgmail commented Jan 28, 2022

tsadigovAgmail commented Jan 28, 2022

tolusha commented Jan 28, 2022

tsadigovAgmail commented Jan 28, 2022

tsadigovAgmail commented Jan 28, 2022 •

edited

Loading

tsadigovAgmail commented Jan 28, 2022

tsadigovAgmail commented Jan 28, 2022

tsadigovAgmail commented Jan 29, 2022

tolusha commented Jan 31, 2022

tsadigovAgmail commented Jan 31, 2022

tsadigovAgmail commented Feb 1, 2022

tolusha commented Feb 1, 2022

tsadigovAgmail commented Feb 2, 2022

tolusha commented Feb 2, 2022

tsadigovAgmail commented Feb 5, 2022 •

edited

Loading

tsadigovAgmail commented Feb 7, 2022

tolusha commented Feb 10, 2022

tolusha commented Feb 10, 2022

tsadigovAgmail commented Mar 7, 2022

hellomaybe commented Jun 7, 2022

thenets commented Nov 8, 2022

tolusha commented Nov 9, 2022 •

edited

Loading

error starting with out of the box configs #21076

error starting with out of the box configs #21076

Comments

tsadigovAgmail commented Jan 26, 2022

Summary

Relevant information

l0rd commented Jan 26, 2022

tsadigovAgmail commented Jan 27, 2022 • edited Loading

tsadigovAgmail commented Jan 27, 2022 • edited Loading

tolusha commented Jan 27, 2022

tsadigovAgmail commented Jan 28, 2022

tolusha commented Jan 28, 2022 • edited Loading

tsadigovAgmail commented Jan 28, 2022

tsadigovAgmail commented Jan 28, 2022

tolusha commented Jan 28, 2022

tsadigovAgmail commented Jan 28, 2022

tsadigovAgmail commented Jan 28, 2022 • edited Loading

tsadigovAgmail commented Jan 28, 2022

tsadigovAgmail commented Jan 28, 2022

tsadigovAgmail commented Jan 29, 2022

tolusha commented Jan 31, 2022

tsadigovAgmail commented Jan 31, 2022

tsadigovAgmail commented Feb 1, 2022

tolusha commented Feb 1, 2022

tsadigovAgmail commented Feb 2, 2022

tolusha commented Feb 2, 2022

tsadigovAgmail commented Feb 5, 2022 • edited Loading

tsadigovAgmail commented Feb 7, 2022

tolusha commented Feb 10, 2022

tolusha commented Feb 10, 2022

tsadigovAgmail commented Mar 7, 2022

hellomaybe commented Jun 7, 2022

thenets commented Nov 8, 2022

tolusha commented Nov 9, 2022 • edited Loading

tsadigovAgmail commented Jan 27, 2022 •

edited

Loading

tsadigovAgmail commented Jan 27, 2022 •

edited

Loading

tolusha commented Jan 28, 2022 •

edited

Loading

tsadigovAgmail commented Jan 28, 2022 •

edited

Loading

tsadigovAgmail commented Feb 5, 2022 •

edited

Loading

tolusha commented Nov 9, 2022 •

edited

Loading