Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cannot create database in a k8s deployment #1614

Open
brent-artesion opened this issue May 23, 2024 · 3 comments
Open

Cannot create database in a k8s deployment #1614

brent-artesion opened this issue May 23, 2024 · 3 comments
Assignees

Comments

@brent-artesion
Copy link

ArcadeDB Version:

v.24.5.1-SNAPSHOT (build 6841fe3/1716312977604/main)

OS and JDK Version:

Linux 5.15.0-105-generic - OpenJDK 64-Bit Server VM 11.0.23 (Temurin-11.0.23+9)

Expected behavior

Database created

Actual behavior

Error message: "Internal error Cannot create directory '/mnt/data0/databases/test'

Steps to reproduce

Kubernetes deployment

  • Access the admin ui
  • Login as root
  • Navigate to the database screen
  • Click "+ Create" button
  • Create New Database dialog screen
  • Enter database name "test"
  • Click "Send"

My cluster is k3s installed on a three node on-prem cluster. The above error occurs when I use a Longhorn storage class.

I get different error when I use local storage classes of local-path or openebs-hostpath. In those cases the error is:

"Internal error class com.arcadedb.database.LocalDatabase cannot be cast to class com.arcadedb.server.ha.ReplicatedDatabase (com.arcadedb.database.LocalDatabase and com.arcadedb.server.ha.ReplicatedDatabase are in unnamed module of loader 'app')"

@lvca
Copy link
Contributor

lvca commented May 23, 2024

Do you have the full stack trace of the exception?

@lvca lvca self-assigned this May 23, 2024
@brent-artesion
Copy link
Author

brent-artesion commented May 24, 2024

There seems to be two different issues and perhaps you'd want to split them into two separate issues.

  1. This is the exception when using a longhorn storage class for the pvc

here's what's in the logs when the pods start up for the first time

│ arcadedb-0 2024-05-24 18:06:52.773 INFO  [ArcadeDBServer] <arcadedb-0> Creating default database 'Universe'...Exception in thread "main" com.arcadedb.exception.DatabaseOperationE │
│ xception: Cannot create directory '/mnt/data0/databases/Universe'                                                                                                                  │
│ arcadedb-0     at com.arcadedb.database.LocalDatabase.create(LocalDatabase.java:205)                                                                                               │
│ arcadedb-0     at com.arcadedb.database.DatabaseFactory.create(DatabaseFactory.java:90)                                                                                            │
│ arcadedb-0     at com.arcadedb.server.ArcadeDBServer.createDatabase(ArcadeDBServer.java:355)                                                                                       │
│ arcadedb-0     at com.arcadedb.server.ArcadeDBServer.loadDefaultDatabases(ArcadeDBServer.java:577)                                                                                 │
│ arcadedb-0     at com.arcadedb.server.ArcadeDBServer.start(ArcadeDBServer.java:158)                                                                                                │
│ arcadedb-0     at com.arcadedb.server.ArcadeDBServer.main(ArcadeDBServer.java:106)                                                                                                               

Also, when I exec into the pod/container only /mnt/data0 exists. i.e. databases directory is not there

  1. This is the exception creating the database when using a local-path or a openebs-hostpath
2024-05-24 18:19:44.040 INFO  [ArcadeDBServer] <arcadedb-1> Studio web tool available at http://arcadedb-1.arcadedb.default.svc.cluster.local:2480 <arcadedb-1> Error on command execution (PostServerCommandHandler)
java.lang.ClassCastException: class com.arcadedb.database.LocalDatabase cannot be cast to class com.arcadedb.server.ha.ReplicatedDatabase (com.arcadedb.database.LocalDatabase and com.arcadedb.server.ha.ReplicatedDatabase are in unnamed module of loader 'app')
	at com.arcadedb.server.http.handler.PostServerCommandHandler.createDatabase(PostServerCommandHandler.java:170)
	at com.arcadedb.server.http.handler.PostServerCommandHandler.execute(PostServerCommandHandler.java:93)
	at com.arcadedb.server.http.handler.AbstractServerHttpHandler.handleRequest(AbstractServerHttpHandler.java:127)
	at io.undertow.server.Connectors.executeRootHandler(Connectors.java:393)
	at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:859)
	at org.jboss.threads.ContextHandler$1.runWith(ContextHandler.java:18)
	at org.jboss.threads.EnhancedQueueExecutor$Task.run(EnhancedQueueExecutor.java:2513)
	at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1538)
	at org.xnio.XnioWorker$WorkerThreadFactory$1$1.run(XnioWorker.java:1282)
	at java.base/java.lang.Thread.run(Thread.java:829)

What also shows up in the logs and IDK if it's related or not is an error electing a leader, maybe?

2024-05-24 18:19:48.641 INFO  [HAServer] Starting election of local server asking for votes from [arcadedb-0.arcadedb.default.svc.cluster.local:2424] (turn=1 retry=0 lastReplicationMessage=-1 configuredServers=1 majorityOfVotes=1)Error contacting server arcadedb-0.arcadedb.default.svc.cluster.local:2424 for election
java.net.UnknownHostException: arcadedb-0.arcadedb.default.svc.cluster.local
	at java.base/java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:229)
	at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
	at java.base/java.net.Socket.connect(Socket.java:609)
	at com.arcadedb.network.binary.ChannelBinaryClient.<init>(ChannelBinaryClient.java:46)
	at com.arcadedb.server.ha.HAServer.createNetworkConnection(HAServer.java:1008)
	at com.arcadedb.server.ha.HAServer.startElection(HAServer.java:1116)
	at java.base/java.lang.Thread.run(Thread.java:829)

To get this to deploy on a local, on-prem, cluster I did change the arcadedb-http service type to ClusterIP because LoadBalancer doesn't work in such a deploy (k3s) where we use a reverse proxy to expose services.

@yurymkomarov
Copy link

yurymkomarov commented Jun 25, 2024

@brent-artesion

  1. you need to set spec.template.spec.securityContext.fsGroup: 1000

  2. pass all flags as ARCADEDB_SETTINGS variable, cause for some reasons service ignores them from command (you can see that in init log with list of all params)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants