ALTER DATABASE command does not get replicated and gets OSecurityAccessException that always references 'writer' user

[dalegarv]

OrientDB Version: 2.2.17 Enterprise

Java Version: 1.8

OS: SLES

Expected behavior

Running OrientDB distributed with 2 nodes, an alter database command run from node1 with root or admin or any user with update database permissions should replicate to node2 without an error. For example ALTER DATABASE DATEFORMAT 'yyyy'

Actual behavior

The change is only made on node1 an exception like the following is thrown:
com.orientechnologies.orient.core.exception.OSecurityAccessException: User 'writer' does not have permission to execute the operation 'Update' against the resource: ResourceGeneric [name=DATABASE, legacyName=database].null DB name="Entitlements-BDD"

The exception references user 'writer' no matter what user is actually logged in to node1 and for some reason has ".null" as the database resource. The problem doesn't happen from node2 and it is currently only happening in one database and not others.

Steps to reproduce

1. Setup OrientDB with 2 nodes
2. Run node1 in single mode and use the import command to create a database
3. Start node1 and 2 in distributed mode which will replicate the database
4. Run an alter database command from node1
   We have 2 databases that were created this way, and only 1 has the problem and the problem only happens on node1 so it may not be easy to reproduce.

[santo-it]

Hi,

Thanks for opening this issue. I have tried to reproduce it on Window 10, with a 4-nodes cluster but wasn't able.

I executed ALTER DATABASE DATEFORMAT 'yyyy' from node 1 and all other nodes shown correct value. Then I executed ALTER DATABASE DATEFORMAT 'yyyy-mm-dd' from node 4 and verified it replicated correctly to the other nodes. I was connected with the admin user.

I have created an empty database (instead of doing an import in single mode). We will try to test the scenario when an import from JSON is made.

If you have any additional details on how to reproduce this, please let us know.

Many thanks,

[dalegarv]

Thanks for researching this issue. The only other details I can think of is that after performing the import and restarting the cluster, we run a console script remotely connected with with a server user. The script removes the admin user from the database using DELETE from OUser where name = 'admin', and then it runs the ALTER commands. The script was successful one time, but then it started getting the exception the second time it ran. Once the exception started, we were no longer to run any ALTER commands on the database from node 1 with any user. We also tried removing the writer user completely from the database using node 2, and the exception still references a writer user even though none exists.

We've been able to successfully run this script multiple times against the same database using node 2, and also multiple times using node 1 against another database that was created using the same process with the same import JSON. So this appears to be a one off problem.

I was hoping that maybe you could tell from the code a condition that would cause this exception that always references the writer user even when there isn't one and has the database resource as .null If not, then I guess you can close the issue as not reproducible and we'll reopen if it happens again.

[dalegarv]

We're now seeing this same error in another environment, but this time the error happens on both the nodes in the cluster so we cannot perform any alter database commands. Do you have any idea what could cause this problem?

[dalegarv]

I found that I also get a ‘writer’ error when I try to add a property to a class on either node while logged in as the root user:

CREATE PROPERTY AuditEvent.deletedAt DATETIME (MANDATORY FALSE, NOTNULL FALSE, READONLY FALSE)

com.orientechnologies.orient.core.exception.OSecurityAccessException: User 'writer' does not have permission to execute the operation 'Update' against the resource: ResourceGeneric [name=CLUSTER, legacyName=database.cluster].internal DB name="Entitlements-IPE"

The property is added to the node where I run the command, but it’s not replicated. I’ve tried changing the writer role so that it has all the cluster and cluster.internal permissions on both nodes, but I still get the error.

[dalegarv]

We have another database that has this problem. The first database now doesn't even show up in the list of databases even though the directory still exists.

[RAndrews137]

I believe this issue is no longer occurring. @dalegarv, can you please confirm. If so, can you please mark this ticket as closed?

[dalegarv]

hi @RAndrews137, we've only seen this error on databases where we used the import work around where we imported in single mode and then restarted the cluster in distributed mode. We've stopped using the import all together and haven't seen the issue since. Do you still think we should close it?

[RAndrews137]

@dalegarv, I agree the import issue still likely exists. @SDIPro, this ticket is not urgent at this moment since we have a workaround. However, being able to perform an import into a clustered environment is still core functionality for most database systems and would still be a good area for OrientDB to improve upon for the future. Thanks

[santo-it]

Hi @RAndrews137,

Absolutely. There's an enhancement open already for this: Issue #7268 . I believe this one can now be closed - and Issue #7268 will remain open.

Thanks,

[dalegarv]

hi @santo-it, we actually saw this error again recently on a database that was not created using an import. The two nodes were running on the same VM in a test environment, and when we got the error we saw a socket exception preventing the nodes from replicating. When we restarted the nodes, the error went away. Could this be a case of a bad error message?

[santo-it]

Hi @dalegarv,

It could be. Frankly we haven't been able to reproduce this yet.

Thanks,

[lvca]

If there was a network error, it's absolutely possible that was the cause. WDYT if we close this issue and if you guys see this again you reopen it?

[dalegarv]

Hi Luca, It’s true we haven’t seen this problem much lately, especially since we stopped doing imports to create our databases. Would it be possible to get a better error message when this happens which indicates the replication is not working? The problem doesn’t seem to have anything to do with security and the error message was referencing the writer user even when there was no writer user in the database. S. Dale Garvey Principal Architect - Applications Portal Integration Development 770.360.2071 (External) 445-2071 (Internal) From: Luca Garulli <notifications@github.com<mailto:notifications@github.com>> Reply-To: orientechnologies/orientdb <reply@reply.github.com<mailto:reply@reply.github.com>> Date: Friday, July 21, 2017 at 5:57 AM To: orientechnologies/orientdb <orientdb@noreply.github.com<mailto:orientdb@noreply.github.com>> Cc: "Dale (CORP) Garvey" <Dale.Garvey@ADP.com<mailto:Dale.Garvey@ADP.com>>, Mention <mention@noreply.github.com<mailto:mention@noreply.github.com>> Subject: Re: [orientechnologies/orientdb] ALTER DATABASE command does not get replicated and gets OSecurityAccessException that always references 'writer' user (#7295) If there was a network error, it's absolutely possible that was the cause. WDYT if we close this issue and if you guys see this again you reopen it? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<#7295 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AZbL_D_s2Y2QhX-D0h_O_-8CzULNML7nks5sQHYfgaJpZM4MxkIO>.

…

---------------------------------------------------------------------- This message and any attachments are intended only for the use of the addressee and may contain information that is privileged and confidential. If the reader of the message is not the intended recipient or an authorized representative of the intended recipient, you are hereby notified that any dissemination of this communication is strictly prohibited. If you have received this communication in error, notify the sender immediately by return email and delete the message and any attachments from your system.

[mxns]

I am having a similar issue. In a two node scenario, a DELETE on one node yields the same error/debug message. The record seems to be deleted at the node where the delete is executed, but not on the other. If there was something wrong with the user permissions, that would mean the record should not get deleted at all, right?

A workaround is to use the "root" user, but I would ofc prefer to get the permissions right. Note that the database has not been imported.

The whole thing is described on this StackOverflow entry.