Remove horreum.db.secret for good and fix ducumentation

docs/DEVELOPMENT.md

@@ -174,7 +174,6 @@ You can use an existing backup of the database (PostgreSQL 13+) by a command lik
 ```bash
 mvn  quarkus:dev -pl '!horreum-integration-tests' \
   -Dhorreum.dev-services.postgres.database-backup=/opt/databases/horreum-prod-db/ \
-  -Dhorreum.db.secret='M3g45ecr5t!' \
   -Dhorreum.dev-services.keycloak.db-password='prod-password' \
   -Dhorreum.dev-services.keycloak.admin-password='ui-prod-password' \
   -Dquarkus.datasource.username=user \
@@ -193,8 +192,6 @@ horreum.dev-services.keycloak.db-password=<keycloak-user-password>
 horreum.dev-services.keycloak.admin-username=<keycloak-admin-name>
 horreum.dev-services.keycloak.admin-password=<keycloak-admin-password>
 
-horreum.db.secret=<db-secret>
-
 quarkus.datasource.username=<horreum-user-name>
 quarkus.datasource.password=<horreum-user-password>
 

docs/SECURITY.md

@@ -2,17 +2,11 @@
 
 Security uses RBAC with authz and authn provided by Keycloak server, and heavily relies on row-level security (RLS) in the database.
 There should be two DB users (roles); `dbadmin` who has full access to the database, and `appuser` with limited access.
-`dbadmin` should set up DB structure - tables with RLS policies and grant RW access to all tables but `dbsecret` to `appuser`.
+`dbadmin` should set up DB structure - tables with RLS policies and grant RW access to all tables to `appuser`.
 When the application performs a database query, impersonating the authenticated user, it invokes `SET horreum.userroles = '...'`
 to declare all roles the user has based on information from Keycloak. RLS policies makes sure that the user cannot read or modify
 anything that does not belong to this user or is made available to him.
 
-As a precaution against bug leaving SQL-level access open the `horreum.userroles` granting the permission are not set in plaintext;
-the format of the setting is `role1:seed1:hash1,role2:seed2:hash2,...` where the `hash` is SHA-256 of combination of role, seed
-and hidden passphrase. This passphrase is set in `application.properties` under key `horreum.db.secret`, and in database as the only
-record in table `dbsecret`. The user `appuser` does not have access to that table, but the security-defined functions used
-in table policies can fetch it, compute the hash again and validate its correctness.
-
 We define 3 levels of access to each row:
 
 - public: available even to non-authenticated users (for reading)

docs/site/content/en/docs/Deployment/bare-metal.md

@@ -89,8 +89,6 @@ QUARKUS_DATASOURCE_MIGRATION_PASSWORD=Curr3ntAdm!nPwd
 QUARKUS_DATASOURCE_JDBC_ADDITIONAL-JDBC-PROPERTIES_SSLROOTCERT=server.crt
 # As an alternative, certificate validation can be disabled with
 # QUARKUS_DATASOURCE_JDBC_ADDITIONAL-JDBC-PROPERTIES_SSLMODE=require
-# Secret generated during database setup: run `SELECT * FROM dbsecret` as DB admin
-HORREUM_DB_SECRET=xxxxxxxxxxxxxxxxxxxxxxxxx
 
 # --- KEYCLOAK ---
 # This URL must be accessible from Horreum, but does not have to be exposed to the world

docs/site/content/en/docs/Tasks/upload-run/index.md

@@ -10,13 +10,13 @@ Horreum accepts any valid **JSON** as the input. To get maximum out of Horreum,
 There are two principal ways to authorize operations:
 
 - Authentication against OIDC provider (Keycloak): This is the standard way that you use when accessing Horreum UI - you use your credentials to get a JSON Web Token (JWT) and this is stored in the browser session. When accessing Horreum over the REST API you need to use this for [Bearer Authentication](https://datatracker.ietf.org/doc/html/rfc6750#section-2.1). The authorization is based on the teams and roles within those teams that you have.
-- Horreum Tokens: In order to provide access to non-authenticated users via link, or let automated scripts perform tasks Horreum can generate a random token consisting of 80 hexadecimal digits. This token cannot be used in the `Authorization` header; operations that support tokens usually accept `token` parameter.
+- Horreum API Keys: See more in [API keys](/docs/tasks/api-keys). These replace the so called "Horreum Tokens" that were used in the past in operations that accepted a `token` parameter. 
 
 If you're running your tests in Jenkins you can skip a lot of the complexity below using [Horreum Plugin](https://plugins.jenkins.io/horreum/). This plugin supports both Jenkins Pipeline and Freeform jobs.
 
 ## Getting JWT token
 
-New data can be uploaded into Horreum only by authorized users. We recommend setting up a separate user account for the load-driver (e.g. [Hyperfoil](https://hyperfoil.io)) or CI toolchain that will upload the data as part of your benchmark pipeline. This user must have the permission to upload for given team, e.g. if you'll use `dev-team` as the owner this role is called `dev-uploader` and it is a composition of the team role (`dev-team`) and `uploader` role. You can read more about user management [here](/docs/concepts/users).
+New data can be uploaded into Horreum only by authorized users. This user must have the permission to upload for given team, e.g. if you'll use `dev-team` as the owner this role is called `dev-uploader` and it is a composition of the team role (`dev-team`) and `uploader` role. You can read more about user management [here](/docs/concepts/users).
 
 ```bash
 TOKEN=$(curl -s http://localhost:8180/realms/horreum/protocol/openid-connect/token \
@@ -44,12 +44,6 @@ TOKEN=$(curl -s http://localhost:8180/realms/horreum/protocol/openid-connect/tok
 
 Note that the offline token also expires eventually, by default after 30 days.
 
-## Getting Horreum token
-
-In order to retrieve an upload token you need to navigate to particular Test configuration page, switch to tab 'Access' and push the 'Add new token' button, checking permissions for 'Read' and 'Upload'. The token string will be displayed only once; if you lose it please revoke the token and create a new one.
-
-This token should not be used for Bearer Authentication (do not use it in the `Authorization` HTTP header) as in the examples below; instead you need to append `&token=<horreum-token>` to the query.
-
 ## Uploading the data
 
 There are several mandatory parameters for the upload:

docs/site/content/en/openapi/openapi.yaml

@@ -1841,10 +1841,6 @@ paths:
         schema:
           format: int32
           type: integer
-      - name: token
-        in: query
-        schema:
-          type: string
       responses:
         "200":
           description: OK
@@ -2188,7 +2184,7 @@ paths:
       parameters:
       - name: id
         in: path
-        description: Test ID to revoke token
+        description: Test ID to update
         required: true
         schema:
           format: int32

horreum-api/src/main/java/io/hyperfoil/tools/horreum/api/services/TestService.java

@@ -56,7 +56,7 @@ public interface TestService {
     @GET
     @Path("{id}")
     @Operation(description = "Retrieve a test by id")
-    Test get(@PathParam("id") int id, @QueryParam("token") String token);
+    Test get(@PathParam("id") int id);
 
     @GET
     @Path("byName/{name}")
@@ -123,7 +123,7 @@ TestListing summary(@QueryParam("roles") String roles, @QueryParam("folder") Str
     // TODO: it would be nicer to use @FormParams but fetchival on client side doesn't support that
     @Operation(description = "Update the Access configuration for a Test")
     @Parameters(value = {
-            @Parameter(name = "id", required = true, description = "Test ID to revoke token", example = "101"),
+            @Parameter(name = "id", required = true, description = "Test ID to update", example = "101"),
             @Parameter(name = "owner", required = true, description = "Name of the new owner", example = "perf-team"),
             @Parameter(name = "access", required = true, description = "New Access level for the Test", example = "0")
     })

horreum-backend/src/main/java/io/hyperfoil/tools/horreum/svc/RunServiceImpl.java

@@ -344,7 +344,7 @@ public void updateAccess(int id, String owner, Access access) {
         }
     }
 
-    @PermitAll // all because of possible token-based upload
+    @RolesAllowed(Roles.UPLOADER)
     @WithRoles
     @Transactional
     @Override
@@ -430,7 +430,7 @@ public Response addRunFromData(String start, String stop, String test, String ow
         return addRunFromData(start, stop, test, owner, access, schemaUri, description, dataNode.toString(), metadataNode);
     }
 
-    @PermitAll // all because of possible token-based upload
+    @RolesAllowed(Roles.UPLOADER)
     @Transactional
     @WithRoles
     Response addRunFromData(String start, String stop, String test,

horreum-backend/src/main/java/io/hyperfoil/tools/horreum/svc/TestServiceImpl.java

@@ -125,7 +125,7 @@ public void delete(int id) {
     @Override
     @WithRoles
     @PermitAll
-    public Test get(int id, String token) {
+    public Test get(int id) {
         TestDAO test = TestDAO.find("id", id).firstResult();
         if (test == null) {
             throw ServiceException.notFound("No test with name " + id);

horreum-backend/src/main/resources/db/changeLog.xml

@@ -4659,6 +4659,7 @@
             DROP FUNCTION has_read_token;
             DROP FUNCTION has_modify_token;
             DROP FUNCTION has_upload_token;
+            DROP FUNCTION auth_suffix;
         </sql>
         <dropColumn tableName="run" columnName="token" />
         <dropColumn tableName="schema" columnName="token" />

horreum-backend/src/main/resources/horreum.sh

@@ -6,10 +6,6 @@ if [ -n "$QUARKUS_DATASOURCE_USERNAME" ]; then
   # to Liquibase changeLog.xml
   JAVA_OPTIONS="$JAVA_OPTIONS -Dquarkus.datasource.username=$QUARKUS_DATASOURCE_USERNAME"
 fi
-if [ -n "$HORREUM_DB_SECRET" ]; then
-  # Same as above, for Liquibase.
-  JAVA_OPTIONS="$JAVA_OPTIONS -Dhorreum.db.secret=$HORREUM_DB_SECRET"
-fi
 
 echo "Starting Horreum with JAVA_OPTIONS: $JAVA_OPTIONS"
 java $JAVA_OPTIONS -jar quarkus-run.jar

horreum-backend/src/test/java/io/hyperfoil/tools/horreum/svc/BaseServiceTest.java

@@ -882,7 +882,6 @@ protected HashMap<String, List<JsonNode>> dumpDatabaseContents() {
                 "SELECT tablename FROM pg_catalog.pg_tables WHERE schemaname = 'public';").getResultList();
         tables.remove("databasechangelog");
         tables.remove("databasechangeloglock");
-        tables.remove("dbsecret");
         tables.remove("view_recalc_queue");
         tables.remove("label_recalc_queue");
         tables.remove("fingerprint_recalc_queue");

horreum-backend/src/test/java/io/hyperfoil/tools/horreum/svc/RunServiceTest.java

@@ -1,6 +1,13 @@
 package io.hyperfoil.tools.horreum.svc;
 
-import static org.junit.jupiter.api.Assertions.*;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertInstanceOf;
+import static org.junit.jupiter.api.Assertions.assertNotEquals;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+import static org.junit.jupiter.api.Assertions.fail;
 
 import java.io.IOException;
 import java.util.ArrayList;
@@ -33,15 +40,25 @@
 import io.hyperfoil.tools.horreum.api.SortDirection;
 import io.hyperfoil.tools.horreum.api.alerting.ChangeDetection;
 import io.hyperfoil.tools.horreum.api.alerting.Variable;
-import io.hyperfoil.tools.horreum.api.data.*;
+import io.hyperfoil.tools.horreum.api.data.Access;
+import io.hyperfoil.tools.horreum.api.data.Dataset;
+import io.hyperfoil.tools.horreum.api.data.ExperimentComparison;
+import io.hyperfoil.tools.horreum.api.data.ExperimentProfile;
+import io.hyperfoil.tools.horreum.api.data.ExportedLabelValues;
 import io.hyperfoil.tools.horreum.api.data.Extractor;
+import io.hyperfoil.tools.horreum.api.data.Label;
+import io.hyperfoil.tools.horreum.api.data.Schema;
+import io.hyperfoil.tools.horreum.api.data.Test;
+import io.hyperfoil.tools.horreum.api.data.TestExport;
+import io.hyperfoil.tools.horreum.api.data.Transformer;
 import io.hyperfoil.tools.horreum.api.data.changeDetection.ChangeDetectionModelType;
 import io.hyperfoil.tools.horreum.api.internal.services.AlertingService;
 import io.hyperfoil.tools.horreum.api.services.DatasetService;
 import io.hyperfoil.tools.horreum.api.services.ExperimentService;
 import io.hyperfoil.tools.horreum.api.services.RunService;
 import io.hyperfoil.tools.horreum.bus.AsyncEventChannels;
-import io.hyperfoil.tools.horreum.entity.data.*;
+import io.hyperfoil.tools.horreum.entity.data.DatasetDAO;
+import io.hyperfoil.tools.horreum.entity.data.RunDAO;
 import io.hyperfoil.tools.horreum.mapper.DatasetMapper;
 import io.hyperfoil.tools.horreum.server.CloseMe;
 import io.hyperfoil.tools.horreum.test.HorreumTestProfile;

horreum-client/src/main/java/io/hyperfoil/tools/RunServiceExtension.java

@@ -59,16 +59,6 @@ public Object getMetadata(int id, String schemaUri) {
     //      return delegate.queryData(id, jsonpath, schemaUri, array);
     //   }
 
-    @Override
-    public String resetToken(int id) {
-        return delegate.resetToken(id);
-    }
-
-    @Override
-    public String dropToken(int id) {
-        return delegate.dropToken(id);
-    }
-
     @Override
     public void updateAccess(int id, String owner, Access access) {
         delegate.updateAccess(id, owner, access);

horreum-client/src/main/java/io/hyperfoil/tools/horreum/api/client/RunService.java

@@ -57,14 +57,6 @@ public interface RunService {
     //                         @QueryParam("uri") String schemaUri,
     //                         @QueryParam("array") @DefaultValue("false") boolean array);
 
-    @POST
-    @Path("{id}/resetToken")
-    String resetToken(@PathParam("id") int id);
-
-    @POST
-    @Path("{id}/dropToken")
-    String dropToken(@PathParam("id") int id);
-
     @POST
     @Path("{id}/updateAccess")
     // TODO: it would be nicer to use @FormParams but fetchival on client side doesn't support that

horreum-integration-tests/src/test/java/io/hyperfoil/tools/horreum/it/HorreumClientIT.java

@@ -25,6 +25,7 @@
 
 import jakarta.ws.rs.BadRequestException;
 import jakarta.ws.rs.NotAuthorizedException;
+import jakarta.ws.rs.core.Response;
 
 import org.junit.jupiter.api.Assertions;
 
@@ -103,6 +104,30 @@ public void testApiKeys() {
         }
     }
 
+    @org.junit.jupiter.api.Test
+    public void testApiKeysPrivateUpload() throws JsonProcessingException {
+        String theKey = horreumClient.userService.newApiKey(new UserService.ApiKeyRequest("private upload key", USER));
+
+        horreumClient.testService.updateAccess(dummyTest.id, dummyTest.owner, Access.PRIVATE);
+        try (HorreumClient apiClient = new HorreumClient.Builder()
+                .horreumUrl("http://localhost:".concat(System.getProperty("quarkus.http.test-port")))
+                .horreumApiKey(theKey)
+                .build()) {
+
+            Run run = new Run();
+            run.start = Instant.now();
+            run.stop = Instant.now();
+            run.testid = -1; // should be ignored
+            run.data = new ObjectMapper().readTree(resourceToString("data/config-quickstart.jvm.json"));
+            run.description = "Test description";
+            try (Response response = apiClient.runService.add(dummyTest.name, dummyTest.owner, Access.PRIVATE, run)) {
+                assertEquals(200, response.getStatus());
+            }
+        } finally {
+            horreumClient.testService.updateAccess(dummyTest.id, dummyTest.owner, Access.PUBLIC);
+        }
+    }
+
     @org.junit.jupiter.api.Test
     public void testAddRunFromData() throws JsonProcessingException {
         JsonNode payload = new ObjectMapper().readTree(resourceToString("data/config-quickstart.jvm.json"));