feat!: Remove legacy Sipi upload mechanism (DEV-4260) (#3414)

docs/03-endpoints/api-v2/editing-values.md

@@ -302,10 +302,7 @@ For example, to get a JPG thumbnail image that is 150 pixels wide, you would add
 
 ### Upload Files to DSP-INGEST
 
-Support for DSP-INGEST is in its early stage and currently mainly intended for ingesting large amounts of data.
-When a file has been ingested through DSP-INGEST,
-it is necessary to send the header `X-Asset-Ingested`
-along with the request to create the file value resource in DSP-API.
+Support for uploads with DSP-INGEST is now the preferred method of uploading files (and also for ingesting large amounts of data internally).
 
 ### Submit A File Value to DSP-API
 

docs/05-internals/design/api-v2/sipi.md

@@ -18,57 +18,9 @@ found in `sipi/scripts` in the DSP-API source tree.
 
 Each of these scripts expects a [JSON Web Token](https://jwt.io/) in the
 URL parameter `token`. In all cases, the token must be signed by DSP-API,
-it must have an expiration date and not have expired, its issuer must equal 
-the hostname and port of the API, and its audience must include `Sipi`. 
+it must have an expiration date and not have expired, its issuer must equal
+the hostname and port of the API, and its audience must include `Sipi`.
 The other contents of the expected tokens are described below.
-
-### upload.lua
-
-The `upload.lua` script is available at Sipi's `upload` route. It processes one
-or more file uploads submitted to Sipi. It converts uploaded images to JPEG 2000
-format, and stores them in Sipi's `tmp` directory. The usage of this script is described in
-[Upload Files to Sipi](../../../03-endpoints/api-v2/editing-values.md#upload-files-to-sipi).
-
-### upload_without_processing.lua
-
-The `upload_without_processing.lua` script is available at Sipi's `upload_without_processing` route. 
-It receives files submitted to Sipi but does not process them. 
-Instead, it stores them as is in Sipi's `tmp` directory.
-
-### store.lua
-
-The `store.lua` script is available at Sipi's `store` route. It moves a file
-from temporary to permanent storage. It expects an HTTP `POST` request containing
-`application/x-www-form-urlencoded` data with the parameters `prefix` (the
-project shortcode) and `filename` (the internal Sipi-generated filename of the file
-to be moved).
-
-The JWT sent to this script must contain the key `knora-data`, whose value
-must be a JSON object containing:
-
-- `permission`: must be `StoreFile`
-- `prefix`: the project shortcode submitted in the form data
-- `filename`: the filename submitted in the form data
-
-### delete_temp_file.lua
-
-The `delete_temp_file.lua` script is available at Sipi's `delete_temp_file` route.
-It is used only if DSP-API rejects a file value update request. It expects an
-HTTP `DELETE` request, with a filename as the last component of the URL.
-
-The JWT sent to this script must contain the key `knora-data`, whose value
-must be a JSON object containing:
-
-- `permission`: must be `DeleteTempFile`
-- `filename`: must be the same as the filename submitted in the URL
-
-### clean_temp_dir.lua
-
-The `clean_temp_dir.lua` script is available at Sipi's `clean_temp_dir` route.
-When called, it deletes old temporary files from `tmp` and (recursively) from any subdirectories. 
-The maximum allowed age of temporary files can be set in Sipi's configuration file, 
-using the parameter `max_temp_file_age`, which takes a value in seconds.
-
 The `clean_temp_dir` route requires basic authentication.
 
 ## SipiConnector
@@ -78,38 +30,3 @@ with Sipi. It blocks while processing each request, to ensure that the number of
 concurrent requests to Sipi is not greater than
 `akka.actor.deployment./storeManager/iiifManager/sipiConnector.nr-of-instances`.
 If it encounters an error, it returns `SipiException`.
-
-## The Image File Upload Workflow
-
-1. The client uploads an image file to the `upload` route, which runs
-  `upload.lua`. The image is converted to JPEG 2000 and stored in Sipi's `tmp`
-  directory. In the response, the client receives the JPEG 2000's unique,
-  randomly generated filename.
-2. The client submits a JSON-LD request to a DSP-API route (`/v2/values` or `/v2/resources`)
-   to create or change a file value. The request includes Sipi's internal filename.
-3. During parsing of this JSON-LD request, a `StillImageFileValueContentV2`
-   is constructed to represent the file value. During the construction of this
-   object, a `GetFileMetadataRequestV2` is sent to `SipiConnector`, which
-   uses Sipi's built-in `knora.json` route to get the rest of the file's
-   metadata.
-4. A responder (`ResourcesResponderV2` or `ValuesResponderV2`) validates
-   the request and updates the triplestore. (If it is `ResourcesResponderV2`,
-   it asks `ValuesResponderV2` to generate SPARQL for the values.)
-5. The responder that did the update calls `ValueUtilV2.doSipiPostUpdate`.
-   If the triplestore update was successful, this method sends
-   `MoveTemporaryFileToPermanentStorageRequestV2` to `SipiConnector`, which
-   makes a request to Sipi's `store` route. Otherwise, the same method sends
-   `DeleteTemporaryFileRequestV2` to `SipiConnector`, which makes a request
-   to Sipi's `delete_temp_file` route.
-
-If the request to DSP-API cannot be parsed, the temporary file is not deleted
-immediately, but it will be deleted during the processing of a subsequent
-request by Sipi's `upload` route.
-
-If Sipi's `store` route fails, DSP-API returns the `SipiException` to the client.
-In this case, manual intervention may be necessary to restore consistency
-between DSP-API and Sipi.
-
-If Sipi's `delete_temp_file` route fails, the error is not returned to the client,
-because there is already a DSP-API error that needs to be returned to the client.
-In this case, the Sipi error is simply logged.

integration/src/test/resources/sipi.docker-config.lua

@@ -181,16 +181,6 @@ fileserver = {
 -- Custom routes. Each route is an URL path associated with a Lua script.
 --
 routes = {
-    {
-        method = 'POST',
-        route = '/upload',
-        script = 'upload.lua'
-    },
-    {
-        method = 'POST',
-        route = '/store',
-        script = 'store.lua'
-    },
     {
         method = 'DELETE',
         route = '/delete_temp_file',

integration/src/test/scala/org/knora/sipi/SipiClientTestDelegator.scala

@@ -10,11 +10,8 @@ import zio.ULayer
 import zio.ZLayer
 import zio.nio.file.Path
 
-import org.knora.webapi.messages.store.sipimessages.DeleteTemporaryFileRequest
-import org.knora.webapi.messages.store.sipimessages.MoveTemporaryFileToPermanentStorageRequest
 import org.knora.webapi.messages.store.sipimessages.SipiGetTextFileRequest
 import org.knora.webapi.messages.store.sipimessages.SipiGetTextFileResponse
-import org.knora.webapi.messages.v2.responder.SuccessResponseV2
 import org.knora.webapi.slice.admin.api.model.MaintenanceRequests.AssetId
 import org.knora.webapi.slice.admin.domain.model.KnoraProject
 import org.knora.webapi.slice.admin.domain.model.User
@@ -45,15 +42,6 @@ case class SipiServiceTestDelegator(
     if (whichSipi.useLive) { live }
     else { mock }
 
-  /**
-   * Asks Sipi for metadata about a file in the tmp folder, served from the 'knora.json' route.
-   *
-   * @param filename the path to the file.
-   * @return a [[FileMetadataSipiResponse]] containing the requested metadata.
-   */
-  override def getFileMetadataFromSipiTemp(filename: String): Task[FileMetadataSipiResponse] =
-    sipiService.getFileMetadataFromSipiTemp(filename)
-
   /**
    * Asks DSP-Ingest for metadata about a file in permanent location, served from the 'knora.json' route.
    *
@@ -67,26 +55,6 @@ case class SipiServiceTestDelegator(
   ): Task[FileMetadataSipiResponse] =
     sipiService.getFileMetadataFromDspIngest(shortcode, assetId)
 
-  /**
-   * Asks Sipi to move a file from temporary storage to permanent storage.
-   *
-   * @param moveTemporaryFileToPermanentStorageRequestV2 the request.
-   * @return a [[SuccessResponseV2]].
-   */
-  override def moveTemporaryFileToPermanentStorage(
-    moveTemporaryFileToPermanentStorageRequestV2: MoveTemporaryFileToPermanentStorageRequest,
-  ): Task[SuccessResponseV2] =
-    sipiService.moveTemporaryFileToPermanentStorage(moveTemporaryFileToPermanentStorageRequestV2)
-
-  /**
-   * Asks Sipi to delete a temporary file.
-   *
-   * @param deleteTemporaryFileRequestV2 the request.
-   * @return a [[SuccessResponseV2]].
-   */
-  override def deleteTemporaryFile(deleteTemporaryFileRequestV2: DeleteTemporaryFileRequest): Task[SuccessResponseV2] =
-    sipiService.deleteTemporaryFile(deleteTemporaryFileRequestV2)
-
   /**
    * Asks Sipi for a text file used internally by Knora.
    *

integration/src/test/scala/org/knora/webapi/E2ESpec.scala

@@ -151,8 +151,8 @@ abstract class E2ESpec
   protected def getResponseAsJsonLD(request: HttpRequest): JsonLDDocument =
     UnsafeZioRun.runOrThrow(ZIO.serviceWithZIO[TestClientService](_.getResponseJsonLD(request)))
 
-  protected def uploadToSipi(loginToken: String, filesToUpload: Seq[FileToUpload]): SipiUploadResponse =
-    UnsafeZioRun.runOrThrow(ZIO.serviceWithZIO[TestClientService](_.uploadToSipi(loginToken, filesToUpload)))
+  protected def uploadToIngest(loginToken: String, filesToUpload: Seq[FileToUpload]): SipiUploadResponse =
+    UnsafeZioRun.runOrThrow(ZIO.serviceWithZIO[TestClientService](_.uploadToIngest(loginToken, filesToUpload)))
 
   protected def responseToJsonLDDocument(httpResponse: HttpResponse): JsonLDDocument = {
     val responseBodyFuture: Future[String] =

integration/src/test/scala/org/knora/webapi/ITKnoraLiveSpec.scala

@@ -171,28 +171,13 @@ abstract class ITKnoraLiveSpec
         .getOrThrowFiberFailure()
     }
 
-  protected def uploadToSipi(loginToken: String, filesToUpload: Seq[FileToUpload]): SipiUploadResponse =
+  protected def uploadToIngest(loginToken: String, filesToUpload: Seq[FileToUpload]): SipiUploadResponse =
     Unsafe.unsafe { implicit u =>
       runtime.unsafe
         .run(
           for {
             testClient <- ZIO.service[TestClientService]
-            result     <- testClient.uploadToSipi(loginToken, filesToUpload)
-          } yield result,
-        )
-        .getOrThrow()
-    }
-
-  protected def uploadWithoutProcessingToSipi(
-    loginToken: String,
-    filesToUpload: Seq[FileToUpload],
-  ): SipiUploadWithoutProcessingResponse =
-    Unsafe.unsafe { implicit u =>
-      runtime.unsafe
-        .run(
-          for {
-            testClient <- ZIO.service[TestClientService]
-            result     <- testClient.uploadWithoutProcessingToSipi(loginToken, filesToUpload)
+            result     <- testClient.uploadToIngest(loginToken, filesToUpload)
           } yield result,
         )
         .getOrThrow()

integration/src/test/scala/org/knora/webapi/core/LayersTest.scala

@@ -76,6 +76,7 @@ import org.knora.webapi.slice.shacl.api.ShaclApiModule
 import org.knora.webapi.store.iiif.IIIFRequestMessageHandler
 import org.knora.webapi.store.iiif.IIIFRequestMessageHandlerLive
 import org.knora.webapi.store.iiif.api.SipiService
+import org.knora.webapi.store.iiif.impl.SipiServiceLive
 import org.knora.webapi.store.triplestore.api.TriplestoreService
 import org.knora.webapi.store.triplestore.impl.TriplestoreServiceLive
 import org.knora.webapi.store.triplestore.upgrade.RepositoryUpdater
@@ -238,6 +239,7 @@ object LayersTest {
       ListsService.layer,
       IIIFRequestMessageHandlerLive.layer,
       IriService.layer,
+      SipiServiceLive.layer,
     )
 
   private val commonLayersForAllIntegrationTests =