commit 2404eff

Author: Florian Schade <f.schade@icloud.com> Date: Tue Dec 13 14:22:41 2022 +0100 [full-ci] experimental search backport (#5221) * experimental search backport fix basic extractor resource name move escapeQuery regex into global variable minor pr review changes rename DebounceDuration env variable add document title and content when rebuilding bleve resource Co-authored-by: David Christofas <dchristofas@owncloud.com>
owncloud · Dec 13, 2022 · 8c55d3a · 8c55d3a
1 parent ad04e4e
commit 8c55d3a
Show file tree

Hide file tree

Showing 3 changed files with 89 additions and 28 deletions.
diff --git a/services/_includes/adoc/search_configvars.adoc b/services/_includes/adoc/search_configvars.adoc
@@ -177,24 +177,6 @@ a| [subs=-attributes]
 a| [subs=-attributes]
 Path/File name for the TLS certificate key (in PEM format) for the server certificate to use for the grpc services.
 
-a|`SEARCH_DATA_PATH` +
-
-a| [subs=-attributes]
-++string ++
-a| [subs=-attributes]
-++~/.ocis/search ++
-a| [subs=-attributes]
-The directory where the filesystem storage will store search data. If not definied, the root directory derives from $OCIS_BASE_DATA_PATH:/search.
-
-a|`SEARCH_REINDEX_DEBOUNCE_DURATION` +
-
-a| [subs=-attributes]
-++int ++
-a| [subs=-attributes]
-++1000 ++
-a| [subs=-attributes]
-The duration in milliseconds the reindex debouncer waits before triggering a reindex of a space that was modified.
-
 a|`REVA_GATEWAY` +
 
 a| [subs=-attributes]
@@ -258,14 +240,33 @@ a| [subs=-attributes]
 a| [subs=-attributes]
 The clusterID of the event system. The event system is the message queuing service. It is used as message broker for the microservice architecture. Mandatory when using NATS as event system.
 
-a|`SEARCH_EVENTS_GROUP` +
+a|`STORAGE_USERS_OCIS_ASYNC_UPLOADS` +
+`SEARCH_EVENTS_ASYNC_UPLOADS` +
 
 a| [subs=-attributes]
-++string ++
+++bool ++
 a| [subs=-attributes]
-++search ++
+++false ++
+a| [subs=-attributes]
+Enable asynchronous file uploads.
+
+a|`SEARCH_EVENTS_NUM_CONSUMERS` +
+
+a| [subs=-attributes]
+++int ++
+a| [subs=-attributes]
+++0 ++
+a| [subs=-attributes]
+number of event consumers per service instance
+
+a|`SEARCH_EVENTS_REINDEX_DEBOUNCE_DURATION` +
+
+a| [subs=-attributes]
+++int ++
 a| [subs=-attributes]
-The customer group of the service. One group will only get one copy of an event
+++1000 ++
+a| [subs=-attributes]
+The duration in milliseconds the reindex debouncer waits before triggering a reindex of a space that was modified.
 
 a|`OCIS_INSECURE` +
 `SEARCH_EVENTS_TLS_INSECURE` +
@@ -296,6 +297,52 @@ a| [subs=-attributes]
 a| [subs=-attributes]
 Enable TLS for the connection to the events broker. The events broker is the ocis service which receives and delivers events between the services..
 
+a|`SEARCH_ENGINE_TYPE` +
+
+a| [subs=-attributes]
+++string ++
+a| [subs=-attributes]
+++bleve ++
+a| [subs=-attributes]
+Defines which search engine to use.
+
+a|`SEARCH_ENGINE_BLEVE_DATA_PATH` +
+
+a| [subs=-attributes]
+++string ++
+a| [subs=-attributes]
+++~/.ocis/search ++
+a| [subs=-attributes]
+Path for the search persistence directory.
+
+a|`SEARCH_EXTRACTOR_TYPE` +
+
+a| [subs=-attributes]
+++string ++
+a| [subs=-attributes]
+++basic ++
+a| [subs=-attributes]
+Defines the content extraction engine.
+
+a|`OCIS_INSECURE` +
+`SEARCH_EXTRACTOR_CS3SOURCE_INSECURE` +
+
+a| [subs=-attributes]
+++bool ++
+a| [subs=-attributes]
+++false ++
+a| [subs=-attributes]
+Ignore untrusted SSL certificates when connecting to the CS3 source.
+
+a|`SEARCH_EXTRACTOR_TIKA_TIKA_URL` +
+
+a| [subs=-attributes]
+++string ++
+a| [subs=-attributes]
+++http://127.0.0.1:9998 ++
+a| [subs=-attributes]
+URL of the tika server.
+
 a|`OCIS_MACHINE_AUTH_API_KEY` +
 `SEARCH_MACHINE_AUTH_API_KEY` +
 

diff --git a/services/_includes/search-config-example.yaml b/services/_includes/search-config-example.yaml
@@ -22,8 +22,6 @@ grpc:
  enabled: false
  cert: ""
  key: ""
-data_path: ~/.ocis/search
-debounce_duration: 1000
 reva:
  address: 127.0.0.1:9142
  tls:
@@ -35,8 +33,19 @@ grpc_client_tls:
 events:
  endpoint: 127.0.0.1:9233
  cluster: ocis-cluster
- group: search
+ async_uploads: false
+ num_consumers: 0
+ debounce_duration: 1000
  tls_insecure: false
  tls_root_ca_certificate: ""
  enable_tls: false
+engine:
+ type: bleve
+ bleve:
+ data_path: ~/.ocis/search
+extractor:
+ type: basic
+ cs3_allow_insecure: false
+ tika:
+ tika_url: http://127.0.0.1:9998
 machine_auth_api_key: ""
diff --git a/services/_includes/search_configvars.md b/services/_includes/search_configvars.md
@@ -18,17 +18,22 @@
 | OCIS_GRPC_TLS_ENABLED | bool | false | Activates TLS for the grpc based services using the server certifcate and key configured via OCIS_GRPC_TLS_CERTIFICATE and OCIS_GRPC_TLS_KEY. If OCIS_GRPC_TLS_CERTIFICATE is not set a temporary server certificate is generated - to be used with OCIS_GRPC_CLIENT_TLS_MODE=insecure.|
 | OCIS_GRPC_TLS_CERTIFICATE | string | | Path/File name of the TLS server certificate (in PEM format) for the grpc services.|
 | OCIS_GRPC_TLS_KEY | string | | Path/File name for the TLS certificate key (in PEM format) for the server certificate to use for the grpc services.|
-| SEARCH_DATA_PATH | string | ~/.ocis/search | The directory where the filesystem storage will store search data. If not definied, the root directory derives from $OCIS_BASE_DATA_PATH:/search.|
-| SEARCH_REINDEX_DEBOUNCE_DURATION | int | 1000 | The duration in milliseconds the reindex debouncer waits before triggering a reindex of a space that was modified.|
 | REVA_GATEWAY | string | 127.0.0.1:9142 | The CS3 gateway endpoint.|
 | OCIS_GRPC_CLIENT_TLS_MODE | string | | TLS mode for grpc connection to the go-micro based grpc services. Possible values are 'off', 'insecure' and 'on'. 'off': disables transport security for the clients. 'insecure' allows to use transport security, but disables certificate verification (to be used with the autogenerated self-signed certificates). 'on' enables transport security, including server ceritificate verification.|
 | OCIS_GRPC_CLIENT_TLS_CACERT | string | | Path/File name for the root CA certificate (in PEM format) used to validate TLS server certificates of the go-micro based grpc services.|
 | OCIS_GRPC_CLIENT_TLS_MODE | string | | TLS mode for grpc connection to the go-micro based grpc services. Possible values are 'off', 'insecure' and 'on'. 'off': disables transport security for the clients. 'insecure' allows to use transport security, but disables certificate verification (to be used with the autogenerated self-signed certificates). 'on' enables transport security, including server ceritificate verification.|
 | OCIS_GRPC_CLIENT_TLS_CACERT | string | | Path/File name for the root CA certificate (in PEM format) used to validate TLS server certificates of the go-micro based grpc services.|
 | SEARCH_EVENTS_ENDPOINT | string | 127.0.0.1:9233 | The address of the event system. The event system is the message queuing service. It is used as message broker for the microservice architecture.|
 | SEARCH_EVENTS_CLUSTER | string | ocis-cluster | The clusterID of the event system. The event system is the message queuing service. It is used as message broker for the microservice architecture. Mandatory when using NATS as event system.|
-| SEARCH_EVENTS_GROUP | string | search | The customer group of the service. One group will only get one copy of an event|
+| STORAGE_USERS_OCIS_ASYNC_UPLOADS<br/>SEARCH_EVENTS_ASYNC_UPLOADS | bool | false | Enable asynchronous file uploads.|
+| SEARCH_EVENTS_NUM_CONSUMERS | int | 0 | number of event consumers per service instance|
+| SEARCH_EVENTS_REINDEX_DEBOUNCE_DURATION | int | 1000 | The duration in milliseconds the reindex debouncer waits before triggering a reindex of a space that was modified.|
 | OCIS_INSECURE<br/>SEARCH_EVENTS_TLS_INSECURE | bool | false | Whether to verify the server TLS certificates.|
 | SEARCH_EVENTS_TLS_ROOT_CA_CERTIFICATE | string | | The root CA certificate used to validate the server's TLS certificate. If provided SEARCH_EVENTS_TLS_INSECURE will be seen as false.|
 | OCIS_EVENTS_ENABLE_TLS<br/>SEARCH_EVENTS_ENABLE_TLS | bool | false | Enable TLS for the connection to the events broker. The events broker is the ocis service which receives and delivers events between the services..|
+| SEARCH_ENGINE_TYPE | string | bleve | Defines which search engine to use.|
+| SEARCH_ENGINE_BLEVE_DATA_PATH | string | ~/.ocis/search | Path for the search persistence directory.|
+| SEARCH_EXTRACTOR_TYPE | string | basic | Defines the content extraction engine.|
+| OCIS_INSECURE<br/>SEARCH_EXTRACTOR_CS3SOURCE_INSECURE | bool | false | Ignore untrusted SSL certificates when connecting to the CS3 source.|
+| SEARCH_EXTRACTOR_TIKA_TIKA_URL | string | http://127.0.0.1:9998 | URL of the tika server.|
 | OCIS_MACHINE_AUTH_API_KEY<br/>SEARCH_MACHINE_AUTH_API_KEY | string | | Machine auth API key used to validate internal requests necessary for the access to resources from other services.|