Add new Solr fields via an API call (investigation) #5989

pkiraly · 2019-07-03T15:25:48Z

When it comes to adding custom metadata block, there is a manual step involved: adding Solr fields to schema.xml, however Solr provides an API (Schema API) to make this manual step unnecessary, however it invloves some changes in Dataverse.
Note: this ticket is about investigation instead on implementation. First we have to understand every aspect of this change to make sure the existing technologies are reliable and fully support the request.

Solr has a Schema API, which lets you to modify the Solr schema (the list of fields and their properties). Solr can handle the schema in two different ways, and it can be controlled in the solrconfig.xml file. There is a "classic" way, which is based on schema.xml file, and a newer way, called managed schema (its materialization is the "managed-schema" file, and it is editable via the Solr user interface or via API, but it is not advised to edit this file manually).

In the Dataverse provided solrconfig.xml you have this:

The schema API doesn't work with the ClassicIndexSchemaFactory. If you try, Solr returns an error message: "schema is not editable". To enable Schema API, we have to change this setting:

Set ManagedIndexSchemaFactory in solrconfig.xml:

 <schemaFactory class="ManagedIndexSchemaFactory"/>

After this you have to restart Solr, and the Schema API will work this way:

curl -X POST -H 'Content-type:application/json' \
  http://localhost:8983/api/cores/collection1/schema --data-binary '{
  "add-field":{
     "name":"title", "type":"text_en", "multiValued":false,
     "stored":true, "indexed":true
  },
  "add-copy-field":{"source":"title", "dest":"_text_", "maxChars":"3000"}
}'

The details of the Schema API can be found here:
https://lucene.apache.org/solr/guide/7_3/schema-api.html

The details of change from classic schema:

https://lucene.apache.org/solr/guide/7_3/schema-factory-definition-in-solrconfig.html#SchemaFactoryDefinitioninSolrConfig-Switchingfromschema.xmltoManagedSchema

The problems:
The documentation says: "Once Solr is restarted and it detects that a schema.xml file exists, but the managedSchemaResourceName file (i.e., “managed-schema”) does not exist, the existing schema.xml file will be renamed to schema.xml.bak and the contents are re-written to the managed schema file." When I tried it, the schema.xml were not copied, and not renamed. However since the same searches, even fielded searches are working.
When I use Schema API to retrieve fields, it contains only the default Solr fields, and not those Dataverse added via schema.xml.

I asked help from a Solr expert.

(I added @4tikhonov as watcher)

The text was updated successfully, but these errors were encountered:

pdurbin · 2019-07-03T15:41:22Z

@pkiraly thanks for opening this issue! Creating fields in Solr programmatically via API would be a huge improvement over what we do know, which is to manually update schema.xml from time to time.

As you say, it would be especially useful when custom metadata blocks are created. The documented procedure at http://guides.dataverse.org/en/4.15/admin/metadatacustomization.html#updating-the-solr-schema (screenshot below) if quite manual.

Please let me know if I can help at all. Thanks again.

4tikhonov · 2019-07-03T15:49:09Z

Hi @pdurbin,

Yes, it's on the Roadmap of SSHOC DataverseEU project and we'll try to find a solution together. But probably someone already knows "how to unlock the closed door".

pkiraly · 2019-07-03T15:58:30Z

@pdurbin

It would be great help to figure it out if it generally doesn't do what is described in Solr manual or it is just for me (due to some confounding factors of my system configuration).

To do the experiment, do the following steps (on a test machine).

make a copy of your existing Solr instance
stop Solr
on solrconfig.xml find this line:

  <schemaFactory class="ClassicIndexSchemaFactory"/>

and replace to this:

  <schemaFactory class="ManagedIndexSchemaFactory"/>

restart Solr
check if schema.xml.bak file were created and if the Dataverse related fields have been copied to managed-schema file.
report the result here

pdurbin · 2019-07-03T20:26:06Z

@pkiraly one observation is that even before I do anything there's a managed-schema file at the following location:

/usr/local/solr/server/solr/collection1/conf/managed-schema

I made a copy of it with this:

cp -a /usr/local/solr/server/solr/collection1/conf/managed-schema /usr/local/solr/server/solr/collection1/conf/managed-schema.backup.pdurbin

I'm confirming if Solr is up or down with these:

curl http://localhost:8983/solr/collection1/schema/fields

systemctl status solr

Stopping Solr with this:

systemctl stop solr

Backing up the file before editing:

cp -a /usr/local/solr/server/solr/collection1/conf/solrconfig.xml /usr/local/solr/server/solr/collection1/conf/solrconfig.xml.backup.pdurbin

Start Solr again:

systemctl start solr

A file named schema.xml.bak was not created. Nothing from this:

find /usr/local/solr | grep bak

No change to /usr/local/solr/server/solr/collection1/conf/managed-schema . This diff shows no changes:

diff /usr/local/solr/server/solr/collection1/conf/managed-schema /usr/local/solr/server/solr/collection1/conf/managed-schema.backup.pdurbin

pkiraly · 2019-07-04T07:03:07Z

I asked @erikhatcher a well known Lucene/Solr contributor, author and speaker.

He suggested that if this issue occurs, run the following procedure:

stop Solr
cp schema.xml managed-schema
start Solr

I tried it, and it works.

pdurbin · 2019-09-05T06:30:58Z

@joelmarkanderson would benefit from a solution. He recently reported the following at https://groups.google.com/d/msg/dataverse-community/lr26VTP8lhs/5JoZ-IdnBQAJ

"I have successfully populated a controlled vocabulary metadata block, and the list of 38 Values correctly shows under the "Add + Edit Metadata" configuration screen. However, selecting and saving a tag results in an webpage error message: "Error – The metadata could not be updated. If you believe this is an error, please contact Support for assistance."

...

Caused by: org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://localhost:8983/solr/collection1: ERROR: [doc=dataset_758_draft] unknown field 'tag' "

@pkiraly I have not yet tried your latest suggestion above. Mostly I'm just posting the error above so people can find this issue in the future.

pdurbin · 2019-09-05T11:36:38Z

@pkiraly heads up that as a stop gap measure @poikilotherm and I have cooked up a new plan in a new issue: Make Solr schema.xml configuration more flexible, still using Classic Schema Factory #6142

You (and others) are also welcome to read our conversation about it at http://irclog.iq.harvard.edu/dataverse/2019-09-05#i_104531

To be clear, Oliver and I and others still want the solution you are proposing in this issue. We just think the proposal in the other issue will be less effort. It's a short term solution. Your idea (this issue) is the longer term solution. 😄

poikilotherm · 2019-09-05T11:39:08Z

Thanks @pdurbin for cross-referencing 👍 As you already outlined, I'm totally with you @pkiraly that it makes sense to switch to managed schema factory in the long run.

pkiraly · 2019-10-16T09:11:37Z

Is there anybody who could reproduce the process I suggested (see my comments #5989 (comment) and before that)?

@pdurbin Do you have some label for "help needed"? I do not have right to add labels.

pdurbin · 2019-10-16T11:17:45Z

@pkiraly I have not tried playing with Managed Schema. I can add some "help wanted" labels.

Maybe @poikilotherm can help? I believe that the future pull request will be a doc change and some scripts that add fields to the Solr schema dynamically based on the fields metadata blocks that have been loaded into Dataverse.

pdurbin · 2019-10-16T11:19:54Z

By the way, if anyone wants a real custom metadata block to play with, a new one called "codemeta.tsv" is attached to the "CodeMeta-Metadata for Software and displayFormat for controlledVocabularies" thread at https://groups.google.com/d/msg/dataverse-community/nDMbMv4fKf4/P5YxHJzDBgAJ

…xml definition. IQSS#5989

It remains unclear why this data transfer object, used for file asset facets only so far has ever been treated as a bean. The facet labels are used to render facet query links within the Web UI and retrieved from a `Map` in `DatasetPage` UI backing bean.

…eldType IQSS#5989 Moving: while living inside the search package, the functionality is much extended to be used for validation and schema definition. The schema related stuff needs to live on its own. Renaming: make the class more representative of Solr terminology

The SolrFieldProperty class is used as a POJO to wrap Boolean or String properties of Solr <field>, <fieldType> and <dynamicField> definitions.

IQSS#5989

This is a base class to depict <field>, <dynamicField> and <fieldType> in implementing subclasses.

Introducing SolrFieldType to be kind of an enum class of available types. As Java enums do not allow extending a base class, using the good old pre Java 5 style here. Adding a field "ALL" to make all types available as `List` via reflection.

…xml definition. IQSS#5989

It remains unclear why this data transfer object, used for file asset facets only so far has ever been treated as a bean. The facet labels are used to render facet query links within the Web UI and retrieved from a `Map` in `DatasetPage` UI backing bean.

…eldType IQSS#5989 Moving: while living inside the search package, the functionality is much extended to be used for validation and schema definition. The schema related stuff needs to live on its own. Renaming: make the class more representative of Solr terminology

…S#5989 Instead of code duplication, reuse the enum present in DatasetFieldType to match the metadata block data types with Solr data types present in SolrFieldType. Makes it much easier to transform the entities to the necessary SolrStaticFields for the schema validation / management.

This minor refactoring of the dataverse.search.SolrField class allows easier mapping from the database model domain to the schema model domain. It unifies the usage across both worlds inside Dataverse.

@pkiraly

…5989 Per @pkiraly, the standard configurations should be more easy to read for humans by using self-explainatory names instead of abbrevs.

…ype IQSS#5989 This commit introduces a few changes: - The type enum in DatasetFieldType is enhanced to map to SolrFieldType types of Solr fields. - The retrieval of dataverse.search.SolrField from a DatasetFieldType is much simplified due to this usage of SolrFieldType in both areas of the code. - As there might be no mapping existant for some types (like email), the DatasetFieldType.getSolrField() has been refactored to return an Optional<SolrField>. All usages of the method have been updated aligned to this change.

@pkiraly

This EJB stores the Solr schema we need to follow as a single source of truth. We will rely on this in-memory model to validate, update and manage the schema inside the backing Solr instance. It will be usable via API to reload the schema model from the database (and code) plus will do so automatically during startup of Dataverse. This is necessary to have empty Solr instances bootstrapped by us. Thanks to @pkiraly @pdurbin and @rtreacy most of this code was done during a "hacky friday" code-with-me session.

…d present in the default schema) IQSS#5989

…del map IQSS#5989

…olrJ field Map<String,Object> IQSS#5989

…IQSS#5989

…elds from Solr IQSS#5989 Some dynamic field definition in Solr are present by default, but completely unused by us. We ignore those, as we don't want to depict all the types they require in SolrFieldType.

This adds a minimal prototype to work with the schema retrieved from Solr. Does not yet do any comparison, but shows good results in converting the schema present in Solr into the comparable in-memory model.

pdurbin · 2022-10-14T03:10:42Z

Hmm. I see some commits from a little over a year ago from @poikilotherm above.

I think it's safe to say that at least @pkiraly @poikilotherm and I (and probably others) are still interested in this but no one has had the time to code it up.

cmbz · 2024-08-20T15:22:13Z

To focus on the most important features and bugs, we are closing issues created before 2020 (version 5.0) that are not new feature requests with the label 'Type: Feature'.

If you created this issue and you feel the team should revisit this decision, please reopen the issue and leave a comment.

pdurbin assigned pkiraly Jul 8, 2019

poikilotherm mentioned this issue Sep 5, 2019

Make Solr schema.xml configuration more flexible, still using Classic Schema Factory #6142

Closed

poikilotherm mentioned this issue Sep 10, 2019

6142 - Flexible Solr schema deployment #6146

Merged

2 tasks

pdurbin added Help Wanted: Code Mentor: pdurbin labels Oct 16, 2019

poikilotherm mentioned this issue Oct 20, 2020

Documentation: Clarify how managed/classic schema.xml works, use cases, how to operationalize it. #7306

Closed

poikilotherm mentioned this issue Mar 8, 2021

Solr 8.8 upgrade - remaining issues with solrconfig.xml #7662

Open

poikilotherm added a commit to poikilotherm/dataverse that referenced this issue Mar 31, 2021

refactor(index): make SolrType a better representation of the schema.…

9341bdf

…xml definition. IQSS#5989

poikilotherm added a commit to poikilotherm/dataverse that referenced this issue Apr 14, 2021

refactor(index): make SolrFieldProperty an enum of possible properties.

103493c

IQSS#5989

poikilotherm added a commit to poikilotherm/dataverse that referenced this issue Apr 14, 2021

feature(index): introduce base class for Solr fields IQSS#5989

c28b252

This is a base class to depict <field>, <dynamicField> and <fieldType> in implementing subclasses.

poikilotherm mentioned this issue Apr 14, 2021

6070 Update Apache Commons Lang3 #7801

Merged

poikilotherm added a commit to poikilotherm/dataverse that referenced this issue Apr 19, 2021

refactor(index): make SolrType a better representation of the schema.…

caa8fa6

…xml definition. IQSS#5989

poikilotherm mentioned this issue May 7, 2021

Using displayFormat for displaying, indexing and searching compound field values #7856

Open

poikilotherm added a commit to poikilotherm/dataverse that referenced this issue May 21, 2021

fix(index): fix failing test for DatasetFieldType IQSS#5989

32bc9ea

pdurbin mentioned this issue May 24, 2021

Quick fix for #7864 (Simple search of most metadata fields broken) #7865

Merged

This was referenced Jun 22, 2021

Custom Metadata: support boolean values #7961

Open

Custom Metadata: support time values #7960

Open

poikilotherm added a commit to poikilotherm/dataverse that referenced this issue Aug 26, 2021

feat(index): add many more SolrFieldTypes usable within the index (an…

8f1866e

…d present in the default schema) IQSS#5989

poikilotherm added a commit to poikilotherm/dataverse that referenced this issue Aug 26, 2021

feat(index): add conversion from SolrJ field maps to our SolrField mo…

b5a6c9b

…del map IQSS#5989

poikilotherm added a commit to poikilotherm/dataverse that referenced this issue Aug 26, 2021

feat(index): make SolrStaticField + SolrDynamicField buildable from S…

15e1034

…olrJ field Map<String,Object> IQSS#5989

poikilotherm added a commit to poikilotherm/dataverse that referenced this issue Aug 26, 2021

build(deps): update Testcontainers to 1.16.0 + add TC Solr support for …

1106710

…IQSS#5989

This was referenced Sep 1, 2021

Decoupling TSV metadata schema parsing from API endpoint #8085

Open

8085 decoupling tsv #8086

Closed

mreekie added the bk2211 label Nov 1, 2022

poikilotherm mentioned this issue Nov 30, 2022

Feature Request/Idea: Nested compound fields #9200

Open

mreekie removed the bk2211 label Jan 11, 2023

pdurbin added Feature: Search/Browse Type: Suggestion an idea User Role: Hackathon Participant Core developers, frequent contributors, drive-by contributors, etc. labels Oct 8, 2023

cmbz closed this as completed Aug 20, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add new Solr fields via an API call (investigation) #5989

Add new Solr fields via an API call (investigation) #5989

pkiraly commented Jul 3, 2019 •

edited

Loading

pdurbin commented Jul 3, 2019

4tikhonov commented Jul 3, 2019

pkiraly commented Jul 3, 2019 •

edited

Loading

pdurbin commented Jul 3, 2019

pkiraly commented Jul 4, 2019

pdurbin commented Sep 5, 2019

pdurbin commented Sep 5, 2019

poikilotherm commented Sep 5, 2019

pkiraly commented Oct 16, 2019

pdurbin commented Oct 16, 2019

pdurbin commented Oct 16, 2019 •

edited

Loading

pdurbin commented Oct 14, 2022

cmbz commented Aug 20, 2024

Add new Solr fields via an API call (investigation) #5989

Add new Solr fields via an API call (investigation) #5989

Comments

pkiraly commented Jul 3, 2019 • edited Loading

pdurbin commented Jul 3, 2019

4tikhonov commented Jul 3, 2019

pkiraly commented Jul 3, 2019 • edited Loading

pdurbin commented Jul 3, 2019

pkiraly commented Jul 4, 2019

pdurbin commented Sep 5, 2019

pdurbin commented Sep 5, 2019

poikilotherm commented Sep 5, 2019

pkiraly commented Oct 16, 2019

pdurbin commented Oct 16, 2019

pdurbin commented Oct 16, 2019 • edited Loading

pdurbin commented Oct 14, 2022

cmbz commented Aug 20, 2024

pkiraly commented Jul 3, 2019 •

edited

Loading

pkiraly commented Jul 3, 2019 •

edited

Loading

pdurbin commented Oct 16, 2019 •

edited

Loading