Test and move from Solr 8.X to Solr 9.X #9260

carlsonp · 2023-01-04T20:02:56Z

Overview of the Feature Request

Solr 8.X, specifically, 8.11.2 according to this page: "is the last release in the 8.x series." This means it likely won't be receiving any more updates, features, or patches. This also may have implications from a security perspective. The Dataverse Solr schema and major changes should be reviewed and tested.

Evaluate and update Solr schema if needed
Smoke-test with Solr 9.X
Update documentation
Document migration steps?

What kind of user is the feature intended for?

Sysadmin and developers

What inspired the request?

What existing behavior do you want changed?

Any brand new behavior do you want to add to Dataverse?

Any related open or closed issues to this feature request?

The text was updated successfully, but these errors were encountered:

pdurbin · 2023-01-17T15:50:48Z

@carlsonp hi! Thanks for pointing this out. As I said, Solr moves quickly! Again if you're interested in making a pull request, you might be able to follow the pattern of Solr upgrades in previous PRs:

pdurbin · 2023-02-07T20:28:04Z

Not sizing this for now. Hoping we can help @carlsonp make a pull request. 😄 Please let us know!

Please note that this issue and PR are related:

This might be somewhat easy to test by bumping the Solr version in dataverse-ansible (@donsizemore would know).

carlsonp · 2023-02-23T22:21:31Z

Migration away from Solr 8.X would help us in our intended deployment of Dataverse to Azure commercial. We're getting a lot of pushback on vulnerabilities both false-positives in the eyes of the Apache foundation as well as other vulnerabilities as CVEs. From scanning tools, they're picking up these CVEs and our Cybersecurity team wants them to be patched. We aren't interested in forking or patching upstream Solr so it seems Solr 9.X is a better longer-term solution. When I push and scan Solr 9.X there are less CVEs listed. This could be a major hold-up or full blocker in our ability to run Dataverse.

pdurbin · 2023-02-28T19:42:01Z

@carlsonp makes sense. Would you be able to email a list of Solr CVEs your tool is picking for Solr 8.11.1 (the version we use currently) to security@dataverse.org?

mreekie · 2023-02-28T21:30:30Z

Grooming:

Including as part of Payara 6 upgrade deliverable

donsizemore · 2023-03-06T16:03:42Z

@pdurbin I just heard from our security group about my brand-new private Dataverse instances. Per Solr's security page https://solr.apache.org/security.html we're not realistically vulnerable to the CVEs in question, but I opened https://github.com/IQSS/dataverse-security/issues/79 in response to the newer CVE.

carlsonp · 2023-03-06T23:14:42Z

@carlsonp makes sense. Would you be able to email a list of Solr CVEs your tool is picking for Solr 8.11.1 (the version we use currently) to security@dataverse.org?

This is complete.

cmbz · 2023-05-15T18:54:27Z

Prioritizing Payara 6 release
Sizing needed

cmbz · 2023-05-17T15:34:40Z

Sized as 33
Note: Has some requirements that might impact upgrade

Update all Solr references from 8.11.1 to 9.3.0 and make sure you can spin up containers. schema.xml was copied from solr-9.3.0/server/solr/configsets/_default/conf/managed-schema.xml (Note managed-schema.xml as there is no schema.xml upstream.) solrconfig.xml was copied from solr-9.3.0/server/solr/configsets/_default/conf/solrconfig.xml API tests will fail, of course, because we have not yet added our Dataverse-specific config. We'll be moving it from these files and then deleting them: conf/solr/8.11.1/schema.xml conf/solr/8.11.1/solrconfig.xml

In an earlier commit, a vanilla 9.3.0/schema.xml was added. Now we add the Dataverse-specific fields we need. We also delete the old 8.11.1/schema.xml file.

Note that even with these tweaks there are still errors. The current error is "possible analysis error".

"To improve security, XSLTResponseWriter has been moved to the scripting Module instead of shipping as part of Solr core. This module needs to be enabled explicitly." -- https://github.com/apache/solr/blob/main/solr/solr-ref-guide/modules/upgrade-notes/pages/major-changes-in-solr-9.adoc See also https://stackoverflow.com/questions/74118537/solr-9-org-apache-solr-common-solrexceptionorg-apache-solr-common-solrexcepti

pdurbin · 2023-08-16T21:18:47Z

@carlsonp I finally made a PR!

Upgrade Solr to 9.3.0 #9787

Please come back! ❤️ It's even containerized!!!11! 😄 🚀 🎉

Conflicts: downloads/download.sh scripts/vagrant/setup.sh

carlsonp · 2023-08-20T00:05:55Z

Awesome! :D

This reverts commit 8d37b0f.

We don't need the scripting module anymore.

pdurbin mentioned this issue Feb 7, 2023

Upgrade to Payara 6 and from EE 8 to EE 10 (end of Payara 5 Community Edition security patches Q2/2022) #8305

Closed

mreekie added the D: Payara 6 Upgrade Issues and PRs about the move to Jakarta EE 10 + Payara 6 label Feb 28, 2023

scolapasta added this to the 6.0 milestone May 15, 2023

cmbz added the Size: 30 A percentage of a sprint. 21 hours. (formerly size:33) label May 17, 2023

pdurbin self-assigned this Aug 14, 2023

pdurbin added a commit that referenced this issue Aug 15, 2023

add Dataverse fields to 9.3.0 Solr schema.xml #9260

055597f

In an earlier commit, a vanilla 9.3.0/schema.xml was added. Now we add the Dataverse-specific fields we need. We also delete the old 8.11.1/schema.xml file.

pdurbin added a commit that referenced this issue Aug 15, 2023

additional Solr 9 config tweaks #9260

df00469

Note that even with these tweaks there are still errors. The current error is "possible analysis error".

pdurbin added a commit that referenced this issue Aug 16, 2023

replace vanilla Solr 9.3.0 config with our 8.11.1 #9260

5a68632

pdurbin added a commit that referenced this issue Aug 16, 2023

add Solr release note #9260

bb01c6e

pdurbin mentioned this issue Aug 16, 2023

Upgrade Solr to 9.3.0 #9787

Merged

pdurbin removed their assignment Aug 16, 2023

pdurbin added a commit that referenced this issue Aug 18, 2023

Merge branch 'develop' into 9260-solr930 #9260

e6928ea

Conflicts: downloads/download.sh scripts/vagrant/setup.sh

pdurbin added a commit that referenced this issue Aug 21, 2023

Merge branch 'develop' into 9260-solr930 #9260

c7e2e23

pdurbin added a commit that referenced this issue Aug 22, 2023

double sleep in test #9260

926e6ea

pdurbin added a commit that referenced this issue Aug 23, 2023

explain more about Solr modules #9260

8d37b0f

pdurbin added a commit that referenced this issue Aug 23, 2023

Merge branch 'develop' into 9260-solr930 #9260

c21d8e6

pdurbin added a commit that referenced this issue Aug 24, 2023

Revert "explain more about Solr modules #9260" #9260

e5be4be

This reverts commit 8d37b0f.

pdurbin added a commit that referenced this issue Aug 24, 2023

rm scripting module, XSLTResponseWriter removed #9260

8666472

We don't need the scripting module anymore.

pdurbin added a commit that referenced this issue Aug 24, 2023

update release note, fresh index a must #9260

022bf06

pdurbin added a commit that referenced this issue Aug 24, 2023

delete :PublicInstall setting before and after test run #9260

137b1ba

pdurbin added a commit that referenced this issue Aug 24, 2023

update SolrJ to 9.3.0 #9260

81490f8

pdurbin added a commit that referenced this issue Aug 28, 2023

clarify supported version #9260

76902ed

pdurbin added a commit that referenced this issue Aug 28, 2023

handle custom metadata blocks #9260

2a74c6b

pdurbin added a commit that referenced this issue Aug 28, 2023

Merge branch 'develop' into 9260-solr930 #9260

e22e359

pdurbin added a commit that referenced this issue Aug 28, 2023

more detailed Solr release note #9260

be34440

pdurbin added a commit that referenced this issue Aug 28, 2023

fix reindex command #9260

2308c10

kcondon closed this as completed in #9787 Aug 28, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Test and move from Solr 8.X to Solr 9.X #9260

Test and move from Solr 8.X to Solr 9.X #9260

carlsonp commented Jan 4, 2023

pdurbin commented Jan 17, 2023

pdurbin commented Feb 7, 2023

carlsonp commented Feb 23, 2023 •

edited

Loading

pdurbin commented Feb 28, 2023

mreekie commented Feb 28, 2023 •

edited

Loading

donsizemore commented Mar 6, 2023 •

edited

Loading

carlsonp commented Mar 6, 2023

cmbz commented May 15, 2023

cmbz commented May 17, 2023

pdurbin commented Aug 16, 2023

carlsonp commented Aug 20, 2023

Test and move from Solr 8.X to Solr 9.X #9260

Test and move from Solr 8.X to Solr 9.X #9260

Comments

carlsonp commented Jan 4, 2023

pdurbin commented Jan 17, 2023

pdurbin commented Feb 7, 2023

carlsonp commented Feb 23, 2023 • edited Loading

pdurbin commented Feb 28, 2023

mreekie commented Feb 28, 2023 • edited Loading

donsizemore commented Mar 6, 2023 • edited Loading

carlsonp commented Mar 6, 2023

cmbz commented May 15, 2023

cmbz commented May 17, 2023

pdurbin commented Aug 16, 2023

carlsonp commented Aug 20, 2023

carlsonp commented Feb 23, 2023 •

edited

Loading

mreekie commented Feb 28, 2023 •

edited

Loading

donsizemore commented Mar 6, 2023 •

edited

Loading