Multi-file downloads not working in the UI in 6.1 with "CC BY 4.0", possibly other licenses #10251
Comments
lmaylein
changed the title
|
@lmaylein just to double-check, does your Solr jetty.xml increase requestHeaderSize from 8192 to 102400? |
|
@donsizemore: |
As in, not related to the other, solr error message? |
At least it doesn't always appear when I try to download several files at the same time. |
|
I have just found another dataset where the download worked. Maybe the problem is related to this error message: Examples: |
|
FWIW: The bug in the stacktrace above is that some download path has a null downloadType and assumes it is non-null. I added that with the Globus PR. It would be easy to make that null-safe. I'm not sure how that relates to intermittent external zipper problems though. |
|
At least the download seems to work for all datasets with "Custom Dataset Terms" and a value at "Terms of Use". |
|
Very interesting. Also, it sounds like it may not be related to the zipper specifically(?). |
|
Is there anything in the server.log immediately after an attempt to download multiple files in datasets where it isn't working? |
At least we have not found any other errors so far. |
Immediately after the download attempt, I cannot find any message at all in the server.log. Nothing that has any relation to the time of the download attempt. |
|
When clicking on "Download" the browser initiates three requests: POST https://heidata.uni-heidelberg.de/dataset.xhtml?persistentId=doi:10.11588/data/DDJFGS (Status 200) GET https://heidata.uni-heidelberg.de/jakarta.faces.resource/faces.js.xhtml?ln=jakarta.faces (Status 200) POST https://heidata.uni-heidelberg.de/dataset.xhtml?persistentId=doi:10.11588/data/DDJFGS (Status 200) If I do the same with a dataset where the download works (Custom Dataset Terms), I get the same three requests, but then the modal window appears with the dataset terms, which is not the case with the other datasets. |
|
Thank you. |
|
When it comes to multi-file downloads working in some datasets vs. not working in others - is it just a matter of "working when the terms-of-use popup is required; not working otherwise"? |
Correct. No entry was created the database table. |
It seems so. But I'm not sure whether a popup shouldn't also appear in the other cases (e.g. https://doi.org/10.11588/data/DDJFGS). I have also checked the browser console to see if a JavaScript error is being logged there. But this is not the case. |
|
These messages apparently also appeared in our payara5 (dataverse 5.14) server.log: _[2024-01-18T09:51:28.343+0100] [Payara 5.2022.3] [INFO] [] [edu.harvard.iq.dataverse.DatasetVersion] [tid: _ThreadID=105 ThreadName=http-thread-pool::jk- |
|
Funnily enough, after pressing the download button (where nothing happens in the user interface), the image for the license button is loaded (which would presumably be displayed in the popup): |
|
... and after pressing the download button, the the relevant
seems to be filled with the correct content (e.g. buttons) but it is still hidden. If I force my browser to show this dialog, the "Accept" button works and the data is downloaded. |
|
Since this message from the server log refers to the guestbook-terms-popup, it probably does have something to do with the problem: |
I withdraw this statement see #10253 |
|
Thank you for all the details. |
What I really meant was, I was wondering if this was not a zipper problem per se, but a general multi-file download problem. I.e., if the problem(s) you were seeing would be there regardless of whether the external zipper was configured or not. |
|
For the record, I haven't been able to reproduce this on my own dev. instance, so far. But I'm going to try with a cloned copy of our prod. instance. (Our real prod. instance hasn't been upgraded to 6.1 yet) |
Reload of https://heidata.uni-heidelberg.de/dataset.xhtml?persistentId=doi:10.11588/data/UUMEDI => same problem |
https://doi.org/10.11588/data/DDJFGS (for example) |
And I just tried the "working dataset" - RAS7U7, downloaded a 2 file zip, and it was clearly produce internally, without the zipper. |
When I download individual, single files from this datasets, I don't get any terms of use popups either, and the downloads are working. Are there actually terms of use and/or custom guestbook enabled in that dataset that I'm supposed to be seeing? |
|
No, there are no guestbooks enabled and this is how the terms edit form looks like (this is the default in our datasets): In this case, should the download start without a popup (with the license information)? |
|
If you're able to try with a fix re: #10251 (comment) i.e. just switching to check |
Unfortunately, I did not understand how I could check this. Is downloadType related to an attribute in the database so I could set it to a defined value? |
|
Sorry - this would be a code change. If you're not set up to build/test a new war file, we can try it when we have a reproducible test case (not sure if we already do or not as I've been skimming the comments). |
|
Yes, I just figured out how to reproduce this - and it's ridiculously easy! Changing it back to "CC 1.0" fixes it just as instantly. I'll try to get more developers to look into this tomorrow. |
|
Also, confirming that yes, with the "CC BY 4.0" license selected, in v6.0 the terms popup IS shown to the user on all downloads, single or multi-file. [edit: this was on my own dev. instance, with the instance-wide default still being "CC 1.0". the Heildelberg instance is using "CC BY 4.0" as the default, so their popup behavior would be different] |
landreev
changed the title
|
Will there be a new release including this patch? Or maybe you can build a new war in advance, which I can then test with our installation? The lack of download options is a major problem for our users. |
This (whether to release 6.1.1 with these fixes) is being discussed, and the decision is going to be made shortly. But we will definitely offer a patch as soon as we can, whether as a 6.1.1, or simply an updated .war file incorporating both this, and the fixes for the thumbnails issues (#10220). |
|
... and yes, thank you for offering to deploy and test a patched war file, I'll let you know as soon as it's ready. |
|
A patched war file with the fixes for this, and the issues with thumbnails (#10220) is available at https://github.com/donsizemore/dataverse_backports/releases/tag/6.1_20240126 (thank you, @donsizemore!). The bug fixes have already gone through our normal testing/QA process. We are still offering it as a prototype at this point, so, if you choose to deploy it, please test it carefully and let us know asap if you encounter any problems. |
|
Deployed on https://heidata.uni-heidelberg.de |
|
... both fixed bugs tested (Download: https://doi.org/10.11588/data/DDJFGS, Icons: https://heidata.uni-heidelberg.de/dataverse/arthistoricum). |
* Switch to per-pid-provider settings * partial refactor towards non-bean providers * ~auto refactor name/package, remove bean status * remove Util class * rename * add factories for all, fix providers, etc. * unmanaged providers * add getters * add name to cnstr, add cnstr for unmanaged, add auth/shoulder checks * update permalinks, add separator setting * no arg constructor * add unmanaged providers * check canManagePid instead * replace getBean(), compiles except for tests * update tests, comment out ones that are TBD * add clear method for testing * bugs - remove dup authority, fix name, add auth/sep/shoulder tests * make managed/excluded lists optional * fix name in generated pids * move setup to berforeall, add test of second permaprovider * provider name->id * adding label, more name->id * providerName->providerId * add factory map, lookups, add factory, perma parsing tests * first datacite parsing test/fix id in pid * rename class * move auth/shoulder check to lower level method * fix ids, fix managed list optional in fake * add effective pid generator logic * add effective pid generator tests * fix param order * fix perma handling of managed/excluded entries * add managed/excluded tests, cleanup * update pidprovider discovery to get effective one when necessary * replace all refs to global protocol/auth/shoulder settings xcept one keeping the new PidProviderFactoryBean.getDefaultPidGenerator() for now as a possible way to stay ~backward comaptible All the rest - tried to find the appropriate PidProvider to supply the values * first UI for setting Pid generator * typo * flyway script to add pid spec column * @autoservice and public class for loader discovery * minor cleanup/refactor * verify protocol/auth are set/match the provider plus cleanup * only call getGlobalId() when one should exist * force all calls to create identifier to set protocol/auth as well * move template to match refactor * require superuser to change PidProvider * cleanup * check can create method * make fake provider create file pids * typo - fix UI * return default instead of null for UI * unrelated - logic fix * partial support for legacy config - FAKE and DatCite - for testing * cleanup * style fail * fix test - don't reset list of providers * allow old aliases * reverse logic in datacite legacy creator, add null check * fix lookups, update test, test DataCite legacy * missing if! * disable obsolete test * updated docs * add test urls as default * cleanup -remove unused imports * unrelated link fix * fix for #10251 - sync terms popup required code * API calls for getting provider info and changing PID Generators * api docs * change level for entries to fix build error * typo in refs * fix indents * more bad refs * support for legacy hdl, perma, ezid * new packages for everyone! (refactor) * unused imports * fix cut/paste issues * add deprecation info * Apply suggestions from code review Co-authored-by: Oliver Bertuch <poikilotherm@users.noreply.github.com> * reorg/update imports * revert 2e41b9e * deprecate old settings * Change error handling and warnings per review * Add testing for a valid PID generator as a config test * formatting, switch if /else logic per review * add deprecation * move pid provider's dir setting to spi scope * change flyway name, tweak release note, delete unused test class * temporary flyway change * use new settings in install * Revert "temporary flyway change" This reverts commit 7106ef6. * fix rest api setting * handle spaces in the pidproviders setting * add note in Harvard setup * refactoring/cleaning DataCite provider, drop cache * moving XmlMetadataTemplate to doi package * missing import * move xml file to match package * minor fixes, make getPidStatus visible in test * disabled test of DPI lifecycle * update installer/docs to not talk about a partial DataCite test setup * remove legacy setting * indent issue * missing ) * fix setting name * remove obsolete settings * add defaults * add valid fake pid setup for docker * also adding pid config to the -dev yml * Update docker-compose-dev.yml Co-authored-by: Steven Winship <39765413+stevenwinship@users.noreply.github.com> * Update docker/compose/demo/compose.yml Co-authored-by: Steven Winship <39765413+stevenwinship@users.noreply.github.com> * Update docker-compose-dev.yml Co-authored-by: Steven Winship <39765413+stevenwinship@users.noreply.github.com> * Update docker/compose/demo/compose.yml Co-authored-by: Steven Winship <39765413+stevenwinship@users.noreply.github.com> --------- Co-authored-by: Oliver Bertuch <poikilotherm@users.noreply.github.com> Co-authored-by: qqmye <qqmye@BOOK-2CB3G91HHU> Co-authored-by: Steven Winship <39765413+stevenwinship@users.noreply.github.com>
(cherry picked from commit 5292682)
|
Another report of this bug came in today: https://dataverse.zulipchat.com/#narrow/stream/378866-troubleshooting/topic/enabling.20guestbook.20at.20request.20for.20instance.20or.20collection/near/454931219 The solution should be to upgrade to 6.2 or later, of course. |
(cherry picked from commit 5198416)
After upgrading from Dataverse 5.14 to 6.1 (via 6.0), the zip download (standalone zipper) is no longer available in our dataverse installation https://heidata.uni-heidelberg.de. What could have gone wrong during the upgrade?
I have set http://localhost:8080/api/admin/settings/:CustomZipDownloadServiceUrl again. But this did not help.
BTW: There are a lot of such messages in our server.log:
[2024-01-22T13:23:35.522+0100] [Payara 6.2023.8] [INFO] [] [edu.harvard.iq.dataverse.search.SearchIncludeFragment] [tid: _ThreadID=101 _ThreadName=http-thread-pool::jk-connector
(1)] [timeMillis: 1705926215522] [levelValue: 800] [[ Search Syntax Error: Error from server at http://localhost:8983/solr/collection1: Query contains too many nested clauses; maxClauseCount is set to 1024]]
And then I find these messages in the server.log. However, they are not related in terms of time.
[2024-01-22T13:35:57.060+0100] [Payara 6.2023.8] [SEVERE] [] [jakarta.enterprise.resource.webcontainer.faces.context] [tid: _ThreadID=103 ThreadName=http-thread-pool::jk-connector(3)] [timeMillis: 1705926957060] [levelValue: 1000] [[
jakarta.ejb.EJBException: Cannot invoke "String.equals(Object)" because "downloadType" is null
jakarta.ejb.EJBException: Cannot invoke "String.equals(Object)" because "downloadType" is null
at com.sun.ejb.containers.EJBContainerTransactionManager.processSystemException(EJBContainerTransactionManager.java:723)
at com.sun.ejb.containers.EJBContainerTransactionManager.completeNewTx(EJBContainerTransactionManager.java:652)
at com.sun.ejb.containers.EJBContainerTransactionManager.postInvokeTx(EJBContainerTransactionManager.java:482)
at com.sun.ejb.containers.BaseContainer.postInvokeTx(BaseContainer.java:4601)
at com.sun.ejb.containers.BaseContainer.postInvoke(BaseContainer.java:2134)
at com.sun.ejb.containers.BaseContainer.postInvoke(BaseContainer.java:2104)
at com.sun.ejb.containers.EJBLocalObjectInvocationHandler.invoke(EJBLocalObjectInvocationHandler.java:220)
at com.sun.ejb.containers.EJBLocalObjectInvocationHandlerDelegate.invoke(EJBLocalObjectInvocationHandlerDelegate.java:90)
at jdk.proxy73/jdk.proxy73.$Proxy411.writeGuestbookAndStartBatchDownload(Unknown Source)
at edu.harvard.iq.dataverse.EJB31_Generated__FileDownloadServiceBean__Intf____Bean.writeGuestbookAndStartBatchDownload(Unknown Source)
at edu.harvard.iq.dataverse.FileDownloadHelper.writeGuestbookAndStartDownload(FileDownloadHelper.java:88)
at edu.harvard.iq.dataverse.FileDownloadHelper$Proxy$$$_WeldClientProxy.writeGuestbookAndStartDownload(Unknown Source)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:568)
at jakarta.el.ELUtil.invokeMethod(ELUtil.java:227)
at jakarta.el.BeanELResolver.invoke(BeanELResolver.java:481)
at jakarta.el.CompositeELResolver.invoke(CompositeELResolver.java:198)
at org.glassfish.expressly.parser.AstValue.invoke(AstValue.java:245)
at org.glassfish.expressly.MethodExpressionImpl.invoke(MethodExpressionImpl.java:248)
at org.jboss.weld.module.web.util.el.ForwardingMethodExpression.invoke(ForwardingMethodExpression.java:40)
at org.jboss.weld.module.web.el.WeldMethodExpression.invoke(WeldMethodExpression.java:50)
at com.sun.faces.facelets.el.TagMethodExpression.invoke(TagMethodExpression.java:70)
at jakarta.faces.event.MethodExpressionActionListener.processAction(MethodExpressionActionListener.java:111)
at jakarta.faces.event.ActionEvent.processListener(ActionEvent.java:72)
at jakarta.faces.component.UIComponentBase.broadcast(UIComponentBase.java:481)
at jakarta.faces.component.UICommand.broadcast(UICommand.java:197)
at jakarta.faces.component.UIViewRoot.broadcastEvents(UIViewRoot.java:858)
at jakarta.faces.component.UIViewRoot.processApplication(UIViewRoot.java:1332)
at com.sun.faces.lifecycle.InvokeApplicationPhase.execute(InvokeApplicationPhase.java:56)
at com.sun.faces.lifecycle.Phase.doPhase(Phase.java:72)
at com.sun.faces.lifecycle.LifecycleImpl.execute(LifecycleImpl.java:159)
at jakarta.faces.webapp.FacesServlet.executeLifecyle(FacesServlet.java:691)
at jakarta.faces.webapp.FacesServlet.service(FacesServlet.java:449)
at org.apache.catalina.core.StandardWrapper.service(StandardWrapper.java:1554)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:331)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:211)
at org.glassfish.tyrus.servlet.TyrusServletFilter.doFilter(TyrusServletFilter.java:83)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:253)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:211)
at org.ocpsoft.rewrite.servlet.RewriteFilter.doFilter(RewriteFilter.java:226)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:253)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:211)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:257)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:166)
at org.apache.catalina.core.StandardPipeline.doInvoke(StandardPipeline.java:757)
at org.apache.catalina.core.StandardPipeline.invoke(StandardPipeline.java:577)
at com.sun.enterprise.web.WebPipeline.invoke(WebPipeline.java:99)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:158)
at org.apache.catalina.connector.CoyoteAdapter.doService(CoyoteAdapter.java:372)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:239)
at com.sun.enterprise.v3.services.impl.ContainerMapper$HttpHandlerCallable.call(ContainerMapper.java:520)
at com.sun.enterprise.v3.services.impl.ContainerMapper.service(ContainerMapper.java:217)
at org.glassfish.grizzly.http.server.HttpHandler.runService(HttpHandler.java:174)
at org.glassfish.grizzly.http.server.HttpHandler.doHandle(HttpHandler.java:153)
at org.glassfish.grizzly.http.server.HttpServerFilter.handleRead(HttpServerFilter.java:196)
at org.glassfish.grizzly.filterchain.ExecutorResolver$9.execute(ExecutorResolver.java:88)
at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:246)
at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:178)
at org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:118)
at org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:96)
at org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:51)
at org.glassfish.grizzly.nio.transport.TCPNIOTransport.fireIOEvent(TCPNIOTransport.java:510)
at org.glassfish.grizzly.strategies.AbstractIOStrategy.fireIOEvent(AbstractIOStrategy.java:82)
at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.run0(WorkerThreadIOStrategy.java:83)
at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy$WorkerThreadRunnable.run(WorkerThreadIOStrategy.java:101)
at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:535)
at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:515)
at java.base/java.lang.Thread.run(Thread.java:842)
Caused by: java.lang.NullPointerException: Cannot invoke "String.equals(Object)" because "downloadType" is null
at edu.harvard.iq.dataverse.FileDownloadServiceBean.redirectToDownloadAPI(FileDownloadServiceBean.java:319)
at edu.harvard.iq.dataverse.FileDownloadServiceBean.writeGuestbookAndStartBatchDownload(FileDownloadServiceBean.java:128)
at edu.harvard.iq.dataverse.FileDownloadServiceBean.writeGuestbookAndStartBatchDownload(FileDownloadServiceBean.java:97)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:568)
at org.glassfish.ejb.security.application.EJBSecurityManager.runMethod(EJBSecurityManager.java:588)
at org.glassfish.ejb.security.application.EJBSecurityManager.invoke(EJBSecurityManager.java:408)
at com.sun.ejb.containers.BaseContainer.invokeBeanMethod(BaseContainer.java:4835)
at com.sun.ejb.EjbInvocation.invokeBeanMethod(EjbInvocation.java:653)
at com.sun.ejb.containers.interceptors.AroundInvokeChainImpl.invokeNext(InterceptorManager.java:834)
at com.sun.ejb.EjbInvocation.proceed(EjbInvocation.java:603)
at com.sun.ejb.containers.interceptors.SystemInterceptorProxy.doCall(SystemInterceptorProxy.java:163)
at com.sun.ejb.containers.interceptors.SystemInterceptorProxy.aroundInvoke(SystemInterceptorProxy.java:140)
at jdk.internal.reflect.GeneratedMethodAccessor229.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:568)
at com.sun.ejb.containers.interceptors.AroundInvokeInterceptor.intercept(InterceptorManager.java:888)
at com.sun.ejb.containers.interceptors.AroundInvokeChainImpl.invokeNext(InterceptorManager.java:833)
at com.sun.ejb.EjbInvocation.proceed(EjbInvocation.java:603)
at org.jboss.weld.module.ejb.AbstractEJBRequestScopeActivationInterceptor.aroundInvoke(AbstractEJBRequestScopeActivationInterceptor.java:72)
at org.jboss.weld.module.ejb.SessionBeanInterceptor.aroundInvoke(SessionBeanInterceptor.java:52)
at jdk.internal.reflect.GeneratedMethodAccessor209.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:568)
at com.sun.ejb.containers.interceptors.AroundInvokeInterceptor.intercept(InterceptorManager.java:888)
at com.sun.ejb.containers.interceptors.AroundInvokeChainImpl.invokeNext(InterceptorManager.java:833)
at com.sun.ejb.containers.interceptors.InterceptorManager.intercept(InterceptorManager.java:375)
at com.sun.ejb.containers.BaseContainer.__intercept(BaseContainer.java:4807)
at com.sun.ejb.containers.BaseContainer.intercept(BaseContainer.java:4795)
at com.sun.ejb.containers.EJBLocalObjectInvocationHandler.invoke(EJBLocalObjectInvocationHandler.java:212)
... 63 more
]]
The text was updated successfully, but these errors were encountered: