[fetcher] howto ignore ssl errors

[mishako]

Issue by pumpadump
Thursday Feb 04, 2016 at 20:56 GMT
Originally opened as https://github.com/rometools/rome-fetcher/issues/22

---

Hi,

I have a use case where i would like to retrieve a rss feed from sites that have invalid or broken certificates. Is there a possibility to configure the feedfetcher to ignore all ssl exceptions?

[mishako]

Comment by PatrickGotthard
Friday Feb 05, 2016 at 07:12 GMT

---

Hi pumpadump,

could you please give us the Stacktrace and the URL of the feed?

Regards,
Patrick

[pumpadump]

Hi Patrick,
i have the several feeds that fail like this:

javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
    at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
    at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1949)
    at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:302)
    at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:296)
    at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1509)
    at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:216)
    at sun.security.ssl.Handshaker.processLoop(Handshaker.java:979)
    at sun.security.ssl.Handshaker.process_record(Handshaker.java:914)
    at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1062)
    at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1375)
    at sun.security.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:747)
    at sun.security.ssl.AppOutputStream.write(AppOutputStream.java:123)
    at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
    at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
    at org.apache.commons.httpclient.HttpConnection.flushRequestOutputStream(HttpConnection.java:828)
    at org.apache.commons.httpclient.HttpMethodBase.writeRequest(HttpMethodBase.java:2116)
    at org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java:1096)
    at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:398)
    at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171)
    at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)
    at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323)
    at com.rometools.fetcher.impl.HttpClientFeedFetcher.retrieveFeed(HttpClientFeedFetcher.java:242)
    at com.rometools.fetcher.impl.HttpClientFeedFetcher.retrieveFeed(HttpClientFeedFetcher.java:157)
...

Affected feeds:

https://carnegiescience.edu/rss/pressrelease
https://www.acid21.com/rss.xml
https://www.affiliprint.com/de/feed/
https://blog.opo.de/feed/
https://ilovespa.de/feed/
https://inpetso.de/feed/
https://www.tektoria.de/feed/
https://www.volkerkoenig.de/feed/

I also get this exception:

javax.net.ssl.SSLException: Received fatal alert: internal_error
    at sun.security.ssl.Alerts.getSSLException(Alerts.java:208)
    at sun.security.ssl.Alerts.getSSLException(Alerts.java:154)
    at sun.security.ssl.SSLSocketImpl.recvAlert(SSLSocketImpl.java:2023)
    at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1125)
    at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1375)
    at sun.security.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:747)
    at sun.security.ssl.AppOutputStream.write(AppOutputStream.java:123)
    at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
    at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
    at org.apache.commons.httpclient.HttpConnection.flushRequestOutputStream(HttpConnection.java:828)
    at org.apache.commons.httpclient.HttpMethodBase.writeRequest(HttpMethodBase.java:2116)
    at org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java:1096)
    at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:398)
    at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:171)
    at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:397)
    at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:323)
    at com.rometools.fetcher.impl.HttpClientFeedFetcher.retrieveFeed(HttpClientFeedFetcher.java:242)
    at com.rometools.fetcher.impl.HttpClientFeedFetcher.retrieveFeed(HttpClientFeedFetcher.java:157)

for these feeds:

https://pascalebeier.de/feed/
https://hikinginfinland.com/atom.xml
https://e-commerce-blog.de/feed/
https://diemodernekatze.de/shop/feed/

Most of them seem to work in a standard web browser.

Java Version is:

java version "1.8.0_72"
Java(TM) SE Runtime Environment (build 1.8.0_72-b15)
Java HotSpot(TM) 64-Bit Server VM (build 25.72-b15, mixed mode)

Rome Version is:
1.5.1

[PatrickGotthard]

Hi @pumpadump,

currently we are discussing about the future of rome fetcher. The simplest/fastest solution for your problem will be that you download the feed yourself (without rome-fetcher) using e.g. Apache HttpClient and then parse the feed with rome.

Is that a viable solution for you?

Regards,
Patrick

[mishako]

The first error is not caused by Rome. Same exception is thrown when running this code:
new java.net.URL("https://blog.opo.de/feed/").openConnection().getInputStream();
Not sure why though. I guess any Java program will fail downloading this URL.

I can't reproduce the second exception.

[pumpadump]

I know that these errors are not caused by Rome. But if i build my own feed fetching i can circumvent them for example with configuring the httpclient builder like that:

SSLContextBuilder builder = new SSLContextBuilder();
builder.loadTrustMaterial(null, new TrustSelfSignedStrategy());
SSLConnectionSocketFactory sslsf = new SSLConnectionSocketFactory(builder.build());
HttpClientBuilder ClientBuilder = HttpClients.custom().setSSLSocketFactory(sslsf);

But if i want to use rome i cannot configure the http client.
@PatrickGotthard
Yes its a viable solution, i just thought it would be nice if i could configure this aspect in rome directly.

[PatrickGotthard]

This is a case where rome-fetcher is too inflexible. Please have a look at #276, we will remove support for rome-fetcher in the next major version.