Redirection error in latest (2022-09) ifixit zim (English and Russian version) #826

kelson42 · 2022-10-03T10:11:24Z

From ifixit created by laggykiller: openzim/ifixit#86

The latest (2022-09) ifixit zim (English and Russian version) has redirection error ('redirected you too many times')

The zim in question is here:
https://download.kiwix.org/zim/ifixit/ifixit_en_all_2022-09.zim
https://download.kiwix.org/zim/ifixit/ifixit_ru_all_2022-09.zim

You can test it out now from here:
https://library.kiwix.org/ifixit_en_all_2022-09
https://library.kiwix.org/ifixit_ru_all_2022-09

The previous versions (2022-06) are normal:
https://download.kiwix.org/zim/ifixit/ifixit_en_all_2022-06.zim
https://download.kiwix.org/zim/ifixit/ifixit_ru_all_2022-06.zim

Other languages are not affected

When viewed from kiwix-serve, it produces redirection error:

When viewed from desktop application (e.g. Windows), the homepage is broken:

kelson42 · 2022-10-03T10:11:26Z

@kelson42 this might be a kiwix-serve issue. The ZIM entries look correct:

# this is OK. that's that's what it always looks like
zim.main_entry.is_redirect
> True
zim.main_entry.get_redirect_entry()
> Entry(url=Main-Page, title=Main-Page)

# here we see that we redirect from Main-Page to home/home. Again, that's normal. We do it everywhere
zim.main_entry.get_redirect_entry().is_redirect
> True
zim.main_entry.get_redirect_entry().get_redirect_entry()
> Entry(url=home/home, title=iFixit: The Free Repair Manual)

Problem revolves around the handling of /:

curl -I http://192.168.5.80:9999/ifixit_en_all_2022-09/
HTTP/1.1 302 Found
Connection: close
Content-Length: 0
Location: /ifixit_en_all_2022-09/
Access-Control-Allow-Origin: *
Cache-Control: no-cache, no-store, must-revalidate
Date: Mon, 03 Oct 2022 09:55:23 GMT

======================
Requesting :
full_url  : /ifixit_en_all_2022-09/
method    : OTHER (1)
version   : HTTP/1.1
request#  : 0
headers   :
 - accept : '*/*'
 - host : '192.168.5.80:9999'
 - user-agent : 'curl/7.79.1'
arguments :
Parsed :
full_url: /ifixit_en_all_2022-09/
url   : /ifixit_en_all_2022-09/
acceptEncodingDeflate : 0
has_range : 0
is_valid_url : 1
.............
** running handle_content
Response :
httpResponseCode : 302
headers :
 - Location: '/ifixit_en_all_2022-09/'
 - Access-Control-Allow-Origin: '*'
 - Cache-Control: 'no-cache, no-store, must-revalidate'
Request time : 0.005543s
----------------------

Accessing the home/home entry directly works as expected

curl -I http://192.168.5.80:9999/ifixit_en_all_2022-09/home/home
HTTP/1.1 200 OK
Connection: close
Content-Length: 21948
Content-Type: text/html
Access-Control-Allow-Origin: *
ETag: "1664790915539257/c"
Cache-Control: max-age=2723040, public
Date: Mon, 03 Oct 2022 09:57:59 GMT

======================
Requesting :
full_url  : /ifixit_en_all_2022-09/home/home
method    : OTHER (1)
version   : HTTP/1.1
request#  : 1
headers   :
 - accept : '*/*'
 - host : '192.168.5.80:9999'
 - user-agent : 'curl/7.79.1'
arguments :
Parsed :
full_url: /ifixit_en_all_2022-09/home/home
url   : /ifixit_en_all_2022-09/home/home
acceptEncodingDeflate : 0
has_range : 0
is_valid_url : 1
.............
** running handle_content
Found home/home
mimeType: text/html
Response :
httpResponseCode : 200
headers :
 - Content-Type: 'text/html'
 - Access-Control-Allow-Origin: '*'
 - ETag: '"1664791208912869/c"'
 - Cache-Control: 'max-age=2723040, public'
Request time : 0.003537s
----------------------

Kiwix-JS is not affected by this bug, nor is kiwix-desktop macOS (97)

kelson42 · 2022-10-03T10:11:26Z

Both were created using openzim/ifixit:0.2.1 so can't be related to a code change anywhere.

kelson42 · 2022-10-03T10:11:26Z

@rgaudin @benoit74 I strongly suspect this is linked to something which has changed upstream. Anyway the problem is accute! We should disable the recipes and remove the problematic ZIM files from the repo!

kelson42 · 2022-10-03T10:11:26Z

@rgaudin @benoit74 Indeed, looks super serious, the homepage seems to redirect on itself!!!

kelson42 · 2022-10-03T10:11:26Z

This is weird, I did not released any change in the scraper since previous
version.
Did you released any big changes in any underlying library ?

kelson42 · 2022-10-03T10:13:26Z

@veloman-yunkan I have migrated the ticket from the iFixit repo (but the comments are not in the right order). This is a blocker for 12.0.0 IMO, pretty unclear why such a problem has been unoticed so far.

rgaudin · 2022-10-03T10:37:56Z

Confirming that the 2022-06 version is not affected

❯ curl -I http://192.168.5.80:9999/ifixit_en_all_2022-06/
HTTP/1.1 302 Found
Connection: close
Content-Length: 0
Location: /ifixit_en_all_2022-06/home/home
Access-Control-Allow-Origin: *
Cache-Control: no-cache, no-store, must-revalidate
Date: Mon, 03 Oct 2022 10:33:27 GMT

❯ curl -I http://192.168.5.80:9999/ifixit_en_all_2022-06/home/home
HTTP/1.1 200 OK
Connection: close
Content-Length: 21948
Content-Type: text/html
Access-Control-Allow-Origin: *
ETag: "1664793096418630/c"
Cache-Control: max-age=2723040, public
Date: Mon, 03 Oct 2022 10:33:37 GMT

ZIM entries are similar:

zim.main_entry.is_redirect
> True
zim.main_entry.get_redirect_entry()
> Entry(url=Main-Page, title=Main-Page)

zim.main_entry.get_redirect_entry().is_redirect
> True
zim.main_entry.get_redirect_entry().get_redirect_entry()
> Entry(url=home/home, title=iFixit: The Free Repair Manual)

Metadata are similar as well (not that it should matter)

{'Counter': 'application/javascript=2;application/vnd.ms-fontobject=1;font/sfnt=1;font/woff=1;font/woff2=6;image/gif=3;image/jpeg=15;image/png=24;image/svg+xml=38;image/vnd.microsoft.icon=1;image/webp=409652;text/css=18;text/html=173708;text/x-component=1',
 'Creator': 'iFixit',
 'Date': '2022-06-16',
 'Description': "iFixit is a global community of people helping each other repair things. Let's fix the world, one device at a time. Troubleshoot with experts in the Answers forum—and build your own how-to guides to share with the world. Fix your Apple and Android devices—and buy all the parts and tools needed for your DIY repair projects.",
 'FaviconPath': 'illustration',
 'Language': 'eng',
 'Name': 'ifixit_en_all',
 'Publisher': 'openZIM',
 'Tags': 'iFixit;_videos:yes;_pictures:yes;_category:iFixit',
 'Title': 'iFixit: The Free Repair Manual'}

{'Counter': 'application/javascript=2;application/vnd.ms-fontobject=1;font/sfnt=1;font/woff=1;font/woff2=6;image/gif=3;image/jpeg=15;image/png=24;image/svg+xml=38;image/vnd.microsoft.icon=1;image/webp=417315;text/css=18;text/html=178324;text/x-component=1',
 'Creator': 'iFixit',
 'Date': '2022-09-14',
 'Description': "iFixit is a global community of people helping each other repair things. Let's fix the world, one device at a time. Troubleshoot with experts in the Answers forum—and build your own how-to guides to share with the world. Fix your Apple and Android devices—and buy all the parts and tools needed for your DIY repair projects.",
 'FaviconPath': 'illustration',
 'Language': 'eng',
 'Name': 'ifixit_en_all',
 'Publisher': 'openZIM',
 'Tags': '_pictures:yes;_category:iFixit;_videos:yes;iFixit',
 'Title': 'iFixit: The Free Repair Manual'}

veloman-yunkan · 2022-10-04T09:10:13Z

The issue with ifixit_en_all_2022-09.zim is that it contains an item at the empty path "":

$ zimdump list --url="" ifixit_en_all_2022-09.zim 
path: 
* title:          Sales Policies
* idx:            0
* type:           item
* mime-type:      text/html
* item size:      4547

Currently in kiwix-serve the main entry is returned for an empty ("") or root ("/") path only if no entry exists at that path in the ZIM file:

libkiwix/src/tools/archiveTools.cpp

Lines 161 to 171 in f6ae75e

    
           zim::Entry getEntryFromPath(const zim::Archive& archive, const std::string& path) 
        
           { 
        
             try { 
        
               return archive.getEntryByPath(path); 
        
             } catch (zim::EntryNotFound& e) { 
        
               if (path.empty() || path == "/") { 
        
                 return archive.getMainEntry(); 
        
               } 
        
             } 
        
             throw zim::EntryNotFound("Cannot find entry for non empty path"); 
        
           }

rgaudin · 2022-10-04T09:11:35Z

Ah! Very interesting ; I did not know that. Thanks. We'll fix the scraper. Leaving this open to discuss whether we still want that (weird?) behavior

veloman-yunkan · 2022-10-04T09:22:08Z

In any case, there is a bug in InternalServer::handle_content() which assumes that the entry fetched for an empty path ~~is a redirect entry~~ has to be converted to an HTTP redirect:

libkiwix/src/server/internalServer.cpp

Lines 1045 to 1050 in f6ae75e

    
           auto entry = getEntryFromPath(*archive, urlStr); 
        
           if (entry.isRedirect() || urlStr.empty()) { 
        
             // If urlStr is empty, we want to mainPage. 
        
             // We must do a redirection to the real page. 
        
             return build_redirect(bookName, getFinalItem(*archive, entry)); 
        
           }

Before this fix the root URL for a book was assumed to resolve to the main page. This was not true for ZIM files containing an entry at an empty path or with a path equal to "/", resulting in issue #826. The logic behind this behaviour is found in `kiwix::getEntryFromPath()`. The fix to that issue is a little more general and will result in an HTTP redirect in any case where `kiwix::getEntryFromPath(zim, path)` returns an entry with a real path different from the requested one. In particular, this will affect the behaviour on ZIM files with the old namespace scheme, where the requested resource - if not found - is also looked up in the 'A', 'I', 'J', and/or '-' namespaces. Now instead of returning the contents of that other resource an HTTP redirect response will be sent.

kelson42 · 2022-10-04T14:07:36Z

I guess we can close this now.

kelson42 added the bug label Oct 3, 2022

kelson42 added this to the 12.0.0 milestone Oct 3, 2022

kelson42 assigned veloman-yunkan Oct 3, 2022

kelson42 mentioned this issue Oct 3, 2022

Release 12.0.0 #810

Closed

13 tasks

veloman-yunkan mentioned this issue Oct 4, 2022

New logic for generating HTTP-redirects #827

Merged

mgautierfr closed this as completed in #827 Oct 4, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Redirection error in latest (2022-09) ifixit zim (English and Russian version) #826

Redirection error in latest (2022-09) ifixit zim (English and Russian version) #826

kelson42 commented Oct 3, 2022

kelson42 commented Oct 3, 2022

kelson42 commented Oct 3, 2022

kelson42 commented Oct 3, 2022

kelson42 commented Oct 3, 2022

kelson42 commented Oct 3, 2022

kelson42 commented Oct 3, 2022

rgaudin commented Oct 3, 2022

veloman-yunkan commented Oct 4, 2022 •

edited

Loading

rgaudin commented Oct 4, 2022

veloman-yunkan commented Oct 4, 2022 •

edited

Loading

kelson42 commented Oct 4, 2022

Redirection error in latest (2022-09) ifixit zim (English and Russian version) #826

Redirection error in latest (2022-09) ifixit zim (English and Russian version) #826

Comments

kelson42 commented Oct 3, 2022

kelson42 commented Oct 3, 2022

kelson42 commented Oct 3, 2022

kelson42 commented Oct 3, 2022

kelson42 commented Oct 3, 2022

kelson42 commented Oct 3, 2022

kelson42 commented Oct 3, 2022

rgaudin commented Oct 3, 2022

veloman-yunkan commented Oct 4, 2022 • edited Loading

rgaudin commented Oct 4, 2022

veloman-yunkan commented Oct 4, 2022 • edited Loading

kelson42 commented Oct 4, 2022

veloman-yunkan commented Oct 4, 2022 •

edited

Loading

veloman-yunkan commented Oct 4, 2022 •

edited

Loading