Skip to content
This repository has been archived by the owner on Sep 16, 2019. It is now read-only.

Adds option to override the theme's icons through monochrome icons and fixes string encoding issues #25

Closed
wants to merge 3 commits into from
Closed

Adds option to override the theme's icons through monochrome icons and fixes string encoding issues #25

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
1ea9899
adds option to override theme's own through monochrome icons
kostrykin Aug 2, 2016
8e0cec1
adds support for Zotero's strange string encodings
kostrykin Aug 2, 2016
8996381
fixes version from 2.2.1 to 2.1.1
kostrykin Aug 2, 2016
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion libqnotero/qnotero.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@ class Qnotero(QMainWindow, UiLoader):

"""The main class of the Qnotero GUI"""

version = '2.1.0'
version = '2.2.1'

def __init__(self, systray=True, debug=False, reset=False, parent=None):

Expand Down
28 changes: 28 additions & 0 deletions libzotero/libzotero.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,33 @@
import time
from libzotero.zotero_item import zoteroItem as zotero_item

def decode_zotero_str(raw, max_iters=3):
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For me, this completely breaks the PDF attachments. This appears to be due to my own weird way of trying to fix the encoding. If I comment the line below out (from LibZotero.update()), it works again on my system.

item_attachment = item_attachment.encode('latin-1').decode('utf-8')

However, the fact that a patch that for you seems to fix things, breaks things for me makes me a bit worried. On what operating system do you work? It's not unthinkable that the encoding differs between operating systems.

Copy link
Contributor Author

@kostrykin kostrykin Oct 19, 2016 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm on Linux. I have that fork running on 3 different systems,

  • two of which are Ubuntu 14.04 based Elementary OS Freya
  • and one is Ubuntu 16.04 based Elementary OS Loki.

What operating system are you on?

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What operating system are you on?

I'm on Ubuntu 16.04.

It has been observerd that Zotero encodes its strings sometimes twice recursively. The modifications of this commit try to cope with that by decoding each string up to three times.

What makes you think this? Is this a conclusion based on the fact the decoding multiple times resolves the encoding issues, at least in some cases? Or is it a known bug in the Zotero code?

I'll think about this. The combination of my odd hack (clearly dating from before understood how encoding really works) and your fix is nonsensical; but it may be that your fix on its own is correct. But I'd like to do some more tests before I push another fix that actually doesn't fix things.

(As you may have noticed, I don't have that much time to work Qnotero anymore. But I'll get to it!)

Copy link
Contributor Author

@kostrykin kostrykin Oct 19, 2016 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sounds very reasonable!

To answer your question: My fix was based on the observation that Zotero encoded 'Öf' (a two-letters string, where the first letter is "Ö", that is a non-ASCII) as the five-bytes sequence \xc3\x83\xc2\x96\x66. This is strange, because UTF-8 encodes ASCII letters as single bytes and non-ASCII characters as a sequence of two bytes. Empirically, I ruled out the possibility of that being a UTF-16 encoding. So why are two letters encoded as five bytes?

In [14]: '\xc3\x83\xc2\x96\x66'.decode('utf-8')
Out[14]: u'\xc3\x96f'

Turns out, that \xc3\x83\xc2\x96\x66 is the UTF-8 encoding for the three-characters-string \xC3\x96f (note the f at the end). The first two bytes \xC3\x96 are in turn the UTF-8 representation of Ö. So I suspect that this is an odd behaviour from Zotero.

Update: In an earlier version of this, I miscounted the number of bytes in the exemplary sequence and wrote that it were 4, although it are 5.


"""
Decodes a string from the Zotero database.

There seems to be an issue with the way, Zotero encodes
strings in its database: For some reason, Zotero encodes
them *twice* with UTF-8, at least sometimes. This
functions aims to decode such strings, even if they were
encoded more than twice.

For example, Zotero was observed to encode

u'Öf' as b'\xc3\x83\xc2\x96\x66',

which is the UTF-8 encoding for u'\xC3\x96f', where
u'\xC3\x96' is *in turn* an UTF-8 representation of u'Ö'.
"""

result = str(raw, 'utf-8')
for iter in range(max_iters):
try:
result = str(bytes([ord(c) for c in result]), 'utf-8')
except:
break
return result

class LibZotero(object):

"""
Expand Down Expand Up @@ -170,6 +197,7 @@ def update(self, force=False):
# Copy the zotero database to the gnotero copy
shutil.copyfile(self.zotero_database, self.gnotero_database)
self.conn = sqlite3.connect(self.gnotero_database)
self.conn.text_factory = decode_zotero_str
self.cur = self.conn.cursor()
# First create a list of deleted items, so we can ignore those later
deleted = []
Expand Down