column description bug

[reza1615]

In the column description, these metric have bugs

• Rows w/ Hidden char: count Arabic charters as hidden char for example count تست as hidden which is not hidden
• Rows w/ accent: count as
  • Num
  • Accent
  • Hidden
• Rows w/ number: doesn't count number form example if a row is foo123 it doesn't count it as num

Test: on http://alphatechadmin.pythonanywhere.com/dtale/main/4

• In the str_val modify cells as
  • 123
  • تست
  • foo123
  • fooÀ
  • ZWNJ

The correct column description should be

• numeric chars: 2
• accent chars: 1
• hidden char: 1

I checked the regexes in the #335 and it was ok
https://regex101.com/r/GPxA1F/1
https://regex101.com/r/GPxA1F/2

[aschonfeld]

So the issue with numeric chars was just a front-end mapping issue. Accent charts seems to be fine using the data you gave me:

This is the regex I'm using to compute hidden characters:

from string import printable
df['var1'].str.count(r'[^{}]+'.format(printable)`).astype(bool).sum()

Looks like it is saying that تست, fooÀ & ZWNJ contain hidden characters. Is there something wrong with string.printable?

[reza1615]

printable just check ASCII table so it is perfect for English language and doesn't support accent characters and Unicode characters

• for numeric chars please use [\d]+ it supports all languages numbers like ۱۲۳123
• for hidden chars please use this regex

printable =r'\w \!\"#\$%&\'\(\)\*\+,\-\./:;<»«؛،ـ\=>\?@\[\\\]\^_\`\{\|\}~'
df['var1'].str.count(r'[^{}]+'.format(printable)`).astype(bool).sum()

test the regex: https://regex101.com/r/cF5AVW/6

[reza1615]

Also please update the Cleaning function (remove hidden characters)

[reza1615]

And remove numbers cleaning function

[aschonfeld]

fixed in v1.31.0