IMAP error after upgrade to 6.8.2

[LoserFriendly]

Hi there, today I've tried to upgrade Parsedmarc from version 6.4.1 to version 6.8.2. I've installed geoipupdate as required, I've also installed libxml2-dev and libxslt-dev because they were missing on my system (Ubuntu server 18.04.4 LTS). Elasticsearch/Kibana version is 6.8.6.
I receive the reports in the INBOX of a local user (dmparser) which reads them via IMAP locally, runs Parsedmarc and sends the results to Elasticsearch (on the same machine).
The "mail" folder in the home directory of the user contains Archive, Archive.Aggregate, Archive.Forensic, Archive.Invalid and Sent "folders". The INBOX is in /var/spool/mail/dmparser as usual. I'm running Parsedmarc in a pypy3 virtualenv, version pypy3.5-6.0.0.
This is my parsedmarc.ini:

[general]
save_aggregate = True
save_forensic = True
#strip_attachments_payloads = True
nameservers = 193.205.207.25,193.205.215.6
debug = True
#silent= False
output = /home/dmparser/parse_output
log_file = /var/log/parsedmarc.log

[imap]
host = 192.168.206.71
port = 143
ssl = False
user = dmparser
password =
watch = True

[elasticsearch]
hosts = 127.0.0.1:9200
ssl = False
index_suffix = unitntest
number_of_replicas = 0
number_of_shards = 1

Parsedmarc seems to issue the IMAP commands correctly:

Feb 5 14:47:11 smtp-test dovecot: auth: Debug: auth client connected (pid=28458)
Feb 5 14:47:11 smtp-test dovecot: auth: Debug: client in: AUTH#0111#011PLAIN#011service=imap#011secured#011session=LsywZNSd4p3AqM5H#011lip=192.168.206.71#011rip=192.168.206.71#011lport=143#011rport=40418#011resp=
Feb 5 14:47:11 smtp-test dovecot: auth-worker(28459): Debug: Loading modules from directory: /usr/lib/dovecot/modules/auth
Feb 5 14:47:11 smtp-test dovecot: auth-worker(28459): Debug: Module loaded: /usr/lib/dovecot/modules/auth/lib20_auth_var_expand_crypt.so
Feb 5 14:47:11 smtp-test dovecot: auth-worker(28459): Debug: passwd-file /etc/dovecot/users: Read 6 users in 0 secs
Feb 5 14:47:11 smtp-test dovecot: auth-worker(28459): Debug: pam(dmparser,192.168.206.71,): lookup service=imap
Feb 5 14:47:11 smtp-test dovecot: auth-worker(28459): Debug: pam(dmparser,192.168.206.71,): #1/1 style=1 msg=Password:
Feb 5 14:47:11 smtp-test dovecot: auth: Debug: client passdb out: OK#0111#011user=dmparser
Feb 5 14:47:11 smtp-test dovecot: auth: Debug: master in: REQUEST#0112491809793#01128458#0111#0110db591db74d2fbba9833cb551bf427f8#011session_pid=28460#011request_auth_token
Feb 5 14:47:11 smtp-test dovecot: auth-worker(28459): Debug: passwd(dmparser,192.168.206.71,): lookup
Feb 5 14:47:11 smtp-test dovecot: auth: Debug: master userdb out: USER#0112491809793#011dmparser#011system_groups_user=dmparser#011uid=1002#011gid=1002#011home=/home/dmparser#011auth_token=3cf7f1596a8ffb0005fa61858cced34cd2ed603f
Feb 5 14:47:11 smtp-test dovecot: imap-login: Login: user=, method=PLAIN, rip=192.168.206.71, lip=192.168.206.71, mpid=28460, secured, session=
Feb 5 14:47:11 smtp-test dovecot: imap(dmparser): Connection closed (CREATE finished 0.010 secs ago) in=152 out=1623

but in the parsedmarc.log I see:
2020-02-05 14:47:11,510 - ERROR - [cli.py:606] - IMAP Error: create failed: [CANNOT] Mailbox doesn't allow inferior mailboxes (0.001 + 0.000 secs).

and the messages in the INBOX are not processed, probably because the service doesn't even start:

root@smtp-test:~# service parsedmarc status
● parsedmarc.service - parsedmarc mailbox watcher
Loaded: loaded (/etc/systemd/system/parsedmarc.service; enabled; vendor preset: enabled)
Active: activating (auto-restart) (Result: exit-code) since Wed 2020-02-05 15:02:18 CET; 4min 14s ago
Docs: https://domainaware.github.io/parsedmarc/
Process: 28636 ExecStart=/usr/local/bin/parsedmarc -c /etc/parsedmarc.ini (code=exited, status=1/FAILURE)
Main PID: 28636 (code=exited, status=1/FAILURE)

Anyone could point me in the right direction, please? I'm restoring the snapshot I took before upgrading in the meantime.

Any help is really appreciated, thank you
Sebastiano

[seanthegeek]

Hi,

I don't think upgrading/downgrading parsedmarc is going to solve this issue

Here's what I found https://help.bittitan.com/hc/en-us/articles/115008253328-Mailbox-doesn-t-allow-inferior-mailboxes

Due to a defective implementation, the Destination IMAP server does not support the idea that folders may contain both messages and subfolders. In other terms, the Destination may allow you to have a folder A containing a subfolder B, or a folder A containing email items, but not both at the same time. So, if folder A contains email items, trying to create subfolder B results in a "Mailbox doesn't allow inferior mailboxes" errors.

Resolution:
One solution is to ensure that each folder that contains subfolders does not contain any email items. Review your Source mailbox's folder structure and compare it to the Destination's. If any folder at the Source has subfolders, and the Destination folder does not yet have these subfolders but does contain email items, consider either manually creating the subfolders (although this may fail with the same error), or move contained email items to a new location. Also consider upgrading the IMAP system to see if a more recent version solves the problem.

[LoserFriendly]

Hi,

I don't think upgrading/downgrading parsedmarc is going to solve this issue

Here's what I found https://help.bittitan.com/hc/en-us/articles/115008253328-Mailbox-doesn-t-allow-inferior-mailboxes

Due to a defective implementation, the Destination IMAP server does not support the idea that folders may contain both messages and subfolders. In other terms, the Destination may allow you to have a folder A containing a subfolder B, or a folder A containing email items, but not both at the same time. So, if folder A contains email items, trying to create subfolder B results in a "Mailbox doesn't allow inferior mailboxes" errors.

Resolution:
One solution is to ensure that each folder that contains subfolders does not contain any email items. Review your Source mailbox's folder structure and compare it to the Destination's. If any folder at the Source has subfolders, and the Destination folder does not yet have these subfolders but does contain email items, consider either manually creating the subfolders (although this may fail with the same error), or move contained email items to a new location. Also consider upgrading the IMAP system to see if a more recent version solves the problem.

Thaks Sean for the prompt answer. I came across that article too, but I thought that if everything used to work fine before the upgrade and nothing was working after it, probably the cause were the upgrade itself.
Today (I'm in UTC+1 timezone) I was planning to start the upgrade again and remove everything from the "mail" directory of the dmparser user (the one who receives the reports). While upgrading Parsedmarc though, I noticed three errors I hadn't noticed before:
ERROR: mailsuite 1.5.4 has requirement mail-parser==3.11.0, but you'll have mail-parser 3.12.0 which is incompatible.
ERROR: mailsuite 1.5.4 has requirement six==1.13.0, but you'll have six 1.11.0 which is incompatible.
ERROR: mail-parser 3.12.0 has requirement six==1.14.0, but you'll have six 1.11.0 which is incompatible.
Although at the end of the upgrade, every package seems to get installed, I don't know if these errors may cause a misfunction in Parsedmarc.

Anyway, I removed the entire "mail" directory in the user's home directory and let Alpine create it again from scratch, but the error related to IMAP not supporting "inferior directories" is still there, even if there isn't any other IMAP folder than Archive at the moment in the "mail" directory. Since the OS is fully updated to the latest patches, I'm a bit stuck. The only possible solution I can think of is reinstalling the virtuaenv with the new release of pypy and see if this solves the problem.

[LoserFriendly]

Ok, I've just tried to install everything from scratch. I've removed the /opt/venvs directory and created it againg, I've downloaded and used the latest pypy build (PyPy3.6 v7.3.0), I've removed every directory inside the home directory of the user which receives the reports, I've removed the /var/mail/ file, I've installed Parsedmarc following the instructions given in the docs but two errors keep showing up:
ERROR: mailsuite 1.5.4 has requirement mail-parser==3.11.0, but you'll have mail-parser 3.12.0 which is incompatible.
ERROR: mailsuite 1.5.4 has requirement six==1.13.0, but you'll have six 1.14.0 which is incompatible.
Since they seem to be non-blocking, I went on: I stopped and started postfix and dovecot, finally started Parsedmarc service but:
2020-02-07 10:20:17,750 - ERROR - [cli.py:606] - IMAP Error: create failed: [CANNOT] Mailbox doesn't allow inferior mailboxes (0.001 + 0.000 secs).

Is there a way to increase the verbosity of the debug messages? I can't understand what mailbox it's trying to create unsuccessfully. Eventually, could I upgrade Parsedmarc step by step from 6.4.1 to 6.8.2 using the intermediate versions? Are old versions still available?
Thanks in advance for any suggestion.

[seanthegeek]

When IMAP talks about mailboxes it means folders.

You can install whatever version of parsedmarc you like.

For example

sudo -H pip install -U parsedmarc==6.4.1

[LoserFriendly]

I got it working, thanks for the support. Up to version 6.4.1, Parsedmarc saved the reports in IMAP folders called Archive. Aggregate, Archive.Forensic and Archive.Invalid. Starting from version 6.5.0, it expects to find Archive/Aggregate, Archive/Forensic and Archive/Ivalid, but Archive was automatically created by IMAP as a folder containing e-mail messages... that's where the "Mailbox doesn't allow inferior mailboxes" came from. I've deleted the Archive folder and recreated it as a folder containing only subfolders. This solved the problem.
Upgrading directly to 6.8.2 version still gave the errors related to mailsuite requirements, but upgrading step by step from 6.5 to 6.8.2 through every major release solved this issue too.