IMAP Namespacing Change

[StarkZarn]

After a recent update, I started receiving the following error when parsedmarc tries to retrieve reports from my inbox.

parsedmarc     |    ERROR:cli.py:1406:Mailbox Error
parsedmarc     | Traceback (most recent call last):
parsedmarc     |   File "/usr/local/lib/python3.12/site-packages/parsedmarc/cli.py", line 1385, in _main
parsedmarc     |     reports = get_dmarc_reports_from_mailbox(
parsedmarc     |               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
parsedmarc     |   File "/usr/local/lib/python3.12/site-packages/parsedmarc/__init__.py", line 1445, in get_dmarc_reports_from_mailbox
parsedmarc     |     connection.create_folder(archive_folder)
parsedmarc     |   File "/usr/local/lib/python3.12/site-packages/parsedmarc/mail/imap.py", line 38, in create_folder
parsedmarc     |     self._client.create_folder(folder_name)
parsedmarc     |   File "/usr/local/lib/python3.12/site-packages/mailsuite/imap.py", line 305, in create_folder
parsedmarc     |     imapclient.IMAPClient.create_folder(self, folder_path)
parsedmarc     |   File "/usr/local/lib/python3.12/site-packages/imapclient/imapclient.py", line 1031, in create_folder
parsedmarc     |     return self._command_and_check(
parsedmarc     |            ^^^^^^^^^^^^^^^^^^^^^^^^
parsedmarc     |   File "/usr/local/lib/python3.12/site-packages/imapclient/imapclient.py", line 1753, in _command_and_check
parsedmarc     |     self._checkok(command, typ, data)
parsedmarc     |   File "/usr/local/lib/python3.12/site-packages/imapclient/imapclient.py", line 1759, in _checkok
parsedmarc     |     self._check_resp("OK", command, typ, data)
parsedmarc     |   File "/usr/local/lib/python3.12/site-packages/imapclient/imapclient.py", line 1636, in _check_resp
parsedmarc     |     raise exceptions.IMAPClientError(
parsedmarc     | imaplib.IMAP4.error: create failed: Client tried to access nonexistent namespace. (Mailbox name should probably be prefixed with: INBOX.) (0.001 + 0.000 secs).

I haven't changed my mail server during this time, so I suspect it's something to do with how parsedmarc is trying to access the folder. In my parsedmarc.ini file I have the following [mailbox] parameters set.

[mailbox]
archive_folder = DMARC_Reports
batch_size = 2

The mail server is a basic postfix/dovecot stack.

Looking at recent issues, I suspect it might have something to do with a change made in #552 to resolve #551 but I don't know for certain.

[seanthegeek]

Well, crap. I'm not runniing these mail servers so I can't test. @gaige, what's your take, since #553 was your PR?

[seanthegeek]

@StarkZarn Can you provide the output of the checkdmarc CLI with the --debug option for the version that was working and the version that is broken. --debug output should tell us the paths of the folders it is trying to create or access.

[gaige]

@StarkZarn do you know which version caused the problems? In particular, I'm also running postfix/dovecot (although on Solaris, which likely isn't your OS), and after version 8.13.0, I started seeing problems with my configuration, which I traced back to the archive folder separator (hence my fix applied in 8.14.0).

The change that I made specifically uses the namespace command in IMAP in order to retrieve the namespace separator so that parsedmarc was no longer using "/" as a separator (still the explicit default), which isn't consistent with the "." separator used by dovecot.

Can you check against version 8.13.0 and see if the problem still occurs for you? If it does, then it's not affected by my fix (although it likely indicates that there needs to be an expansion of my fix for dovecot to handle your use case).

One additional check if you can: if you can log in manually to your IMAP server and check the result of the namespace command, that would help narrow down the case here.

Assuming you're using OpenSSL:

openssl s_client -crlf -connect MY_MAIL_SERVER:993

This will connect to the IMAP server. The trick is going to be ensuring that you can log in. If you have PLAIN auth or LOGIN auth enabled, you should be able to use:

a0 LOGIN <dmarc username> <dmarc password>

To log in, at which point you should be able to issue the namespace command using:

a1 NAMESPACE

This will respond with a list of tuples, looking something like:

* NAMESPACE (("" ".")) NIL NIL
* a2 OK Namespace completed (....)

Based on the error message, you may see something like

* NAMESPACE(("INBOX" ".")) NIL NIL

Which indicates you need an INBOX (or something else) prefix for every mailbox. If this is the case, we may need to make a further update to add a required prefix in cases like yours, which can be done using data from the namespace command.

At any rate, if you can check the prior version (8.13.0) and also get the response to the NAMESPACE command that would be highly useful in tracking this down.

If you can't get the namespace command to work, please take a look at the conf.d/10-mail.conf file in your dovecot config. and pull out the namespace inbox stanza. In particular to see if there's a prefix defined, and also check for other non-commented namespace stanzas. Those are the namespaces that the IMAP command reports on so that's an alternative to getting the data directly via IMAP.

[gaige]

@StarkZarn One other thought for an easy test for the latest version would be to change the archive_folder to INBOX. DMARC_Reports if your dovecot configuration is using an INBOX prefix. That should result in the appropriate nesting, and also would be a further indication that we should update the code to include the base prefix if it's not empty.

[StarkZarn]

@StarkZarn One other thought for an easy test for the latest version would be to change the archive_folder to INBOX. DMARC_Reports if your dovecot configuration is using an INBOX prefix. That should result in the appropriate nesting, and also would be a further indication that we should update the code to include the base prefix if it's not empty.

Hey thanks so much for the fast turn around! I apologize for not having more data for you up front, it's been a busy week. The quoted reply 100% works, so well done on that thought process @gaige.

Do you still need the other data to localize faults or does it make enough sense now? I really appreciate the work you've both put into this, great stack here!

[gaige]

Glad to hear the diagnostics worked. It would still be helpful to get the output of the IMAP namespace command in order to have this work out of the box. If that is possible. Thanks for the quick reply!

[gaige]

@seanthegeek I see two ways of handling this.

1. We can add to the documentation that if the mail server requires a prefix, that the user should add that to the archive_folder and use the INBOX.XXX example
2. I can make another PR and add the prefix and we can change the documentation to indicate that the prefix is automatically added.

Second option may be the most straightforward as it'll require the least amount of work on behalf of anyone adopting it. However, there's going to be an additional question of how we handle shared mailboxes in an IMAP store in the event that there are separate namespaces. I can do a search for the prefix in the archive_folder to match against any of the namespaces and if it starts with a known prefix, then use that namespace; otherwise add the namespace prefix for the Private namespace (the first one, and the only one that guaranteed to be there on non-anonymous IMAP connections).

In that case, the documentation adjustment should be that we support IMAP namespaces and default to the private namespace per the spec.

[jnoordsij]

Note that this might possibly not be a problem with parsedmarc itself, but rather with its dependency mailsuite. I obtained the same error after solely updating the version on that, without using a new parsedmarc version. I've reported it at seanthegeek/mailsuite#10.

[gaige]

Note that this might possibly not be a problem with parsedmarc itself, but rather with its dependency mailsuite. I obtained the same error after solely updating the version on that, without using a new parsedmarc version. I've reported it at seanthegeek/mailsuite#10.

Solid point. The documentation for mailsuite indicates that it should take care of the namespace and separator issues, allowing the use of / as the separator. It looks like the root cause of this is a regression in Mailsuite. If that's fixed, my modification should be reverted, as it will be both redundant and potentially in conflict with the fixed code in Mailsuite.

[seanthegeek]

@gaige You're right. mailsuite would be the better place to fix this. can I get a second set of eyes on seanthegeek/mailsuite#10 ? I only code change had nothing to do with IMAP. seanthegeek/mailsuite@01cb8a8

The relevant IMAP code is here: seanthegeek/mailsuite@01cb8a8

[gaige]

@seanthegeek That's super odd. As you mentioned, that changed shouldn't have triggered the behavior..

[seanthegeek]

I don't pin many dependencies in mailsuite, so I'm wondering if an underlying dependency updated...

[jnoordsij]

By manually applying the changes one by one I've found the cause; see seanthegeek/mailsuite#11.