Mails missing from combined view (dovecot 2.3.21)

[indridieinarsson]

🐛 Bugreport

I have cypht (2.2.0) running as docker container. I have 2 mail accounts, each on a different mail server. One of the mailservers is a self-hosted one (docker mailserver, running dovecot 2.3.21 as imap backend).

Both mail accounts work as expected when clicking the respective inbox in the "Email" drop down.
However, all mails from the self hosted one (dovecot 2.3.21) are missing from all combined views (Everything, Unread , All etc.)

I have poked around a bit, and found the following search query being sent to the imap server
SEARCH (ALL) ALL SINCE "16-Aug-2024" NOT DELETED NOT HEADER X-Auto-Bcc cypht

I didn't quite go so far as to look into the actual reply from the imap server, but at least cypht parses it without any errors, and returns an empty array of messages (and I know there are some messages in this period of time).

If I just commented out some of the code that generates the search query, namely the following lines from hm-imap.php :

if ($only_auto_bcc) {
$fld .= ' HEADER X-Auto-Bcc cypht';
}
if ($exclude_auto_bcc && !mb_strstr($this->server, 'yahoo') && $this->server_supports_custom_headers()) {
$fld .= ' NOT HEADER X-Auto-Bcc cypht';
}

making the search command look like this :
SEARCH (ALL) ALL SINCE "16-Aug-2024" NOT DELETED

After this change, the reply from the imap server is parsed, and the Everything view is populated with emails from both of my accounts.

[indridieinarsson]

Ok - I was a bit too quick to celebrate. When actually looking at the messages, I am getting all messages UNTIL 16-Aug-2024, but not the ones after that.
I must admit, that I don't know a lot (as in nothing) about imap standards and inner workings.

[indridieinarsson]

Getting messages from the wrong time period was just due to me messing with the search command (removed the UID part for some testing, forgot to add it back).

But it seems that my mailserver chokes on the NOT HEADER X-Auto-Bcc cypht part. There is a function that checks for support (server_supports_custom_headers() ). It seems to get a positive answer from the server, the function returns true, but the subsequent search command returns no results.

[marclaporte]

Related MR: #1152 from @Shadow243

[Shadow243]

NOT HEADER X-Auto-Bcc cypht part. There is a function that checks for support (server_supports_custom_headers() ).

Just to confirm a doubt, can you please try to add the condition here: https://github.com/cypht-org/cypht/pull/1152/files#diff-2a16fbb7ff5f2c47ea84a2060f25affdc0fbeb9df48cff9f9438fa7ec705f624R1099

like

if ($only_auto_bcc && $this->server_supports_custom_headers()) {}

[indridieinarsson]

Thanks @Shadow243.
I am already running with the code in the commit you mention, getting no messages from my dovecot mailserver if I run it as is. Only changes I have made is to comment out lines 1102:1104, then the search query returns the messages I expect it to.

I'm a bit uncertain what you mean exactly, could you perhaps specify more precisely what changes you want me to make.

Greetings,
Indriði

[indridieinarsson]

I've looked into this some more, connecting to the server with pythons imaplib. The server is responding with OK and empty results list when searching with
SEARCH (ALL) ALL SINCE "16-Aug-2024" NOT HEADER X-Auto-Bcc cypht
and also when searching with
SEARCH (ALL) ALL SINCE "16-Aug-2024" HEADER X-Auto-Bcc cypht

If searching by custom headers wasn't supported, the server ought to answer with an error message stating that, I think.

This looks very much like the issue is with this (version of the) server, rather than cypht.

[marclaporte]

This looks very much like the issue is with this (version of the) server, rather than cypht.

@indridieinarsson Dovecot has a gigantic install base so this can affect a lot of people. I suggest to report the issue to Dovecot to see where they stand and what we can learn to make Cypht better.

[indridieinarsson]

I've been able to track this down to the dovecot-fts-xapian plugin for dovecot.
The issue is not present if I disable full text search and use the default dovecot index instead (which is not nearly as flexible, and mostly unusable for real-life search in gigantic mailboxes), but re-appears as soon as I enable fts again.

In the source for dovecot-fts-xapian, I found that it is indexing only a selected (hard-coded) subset of email headers, and apparently returning an empty search result list for all searches that involve non-indexed headers.

Closing the issue, as it definitely has nothing to do with cypht, and there hardly is any sensible way to make a workaround either.

[Shadow243]

Closing the issue, as it definitely has nothing to do with cypht, and there hardly is any sensible way to make a workaround either.

That's great. I spent a day looking for it in cypht, I couldn't even reproduce thought.

Can you provide more details so we can document it somewhere?

[marclaporte]

Re-opening because others seem to be getting this: #1259

So maybe we can detect the specific issue, and report an issue upstream? Then, maybe we can put a warning in Cypht?

[indridieinarsson]

Is it confirmed that #1259 is the same issue?

My issue (i.e. #1195) was due to a very specific peculiarity in fts-xapian: It indexes only a fixed set of headers, and returns an empty result set for all search queries involving other headers (i.e. "HEADER X-Auto-Bcc cypht" and "NOT HEADER X-Auto-Bcc cypht" are both empty)
If so, there is a cypht-specific fix here : indridieinarsson/fts-xapian@a4f0edc and here indridieinarsson/fts-xapian@a0d52e7
Running cypht against a dovecot server with this version of fts-xapian is confirmed working.

[marclaporte]

Is it confirmed that #1259 is the same issue?

I don't know but symptoms are similar. One of Cypht's main selling points (compared to almost everything else) is email aggregation. So if there are cases where this fails, we should try to detect and inform the user how to solve.

Can you propose a PR to https://github.com/grosjo/fts-xapian?
grosjo/fts-xapian@master...indridieinarsson:fts-xapian:master

Thanks!