Intrinsic IMAP Issues

There are a few issues relating to IMAP's UI that are constrained by inherent features in the IMAP protocol. There are also some issues which appear to be related to the IMAP protocol, but which are probably just the client doing a poor implementation. This page will describe some of the issues intrinsic to IMAP.

Commented on by Dave Cridland -- DaveCridland - 15 Dec 2003

Multiple Clients

The IMAP spec explicitly requires that more than one client can access the server at a time. This can lead to coherency problems and can make it look like messages are disappearing. Examples:

• Client A and Client B are both logged in to a particular IMAP account. Client A has filters set up to move messages from Fred to the Clients folder. Client B does not. Client B notices a message from Fred and displays it in the inbox. Client A notices the message a few minutes later and moves it from the inbox to the Clients folder. The server notifies Client B that the message has moved, and so it appears to disappear from the Client B inbox.
• Jackie logs in to her work account from her home computer before she goes to work. She downloads a message about the upcoming reorg. Jackie gets to work and logs in from her work computer. She reads the message about the reorg and deletes it. She leaves work and goes home. She sees the message about the reorg in her home computer's local cache. She dials in to her work account, and the reorg message disappears before her eyes.

Note that in addition to two different sets of client filters, a user could also have server-side filters. If the server-side and client-side filters don't match, this could also lead to a confused user. However, we can probably write off this problem as pilot error without too much trouble.

The first is why client filtering is a bad idea, in my opinion. You're asking your mail to be filtered and sorted by something that is not always operating, or might be operating with different configuration or even implementation somewhere else concurrently. Filtering should be, IMHO, done on the server.

The second is easy to avoid - if the message is actually opened and displayed, or otherwise in use, then the client can simply tell the user. If it's simply in a summary, no harm done. The user is probably aware they deleted it anyway. - Dave

Server is authoritative

The way the IMAP spec is written, the server is the authoritative source of messages and their organization. This makes working offline problematic: anything that you do offline must get synched up with the server later. If another client modified messages or their organization in the meantime, the synchronization gets complicated.

If the clients are moving folders about, then yes, it's painful. - Dave

The server doesn't keep track of what the various clients know, so the server can't tell the client what has changed since that client last visited. The client thus must download some information about all of the messages every time the client connects, so the client can see what messages have left and come into all the different mailboxes.

The only information needed is FLAGS, and even then only for the messages actually visible. Future extensions will make this even easier. - Dave

Copy, not move

The IMAP spec does not allow moving messages from one place to another. To "move" a message requires two steps: issuing a COPY command, then a DELETE command. This means that

• Moves are slow. IMAP servers can't just change a pointer reference.
• If there is a failure such that the DELETE doesn't complete, you can end up with two copies of a message.

If there is a client that doesn't notice that flagging the source message deleted failed, then it's buggy. Moves via COPY+DELETE can be equally fast as a "real" move. Consider a system using hard links on Maildir, for instance. - Dave

Expunge, not delete

The IMAP protocol forces deletions to be a two-step process: one command marks a message for deletion, then a second command deletes all messages marked for deletion. (The second step is called "Expunge".) Some clients require the user to perform two actions; Chandler should expunge without user intervention. Update: I found somebody who actually likes doing both steps; I am following up to understand why and whether we can fulfill his desires with other features.

I like it too. It means the client can choose to have a virtualized Trash folder, or display deleted messages inline. Personally, I usually hide them, but sometimes show them inline (If, for instance, I'm looking for a specific mail and I can't recall if I deleted it or not.) It's also dramatically more efficient than EXPUNGing after every +FLAGS.SILENT (\Deleted) you do. - Dave

There is a minor design question about when to expunge. One obvious tack is to expunge immediately after any message is marked for deletion. One could argue that you want to give the user a slight grace period before irrevocably deleting messages. Unfortunately, you can't tell the IMAP server to expunge only messages older than X hours -- expunge deletes all messages marked for deletion. You could also expunge upon close, but there will be someone out there who never closes their IMAP client. You might also get people confused about why they are still over quota when they just deleted a bunch of stuff.

Another option would be to copy marked messages to a local Trash folder, then expunge the copies on the server. People would have to empty the trash manually. The disadvantage of this is that you could potentially take up bandwidth with stuff that people have already indicated that they don't want. (If the user only downloaded the header before deciding they didn't want it, it would be a shame to then download a big huge body...)

The good news is that this is not an incredibly difficult design issue. Most people won't ever notice or care about the difference between the top few options.

New messages

In POP, there is only one stream of messages, which means only one place where messages can show up. With server-side filtering on IMAP, new messages can pop up in many different folders. This means that the client needs to check a number of different folders, which means that it might take a little while for the client to fully understand where there are new messages.

Well, yes. This is what STATUS is for. The only annoyance about STATUS in IMAP is that it only handles one mailbox at a time - so you simply pipeline them all. - Dave

Public folders

While POP, being client-focused, is a very individualistic protocol, IMAP's server nature allows it to be much more collaborative. Public IMAP folders are useful as a replacement for Usenet. This is good. While I haven't heard of it, it's easy to imagine an IMAP interface to RSS feeds as well.

The bad news is that now there are even more places that new messages can show up. There can be so many folders that the client flat cannot examine them all. The client must be able to recognize which folders the user has "subscribed" to and only examine those for new messages. In addition, if there are a lot of newsgroups that the user has a casual interest in, perhaps the user wants to be able to specify separately which folders to subscribe to (i.e. show up in the list of folders) and which to automatically check for new messages in.

Well, IMAP servers already do support the notion of mailbox subscriptions. As for which to check for new messages, I've not noticed a problem with all my folders yet, but perhaps there is a scalability problem there. - Dave

Server busy?

Sometimes, when I (Ducky) drag a message to a different folder in Mozilla, it "bounces off" the folder. I have to grab the message and try again (and sometimes again and again) until the move "takes" and the message moves to the other folder.

I also sometimes get an error when sending a message, where Mozilla tells me that my message was sent but that it couldn't be properly stored in the Sent mailbox on the server.

I don't know why this happens, but I speculate that these things happen when the server is busy and refuses the connection. It would be nice if my client could somehow queue up the failed transactions and keep trying until the server wakes up enough to accept the transaction.

Your client is buggy, then - it should be doing exactly what you describe - batching up the failed transactions - Dave

Namespace

There was an ambiguity in the IMAP spec such that it wasn't clear whether a user could see anything besides their own personal mailboxes (like public mailboxes). There is a commonly-implemented NAMESPACE extension to IMAP that allows a client to discover the prefixes of namespaces used by a server for personal mailboxes, other users' mailboxes, and shared mailboxes. Most IMAP servers implement NAMESPACE. However, there is going to be a server out there someday that doesn't support NAMESPACE, and the user will need to specify various prefixes for shared folders.

Message UIDs don't always persist

The IMAP client and server refer to messages by their UIDs. There are some nice features about UIDs, such that they are required to be strictly ascending in order of when they arrive at the server. Thus, to synchronize, the client can just figure out which UIDs are bigger than the last UID it saw in that mail folder.

Well, you can make things marginally more efficient by looking for what the value of UIDNEXT was for the mailbox, but I don't think it makes any difference in practise. - Dave

Unfortunately, while servers are very strongly encouraged to persist UIDs across sessions, it is not absolutely required. Thus, some servers could regenerate the UIDs every session. I would be a little surprised if any of the major servers regenerate the UIDs, but it is possible.

UIDs can appear to be non-persistent if one client deletes and recreates a folder before the other client finds out. For example, suppose that at work, Jane deletes her "Stuff" (and all the message in it) folder, then creates a new folder named "Stuff". Her home IMAP client doesn't know that the new "Stuff" is different from the old "Stuff". Some of the UIDs in her home cache will match the UIDs of the new messages, so the home IMAP client doesn't download those messages, even if Jane has never seen them.

UIDs are NOT PERSISTENT. They simply have a lifetime longer than a session. In fact, they persist as long as the UIDVALIDITY of the mailbox is unchanged. If it changes, the UID information you have is invalid.

What this means is that the last paragraph describes a buggyg client. Otherwise, it's merely a bit slower if the server is regenerating the UIDs with every mailbox open. - Dave

Server push

The IMAP spec explicitly says that the client must be able to receive status updates at any time. This places higher demands upon the network architecture than if the client controlled the flow of traffic. Tihs doesn't have any UI issues, but does make the job more difficult.

No, it puts less demand on the network architecture, but makes it more complex to implement. But really, not by much. You just do a select() or similar every now and then. It does, however, give you much better response time to changes in the mailbox - Dave.

-- DuckySherwood - 08 Oct 2003

---

I just read the useful IMAP ACL summary by LisaDusseault. I want to comment on the use-case of a Chandler repository as a client of an IMAP server. While the security models may differ, there may be very good reasons that someone may want to implement this. If a Chandler user works in an environment where IMAP shared folders are used the IMAP interoperation feature may be a hard requirement for choice of Chandler over another tool. I believe that if the shared folders are not in fact IMAP folders in Exchange server, they operate in a very similar manner. Unfortunately I don't have an IMAP server implemented at the moment to test with how Evolution handles this. The question of how XimianEvolution handles is is relevant since it represents itself as an open-source drop-in replacement to MicrosoftOutlook client. I talked to someone in a Notes environment on IRC that says while Evolution claims the same thing for Lotus Notes, it's implementation falls short.

-- GrantBowman - 10 Apr 2004

---

IMAP interoperability is a hard requirement for Chandler, at least insofar as Chandler has to be able to synchronize well with IMAP mailboxes while not molesting them. The firmest requirement is in the Westwood timeframe. (Westwood is specifically for higher-education users, who use IMAP heavily.)

-- DuckySherwood - 12 Apr 2004

---

Count me among the folks who actually like the Delete/Expunge two step model. The real solution for this to make it look like Outlook, IMHO, is one that I'm really surprised others don't use, and that's the virtual Trash folder. It's really not too terribly difficult to impliment -- the main folder view simply excludes anything with the Deleted flag set, and the Trash folder shows it. Expunge happens on an "Empty Trash" command, which happens either upon closing the client, or after a period of time, with a prompt to the user. The biggest complication with this is that this effectively makes a virtual Trash folder for every IMAP folder, which means that to view the actual Trash folder would require a FETCH for every mailbox. A compromise could be to use the virtual Trash folder for the INBOX, and for all other mailboxes either use an actual Trash folder, or just mark as deleted.

As far as the limitations of the EXPUNGE command, the UIDPLUS (http://www.rfc-editor.org/rfc/rfc4315.txt) extension covers this, for servers which support it. The major open source servers support it, although I don't know about the commercial ones like Communigate and Exchange.

Dave mentioned the availability of subscriptions above. This is a mixed bag, because while the IMAP protocal provides the subscriptions happily, it never specifies what the client should do with them, and as such, each client treats them differently. This makes using different clients a royal pain. If Chandler wants to do fancy things with it, I'd say try using the ANNOTATE extension for mailboxes, and stick something in there. Alternately, do what Mulberry does (did?) and just record what mailboxes should be checked for new messages in the client preferences, and store these in Cosmo to make it portable.

-- MichaelBacon - 05 Jun 2006