Feature requests about mail

Hello,

I’ve been thinking about this feature request for a while, but I’ve finally found the time to write it down. This request is about how EF handles mail messages, and suggestions to improve it.

At the moment, once a mailbox is imported, the only things one may do are:

  • deleting or moving the mailbox;
  • merging it with another mailbox;
  • tagging or annotating individual messages.

This is great and useful, but it prevents one from importing mailboxes that are not sorted yet, as it’s impossible to move messages around, which is the feature I’m really missing. One could always use an external program that is able to read and write mbox files to do this, but not as soon as mails have been tagged or annotated. I guess this is the major motivation for this feature: as EF needs to keep track of where the mails are in the mbox file to associate it to its tags and annotations, any change to this (by moving or deleting mail) has to be done inside EF.

What I would really like is to be able:

  • to delete a message from a mailbox;
  • to move a message from one box to another;
  • to create a new mailbox from one or several messages.

I realize this is a quite big request, as it requires a bit of mbox manipulation code, so it could be a 2.0 feature.

An other small request would be to show in the preview pane the message list when a mailbox is selected in the list view.

Thanks a lot for taking the time to read this.

Thanks for the suggestions. I definitely plan to add the ability to create new mailboxes from selected messages and to preview the message list in the viewer pane. Moving and deleting messages requires mbox manipulation, as you say, and this is potentially dangerous if the mbox file wasn’t properly formed. (I’ve seen some really strange finds in the wild.) I’ll keep it in mind, though I’m not yet decided on whether the cost-benefit is worth it.

In the meantime, you can freely sort your messages using tags. I think this provides most of the functionality of your first three suggestions, without changing the stored mbox file. And, unlike creating new mailboxes, you can organize the same message in different ways, since each message can have more than one tag.

To me it seems Archival storage should be read only, like in Subversion or Venti (a concept now shared by a variety of professional document archival solutions).

I think that’s true, but it depends a bit on what your conception is of what you’ve done by moving your mail to EF. I moved a bunch of mail to EF in order to speed up Mail, but I still consider it to be in need of sorting/trimming, later.

I had more or less resolved myself to the idea that when later finally comes, I’ll reimport the mboxes to Mail, trim and sort, and then send them back to EF, maybe that time really as an archival copy.

I use MailTags a fair bit, though, and I am not sure that they’ll survive the trip back to Mail (though I greatly appreciate that they made the trip to EF safely). That’s one thing that I’m kind of hoping will have a solution by the time I get around to dealing with this. (If it doesn’t work already, I haven’t tried it, but I don’t have a lot of faith – I’m still using MT 1.2.2 and many of my tagged messages came from long ago, pre-IMAP and pre-MT, so I don’t know if the tag information got embedded in the mail headers in the export to mbox on the way to EF.)

I would welcome a way to edit mboxes in EF, though. Is it difficult to do an integrity check on an mbox file to see if it is ill-formed? (Wouldn’t the mbox display incorrectly in EF already if it were corrupt? It would seem to be indirect evidence of well-formedness that the mbox is safely readable in EF anyway.)

They would not, because the mbox format does not store the MailTags stuff. So I don’t recommend doing that.

My current plan to support trimming of mailboxes within EagleFiler is to let you hide individual messages by pressing Delete and then select the remaining ones and choose a command to copy them into a new mailbox, preserving the tags and notes. This should be much easier than going back and forth through Mail. And it’s much better from an implementation point of view than direct mbox editing, because it will be easier to implement undo, and your data will always been in a consistent state on disk.

It’s almost impossible. The reason is that an invalid mbox will appear to be a valid mbox containing a different number of messages.

Yes, probably you would see an incorrect list of messages in EagleFiler, but this is not the kind of error that it’s easy to see unless you are looking closely.

Anyway, if you’re importing messages from Mail, I wouldn’t worry about the mbox being ill-formed, because EagleFiler will generate it in the correct format.

Sounds good to me, that will do what I want. Hiding alone might do what I need (but I suppose we would want hidden message to also hide from searches unless explicitly told not to).

Perhaps someday down the road there would be a way to construct an mbox file that stores the tags (I think MailTags 2 uses something like an X-MailTags header, so they can be stored on the IMAP server) in the process of copying to a new mailbox, but I’m not in a hurry for that. In fact, if this is do-able, it would actually mean that there’d be added value in moving old email to EF and back into mail to insert the tags into the old pre-MT2 messages. Though maybe Indev will come up with a way to do that all within Mail at some point, I know it was briefly discussed on their forums.

Thinking about it two seconds more, I see that this is perhaps kind of a problem because the X-MailTags header would get stuck in whatever state the tags were in at the time of copying the messages to the new mbox. That is, if one were later to add tags in EF to these messages, we’d be back to the problem of editing existing mboxes if we wanted to keep the X-MailTags header up to date. Perhaps this would have to be limited to an option that’s only called upon on a kind of “export” operation that is not intended to remain in EF afterwards. (Though I know an export function is not now currently necessary for much, given the fact that everything is just a file inside the EF library folder.)

Nothing is easy.

Maybe there could be an explicit “send back to mail” operation?

Perhaps. I could certainly make it export back into Mail’s emlx format. I’m not sure how many people would have a use for that, though.

I am sorta following this thread but not sure what objective – may be I need to read closer.

My own objective here was basically to use EF as a place to dump my older email, so that it wasn’t bogging down Mail, but was still searchable/sortable/etc. And EF does a great job with mail, including the ability to keep MailTags tags attached.

However, in my mind, I wasn’t archiving old mail so much as I was pushing it into a “to deal with later” pile. My own shortcoming, perhaps, but I’m not always good about deleting the messages I’ll never care about again, but I don’t have time now (and I might have time later) to get it into real “archival” shape. So, what I personally was looking for was a way to do my archive trimming within EF.

That was what I was after, and the ensuing discussion suggests that once it is possible to hide messages and then export non-hidden messages to a new mbox within EF (presumably with tags intact), this can be accomplished. So, I think the functionality I was looking for is on the horizon already.

The bit about sending an mbox back to Mail was exploring another way this might be done (send the mbox back to Mail, trim, and send it back to EF), but how well that works, even if EF embedded the tag information in an X-MailTags header during export, would depend a lot on what MailTags does with that information in messages that are not still on an IMAP server. Specifically, whether it would respect (and keep up-to-date) information in the X-MailTags header of a locally stored message. So, probably these things are best dealt with by indev on the MailTags side before any time is invested on the EF side.

Too much information? Anyway, I’ve typed all this now, may as well post it.

With the risk of giving a simplistic response, this seems like a workflow issue more than an import export issue. from the sounds of it you are using EF to archive emails that you may need to refer to in the future but are not on/no longer on your agenda. And the basic principle of the archive is that the only things to be archive are things that will make it to EF
ie to keep it a one way street and tweak the workflow (wet wear)

Mind you, stuff does happen and people do need to resurrect dead items. – Rather than dealing with mbox files though, it should be easy for EF to construct a emlx file the format is very straight forward and have Mail import those. If done in an integrated way, it may be possible to write a plug in for mail that would add an “Import from Eagle Filer” item. The plugin would access the EF database, allow selection of items/groups, and directly import them to a mailbox or write them out to temp emlx files and call on Mail’s import functions. However Michael’s hand would be required there as he knows the structure of EF.

I understand the need to access older mail, but what is your reason for wanting to bring it back into Apple Mail? Just curious.

In EagleFiler 1.1.3, you can select some messages and choose Export, and EagleFiler will save them as .emlx files. You would probably need to put these into a Mailbox.mbox/Messages folder structure in order for Mail to import them, although just by double-clicking a single .emlx file you can view it in Mail.

At some point, I plan to improve the export such that it will generate the .mbox folder for you and so that it will export the tags and notes in MailTags format inside the .emlx files.

This would be great.

I too think these are vital functions, though they carry the risk of corrupting mail messages when modifying the mbox. Nonetheless, they are crucial because archival is an on-going process.

Suppose I have a mailbox folder called Receipts into which I keep tossing in emails of customer receipts. As time passes, that Receipts folder is going to grow voluminously, and I’ll need to break it up and organize its messages into sub-folders (Receipts 1970-1980, Receipts 1980-1990, etc.) in order to keep my archive manageable. To do this, I’ll need to be able to create new folders, move messages—and possibly delete superfluous messages.

I think you’ve got a great product here, Michael. I prefer your design to DevonThink Pro, even if it doesn’t have DevonThink’s bells and whistles. Nonetheless, it needs these vital operations.

I’m sorry to ask what you may think is an obvious question, but why? What are you trying to accomplish? By default, EagleFiler creates a new mailbox each time you import messages. So the receipts would already be separated by time period. If you want to merge multiple mailboxes (e.g. to combine different months of a year), you can.

If you are concerned about having too large a mailbox file, I think you needn’t be—EagleFiler can handle it. If you want more flexibility in how you view the messages, you can merge the mailboxes as you like or freely organize the messages using tag sources.

But suppose I want to organize mailboxes differently than from what I had originally planned when I first started grouping my email into mailbozes in Apple Mail. I need the ability to rename folders, create new folders or subfolders and to move mail messages into a different grouping.

In the Receipts example, suppose I first decide to organize the mailboxes into Receipts 1970 -1979, Receipts 1980- 1988, etc. but then decide I want to group them into: Receipts 1970 - 1974, Receipts 1974 - 1979, Receipts 1980 - 1984, etc; or perhaps I want different date boundaries from that I began with: Receipts 1971 - 1980, Receipts 1981 - 1990, etc. Or suppose I discover that I made a mistake, and found a different email inside a Receipts folder. I need to pull that email message out of the Receipts folder and insert it into a proper location.

Apple Mail provides the ability to create new folders and subfolders, to move around mail messages, and to delete them. As an archiver, EagleFiler should be offering the same type of functionality (without having to export mailboxes back into Apple Mail for further mailbox refinement).

Have you tried doing this with EagleFiler’s tag sources? They work like albums in iPhoto. You can group messages from different mailboxes together, or even put the same message in more than one grouping, e.g. to view your mail in different ways.

I’ve looked at Tagging, but I don’t think Tagging and folders are quite the same thing. To take the of iPhoto and iTunes example and their albums, I don’t think iTunes, for example, scales well for large datasets. Most people have a few hundred songs in their iTunes library; heavy-duty mail users get that much mail in a week. An acquaintance of mine is a DJ who has over 50,000 song titles on his Mac. He finds iTunes and iTunes albums woefully inadequate for the task of maintaining his music collection. Likewise professional photographers do not use iPhoto for their collections, but opt for Picassa, Aperture, iView or the like.

There is a difference between tagging and partitioning. When datasets became very large, they became difficult to manage, even when tagged for grouping or sorting. Mail folders (or mailboxes), however, provide partitioning, which often fits better with the user’s concept of how certain forms of information can be organized.

Just for fun, I tried to emulate tags as a form of nested folders in EagleFiler. I found that tags suffer from namespace collisions. For example, suppose I have Tags:

Receipts
±- Personal
±-±–1970s
±- Business
±-±–1970s

the second 1970s tag under Business cannot be created, because it one was already created under Personal.

Nonetheless, I think folders, with their inherent hierarchical organization and partitioning, are a more natural means of organizing voluminous mail messages than tags, which groups together different properties and associations of mail messages.

Yes, I think the key difference (which is good or bad depending on what you’re trying to do) is that each record can have any number of tags, but exactly one folder.

Yes, at present you would have to do a search for multiple tags in order to narrow down the desired set of records.

Thanks for the comments. As I said a few messages back, I plan to add some features to manipulate the contents of mailboxes. However, I wanted to make sure that people were aware that tagging provides similar functionality today.

My main motivation to have EF do some mailbox manipulation is in fact tagging: as soon as messages are tagged, I’m afraid there is no way to manipulate the mailbox, as using an external tool then reimporting the modified mailbox will lose the tags.