Feature proposal: export library

(I’m not calling this a feature request because this is an half baked idea and I would like to have feedback about it from other users.)

One of the reasons I’m using EagleFiler is because it uses the file system as its document storage. There is however some extra information that is stored in other places and which may represent a significant time investment from users: tags and notes.

A feature I would like to see is a way to export this information in an easy way to deal with. For instance, this could be an html file with a structure corresponding to the folder structure, links to documents, with the tags and notes annotations along the documents (or maybe just links to notes as they are rtf files).

There is (at least) one big issue: how to deal with mail? Individually tagged or annotated mail would be tricky to export, unless there is a simple way to link inside an mbox file.

Please let me know if you think it’s useful, crazy, or impossible. Thanks.

Another approach would be to save the tags and notes (as plain text) in the files’ extended attributes. I’d definitely like to hear more feedback about what people want before I implement a more elaborate export feature.

I’m a bit weary of extended attributes because the synchronization tool I use does not deal with them. In fact, there are not many synchronization tools out there that know how to deal with them. (Some backup tools do, however.)

I agree.

It’s a similar issue to the (mis)handling of Spotlight (Finder) comments in the customizable metadata fields, and/or searching through such fields thread.

And until it’s clearer to me what (if anything) Apple intends to do with extended attributes I’m uncomfortable with third party apps relying on them.

Exporting from DEVONthink preserves metadata in “DEVONtech_storage” XML binary plist files, one for each each enclosing folder. And it uses those files to restore metadata when importing, if they exist.

I haven’t thought this through much but was already wondering if somethingsimilar might make sense for EF, with these hypothetical “EagleFiler_storage” files dynamically created when necessary for preserving metadata (i.e. no explicit export like DT). Basically, like “external” .DS_Store (or ._ ) files instead of “internal” extended attributes. Utilities could be written to process them in useful ways outside EF and they’d be portable to other filesystems/platforms.

Just a bit of brainstorming about what might satisfy brab’s half-baked idea, with enough generality for other purposes.

I like this general idea of a non-database file for metadata interchange.

But do you mean that it should always create these files and keep them up-to-date? They’d have to be updated just about any time you moved or added a record, or whenever you changed the tags. With one file per folder (and you probably wouldn’t want one metadata file per file) that could be a lot of overhead if you have lots of records in each folder or lots of messages in a mailbox. Maybe it would be better to write the metadata files into the existing library structure (no need to export and copy everything) only when specifically requested.

Troof. Given that so much of the data is already made available with the transparent file structure, to dynamically keep updated some metadata file(s) simply for the purposes of export would be massively unworth it.

Yes, this would work (this is what I was thinking with an “Export library” menu option).

I think “Export Library” implies that a copy of the library would be created. So maybe if it were called “Export Metadata,” although that sort of implies that it would ask you where to save the metadata.

You clearly have a point. I’ve thought a bit about other menu names, but could not come up with something short enough (I guess “Annotate Library with Plaintext Metadata” is too long). I guess something obscure like “Unzip Metadata” or “Clone Metadata” is also not really user friendly.

That’s probably best if the overhead would likely be too high with dynamic updates. In that case a “Save metadata when EagleFiler exits” preference could be useful, maybe with a frequency variable (e.g. “N times per day”) for more elaborate tuning.

Thanks for considering this.

I’m disinclined to this kind of preference because it could take a long time to do this when quitting (OS X applications are not supposed to delay quits, plus it could be annoying) and because I think one might want to export the metadata for one library but not others. I still don’t think “Save” is the right word. What do you think of “Backup”? There are a variety of applications that save their own backups in a predetermined location. In this case the backups would have the additional virtue of being property lists that other applications could read.

Longer than launching EF on my iMac G5? :-o

Not sure I understand the factors that would contribute to it taking a “long” time, and that may be a subjective issue (for me) anyway …

DEVONthink can be configured to automatically “Backup & Optimize” the currently open database at different intervals (None/Hourly/Daily/Weekly/Monthly), which happens while the app is running. The delay when that occurs is longer as the database size increases but with it set to Daily the one-time daily interruption isn’t annoying because I know what it’s doing and want the automatic safety net it provides.

Lately NetNewsWire has been taking a “long” time quitting because of article backlog. That’s not annoying since I’m not frequently quitting/restarting it and just switch to another app while it’s busy.

Surely I wouldn’t be the only person wanting automatic EF metadata backups (disabled by default) who’d also be willing to tolerate occasional (configurable) delays for them. Eventually someone will wish they had a metadata backup that they’ve forgotten to create so there ought to be some method for automating it.

and because I think one might want to export the metadata for one library but not others.

Perhaps. Think you’ll ever consider per-library preferences? DEVONthink still lacks per-database prefs, which reduces complexity but is too inflexible/restrictive for me at times.

I still don’t think “Save” is the right word. What do you think of “Backup”?

Backup seems fine, though I hadn’t really considered my preference (pun?) of terminology for it.

There are a variety of applications that save their own backups in a predetermined location.

And frequency, per the earlier DEVONthink example. Btw, I often mention DT because it’s an easy, familiar reference point. More detailed EF/DT comparisons, if any, belong elsewhere, e.g. the EF vs DEVONthink Pro thread.

I’m curious which apps you were thinking of.

In this case the backups would have the additional virtue of being property lists that other applications could read.

Definitely.

Yes, I think that makes more sense than postponing it until quit, and it could then happen in the background.

Sure. Some of the “preferences” in the View menu are per-library, but so far I haven’t found a need for non-boolean ones.

The ones I use most often are BBEdit and QuickBooks.

EagleFiler 1.2 adds a Backup Metadata feature.

This is great news. However I can’t see the “Backup” entry in the File menu, and I could not find the backup itself in the .eflibrary package. Am I missing something?

Holding down the Option key with the File menu open? That’ll change Export… to Backup Metadata (and Close to Close All Windows).

Also, in most cases it shouldn’t be necessary to do this manually, since EagleFiler will automatically backup the metadata when opening the library, and periodically while it’s open.

Thanks for the “alt” suggestion.

What confused me is that I thought I read the metadata was saved in the .eflibrary package, and I finally found it in the files hierarchy.

The metadata is stored in the package, however the backups (the “EagleFiler Metadata.plist” files) are mixed into the Files hierarchy.

Ah, this is much clearer. Thanks.