Advantages of a EF library "separate" from the Finder

I know that EagleFiler saves all of the created and imported files in the Finder, and, therefore, its library is technically not separate from the rest of the Finder’s contents. However, since EF wants to be the one manipulating and moving all the files (not directly through the Finder), its library is in some respects “separate.”

Michael, could you summarize the advantages and disadvantages of having such a library? (vs a database that indexes the Finder’s contents without directly importing them.)

I have done my best to find such info in the forum threads, and here are a few things that I’ve found:

Advantages:

  • the database stays accurate and up-to-date with the library, since all changes are recorded simultaneously (unlike the case if someone where to quit EF and then make number changes to the library with the finder.)

  • EF can use the checksum to verify the files’ integrity.

Disadvantages

  • other tools that might manipulate the Finder’s contents shouldn’t be applied to the EF library.

  • two-way synchronization of libraries should be done through importing through EF rather than copying library components directly.

Of the following, which would be alright to accomplish independent of EF (but on the library contents):

  1. adding or removing spotlight comments
  2. double-clicking the file directly in the finder and editing its contents with a given editor.
  3. editing a file’s name or extension
  4. changing the default application for a file
  5. using an application that doesn’t save to the original file, but rather moves the original to a backup and writes a completely new file (which changes the lower-level id tag of the file – I can’t remember what’s it’s called right now)
  6. changing a file’s label

Thanks.

Well, I think the latter design just wouldn’t work. Without EagleFiler managing the files, it would be hard for it to keep track of them. You wouldn’t be able to add persistent metadata such as tags and notes. EagleFiler would be horribly slow, having to rely on Spotlight and scanning the disk all the time just to stay in sync. This is why virtually all applications opt to put everything in their own database (which gives the app even more control over the data, but doesn’t interoperate as well) or to use a database alongside a managed folder (EagleFiler, Mail, iTunes, iPhoto, Aperture, etc.).

All of these are OK except for 3. The rules are simply that you should not add, remove, or move/rename files except by going through EagleFiler. Viewing the files and editing their contents or other attributes (except the name) are fine.

Well, there are applications out there that do make it work. Together/KIT, DevonThink, and Leap all are able to maintain their respective databases (with metatdata, tags, and notes) without directly importing and taking control of the resources in their library.

So given that there are two feasible methodologies (i.e. import vs index), I guess I was asking as to the advantages/disadvantages of the one implemented in EagleFiler. From your comment above, it seems that speed is a central advantage.

I’m not espousing either method, but I just want to clarify what one could expect from each.

Thanks again,
Jeff

Jeff,

In my experience the simpler, less ‘proprietary’, less heavily-layered, transparent the application, the less of its own manipulation it needs to work, the stabler.

I’ve been so impressed with how much thought has gone into architecting EF around the Finder and its file structures.

In the long run, we’ll always bless Michael for this… we should already :slight_smile:

Yes and no. They all make it possible to annotate files without copying them into a managed folder, but they don’t solve the problem in its full generality. (By which I mean letting you manipulate the application’s hierarchy from the Finder while definitively associating tags and notes with each file.)

By default, Together works like EagleFiler; it copies the files and folders into a managed folder whose contents it tracks with its database. As with EagleFiler, if you break the rules and rename or remove a file, it will get confused. I think it has an option to check for files that you added via the Finder, which is also something I’m planning for EagleFiler.

Together also has another mode where it will import alias files instead of copying the files. This way, you can move or rename the original file, and hopefully the alias will still track it. When you manipulate the file in Together, you are really manipulating the alias file, which lives in its managed folder. This model kind of breaks down because you can’t import a link to a folder—you end up with a new folder (in the managed area) containing some alias files.

I think this style of linking could be implemented for EagleFiler, without much reduction in speed, but I personally don’t like the user experience. I prefer the EagleFiler method where moving a file into a folder in EagleFiler rearranges the corresponding file in the Finder (rather than an alias to the file). It’s simpler when there’s a direct correspondence. Also, it’s nice to know that a library folder is self-contained (e.g. so you can move it) and that, if you have an encrypted library, the documents in the library (not just the aliases) are encrypted. Together avoids this confusion (but creates separate ones) by not encrypting at the library level.

Lastly, my experience from iTunes (in link mode) and iDVD tells me that it’s more reliable to let the application manage the files. You can use aliases, and often everything will be fine, but inevitably after rearranging files, moving to a new disk or Mac, etc. the aliases lose track of the original files. I’ve lost playlists and DVD projects this way, even though it looked like everything was in the right place.

It’s not easy to see what DEVONthink is doing, but my impression is that it copies an alias into its database and uses this to track the original file. Unlike EagleFiler, the folders in DEVONthink exist only within its database. And DEVONthink doesn’t seem to let you edit linked files. So this is basically like Together, only less flexible.

Leap uses aliases (stored in a plist file) to associate tags and notes with files. I think it’s really a different kind of application than EagleFiler. It’s more about overlaying different kinds of views and searches on top of the filesystem than about helping you browse and manipulate the filesystem.

Speed is a big part of it. For example, when you click on the Records source in EagleFiler, it can quickly show you all the files in all the folders. If EagleFiler didn’t import them, but instead was a viewer for the disk contents (like the Finder), it would have to scan all of the library’s folders right then and there (or be out of sync). Also, EagleFiler knows which tags and notes go with which files by virtue of its database; it doesn’t have to constantly match its internal records up with the files on disk to see where the tags go. As I add features, it will become more and more important that EagleFiler keeps track of the files itself.

Second, I think it’s a simple and beautiful model to have the folder structure in the library match that of the Finder. Make changes in EagleFiler, and they’re reflected directly in the Finder. I want to use EagleFiler to organize the actual files on my disk, not to create virtual hierarchies or folder hierarchies full of aliases.

Mark, this goes to the heart of my question actually: I would argue that an application that left the files and folders where I have placed and organized them in the Finder is MORE transparent and one that didn’t need to explicitly import or be used to move and rename files relies on LESS of its own manipulation.

Please understand that I’m not being deliberately contentious. My whole career is built around generating and organizing information, so I carefully consider the tools that I use. (probably too careful -> analysis paralysis) Some of the tools I use currently (Hazel, shell scripts, etc) operate directly on files in the Finder–moving them, renaming them, generating secondary files, etc. Some of this would become difficult or require retooling if those files were also to be managed by EagleFiler. Which is why of course, I’m asking so many questions. :slight_smile:

I’ve been following EF from the very start, and I’m also very impressed by the thought and responsiveness of Michael.

Michael,
Thank you for such a complete reply. You certainly make some cogent points–I will give this some consideration.

Best,
Jeff

Jeff:

I think there are some more advantages to the way EagleFiler does things that Michael didn’t mention.

Keeping the files in the normal filesystem means portability. What happens if I need to take my library to a Windows machine? I can still get my files. What happens if I have to take my library to a machine without EagleFiler on it? I can still get my files.

This actually happens to me a fair bit because I work from home. I work on a laptop and I have libraries of files for each project. When the project is complete, I archive it and upload it to the office server. There have been several times when, while visiting the office, someone has asked me for one file or another. I can still retrieve these files without installing EagleFiler on a computer just for the purpose.

Another big advantage is that you are guaranteed to get the file back out the way it went in. Since EagleFiler isn’t stuffing the file into a database, it isn’t going to get munged. You can pull it out of your library altogether, if you want, with no consequences.

Related to that is the fact that EagleFiler can handle any type of file. It doesn’t have to know its format or be able to display its contents to allow you to track it, add metadata to it, or open it for editing.

Personally, these are the reason why I chose EagleFiler over all the others.

Sure, I agree with that. When there are fewer restrictions it’s more transparent (to the user). I do not think that more transparency means more stability, however. You achieve stability by reducing magic, which I suppose is a different kind of transparency.

Like most things in technology, this is all about tradeoffs. It would be nice if you could use Hazel and shell scripts to operate on your mail or the contents of your iPhoto or Aperture library, move things around, etc. EagleFiler makes similar tradeoffs as those apps, although my aim is to increase transparency where possible. The restrictions will not get tighter over time, and they’ll probably get a bit looser.