EagleFiler is not so comfortable for a bit big library.

Recently, I noticed that EagleFiler is not so comfortable. Currently, my library has over 3100 records with ~2.9 GB.

When importing webarchive, EF is “Importing URL” and then is “Indexing Records”. Both requires about two seconds for each. This happens almost every time for importing webarchive. Moreover, EF sometimes “copies” tags to Spotlight. This also requires about two or four seconds.

According to manual, I always open library and “hide” EF window via OPTION + click other application window and “show” EF window via click EF’s Dock icon or hot-corner.

EF manual seems to suggest that the best way to avoid this phenomenon is to split current library to multi libraries, however, I appreciate to get other advice if I keep ONE library.

Best regards,

I don’t consider that to be a very big library. One of my main ones has 60,000 records.

Importing a URL has nothing to do with library size; it’s the time it takes to download the page (and images, stylesheets, etc.) from the Web site. Indexing speed depends on the amount of text content in the library and may also be affected by your network connection and the site speed, if the Web archive has dynamic content. EagleFiler indexes in the background, so it’s not clear to me why the 2-second delay is bothering you.

The Spotlight tag storage is somewhat fragile, so EagleFiler periodically checks to make sure it’s up-to-date. The more files you have, the longer this will take. Again, this happens in the background so it’s not clear to me what the issue is.

The manual is not really talking about the things that you mentioned above. For 3,100 records I recommend using one library.

Thank you for your kind reply as always.

I am relieved to hear that your library reaches up to 60,000 records.
If 3000 records is not so BIG, my library might have curious action.

When importing webarchive, I can see that EagleFiler do importing at first and do indexing via Activity viewer.

I agree that the former is somehow independent of size of library and I don’t bother it. My concern is for the latter indexing.

The latter indexing seems to depend upon the size of library NOW. This is true when the SAME webarchive is imported to big and small library, that is, the bigger requires more indexing time than the smaller.

Until now, I think that indexing is done in the style of ADDITIONAL and indexing time is definitely independent of library size, and depends only upon the size of each imported webarchive.

Of course, if re-indexing of the whole library is happened for each importing, indexing time has to depend upon size of library.

As I cannot notice this issue until my library size reaches up to 2600 (2 GB) or so, my library may has some troublesome and may require re-index through the library for each indexing.

Anyway, I will try to reindex again.

regards,

Before reindexing, I open EagleFiler.log and find the following log appears for each importing.

2010-03-25 23:04:21,427 - jobs.pyc:1848 Error setting OpenMeta tags: <NSError Domain=NSPOSIXErrorDomain Code=13 UserInfo={
MJTCodeString = “setxattr(path, [key UTF8String], [data bytes], [data length], position, options)”;
MJTFailingKey = “com.apple.metadata:kOMUserTags”;
MJTFile = “/Users/mjt/Documents/Programming/MJTFoundation/Source/NSURL+MJTXattr.m”;
MJTFunction = “-[NSURL(MJTXattr) mjtSetXattrData:forKey:error:]”;
MJTLine = 50;
MJTStackFrames = (
“-[NSURL(MJTXattr) mjtSetXattrData:forKey:error:] (53)”
);
NSFilePath = “/Users/Username/Documents/main/Files/MP540 series \U96fb\U5b50\U30de\U30cb\U30e5\U30a2\U30eb\Uff08\U53d6\U6271\U8aac\U660e\U66f8\Uff09”;
NSLocalizedDescription = “Operation could not be completed. Permission denied”;
NSLocalizedFailureReason = “Permission denied”;
}>

This may be related to current issue?

Regards,

Yes, that’s right. It takes longer to add the same text to a large index than a small one. That’s just the way indexes work. Also, larger indexes will be more affected by filesystem fragmentation. 2 seconds seems within the normal range.

It’s not re-indexing the whole library, just adding the information for the new file.

This means there was a permissions problem that prevented EagleFiler from copying the tags to Spotlight.

I wouldn’t worry unless there’s an error in the log saying that EagleFiler couldn’t open the index file and had to start a new one.

Thank you for your kind reply again.

It takes longer to add the same text to a large index than a small one. That’s just the way indexes work. Also, larger indexes will be more affected by filesystem fragmentation. 2 seconds seems within the normal range.

I completely misunderstand about index work. I imagine indexing is like somehow numbering, and so indexing after importing means that only 3101th info is added on the current library (which infos from 1st to 3100th are established).

You mean that EF inserts imported archive as (say) NEW 2011th into current library and then indexing from 2011th - 3101th will start. This indexing-time depends upon size of library if inserting point is NOT the last.

Any way, I will try to re-index my library.

Regards,

Yes, that’s how it works. (Nevertheless, this process takes longer if the index is larger.) If you’re interested in how the index works, this page explains the basic idea.

No, it does not go back and process the files that are already in the index.

Why? What makes you think that’s necessary?

Thank you for kind reply to my elementary question.

I hope I may understand.

In conclusion, I may expect longer indexing-time than now when my library will increase number of records far more than 3100?

Regards,

Yes, having more records will generally make everything slower (although probably not dramatically so).

My database reaches up to 3800 items (3.3GB, 3700 webarchives) and importing-time is wandering around 2 seconds. This time is not changed from starting day of this thread as you predicted correctly.

Recently, I notice that importing-time seems to be slightly shorter than in that days and my additional action is only to close Activity viewer (as EF manual recommends).

If this shortening is not due to my imagination, I wonder why this viewer window relates to importing time?

regards,

It takes more processor time to display and update the window than not to display it. This will probably only be noticeable when importing many files at once, however.

It takes more processor time to display and update the window than not to display it. This will probably only be noticeable when importing many files at once, however.

Certainly. I tend to import next-webarchive before EagleFiler will finish to import previous-webarchive and so three or four items happen to share importing process.

Thank you for kind reply as always.

Size of my library reaches up to 4700 items (4.2GB, 4500 webarchives) and importing-time is still wandering around 2 seconds.

However, I notice that EF works in more brisk way than in previous post and my additional action is only to increase my iMac memory from 2 to 4 GB (OSX 10.5.8).

Though this may be only my imagination, memory size is related to EF’s brisk way of working?
And we expect more crisp behavior of EagleFiler if I upgrade from Leopard to Snow Leopard?

regards,

EagleFiler itself doesn’t need large amounts of memory. That said, having more memory is probably the easiest way to speed up lots of tasks on your Mac. The more you have, the more applications you can run smoothly together, and the more the OS can use for its disk cache.

I think Snow Leopard is generally a bit faster, but I wouldn’t expect to see a major difference.

Thank you for kind advice as always.

As for (probably) cache issue, I have one more question.

EF’s real memory size reaches up to 180 MB fluently than ever (after increasing from 2 GB to 4 GB) and I tend to reduce this size down to 100 MB or less via memory management utility(iFreeMem).
In my understanding, this utility can open inactive field of EF and this memory field may be related to unused cache. And so, this memory area may not be related to EF’s critical speed (importing, indexing, and searching).
Really, I cannot see any difference after reduction of inactive field of EF.

Is it correct understanding?

Regards,

I don’t know how iFreeMem works, but my guess would be that it purges the disk cache. It’s not clear to me why you would want to do this, since uncached files will have to be read from the slower disk, and Mac OS X is supposed to automatically recycle memory from the cache as needed.

My advice would be not to worry about small differences in performance that are probably not really under your control.

Thank you for your reply to my primitive question.

I don’t know how iFreeMem works, but my guess would be that it purges the disk cache. It’s not clear to me why you would want to do this, since uncached files will have to be read from the slower disk, and Mac OS X is supposed to automatically recycle memory from the cache as needed.

To avoid swapping from memory to disk is the reason why I use ‘iFreeMem’ and reduce(purge) ‘inactive’ memory field. I am afraid of misunderstanding ‘inactive’, however, I dare to say my understanding as follows.

I usually open Ecto, Evernote, EagleFiler, Mail, Scrivener, Clips, ClamXav Sentry, and Safari with many tabs simultaneously.
The inactive memory field may sometimes remain without purge and continue to increase if I continue to use above applications. In worst case, size of ‘inactive’ memory increases up to 3 GB (even when active memory size is only 1 GB) and swapping from memory to disk is started. And so, purging ‘inactive’ memory (including that of EF also) is required in order to avoid unexpected swapping.

In EF’s case, I can reduce real memory of 180 MB to 100 MB via ‘iFreeMem’ and so I guess inactive memory of 80 MB is purged (moved from memory to disk). If EF may NOT require this inactive memory field, I safely use ‘iFreeMem’ to reduce ‘inactive’ memory and avoid unwanted swapping.

Sorry for my poor explanation.

regards,

My understanding is that it’s good to have lots of inactive (yellow) RAM. This means that information is cached and will not need to be read from disk. It shouldn’t cause swapping because when Mac OS X needs to allocate memory it will reclaim the inactive memory before it starts paging out to disk. If you have lots of free (green) memory, that essentially means you’re wasting those RAM chips.

Purging the cache does not move anything to disk because the files are already on the disk.

I may misunderstand role of inactive memory.

Current observation is as follows.
When I reduce EF’s real memory size from 180 MB to 100 MB via ‘iFreeMem’, inactive memory size of 80 MB is changed to free memory and 128 (64+64) MB of swap disk file is created at that time.

Anyway, I will try to understand role of inactive memory more correctly.

Thank you for your kind advice again.

I live with EagleFiler comfortably.

Size of my library reaches up to 9000 items (8GB, 8700 webarchives) and importing-time is still wandering around 2-4 seconds.

Recently, importing some kind of webarchives require 4 seconds or more and I prefer to watch importing process via Activity window. However, this ‘extra’ window seems to bother other windows (say, Safari) sometimes in my case.

Is there any way to display the progress bar of importing (which displays in Activity window) in the status bar in EagleFiler (right-bottom of EF window)?

For example, the status bar displays ‘under importing’ and ‘under reindexing’.