Compressed dmg issue

[BarryWood]

I'm attempting to automate my backups and it looked like DropDMG would fit in perfectly but I'm running into a problem. Here's my procedure:

1. Backup my boot drive to FireWire drive
2. Create a compressed, encrypted disk image of the backup drive
3. The dmg goes on one of several firewire drives that I rotate off site

I've been using this procedure for about a year and it's worked out well. I've got my boot drive backed up and the data is encrypted so that it's secure off site.

I'm trying to automate the process so I can just let it run overnight. I'm now using SuperDuper to create the initial backup and then I have it run a shell script when it completes. The shell script contains this command:

dropdmg -g systembackup /Volumes/Backup

the config "systembackup" creates a ".dmg bzip-compressed" image and I've got the ecryption passphrase set in there.

I had run some small tests and everything was working fine. SuperDuper did the backup and was able to run the shell script and DropDMG did it's thing.

I went ahead and had it do the full backup last night but in the morning I found that DropDMG encountered this error:

Error 1 - hdiutil: convert failed - No space left on device for “Backup”

There was 70gb+ on the target disk and the compressed dmg should only be about 30gb.

I started the DropDMG processes again from the Terminal and now I see what's happening: DropDMG first creates an uncompressed dmg file and then as a separate step it creates a compressed version. Not only does this take a lot of disk space, it also takes a long time to complete. With Disk Utility it will create the compressed dmg reasonably quickly in one step.

What settings do I need to use for DropDMG to get this same, one-step compressed dmg functionality?

[Michael Tsai]

What settings do I need to use for DropDMG to get this same, one-step compressed dmg functionality?

At present, DropDMG cannot create a compressed DMG from a folder in one step. So you need a lot of free disk space to make a big compressed image. I’m working on addressing this for the next version. However, if you set SuperDuper to create its backup in sparse image format, DropDMG would be able to convert that to a compressed and encrypted image in one step.

[BarryWood]

Thanks for the quick reply. That idea may actually allow me automate some of the other steps in the process.

[BarryWood]

Hi Michael,

I was able to get a complete backup last night using the method you described.

It appears, from looking at the SuperDuper log, that the DropDMG step took just under 4 hours to complete.

10:10 pm - Backup started
11:43 pm - DropDMG process started
03:40 am - Rsync command started
03:56 am - complete

The SuperDuper backup is creating an image of my boot partition to a data partition on the same internal volume.

The DropDMG image converts the data partition sparse image to a compressed, encrypted image on a firewire 800 drive.

Is it normal that it should take nearly four hours to convert a 47gb sparse image to a 26.7gb encrypted image on a dual 2.3ghz G5?

[Michael Tsai]

I was able to get a complete backup last night using the method you described.

Great!

Is it normal that it should take nearly four hours to convert a 47gb sparse image to a 26.7gb encrypted image on a dual 2.3ghz G5?

I don’t have such a G5 for testing at the moment, but at first glance that seems reasonable. Bzip2 is an efficient compressor, but it’s very processor-intensive. Zlib creates slightly larger files but is almost an order of magnitude faster. Both will take advantage of multiple processor cores.

[BarryWood]

Would Zlib be any faster than Bzip2 for uncompressing?

What type of compression does Disk Utility use when you use it to make a compressed dmg?

[BarryWood]

Here's another question. When I attempted to mount the disk image that DropDMG converted, after about an hour of verification, the Finder tells me that it might be damaged. If I proceed I get a warning box that says "The following disk images failed to mount" and next to the image name is the message "no mountable file systems".

Can you shed any light on this error?

[Michael Tsai]

Would Zlib be any faster than Bzip2 for uncompressing?

Yes, as I said above it’s much faster—about 8x, I think, the last time I measured.

What type of compression does Disk Utility use when you use it to make a compressed dmg?

I think it uses zlib, tuned for faster/less compression compared with DropDMG’s zlib.

[Michael Tsai]

When I attempted to mount the disk image that DropDMG converted, after about an hour of verification, the Finder tells me that it might be damaged. If I proceed I get a warning box that says "The following disk images failed to mount" and next to the image name is the message "no mountable file systems".

Did you try to convert the image while it was still mounted? That’s the only time I’ve seen that type of error. It’s not a good idea to copy a read-write filesystem while it’s in use.

[BarryWood]

From the SuperDuper log it looks like it doesn't dismount the image that it creates until the very end. That would mean that it's mounted when DropDMG is attempting to convert it.

I'll have to see if it's possible to dismount the image as the first command in the script that SD executes after the backup.

[BarryWood]

Well, I think it worked. I could see from the SuperDuper log that my shell script unmounted the sparse image and that DropDMG did it's thing. Using the Zlib compression only 1.5 hours rather than 3 hours with Bzip2.

I rebooted and then tried to use Disk Utility to restore the compressed image to a firewire drive. It was failing with a nondescript error so instead I just used the Finder to mount the image. When it got through the lengthy process of opening it gave me an alert box with "Disk image you are opening may be damaged" and should it proceed. I told it to go ahead and it actually mounted the image. I was then able to use Disk Utility to restore from the open image to the firewire drive. I was able to boot to the firewire drive and my system came up so it appears to have worked.

I still have the original sparse image that SuperDuper created and it mounts just fine (and rather quickly too) so there's no issue with the source of the DropDMG conversion.

I am a little wary about the Finder warning about the image. Do you have any idea what this was about?


One other note, when I was booted to the disk I restored the image to I ran a repair permissions to see what it would tell me. Here's what it said:

Permissions repair complete
User differs on ., should be 0, owner is 99
Group differs on ., should be 80, group is 99
Permissions differ on ., should be drwxrwxr-t , they are drwxr-xr-x
Owner and group corrected on .
Permissions corrected on .
The privileges have been verified or repaired on the selected volume

Any light you can shed on this one would be this would be appreciated.

[Michael Tsai]

I am a little wary about the Finder warning about the image. Do you have any idea what this was about?

No. I don’t think I’ve ever seen that.

One other note, when I was booted to the disk I restored the image to I ran a repair permissions to see what it would tell me. Here's what it said:

If you want to restore from an image via the Finder, you should mount the image using DropDMG’s Mount Image (with Owners)… command rather than by double-clicking it.

[BarryWood]

That's good to know. I was able to mount the image (with owners) using DropDMG and didn't get any errors reported.

[Jo*]

Hi there,

after searching the forums this thread seemed to fit my concern most. The TO seemed to have a similar approach backing up his data.

I am currently evaluating on how DropDMG could master my aim to securely (double)-backup my files onto an online-backup storage at Mozy. Although the backup data is being encrypted at the Mozy servers, for my peace of mind I'd feel more comfortable with my data being securely encrypted before uploading them.

Similar to the TO I backup my data to an external F/W-drive using SuperDuper!. Additionally I setup SD! to automatically create a sparseimage of my backup using its scheduled backup function.

I would then like to perform DropDMG to create an encrypted DMG from that sparseimage that SD! created. As a last step in that whole procedure, the Mozy UI is then scheduled to update the online storage with that newly created encrypted DMG.

Ideally, the whole process is on an automatic (overnight)schedule, so I won't have to discipline myself for performing the backups manually and can pretty much forget about it.

SD! and the Mozy UI can be scheduled. For DropDMG I will have to use Applescript which I would schedule via an iCal event. Not being very accustomed to Applescript, I just copied the script example from the DropDMG helpviewer (3.2.4) and tried to modify it according to my needs. I didn't get very far :-)

Basic scriot-setup:

• If DropDMG doesn't run, the script should start the application.
• The SuperDuper! sparseimage (named "MozyOnlineBackup.sparseimage") is on an external drive named "Archiv" (same root level as internal Macintosh HD)
• Since the sparseimage is about 1.5 GB in size, I increased the timeout to 4 hrs.
• The DropDMG configuration is called "Mozy_Backup"

-- create a new image from a folder
tell application "DropDMG"
    activate
    -- allow 4 hours before AppleScript times out
    with timeout of 4 * 60 * 60 seconds
        set p to "Archiv/MozyOnlineBackup.dmg" -- POSIX path to source folder
        -- Rather than specifying all the options in AppleScript, you
        -- can save them as a DropDMG configuration and refer to them
        -- that way.
        create image from path p configuration name "Mozy_BackUp"
    end timeout
end tell

Could someone please support me with modifying the Applescript in order to get it work according to my described needs?

Thank you very much!

Jo*

[Michael Tsai]

Although the backup data is being encrypted at the Mozy servers, for my peace of mind I'd feel more comfortable with my data being securely encrypted before uploading them.

Yes, I’d definitely recommend adding your own layer of encryption before giving your backup to a third party.

SD! and the Mozy UI can be scheduled. For DropDMG I will have to use Applescript which I would schedule via an iCal event.

Another option, rather than relying on iCal, would be to use SuperDuper’s “After copy…Run shell script” option to invoke DropDMG. The shell script could then be something like:

#!/bin/sh
dropdmg --config-name "Mozy_BackUp" "/Volumes/Archiv/MozyOnlineBackup.sparseimage"

If DropDMG doesn't run, the script should start the application.

The tell command already launches DropDMG if it isn’t running. The activate command just brings DropDMG to the front

The SuperDuper! sparseimage (named "MozyOnlineBackup.sparseimage") is on an external drive named "Archiv" (same root level as internal Macintosh HD)

Paths need to start with a “/”. Also, you said that SuperDuper was creating a sparseimage file, so I think you’ll want to use something like:

set p to "/Volumes/Archiv/MozyOnlineBackup.sparseimage"

[BarryWood]

For my backup I use the SD "run shell script" feature that was suggested above.

Everything has been running great using this approach.

One thing to be wary of is that if you have updated DropDMG, when your shell script attempt to run you will get a dialog box asking you if you wish to grant keychain access the new version of DropDMG. If this happens your automated backup will stall.

[Michael Tsai]

One thing to be wary of is that if you have updated DropDMG, when your shell script attempt to run you will get a dialog box asking you if you wish to grant keychain access the new version of DropDMG. If this happens your automated backup will stall.

Yes, the way around this is to launch DropDMG and make an encrypted image after updating it. That will make it try to unlock the keychain and get the dialog out of the way. In a future version, DropDMG will be code-signed so that (if you’re running Leopard) you’ll only have to grant it keychain access once (when you first install it).

[Jo*]

Thanks so much to both of you for your quick and elaborate reply!

Although I tried before to get the hang on applescripting I probably didn't even scratch the surface of the matter.

To an experienced developer or scripter that probably seems pretty lame but my talent is definetely not geared up towards command-lining, etc. - I'm more into designing stuff (on the graphical side).

So I don't even know the difference betweeen a shell script and an Applescript :-O

I checked the SD! users manual but there is not much said about shell scripting (although it's an excellent manual - which also has to be said for the DropDMG manual by the way. Very well thought out).

All I pretty much know is that I will have to write a script and save that someplace on my harddrive on which SD! can revert to using the advanced options pane of SD!.

Would you mind by any chance to show me how an adequate script would look like (ideally using the code tags of this forum so I can see a specific order). To be honest I can't even grasp the meaning of the command-lines in Michaels post...

By the way, does my DropDMG configuration as enclosed as png in my former post make sense? Would it make sense to add further encoding and would any further encoding not interrupt the automatic creation of the dmg (in terms of no manual user interference being required)?

I very much appreciate your help!

Jo*

[BarryWood]

Here is the script that I use after SD! does the backup:

#!/bin/tcsh -f

echo "Dismounting image..."
diskutil unmount /Volumes/system-backup

echo "Converting disk image..."
dropdmg -g systembackup /Volumes/Data/system-backup.sparseimage

### Always exit with 0 status
exit 0

The DropDMG configuration called "systembackup" does the encrypting of the disk image for me.

[Michael Tsai]

Would you mind by any chance to show me how an adequate script would look like (ideally using the code tags of this forum so I can see a specific order).

Try this:

tell application "DropDMG"
    with timeout of 4 * 60 * 60 seconds
        set p to "/Volumes/Archiv/MozyOnlineBackup.sparseimage"
        create image from path p configuration name "Mozy_BackUp"
    end timeout
end tell

By the way, does my DropDMG configuration as enclosed as png in my former post make sense?

Yes, although I would uncheck “Auto-open” since it will slow it down but not help anything (in this case).

Also, you might prefer to have it not “Delete after creating” so that SuperDuper doesn’t have to create a whole new image file the next time.

Would it make sense to add further encoding and would any further encoding not interrupt the automatic creation of the dmg (in terms of no manual user interference being required)?

Further encoding wouldn’t interrupt anything, but it would make the process slower unnecessarily.