?

Log in

No account? Create an account
HFS restore - brad's life [entries|archive|friends|userinfo]
Brad Fitzpatrick

[ website | bradfitz.com ]
[ userinfo | livejournal userinfo ]
[ archive | journal archive ]

HFS restore [Aug. 3rd, 2006|01:04 pm]
Brad Fitzpatrick
[Tags|, ]

During the honeymoon I frequently moved photos from my 1GB compact flash card to my laptop, whenever it got close to full. I did this with rsync, because rsync's my homeboy and I know how it works.

This morning I went to do another batch, but this time bravely used the OS X Finder, and in the process unlinked ("deleted", if you will) a bunch of photos because one of the directories had an overlapping name ("DCIM1843"). When it asked,

"Duplicate files, do you want to overwrite or skip or cancel?"

I was like,

"I know for a fact there are no duplicate filenames, so sure, overwrite...."

(because I didn't want to cancel and didn't think about what "skip" might mean, like would it skip that directory?)

Instead it deleted the only copy I had of those 80 photos or so, deleting not just the directory but all the files in it as well. Stupid finder.

So I immediately suspended the machine while I thought, so no filesystem activity occured. Then found this, which gives me hope (but $145), then went and bought an external Firewire drive to image my harddrive, using my MacBook Pro in target mode, using Artur's laptop has a host, moving all data from my laptop (with 'dd') to the firewire drive.

Except I'm only getting 4 MB/s through it, even with two dd's running, one reading, one writing. For 100GB, that's 7.11 hours. Bleh.

Any suggestions to speed this up?
LinkReply

Comments:
(Deleted comment)
[User Picture]From: brad
2006-08-03 08:20 pm (UTC)
Maybe. But not enough space on his laptop. And in the end we have to go to the external drive anyway.
(Reply) (Parent) (Thread)
(Deleted comment)
[User Picture]From: brad
2006-08-03 08:31 pm (UTC)
OS X, yes. Looks fine to me. Macbook Pro in target mode is 400 Mbps connection, and Firewire drive is 800 Mbps connection.

Not sure what I'd look for to be wrong, though.
(Reply) (Parent) (Thread)
[User Picture]From: boggyb
2006-08-03 08:17 pm (UTC)
You are using a sane blocksize with dd? I think the default is stupidly low.

I would have also thought that one dd is faster, as it's a single process (less context switches and buffer copying).
(Reply) (Thread)
[User Picture]From: brad
2006-08-03 08:19 pm (UTC)
Tried different sizes... 8MB, 16MB.... both have the same speed.

Tried one dd and two dds.... no difference. We'd thought 2 would be faster, parallizing reads and writes, keeping both disks busy.
(Reply) (Parent) (Thread)
[User Picture]From: boggyb
2006-08-03 08:21 pm (UTC)
Hmm... is the destination drive particularly fragmented? What's the CPU usage like?
(Reply) (Parent) (Thread)
[User Picture]From: brad
2006-08-03 08:23 pm (UTC)
I'm dding to a block device, not a filesystem, so fs fragmentation isn't an issue. As far as block-level fragmentation (bad block remapping), I'd hope not: it's a brand new disk.
(Reply) (Parent) (Thread)
[User Picture]From: brad
2006-08-03 08:23 pm (UTC)
CPU is like zero.
(Reply) (Parent) (Thread)
[User Picture]From: brad
2006-08-03 08:22 pm (UTC)
Also not mentioned: the pipe buffer size on OS X is probably only like 8-64 kB. So two processes might not help there.

What can I insert in the pipe chain to add an n-MB buffer?
(Reply) (Parent) (Thread)
[User Picture]From: boggyb
2006-08-03 08:38 pm (UTC)
No idea, and probably wouldn't help. You want to minimise the amount of copies of the buffer floating around. I know how I'd go about writing a program to do such a thing under windows (disable buffering in the disk driver, use asynchronous I/O with nice large chunks), but no idea for a mac.
(Reply) (Parent) (Thread)
[User Picture]From: brad
2006-08-03 08:42 pm (UTC)
Buffer copying is the least of my worries right now: CPU usage is virtually zero. I want to maximize "shit going on".
(Reply) (Parent) (Thread)
[User Picture]From: robbat2
2006-08-03 08:44 pm (UTC)
Here's an app that can add a buffer to a pipe.

http://distfiles.gentoo.org/distfiles/buffer-1.19.tgz
And then this checkout so it compiles on modern systems:
http://sources.gentoo.org/viewcvs.py/*checkout*/gentoo-x86/sys-block/buffer/files/1.19-deb-gentoo.patch?rev=1.1

sys-block/buffer in Gentoo.
upstream is long since AWOL.
(Reply) (Parent) (Thread)
[User Picture]From: brad
2006-08-03 08:49 pm (UTC)
Thanks!
(Reply) (Parent) (Thread)
[User Picture]From: grumpy_sysadmin
2006-08-04 12:38 am (UTC)
Okay, you definitely want Schily's dd. Or tar, even. Or NetBSD's pax. (Hell, even Darwin pax is better than their dd...)
(Reply) (Parent) (Thread)
[User Picture]From: brad
2006-08-04 12:54 am (UTC)
tar/pax won't do: I need to recover unlinked HFS files. tar/pax would skip them. The data I need is in-between currently linked files.
(Reply) (Parent) (Thread)
[User Picture]From: grumpy_sysadmin
2006-08-04 01:07 am (UTC)
Too true. It's been a long day. I still recommend Schily's dd, but I don't make any assertions as to whether it'd build on Darwin.
(Reply) (Parent) (Thread)
[User Picture]From: edanya
2006-08-03 08:21 pm (UTC)

what are you talkin about, foo.

image your hard drive? target mode? dd? man, im gonna have to pay attention when mac people talk now :o\

:oP
(Reply) (Thread)
[User Picture]From: taral
2006-08-03 08:48 pm (UTC)
Most UNIX dds suck. And I mean *suck*. You're better off writing your own.
(Reply) (Thread)
[User Picture]From: brad
2006-08-03 08:57 pm (UTC)
Are there any good ones?
(Reply) (Parent) (Thread)
[User Picture]From: taral
2006-08-03 09:29 pm (UTC)
Not that I know of.
(Reply) (Parent) (Thread)
[User Picture]From: grumpy_sysadmin
2006-08-04 12:36 am (UTC)
Schily's dd (usually installed as sdd) is subjectively good. It's frickin' fast, that's for sure. And Schily is nowhere near as difficult to deal with as people make him out to be; he's just very abrupt in English. (Speaking/typing fluent German to him makes him immediately friendly, in my experience.)
(Reply) (Parent) (Thread)
[User Picture]From: robbat2
2006-08-03 09:44 pm (UTC)
I also recommend this application:
http://www.cgsecurity.org/wiki/PhotoRec

I've had great success with it in the past (fully restoring files from 2-3 formats/4-5 years previously).
(Reply) (Thread)
[User Picture]From: brad
2006-08-03 10:39 pm (UTC)
Looks nice. Thanks!
(Reply) (Parent) (Thread)
[User Picture]From: johno
2006-08-03 10:47 pm (UTC)
Were the files you blew away on the HD or the memory card?

If on the card, there are utilities that will "undelete" or "recover" images from the cards.

Start at the card manufacturer.

I know the Lexar utility (which comes on the card when you buy it, saved my photos a couple of times.
(Reply) (Thread)
[User Picture]From: brad
2006-08-03 11:16 pm (UTC)
They're on the HFS+ filesystem.
(Reply) (Parent) (Thread)
From: (Anonymous)
2006-08-04 08:20 am (UTC)

And its on a Mac, right?

Weren't the old ones just moved to the trash? That's what happens when I "overwrite" the old Firefox.app with the new Firefox.app.
(Reply) (Parent) (Thread)
[User Picture]From: brad
2006-08-04 01:24 pm (UTC)

Re: And its on a Mac, right?

Now with a directory apparently. Just reproduced with junk data, replacing a directory, and files/directories didn't go in trash.
(Reply) (Parent) (Thread)
[User Picture]From: terrajen
2006-08-03 11:59 pm (UTC)
I mean it. A honeymoon do-over is necessary.
(Reply) (Thread)
[User Picture]From: grumpy_sysadmin
2006-08-04 12:26 am (UTC)
I've noticed that {Power,Mac}Book-as-Firewire-target performance is:
  1. attrocious, especially with the raw device. (pax(1)--ie, through the FS--was faster the last time I tried),
  2. and an enormous resource hog (ie, anything else I tried to do while doing a large copy that way took minutes to page back in).
I am not impressed.

(I think I was getting better throughput than that, but it was between two PowerPC-based Powerbooks...)
(Reply) (Thread)