?

Log in

No account? Create an account
flash memory - brad's life — LiveJournal [entries|archive|friends|userinfo]
Brad Fitzpatrick

[ website | bradfitz.com ]
[ userinfo | livejournal userinfo ]
[ archive | journal archive ]

flash memory [Oct. 7th, 2004|08:35 am]
Brad Fitzpatrick
I have a handful of CompactFlash cards (8MB, 16MB, 2 x 64MB, 256MB, and 512MB) and USB memory sticks (8MB and 256MB).

They're neat, not having moving parts, and being pretty damn robust, but damn are they slow to write to.

I'm sitting here waiting for 350 MB of files to copy to CompactFlash so I can bring it to work.

I've been thinking about making a directory synchronization tool based on CompactFlash/USB memory stick sneakernet. Have two hosts negotiate what files they have and their checksums, and each day going between home and work (or work and home), you put the media card(s) in and it gets the next 500 MB or whatever of the transfer. After a few days or a week you can move an entire directory, and faster than my shitty slow DSL upstream. And because both sides would keep/generate a database of checksums to locations, rearranging directories wouldn't be a slow operation to sync.

Okay, copy's done, I'm outta here.
LinkReply

Comments:
[User Picture]From: erik
2004-10-07 08:45 am (UTC)
Why not just use USB 2.0 flash drives?
(Reply) (Thread)
[User Picture]From: brad
2004-10-07 09:03 am (UTC)
I think I do, on some computers. I think most flash is slower than USB 2.0's transfer speeds, though.
(Reply) (Parent) (Thread) (Expand)
Nyetski - (Anonymous) Expand
(Deleted comment)
[User Picture]From: brad
2004-10-07 09:00 am (UTC)
Uh, I said the network is slow.

The point of the checksum database and sneaknet transfer of flash memory was to AVOID the network.
(Reply) (Parent) (Thread) (Expand)
[User Picture]From: recompiler
2004-10-07 08:52 am (UTC)
Never underestimate the the transfer speed of a speeding station wagon filled with 16MB flash drives.
(Reply) (Thread)
[User Picture]From: brad
2004-10-07 09:01 am (UTC)
For reals.

I guess the hardcore data mining scientists nowadays just ship each other cheap-o computers around filled with hard drives because the Internet sucks.
(Reply) (Parent) (Thread) (Expand)
[User Picture]From: mart
2004-10-07 09:00 am (UTC)

Possibly stupid suggestion

Why not just do this over the Internet in the background? Have a daemon whose job is to track the changes and propogate them during off-peak transfer times such as when you're driving to work. It can't take that long to push 500MB on a DSL link…

(Reply) (Thread)
[User Picture]From: mart
2004-10-07 09:02 am (UTC)

Re: Possibly stupid suggestion

Oh, and to avoid the angry “I ALREADY SAID THAT” response, I'm proposing that by doing this constantly in the background the speed won't matter since it'll just be happening while you're sleeping, driving, eating or whatever.

(Reply) (Parent) (Thread) (Expand)
From: jeffr
2004-10-07 09:08 am (UTC)
You mean like a multivolume tape backup system? I'm pretty sure this is a well solved problem. ;-)
(Reply) (Thread)
[User Picture]From: brad
2004-10-07 09:12 am (UTC)
I'm sure it is.

I'm also pretty sure it's about a page of Perl.

As I said to somebody else, the part I care most about is that after I rearrange my mp3s, I don't have to re-rsync or re-copy them all, if they're just moving around the filesystem.
(Reply) (Parent) (Thread) (Expand)
(Deleted comment)
[User Picture]From: erik
2004-10-07 09:17 am (UTC)
They even have those nice ones now where you can build backup profiles and press the shiny blue light on the front of the external HD and have it perform the defined backup automatically. Oh, the simplicity of it all.
(Reply) (Parent) (Thread) (Expand)
(Deleted comment)
From: evan
2004-10-07 09:44 am (UTC)
The hard part about remote syncing is you can't have both changing files and changing locations without some sort of metadata. rsync only handles the former, while your checksum project sounds like it only handles the latter (though, of course, that makes sense with MP3s).
(Reply) (Thread)
[User Picture]From: loganb
2004-10-07 11:15 am (UTC)
But since we're in vaporware land already, Brad can make his "bsync" program maintain an index of the available checksums on each endpoint. Then, when the machines are determining the file list to exchange, they can reference their checksum list to determine the files they already have. In true geek, fashion, this is far too complex and there is a much simpler solution.

Of course, this is all far easier than just manually copying a directory tree a day or just leaving a throttled FTP session running in the background.

I personally took the "easy" route and setup a linux router at home that would de-prioritize bulk transfers. About 6 days of LeechFTP solved the problem. :)
(Reply) (Parent) (Thread)
[User Picture]From: dormando
2004-10-07 10:49 am (UTC)
I've settled to using my laptop for that. I take chunks of mp3s and do a quick manual arrangement (I don't change the lineup too often, so...). I took the approach of pretending the problem doesn't exist. I have a subset of mp3s at my workstation at work, and maintain it individually by changing whole sets of directories at once.

The connection at work is also really slow (a highly contended T1), so I'm often grabbing a set of 3+ CD ISO's from home and bringing them in. Only takes a couple minutes to transfer from my fileserver to my laptop, then I'm gone. Since I've bought the DVD burner, I've also started slapping a few gigs of ISO's onto a dvd-r and bringing it in. Then I can do some basic archival in the office, and four DVD-R's is more than my laptop could reasonably hold anyway.
(Reply) (Thread)
[User Picture]From: mendel
2004-10-07 10:54 am (UTC)
You could always just put everything in MySQL, set up replication, and... er, sorry, I thought I was in jwz's journal there for a second.
(Reply) (Thread)