January 11th, 2005

belize

lsof -o

I'm glad I'm not the only one upset that lsof(8) won't show me fd offsets under Linux.

mulix? Help? :-)

I'm currently tarr+gzipping 384GB of data, and I'd like to know percentage complete and current compression ratio.

Guess I need a wrapper.
belize

Parallel compression

Are there any multi-threaded compression algorithms, or at least wrappers/formats for interleaved compression, with variable interleave size?

It'd be nice to take advantage of multiple processors when gzipping 380 GB, while still doing sequential reads, even if the resultant file was non-standard.
belize

smart migrator

I just wrote a database migration script which, after each chunk of data moved, asks the load balancer (Perlbal) the free user queue depth. If more than 10 (less is just noise), it sleeps a second and asks again. Only once the queues are empty does it migrate more data.

End result: it moves data as fast as it can, without affecting page response times.

Collapse )

I've been meaning to make a generic wrapper for any utility, where the wrapper parent watches the load balancers, and the child does its work at full speed, but the parent will occasionally SIGSTOP/SIGCONT it...... haven't got around to that yet. The generic wrapper could even have another pluggable-child as its rate-limit determiner, so anybody could use it.