?

Log in

No account? Create an account
lsof -o - brad's life — LiveJournal [entries|archive|friends|userinfo]
Brad Fitzpatrick

[ website | bradfitz.com ]
[ userinfo | livejournal userinfo ]
[ archive | journal archive ]

lsof -o [Jan. 11th, 2005|11:51 am]
Brad Fitzpatrick
I'm glad I'm not the only one upset that lsof(8) won't show me fd offsets under Linux.

mulix? Help? :-)

I'm currently tarr+gzipping 384GB of data, and I'd like to know percentage complete and current compression ratio.

Guess I need a wrapper.
LinkReply

Comments:
[User Picture]From: gaal
2005-01-11 08:13 pm (UTC)
Simplest way would be a counting cat...

perl -C0 -e 'open my $log, ">somewhere" or die $!; while (read $log, my $buf, 8192) { print $buf; print $log ++$blocknum }' file | tar ... | gzip -c ...
(Reply) (Thread)
[User Picture]From: gaal
2005-01-11 08:14 pm (UTC)
(There wanted to be a newline in that second print, really there did!)
(Reply) (Parent) (Thread)
pv - (Anonymous) Expand
[User Picture]From: brentdax
2005-01-11 08:26 pm (UTC)
One of the few things that's always annoyed me about Unix commands is that few of them have any indication of their progress, even as an option. That's great as a default, since you usually don't want that sort of thing in a shell script, but I'd love it if tar, cp, and a few other commands could give me some indication of how much work was left to do when operating on multiple files. I usually end up going to another window/Konsole tab/VC/SSH session and running something like watch ls -hl destfile.tar.gz, which isn't terribly useful, but at least reassures me that it's really doing something...
(Reply) (Thread)
[User Picture]From: brad
2005-01-11 08:50 pm (UTC)
I've talked to whitaker a few times about patching all the common utils to have a --dbus-progress option that links in to D-BUS and talks about its progress, so other utilities can watch all the current ongoing operations (burns, tars, copies, etc...)
(Reply) (Parent) (Thread) (Expand)
[User Picture]From: taral
2005-01-11 08:34 pm (UTC)
It used to be supported, but that was when lsof was opening /dev/kmem to do its work. Now it uses /proc, but file offset isn't exported via that interface.
(Reply) (Thread)
[User Picture]From: brad
2005-01-11 08:50 pm (UTC)
I figured it was something like that.
(Reply) (Parent) (Thread)
[User Picture]From: taral
2005-01-11 08:36 pm (UTC)
bzip2 -vv will output a line for every 900k it processes...
(Reply) (Thread)
[User Picture]From: ch
2005-01-11 08:38 pm (UTC)
If you stick (gnu) dd in the pipe, you can hit it with SIGUSR1 and it will spew the number of records written so far.
(Reply) (Thread)
[User Picture]From: mendel
2005-01-11 09:59 pm (UTC)
pv - Pipe Viewer - is a terminal-based tool for monitoring the progress of data through a pipeline. It can be inserted into any normal pipeline between two processes to give a visual indication of how quickly data is passing through, how long it has taken, how near to completion it is, and an estimate of how long it will be until completion.
kulik:~$ pv -cN source < linux-2.2.20.tar.bz2 | bzcat | pv -cN bzcat \
         | gzip -9 | pv -cN gzip > linux-2.2.20.tar.gz

   source: 4.48MB 0:02:01 [9.84kB/s] [========>                    ] 29% ETA 0:04:44
    bzcat:   26MB 0:02:01 [ 192kB/s] [                 <=>                         ]
     gzip: 5.58MB 0:02:01 [  64kB/s] [                              <=>            ]
(Reply) (Thread)
[User Picture]From: brad
2005-01-11 10:03 pm (UTC)
Neat!
(Reply) (Parent) (Thread)
From: evan
2005-01-11 11:26 pm (UTC)
That is awesome!
(Reply) (Parent) (Thread)
(Deleted comment)
From: sergent
2005-01-12 06:46 am (UTC)

my fault?

This is probably partially my fault. Vic Abell did the /proc-based lsof port to Linux 2.0 on the machine in my dorm room in 1996 or so. I was slightly responsible for convincing him to use /proc instead of depending on kernel headers, the way I remember it, but it was long enough ago that I probably remember wrong.

Anyway, lsof just stats the files in /proc/PID/fd/FD, so it has no way to report offsets because the information isn't there.

I seem to remember someone having a kernel change that hid this information somewhere at the time, as well as an lsof change that could grok it, but I really have no idea. It can't be very hard.

Actually, look at section 10.2.1 of the lsof FAQ:
ftp://lsof.itap.purdue.edu/pub/tools/unix/lsof/FAQ
which seems to mention said kernel change. The idea as I remember it was to make stat report the size you get back when you stat the "symlinks" in /proc/PID/fd actually have the file offset in it, but it's a bit of a hack. I think that if you have a patch to do this in your kernel, then lsof will magically start supporting the -o flag without needing any more changes. It's a pretty embarrassing hack.
(Reply) (Thread)
[User Picture]From: brad
2005-01-12 07:45 am (UTC)

Re: my fault?

Ah, cool. If lsof already does that lstat hack on the /proc/PID/fd/FD link, a kernel patch shouldn't be that hard, if it doesn't already exist.
(Reply) (Parent) (Thread) (Expand)
[User Picture]From: mulix
2005-01-12 08:53 am (UTC)
Looks like you already got all the help you could need :-)
(Reply) (Thread)
[User Picture]From: brad
2005-01-12 09:09 am (UTC)
You're the one Linux kernel guy I knew for sure read my journal. :-)
(Reply) (Parent) (Thread)