I didn't follow what's running where at all. Maybe a drawing would help?

[Laptop w/ spewserver]  <---------  [ Server w/ disks and spew client ]

I guess there's two "servers" there, which is confusing.

The point is you spew random I/O at the expensive server w/ the expensive disks, which reports it over the network to the laptop running the spewserver. Then you pull the power cable on the expensive server, reboot it, and compare its disk contents with that which the laptop recorded.

From what I understand:

1. Connect hard drive/storage system/blah to computer A.
2. Connect computer A to computer B.
3. Make computer B tell computer A what to write where on the drive/system/blah.
4. At some point, pull the plug on computer A.
5. Compare what should have been written (which is recorded on computer B) to what actually was written (on computer A).
6. ...
7. Profit!

6 = angsty teens put bad poetry on disks

Wow, funniest comment reply all day.

This one is pretty good too.

Ahahaha. Yeah it is.

You'd think.

I'm just as flabbergasted.

Can I get a copy of this little script? My agency is looking at buying large arrays for PII data. This would be a GREAT test app.

If you save my agency money.. Your fed taxes might go down $0.000000000000000001 next year. :)

It's not much longer than my description of it, but email me and I'll try and polish it up a tad for release. It doesn't setup the raw mapping automatically right now, but it could/should, to be more user-friendly.

What was the vendor's response?

It seems that write through is not really implemented on quite a lot of disks. There was a guy who wrote a program that would write a 1 to sector zero, and a 1 to sector 1, and then a 2 to zero and a 2 to sector 2, and so on, pull the plug at some point. guess what sector zero has in it? A zero, NONE of the updates got written, because they were replace before they got stale enough to get written.

I am not able to find the mention though.

I'm not writing to the block device layer with the IO scheduler, though. I made a raw(8) device from a block device so I could bypass all that and actually get stuff to disk.

O_DIRECT

Well, uh, in theory. But it has a history of sucking... each filesystem implements it itself, and each used to have bugs related to O_DIRECT, one of which I found and reported recently (a deadlock in XFS).

I want to test the disks, not filesystem code. That's a separate issue.

That is just it, he did that, the problem is at a hardware level.

Ah. Cool, so I'm not alone. :-)

We don't have a spare to test it, though.

Brad, you're an evil genius. Cool idea.

I decided not to overengineer tooo much for something slightly kinda sorta similar I'm doing at work. Using 8K "pages" it writes incrementing 32-bit integers starting at page->offset(0) and going to the end of the page. It's fairly easy to check for inconsistencies, and it also lets us check for something weird we've seen once or twice with a certain bit of software, where the data is shifted to the right by 4 bytes :/

Mwahahaha. Do they have battery backup?

I don't have all the results back from Matthew yet.

So far he's had success taking out disks from behind the RAID, changing their SCSI settings with scsirastools, and putting them back behind the RAID. Why the RAID card doesn't do this itself is fucking beyond me.