Log in

No account? Create an account
pathetic state of 64-bit availability - brad's life — LiveJournal [entries|archive|friends|userinfo]
Brad Fitzpatrick

[ website | bradfitz.com ]
[ userinfo | livejournal userinfo ]
[ archive | journal archive ]

pathetic state of 64-bit availability [Sep. 9th, 2004|05:33 pm]
Brad Fitzpatrick
Here are the 64-bit options as I understand them, and why they're all depressing:

EMT64 Xeons -- we got two, but the motherboards from SuperMicro were from a faulty manufacturing run and have to be recalled... they passed burn-in, but they'll only boot once before the CMOS battery dies. Unfortunately we can't get replacements because Intel has apparently told SuperMicro and all other vendors that sell Intel's Lindenhurst 800Mhz FSB EMT64 chipset to stop, because Intel's found bugs in the chipset. This might be confidential, but I didn't sign no NDAs. I heard it from a little birdy who heard it from another birdy, and so on, and the birdy telephone game just results in a lot of confusing chirping, so I don't guarantee anything I say here is the truth... just what I decoded.

AMD Opterons -- incredibly spotty availability. some weeks only the high-end ones are available, and only a few, and some weeks only the low-end ones are available, and only a few. plus the motherboards are flaky compared to the robust server ones available for Intel chips.

Itaniums -- see above w/ spotty availability. Apparently these aren't stocked well in the US? Then where are they stocked well?! Plus, who uses Itaniums? But at least it's an official Debian-supported architecture. (amd64 will be soon....)

SPARC -- Solaris OS; Sun pricing. I'd like to run Linux, just for consistency. Solaris 10 did impress me, though, but I don't have time to learn it on a production system we needed 3 weeks ago. I want to learn Solaris 10 in isolation without stress.

G5 Xserve (update) -- only 8GB of memory? oh, and pricey.

In conclusion: Bleh.

We waited quite some time for the EMT64 Xeons, but now we can't even get one to work. I had assumed we could always go AMD (and betray the family that fed me) but that's not even looking like an option.

BTW, the reason we need 64-bit processors is so InnoDB can use tens of GB, and not just ~2GB.... 32-bit address space is just so tight.

But much love to Silicon Mechanics and SuperMicro for working so hard to get this all working for us. A fix might be happening afterall. We'll see. I guess this is what you get on the bleeding edge.

[User Picture]From: paladin3
2004-09-09 05:35 pm (UTC)
xServe G5?

(Reply) (Thread)
[User Picture]From: brad
2004-09-09 05:39 pm (UTC)
Hm, maybe. Linus uses a G5 (ppc64) as his main computer nowadays. Most the benh tree is merged into vanilla.

Thanks for the idea.
(Reply) (Parent) (Thread)
[User Picture]From: feignedapathy
2004-09-09 05:42 pm (UTC)
So I assume Xserves are out due to Apple pricing and OS X underneath it?

Why ya gotta h8? ;)

Debian runs on ppc64. :D
(Reply) (Thread)
[User Picture]From: brad
2004-09-09 05:54 pm (UTC)
I'd run Linux on it, which is well supported. Pricing's not /that/ bad. The main issue is memory. It has 8 slots, but they don't take 2GB sticks, apparently?
(Reply) (Parent) (Thread) (Expand)
[User Picture]From: krow
2004-09-09 06:03 pm (UTC)
You are the first person I have heard complain about opeteron. I am considering picking up an opteron box just for local builds/64 bit play thing.
(Reply) (Thread)
[User Picture]From: lorockia
2004-09-09 06:05 pm (UTC)
What about a Power4+ or Power5 from IBM?

the Entry level computers supports 32GB of ram.

The most impressive thing is the Power5 can support
512MB L3 Cache. (yes MB, and its only the real high end ones)
(Reply) (Thread)
[User Picture]From: jwz
2004-09-09 06:38 pm (UTC)
Have you tried calling your vendors douchebags and then offering sex?

I'm just sayin'...
(Reply) (Thread)
[User Picture]From: ch
2004-09-09 06:56 pm (UTC)
(Disclaimer: I work in HP Labs.)

HP is shipping Opteron (DL145, DL585) and Nocona (G4 - DL360, DL380, DL560, DL580, DL740, DL760, ML350, ML370, ML570) rack servers.

You can stuff 64G in the bigger models.


There's IA64, too, of course.

We (and our obvious two competitors) are first in line for silicon and for assistance from Intel and AMD.

(Reply) (Thread)
[User Picture]From: brad
2004-09-09 09:02 pm (UTC)
That might be the way to go, getting from a big-named vendor that has silicon-access-rights direct from Intel/AMD.
(Reply) (Parent) (Thread) (Expand)
[User Picture]From: peter_zaitsev
2004-09-09 09:17 pm (UTC)


Sad to hear you're so unlucky :)

We have early EMT64 in our lab which had some issues initially but with BIOS update
it runs pretty solid. I do not run stress tests on it however. You would be the first
person I would hear to run it in Production. Opteron is very common however - perhaps
people normally do not have such high mainboard features requirements or might be it is all about family roots :)

Anyway these all early bugs should be fixed in few months as it usually happens with new stuff.

My love is still with Opteron at this point, at least until I got the chance to benchmark 4CPU Opteron vs 4CPU EMT64
(Reply) (Thread)
[User Picture]From: brad
2004-09-09 09:23 pm (UTC)

Re: EMT64

We did like 3 BIOS/eeprom flashes of our EMT64 and it finally all worked, except the leaky diode to the CMOS.

But that's being fixed this weekend and Monday/Tuesday of next week. We should have it burned=in/tested and ready for production on Wednesday.

We'll see.

I guess the AMDs are in stock now, so we could go that route too.
(Reply) (Parent) (Thread)
(Deleted comment)
[User Picture]From: brad
2004-09-10 01:30 pm (UTC)
What I hear about Tyans is that they're hit and miss. Some board models are great, some are flakey.
(Reply) (Parent) (Thread)
[User Picture]From: agreg
2004-09-10 05:02 pm (UTC)


"BTW, the reason we need 64-bit processors is so InnoDB can use tens of GB, and not just ~2GB.... 32-bit address space is just so tight."

Just wondering, what kind of application are you using InnoDB for that requires >10GB RAM?
(Reply) (Thread)
[User Picture]From: brad
2004-09-10 05:42 pm (UTC)

Re: InnoDB

A LiveJournal database cluster.
(Reply) (Parent) (Thread) (Expand)
From: divelog
2004-09-10 08:51 pm (UTC)


64 bit eh? What are you going to do next year when lj needs more than 18 Exabytes of ram? :)
Seriously though, can't you get 32bit machines that take 12G of ram? Can't linux work around that 2GB limitation?
(Reply) (Thread)
[User Picture]From: brad
2004-09-10 11:51 pm (UTC)

Re: So...

Research process address space layout.

32-bit x86 with PAE can address up to 16GB, but with paging... a single process can't.

So you can run four 4GB processes, but not one process (one database) that can use it all at once. (in reality the top 1GB is used by the kernel, so it'd be more like five 3GB processes, but a lot of that is wasted on other stuff too.....)

We do use our 8GB and 16GB 32-bit machines to run tons of parallel memcached instances, but it just doesn't work well for databases.
(Reply) (Parent) (Thread) (Expand)
[User Picture]From: fallingwater
2004-09-24 05:01 am (UTC)
I don't see IBM's POWER5 gear on that list.
Too rich for your blood? ;-)
(Reply) (Thread)
[User Picture]From: brad
2004-09-24 08:09 am (UTC)
I called for a quote, told them where to email it to, and they called back when I wasn't there. Hell if I want to call back and talk to a salesdroid to get a number. They should've just emailed it to me. *grumpy grumpy*

I suppose I should press on with them, though.
(Reply) (Parent) (Thread) (Expand)