The Grumpy Editor goes 64-bit [LWN.net]

Your editor did much of his early programming on a large, 60-bit computer. "Large" as in "you could walk around inside it." Its six-bit character set was challenged by exotic characters - like lower case. But it sure had a fast card reader. Your editor has started a few articles by saying that recent "progress" has made things worse, rather than better, but he won't be saying that this time.

By the early 1980's, 32-bit systems had taken over much of the computing world. And, with certain exceptions, 32 bits has been the way of things for a good two decades. Processor speeds have gone up by three orders of magnitude, as have disk sizes; main memory has grown by even a bit more. But most systems sold today still use 32-bit words and addresses. The fact is, 32 bits suffice for almost every quantity we need to manipulate with computers. The exception, increasingly, is memory. We have hit the point where we are running out of address space. The need to work with ever more memory to run our increasingly bloated applications will eventually push much of the industry over to 64-bit processors.

Your editor decided to be ahead of the curve, for once. So he ordered up a new motherboard and Athlon64 processor. Before the process was done, he also ended up buying a new video card, power supply, and disk drive. In fact, the only original component left in the case (a holdover from when LWN thought it might be a training company) is the diskette drive. But, the new system is now up and running, and your editor has had a chance to get a feel for what the 64-bit world has to offer.

The hardest question, perhaps, was the choice of distribution to run. The new system replaces a Debian unstable box, so Debian was the obvious first choice. The state of the Debian x86_64 port is a little discouraging, however. Installation requires starting with the basic x86 distribution, coming up with 64-bit versions of gcc and glibc, building a new 64-bit kernel, booting that, and piecing together the rest of the system with the other x86_64 packages that have become available. More than ten years ago, your editor converted, by hand, his first Linux box from a.out to ELF binaries; installing Debian x86_64 looks like a similar process. Somehow, what looked like an interesting and instructive adventure in the early 1990's is distinctly less appealing now.

MandrakeSoft and SUSE both offer x86_64 versions of their distributions. The Gentoo port seems to be coming along reasonably well, but some time spent digging through the Gentoo package database shows that much of the software base still lacks x86_64 support. Your editor, in the end, went with the Fedora Core 2 test 2 release, at least for now. FC2t2 gives good visibility into the development process (as do Mandrake and Gentoo), a familiar, Red Hat core, and the ability to play around with some bleeding-edge features like SELinux. It also is designed around the 2.6 kernel, which is an important feature.

When one leaves the x86 mainstream, it does not take long to realize that the well-trodden pathways have been left behind. Mirrors for the x86_64 architecture are relatively scarce and often behind the times. Most applications do not, yet, come prebuilt for this architecture. Documentation on how to get x86_64 systems up and running is minimal. It is all a bit of an adventure.

That said, the FC2t2 distribution works well - as well as could be expected on any architecture for a development release. And the really nice thing about the x86_64 architecture is that most 32-bit x86 binaries work just fine, as long as you have 32-bit versions of the relevant libraries around. That fact alone makes the transition to this architecture relatively easy.

The need for 32-bit libraries complicates system administration, however. An x86_64 Fedora system has many duplicated packages installed, and working with rpm can, occasionally, be a bit confusing. The rpm interface was not, perhaps, designed for dealing with a world where two packages have the same name and version number, but are still distinct. Unless you plan to leave the 32-bit world behind entirely, however, you will need two versions of the libraries. Chances are that most x86_64 systems will want to run 32-bit binaries for some time - in some cases, they perform better, and, in any case, some programs in FC2t2 (e.g. OpenOffice.org) are still built that way.

Building applications can also be a bit of a challenge, at least a first. Quite a few makefiles and configure scripts assume that libraries live in /usr/lib. On a Fedora system, /usr/lib has the 32-bit versions of the libraries; the native versions live in /usr/lib64. A makefile which uses the default gcc (which compiles in 64-bit mode) and tries to explicitly link against things in /usr/lib will fail. Once you learn to recognize this problem, it gets easy to fix.

Your editor was naturally interested in performance issues. To that end, he built a version of bzip2 in both 64-bit and 32-bit mode and compared the results. Both compression and decompression ran about 10% faster in the 64-bit mode. With the x86_64 processor, better performance is generally expected in the native mode, mainly due to the additional registers which are available. The executable size and memory usage in 64-bit mode were larger, but not by much. A second test, using the SoundTouch library yielded a surprise, however: changing the tempo of a large sound file ran in less than 1/5 the time in 32-bit mode. The Athlon64 processor, it would seem, runs certain operations far more slowly in 64-bit mode; your editor has not, yet, had the time to track this one down.

Despite the paucity of mirrors, the glitches, and the surprises, the x86_64 platform makes for a very nice Linux system. The kernel support for this architecture is outstanding, the performance is good, and the expanded address space renders concepts like "high memory" obsolete. After all, we'll never need more memory than can be addressed with 64 bits... Seriously, however, this architecture has helped to realize one of the great promises of Linux: a freedom of choice in hardware as well as software. 64-bit systems are now available at a price even an LWN editor can afford. This editor, who just shifted his old Pentium 450 box over to sacrificial kernel testing duty, is distinctly less grumpy.

(Log in to post comments)

64-bit new - 64-bit old

Posted Apr 14, 2004 19:54 UTC (Wed) by freeio (guest, #9622) [Link]

We can be thankful that some years ago, John "Maddog" Hall convinced DEC to provide an Alpha box to Linus, so many 64-bit issues were solved way back then. Because there have been a continuous stream of (less popular) 64-bit processors which run Linux, the 64-bit code base has matured quite nicely.

This is being written on the currently cheapest of the 64-bit boxes, a 333 MHz Sun Ultra 5, which is running Aurora Linux 1.0 (a port based on RH 7.3 with later kernels and patches). The 64-bit code is mostly limited to kernel space, though, because it adds so little utility on a system which maxes out at 1GB of memory. When I rebuilt apache on our 270 MHz Ultra 5, I compiled it for 64-bit, and it works just fine. The capability to compile 64-bit applications is there, but really there is no advantage for most applications and most users. If one had a Sun 64-bit box that could hold more memory, then perhaps it would be of more beneift.

So hurrah for the 64-bit i86 family processors, which will bring 64-bit processing to the masses. And hurrah for the mainstream 64-bit Linux distributions, which are available _today_. None of this niche stuff anymore!

Marty

The Grumpy Editor goes 64-bit

Posted Apr 14, 2004 20:20 UTC (Wed) by corbet (editor, #1) [Link]

"It seems that 32-bit processors with 36 or 40-bit address buses could prolong the life of 32-bit (instruction) processors for some time."

That, of course, is exactly what modern Pentium processors are. You can put 64GB of memory into such a thing, but that road leads to all the high memory hassles that the kernel hackers have been having so much fun dealing with. Addressing more physical memory can be (and is) done; the real problem is that you need larger virtual addresses. And that really forces a larger word size or your pointer arithmetic slows to a crawl.

Unless, of course, you want to get back into multiple memory models and "near" and "far" addresses. Personally, I don't miss those days at all.

The Grumpy Editor goes 64-bit

Posted Apr 14, 2004 20:38 UTC (Wed) by tjc (guest, #137) [Link]

Unless, of course, you want to get back into multiple memory models and "near" and "far" addresses. Personally, I don't miss those days at all.

Nor do I, and to be honest, I'm not really "up" on the details of memory management in Linux. But it seems to me that going to a 64-bit address space isn't free either. For one thing, if you use hierarchial paging, either the page table is going to be huge, or else you're going to end up with about 6 levels of indirection, which has got to be slow too. My best guess is that they hash the page table somehow?

Anyway, I'll let you see how it goes, and I'll move to 64-bit when it's safe - say in 3 or 4 years. :-)

The Grumpy Editor goes 64-bit

Posted Apr 14, 2004 22:07 UTC (Wed) by smoogen (subscriber, #97) [Link]

Its not really the memory management of the Linux box as much as the memory management of the PC architecture itself. There are limits to where a PCI bus, BIOS, and other things can be 'hacked' to working and each time more memory is extended it gets more hackish than a 'clean-minimalist fix'

The Grumpy Editor goes 64-bit

Posted Apr 14, 2004 20:28 UTC (Wed) by dlang (subscriber, #313) [Link]

the Opteron 64 bit approach is actually simpler to use for both new and legacy software then the 36 or 40 bit memory extentions that Intel is providing, and it performs better as well.

whose who have been around long enough to remember EMS memory on the 8088 machines will find the Intel PAE memory concept very familiar, and there is a GOOD reason why nobody uses EMS memory anymore

2^64 bytes is enough for any human

Posted Apr 15, 2004 7:53 UTC (Thu) by zooko (subscriber, #2589) [Link]

Do you think that we will someday need more memory than can be addressed with 64 bits?

2^64 is 18,446,744,073,709,551,616. That's 17,179,869,184 gigabytes.

Suppose you had some kind of memory storage device which could store 1 gigabyte in a 1 millimeter by 1 millimeter square. Then your whole system would occupy a field 130 meters by 130 meters. Or you could go 3-D -- suppose you have a 1 GB device 1 millimeter by 1 millimeter by 1 millimeter. Then you would need a volume 2.5 meters by 2.5 meters by 2.5 meters to hold enough of these devices to store 2^64 bytes.

And what would you store in there? Let me put it another way -- what's your max bandwidth? How many bytes of data can you meaningfully produce or consume in a second? Let's say that you will someday produce and/or consume 1 GB per second, and you want to store all of this stuff for possible later use. Then this device that you have can store enough of this data for 544 year's worth of uninterrupted, non-repeating production/consumption.

Maybe corporations, AIs, or neo-humans will have use for such amounts of memory, but I don't think that humans will!

2^64 bytes is enough for any human

Posted Apr 15, 2004 11:59 UTC (Thu) by fjf33 (subscriber, #5768) [Link]

Back when people were storing bits in induction coils nobody though that we would need 1 MB of ram and the future computeres were the size of planets. What makes you think that the rate of growth would slow any? It sounds like a couple of Mols of memory cells would do the trick pretty well in a very small space. It just will be different. It may also be that future MMOGs will be simulating things so close to reality that 2^64 would be reasonable.

2^64 bytes is enough for any human

Posted Apr 16, 2004 15:31 UTC (Fri) by giraffedata (subscriber, #1954) [Link]

Back when people were storing bits in induction coils nobody thought...

I'm always suspicious of these statements. Throughout history, people have been smart and imaginative.

Though I was not talking to people about memory sizes in those days, I'm sure anyone who thought about it seriously realized we would need 1MB of memory in a computer.

They quite possibly didn't think we'd ever be able to get it, but that's a whole different discussion.

I guess we could put it another way: Nobody thought in the future we would need 20 bit addresses. (Because we'd never be able to get 1MB of memory behind a single CPU).

2^64 bytes is enough for any human

Posted Apr 20, 2004 10:47 UTC (Tue) by zooko (subscriber, #2589) [Link]

At some point, the "we'll always need more" belief has to be wrong. For example: do you think we'll need more than 2^128 memory? I don't. How many atoms of matter are there in the universe?

2^64 bytes is enough for any human

Posted Apr 22, 2004 18:03 UTC (Thu) by chant (guest, #20286) [Link]

First, imagine some sort of 3-d movie.
Then, an interactive 3-d movie.
Then consider a large-scale 3-d movie; i.e. modelling what happens to each of those atoms in the universe over time.
Or, perhaps, modelling the little tiny components (and sub-components ad nauseum) of even one of those atoms. The possibilities are endless.... and perhaps, storing the 'feel' & phyiscs reacions of physical textures, the bonds between each of those atoms, etc.

Or....on another order of magnitude altogether, the binaries for Windows U[niverse]E.

2^64 bytes is enough for any human

Posted Apr 27, 2004 15:01 UTC (Tue) by slamb (guest, #1070) [Link]

The point is not whether or not it would be useful to have an accurate model of the universe, but that it is impossible. Clearly you can't do better than (or even as well as) harnessing the largest set of good quantum numbers of every particle in the universe. (Your computer has to be in the universe!) So by examining the number of particles in the universe, you can come up with a number that is larger than the maximum possible storage space of a computer. Take the base-2 log and you've got an inexhaustible address space.

I've seen discussions of this number before, but I don't remember the result. But I seem to recall it being less than 2^128; thus, a 128-bit address space is good enough for anyone. Really.

There are other arguments based on thermodynamics limiting the complexity of a computer. Google for discussions of thermodynamics and reversible vs. irreversible computers and you'll see what I mean.

2^64 bytes is enough for any human

Posted Apr 15, 2004 12:26 UTC (Thu) by tjc (guest, #137) [Link]

And just think how long it would take for the POST to count all that! :-) You'd need a pretty fast CPU too..

2^64 bytes is enough for any human

Posted Apr 15, 2004 21:29 UTC (Thu) by wolfrider (guest, #3105) [Link]

--To blazes with the POST; you'd basically have to check that much memory on-the-fly (before malloc returns) and/or in the background.

2^64 bytes is enough for any human

Posted Apr 16, 2004 15:24 UTC (Fri) by giraffedata (subscriber, #1954) [Link]

Do you think that we will someday need more memory than can be addressed with 64 bits?

I do, but the question you've answered is really, will we someday need more than 2^64 bytes of memory in a computer like the ones we have today?

You've made an excellent argument that the answer to that question is no.

On the other hand, I'm sure some day computers will be structured differently, and we will most likely be addressing memory larger than what fits on a motherboard, or even in one building. There is definitely more than 2^64 bytes of information to be stored in the world. We'll struggle for a while with all the usual stopgaps -- paging, multilevel addressing, and such. But eventually we'll break down and get more than 64 bits of address space and go through a transition much like the 32/64 one.

The Grumpy Editor goes 64-bit

Posted Apr 15, 2004 8:06 UTC (Thu) by evgeny (subscriber, #774) [Link]

Nice article. I, too, was faced by the question which distro to choose for the first dual opteron I ordered for our lab. At that time (a few months ago), among "well known" (to me) distros only SUSE officially supported amd64, RH was at beta, but both would cost considerable $$. Not that I'm against commercial distros, but when kinda _forced_ to pay for Linux, I felt offended ;-). Debian/amd64 was even at a scarier state than now (yep, I have an a.out->elf conversion experience as well, and, too, with years such an entertainment seems less attracting...), so I decided to give Gentoo a try. It was at beta then, as well. Surprisingly (it was my first Gentoo experience), the installation was reasonably smooth (and, I dare say, reasonably fast, due to the power of the system). Not without some rough edges, of course. A lot of apps/utilities were masked so I struggled quite a bit with emerge. On the other hand, the box was intended for numbercrunching tasks, so not much beyond base system was needed. Once configured, it runs nicely, updates are easy etc. And Gentoo/amd64 doesn't have the dreadful /lib64 issue.

So, when the time came to installing Linux on my new shiny dual-opteron _workstation_, I hardly hesitated. Well, I haven't been disappointed. But there were issues specific to the workstation install. Many apps are masked, indeed, but most of them are rather "not unmasked" since no one yet reported success. Masks can be easily overriden on the command line (and in the case of success, everyone's encouraged to report to the Gentoo's bugzilla, which usually results in an almost-instant unmasking of the package in the CVS). So now I'm running an almost-pure full-blown 64bit desktop (OO being the only 32bit exception). Mozilla is 64 bits, so the only thing missing is flash in web pages (those who need it badly can run 32bit mozilla/firefox/... builds). For me, it quickly added to the reasons why one should hate closed-source software, and I'm constantly updating my black list of sites whose owners hire brainless web designers putting a flash as the entry page navigation (and these "designers" deserve to be put in the hell forced to program under Windows 2.0 on a 640k AT). The only issue I spent a lot of time was the NVidia (propriety) driver - which resulted in swapping the FX5200 thingy with a Radeon 9200 card supported by XFree out of the box. OpenSource scores one more. (Can't resist temptetation to share my experience with the so much worshipped NVidia support. It seems, all they're doing is accurately redirecting bug reports to /dev/null with a few occasional comments on the BB forums of the kind "be happy Nvidia bothers with you filthy linuxists at all".)

Wow, quite a lengthy comment... Hope it helps others.

The Grumpy Editor goes 64-bit

Posted Apr 15, 2004 14:51 UTC (Thu) by BillyBreen (guest, #3988) [Link]

I second the Gentoo endorsement, though I've been using Gentoo on various AMD and Pentium x86 boxes for the past 3 years. I built my first Athlon 64 box a few weeks ago, and I've been very happy with the Gentoo x86_64 experience thus far. I've emerged a number of 'masked' applications and they've all worked perfectly for me.

The Grumpy Editor goes 64-bit

Posted Apr 15, 2004 14:56 UTC (Thu) by komarek (guest, #7295) [Link]

Our lab started using Gentoo for our ageing Alphas. When we moved to Opterons, we kept using Gentoo. Things have gone relatively well for us. We have no major gripes with Gentoo, but of course there's the occasional grumble. We're using pure 64 bit right now, which isn't a problem since we build all of our research own code from source.

We are stuck on kernel 2.4 because we can't get rw AFS for kernel 2.6. Please, someone, replace AFS! It won't die on its own.

-Paul Komarek

The Grumpy Editor goes 64-bit

Posted Apr 15, 2004 10:24 UTC (Thu) by ewan (subscriber, #5533) [Link]

For floating point number crunching an Itanium is a more obvious
competitor than a Xeon and will confortably outperform an Opteron
even though running at a lower clock rate.

The Grumpy Editor goes 64-bit

Posted Apr 15, 2004 10:35 UTC (Thu) by evgeny (subscriber, #774) [Link]

Itanium is in the different price domain. As to performance, I tried
Itanium-1 and wasn't impressed. Probably Itanium-2 is different.

The Grumpy Editor goes 64-bit

Posted Apr 16, 2004 23:20 UTC (Fri) by ksmathers (subscriber, #2353) [Link]

As to performance, I tried Itanium-1 and wasn't impressed. Probably Itanium-2 is different.

Certainly I2 is improved, but not that much. A disadvantage of EPIC is that if your compiler isn't really good at optimizing for the instruction set, the CPU will spend most of its time idle. Itanium performance depends very heavily on being able to keep the instruction lines full, and since there is no intelligence in the CPU at all, it is up to the compiler to accomplish that parallelism.

Itanium can beat any other CPU out there, but without the parallelism afforded by an extremely customized compiler backend, it can also feel like molasses.

The Grumpy Editor goes 64-bit

Posted Apr 17, 2004 16:06 UTC (Sat) by dvdeug (subscriber, #10998) [Link]

The AMD64 doubles the number of general purpose registers. That may cause a big difference for some 32-bit apps.

The Grumpy Editor goes 64-bit

Posted Apr 26, 2004 18:34 UTC (Mon) by barrygould (guest, #4774) [Link]

You must compile your apps into 64-bit binaries in order to get access to the extra registers.

A 32-bit binary won't be able to use them.

64-bit?

Posted Apr 16, 2004 21:40 UTC (Fri) by RichardRudell (subscriber, #7747) [Link]

Wow. Lots of feedback on what should now be an old topic (-:

Redhat Enterprise Linux 3 was my choice and it works flawlessly. Pricey, yes, but all of the features needed for compute-intensive applications which need more than 32-bits. Robust, solid, and supported by the third-party software I need.

I notice that LWN has never done a piece on how RHEL3, which is a 2.4 kernel, has all of the most useful features of the 2.6 kernel backported to a solid and stable platform. If you need me to list what they are, perhaps a detailed piece analyzing the RHEL3 kernel is more in order ...

[I have no relationship to RedHat other than being a satisfied customer].

BTW: it's not 64-bit addressing. Hypertransport has a physical limitation of 40-bits. Yeah, I couldn't believe it either until I read the specs. 1 TB is a nice improvement over 4 GB, but it's not the 2^64 talked about in an earlier post. Forget virtual address space (currently 48-bits on Opteron), I'm talking about physical address space. There is a hardwired limit which is going to hurt sooner than later.

Worse, because "int" is still 32-bit, we should really call this the "36-bit" machine, i.e., many applications will break because they count the "number of things" using an "int"; if the "thing" is 32-bytes, the "int" overflows at 2^36 bytes. That's only 16 GB which is too near current reality for my liking.

Also, 64-bit mode beats 32-bit mode on Opteron only for "small memory" applications. "Big memory" applications, i.e., why you want 64-bits in the first place, still lose by 5-20%. See, the problem is pointers being 64-bits reduces the effective size of the L1/L2 cache and cause increased traffic to DDR. Typical applications show a 1.6x increase in memory space (each struct being a mix of pointers (64-bit) and ints (32-bit). The extra 8 registers and the new ABI (pass arguments in registers, no frame pointer) help, but you just can't overcome a 60% increase in memory traffic when you really need to use all that memory.

Sincerely,
Richard Rudell

64-bit?

Posted Apr 16, 2004 22:02 UTC (Fri) by RichardRudell (subscriber, #7747) [Link]

Oops. 2^36 is 64 GB. But you can tweak the numbers however you please, make the "thing" 128-bytes instead of 32, and the result is the same. Still too close for comfort.

I would like to add that my field is Computer-Aided Design for Integrated Circuit Design which has been fighting the 4 GB memory barrier for the last 5 years. First came 3 GB on Linux, then 3.5 GB, then 4 GB with the "HUGEMEM" Linux kernel patches (flush TLB on every syscall). The only other solutions have been Sparc-64 and IA64, which have been, hmm, less than price/performance competitive.

64-bit?

Posted Apr 17, 2004 23:39 UTC (Sat) by evgeny (subscriber, #774) [Link]

> Worse, because "int" is still 32-bit, we should really call this the
> "36-bit" machine, i.e., many applications will break because they count
> the "number of things" using an "int"; if the "thing" is 32-bytes, the
> "int" overflows at 2^36 bytes. That's only 16 GB which is too near current
> reality for my liking.

I didn't get it at all. Nor the 32->36 step neither (more importantly) the connection between sizeof(int) and addressable memory - unless you're talking about braindamaged apps casting pointers to ints of some kind.

> Also, 64-bit mode beats 32-bit mode on Opteron only for "small memory"
> applications. "Big memory" applications, i.e., why you want 64-bits in
> the first place, still lose by 5-20%. See, the problem is pointers being
> 64-bits reduces the effective size of the L1/L2 cache and cause increased
> traffic to DDR.

By "big memory" you probably assume not big enough to hit the 2/3/4GB (depending on kernel patch) native limit of x86. Anything above should clearly favor 64bit.

> Typical applications show a 1.6x increase in memory space (each struct
> being a mix of pointers (64-bit) and ints (32-bit). The extra 8 registers
> and the new ABI (pass arguments in registers, no frame pointer) help, but
> you just can't overcome a 60% increase in memory traffic when you really
> need to use all that memory.

Uh? A typical app spends half of its allocation in pointers?! There are such apps certainly, but they are 1) not memory-hungry AND 2) don't need a high CPU<->RAM bandwidth. Anything really memory-intensive should use a contigous memory areas (arrays or some custom mappings) in the first place, in which case the overhead is negligible. The only exception is probably memory debuggers (BTW, one thing I miss badly for x86_64 is valgrind).

64-bit?

Posted Apr 18, 2004 9:21 UTC (Sun) by RichardRudell (subscriber, #7747) [Link]

Sorry, I should have been more clear.

I'm not talking about "brain-damaged" apps which mix int and pointer. I'm talking about a different type of brain-damage -- using an "int" to count, for example, the number of transistors on a chip. This "int" overflows at 2^31 meaning that, regardless of the fact that I can now address 1 TB, I'm going to be hit by a different type of problem before I reach that limit.

This is a different problem from the "make applications 64-bit clean". It is to "upgrade" applications so they use 64-bit ints when necessary.

Re: "uh"; I'm speaking from experience as a user of IC design tools which are (1) memory hungry (4GB-12GB for a single process is not atypical); (2) need high CPU/RAM bandwidth (all of that memory is aggresively used); and (3) show a 60% increase in virtual memory when running in 64-bit mode. I am not attempting to present these as "typical" applications; just that they are important to me ...

I anxiously await Intel's entry into the field with IA32e. The ensuing competition will be great for everyone, both in increasing performance and lowering price.

64-bit?

Posted Apr 27, 2004 15:05 UTC (Tue) by slamb (guest, #1070) [Link]

That's not so bad. They left room in the instructions to address the full 64 bits. They'll be able to replace the hardware with something that supports the full 64 without rewriting the compilers and recompiling all the software.

IIRC, the first "32-bit" Intel machines couldn't actually address 4GB (or even 2GB) of memory, either. But it didn't really cause a problem; by the time people actually wanted to do that, they'd bought new processors. I expect this will be the same situation.