Strange floating point behavior on Xeon

One of the first applications I wrote using the Digipede Framework API was a distributed Mandelbrot app. It's almost a cliche application for a grid computing system--still, it looks pretty and calculates a heck of a lot faster on many machines than on one.

Recently, I was tweaking our calculation code to get better colors out of it. After tweaking it, I noticed something very strange happening: different machines were calculating different colors for the same pixels! Here's a screencap that shows how it manifested itself:


Because different machines calculate different portions of the picture, and those portions may end up right next to each other, it's pretty obvious. The rough edges give you an idea that things just weren't right--I've circled some of them.

At some point, I realized that there was one machine on my testbed in particular that was returning different values than the others. I also noticed that the gradients on the bitmaps from that machine lined up perfectly with the gradients on the bitmaps from the other machines--the colors were just different. Furthermore, the colors were always 1 color off (the colors in my Mandelbrot are stored in an array).



As it turns out, that machine has dual Xeon processors in it, and it's the only one on our network that does. It was pretty clear that the Xeons were causing a problem--and today I decided to pin it town.

To calculate a Mandelbrot set you do a repetitive calculation using complex math. Basically, you start with a complex number z, and you iteratively perform the calculation z = (z * z) + c. You count the number of times you can do that before z is too large to calculate. You then choose your color based on the number of iterations.

I added some code to my calculation object to spit out the results from the calculations. On most machines, it looked like this:

after 0: z = -1.78571428571429 + -1.19047619047619i
after 1: z = -0.0141723356009069 + 3.06122448979592i
after 2: z = -11.1566088075442 + -1.2772455921144i
after 3: z = 121.052849496283 + 27.3089826542847i
after 4: z = 13906.2261232719 + 6610.46985810096i
after 5: z = 149684811.460994 + 183853376.065174i
after 6: z = -1.1396521108449E+16 + 5.50401158655656E+16i
after 7: z = -2.89953366111956E+33 + -1.25453168454679E+33i
after 8: z = 6.83344570443359E+66 + 7.27511369656889E+66i
after 9: z = -6.23129910256229E+132 + 9.94281888781693E+133i
after 10: z = -9.84713565508731E+267 + -1.23913356825186E+267i
after 11: z = Infinity + Infinityi

But on the Xeon machine, it looked like this:

after 0: z = -1.78571428571429 + -1.19047619047619i
after 1: z = -0.0141723356009069 + 3.06122448979592i
after 2: z = -11.1566088075442 + -1.2772455921144i
after 3: z = 121.052849496283 + 27.3089826542847i
after 4: z = 13906.2261232719 + 6610.46985810096i
after 5: z = 149684811.460994 + 183853376.065174i
after 6: z = -1.1396521108449E+16 + 5.50401158655656E+16i
after 7: z = -2.89953366111956E+33 + -1.25453168454679E+33i
after 8: z = 6.83344570443359E+66 + 7.27511369656889E+66i
after 9: z = -6.23129910256229E+132 + 9.94281888781693E+133i
after 10: z = -9.84713565508731E+267 + -1.23913356825186E+267i
after 11: z = NaN + Infinityi
after 12: z = NaN + NaNi

See the difference? After 10 iterations, the calculations were identical. But on the 11th, the first goes to Infinity. The second, though, goes to NaN!

According to the .NET 2.0 documentation on NaN, "This constant is returned when the result of an operation is undefined." And the documentation for Double.PositiveInfinity says "This constant is returned when the result of an operation is greater than MaxValue." Double.NegativeInfinity is, of course, "This constant is returned when the result of an operation is less than MinValue."

MaxValue, by the way, is 1.79769313486232e308. So there's no doubt that the result of squaring z was going to overflow. But I don't see anything to indicate that the result should be NaN--and all of my other machines agree: it should be Infinity.

So is this a problem with the Xeon? The FPU? The .NET 2.0 libraries on my x64 machine? It's a 32-bit app, so I don't see how that would be.

I was able to work around this in code, but it's not the kind of thing developers should have to worry about (see Kim's post on how developers rely on the systems beneath their code).

Still - I'm glad this was only an application drawing pretty pictures. I'd hate to see the calculations on the trajectory of our next Mars probe to go awry based on something like this.

Has anyone else had any discrepancies using .NET 2.0 on the Xeon?

Technorati tags: xeon, mandelbrot, .net20, digipede

MSDN Webcast Resources

Welcome MSDN Webcast attendees!

I said during my webcast today that I'd post a bunch of links and stuff--here they are. Some of these are repeats from my last webcast, but some are brand new.

If you want to download a fully-functional copy of the Digipede Network Developer Edition, just visit www.digipede.net/faster and use the promotion code MSDN.

My first MSDN webcast was called "Object-Oriented Programming for Grid," and is available on demand here.

The paper I mentioned was Web Services and the Grid by Lee Liming of Argonne National Labs. I've referenced this here before. It's a great piece on why web services and grid computing go hand in hand.

I didn't mention this in my webcast, but you also may be interested in reading this article from Ian Foster, one of the fathers of grid computing: Grid's place in the service-oriented architecture.

The blogs I listed were:

Expert Texture: Robert Anderson (CTO of Digipede) on .NET development and grid

A Day in the Life: Kim Greenlee, Digipede Evangelist, on grid computing (including lots of Excel/grid interop)

But if you're interested in distributed computing, you should also check out:

All Things Distributed: Werner Vogel (CTO of Amazon) on, well, all things distributed

Grid Meter: Greg Nawrocki (President of the Globus Consortium) on grid computing

For those who asked about source code: e-mail me! (See link at right)

Update 4/25/06 12:16: Added new links.

Technorati tags: grid computing, digipede, OOP-G

Monday: New York. Tuesday: Everywhere Else!

It's been a busy few weeks at Digipede (which you may have noticed by the small number of posts here).

In addition to planning our next release and some meetings with potential partners, we've been preparing for a couple of events next week.

On Sunday, John and I are going to New York. Microsoft is hosting its fourth annual Financial Services Developer Conference at the Millenium Broadway Hotel. I'm really excited about this--Stevan Vidich of Microsoft has asked us to give a demo during the Capital Markets keynote address Monday morning. It'll be a great opportunity to show off our stuff in front of Manhattan's finest developers--assuming that we can get network connectivity! As usual, it's a pain in the neck to actually get network connectivity at the hotel, which means I may have to give a distributed computing demo without being connected to any computers.

Unfortunately, I'm going to have to miss Tuesday's sessions (I'd really like to see the one by Naresh Bhatia and Stephen Maltzman from Sapient). I'm giving another MSDN webcast Tuesday morning, and I wanted to do that from my office. That all adds up to a hellish itinerary--I won't get to bed Monday night until about 4:30AM Eastern time.

But this should be a great webcast. After giving a good, basic "100" level description of Object Oriented Programming for Grid in my last webcast (you can view that here, by the way), I'm really going to dive in deep in this one. It'll be the most extensive code demo I've ever done. I'll show some practical applications of the Digipede Network, including using it with Excel (which I've shown before) and using it behind a web service and behind a forms app (both new demos).

You haven't registered yet? No fear: register here. Date and time: Tuesday April 25, 11:00 AM PDT.

Technorati tags: excel, digipede

Photo credit: danzo08

Grid Computing and Excel

I get lots of hits on this blog from people searching on "grid computing and Excel" or "cluster computing and Excel," so I thought I'd point people to the Digipede whitepaper page. My colleague Kim recently finished a great whitepaper on using Excel with the Digipede Network; check it out. She describes a bunch of different architectures (depending on your needs and level of expertise), all of which we've implemented in house.

And one of which I'll be demonstrating next Monday at Microsoft's 4th Annual Financial Services Developer Conference in Manhattan.

We've got a couple of other whitepapers up on that whitepaper page as well; if you've been following my blog peripherally but would like to dive in, take a look!

Technorati tags: grid computing, digipede, excel

Resources from MSDN Webcast

After my MSDN webcast, one attendee (ok, it was my wife!) suggested that I put a reading list up on my blog. Thanks, Cindy--good idea (and definitely the first time she suggested that I blog something!). I plan on creating a reading list at some point and putting that in the sidebar. In the meantime, here are the resources I mentioned in my talk today.

The paper I mentioned was Web Services and the Grid by Lee Liming of Argonne National Labs. I've referenced this here before. It's a great piece on why web services and grid computing go hand in hand.

The blogs I listed were:

Expert Texture: Robert Anderson (CTO of Digipede) on .NET development and grid

A Day in the Life: Kim Greenlee, Digipede Evangelist, on grid computing (including lots of Excel/grid interop)

But if you're interested in distributed computing, you should also check out:

All Things Distributed: Werner Vogel (CTO of Amazon) on, well, all things distributed

Grid Meter: Greg Nawrocki (President of the Globus Consortium) on grid computing

[Update 4/11/2006 4:54pm] By the way, the link to the webcast will be up tomorrow. Of course, I'll post it as soon as I get it. The next MSDN Webcast is Tuesday, April 25th at 11:00AM. I'll post a link to that as soon as its listing is correct! Oh, one more thing: there's a Digipede webinar tomorrow morning at 10:00 am PDT; if you're interested, go here for an invitation.

Technorati tags: grid computing, web services

My MSDN Webcast on Tuesday!

My first MSDN webcast is Tuesday April 11th at 11:00AM Pacific time. It's the deepest Object Oriented Programming for Grid talk that I've given outside of Code Camp.

For most of you who are reading this Tuesday morning...what are you waiting for?

Register here!

Technorati tags: OOP-G

Digipede News

I like writing about distributed computing more than I like writing company stuff (I feel so cheap when I do this!), but I also really like to spread good news!

My Gada.be feeds turned up two interesting bits of press about Digipede in the last 24 hours. Just in case you haven't set yours up yet, I thought I'd share them with you:

First, Mario Morejon of CRN magazine wrote a positively glowing piece about the Digipede Network. The headline says it all: Grid Computing Turns .Net Into Enterprise Powerhouse. I won't quote much here--just read the review; it's awesome. He gave us 5 stars!

Second, we were pleased to accept an invitation to Microsoft's BioIT Alliance. The BioIT Alliance is a group of companies "working to further integrate science and technology as a first step toward making personalized medicine a reality." Don Rule put it together--he's a very smart guy, and we're excited to be playing on his team.

Technorati tags: BioIT Alliance, digipede

Vogels on Scalability

Last December I participated in a bit of a debate about scalability. My main point (I was agreeing with Jeremy Wright at the time) was that good software has scalability designed in from the beginning. To quote myself:

...you may actually succeed and build something that people eventually use. Many people. If you are going to make it available on the web, and you're not designing for scalability, then you just aren't planning for success: you're planning for failure.

Well, on Thursday my sentiment was echoed by a pretty powerful voice: Werner Vogels, CTO of Amazon.com.

In his post A Word on Scalability Vogels sums it perfectly:

Why is scalability so hard? Because scalability cannot be an after-thought. It requires applications and platforms to be designed with scaling in mind, such that adding resources actually results in improving the performance or that if redundancy is introduced the system performance is not adversely affected.

Right on! I couldn't have said it better myself (even though I tried).

Vogels goes on to talk about the difficulties inherent in scaling: one he comes up with is the heterogeneity that creeps in when systems are expanded by adding hardware.

Heterogeneity means that some nodes will be able to process faster or store more data than other nodes in a system and algorithms that rely on uniformity either break down under these conditions or underutilize the newer resources.

It's a good point, and it's another compelling reason to use grid behind your SOA or SaaS. Why?

Because when you use a grid infrastructure, you don't have to write the code to handle the heterogeneity. Rather than have to write the algorithms to distribute your code, use a grid system that was designed for that task. It will ensure that your newer, high powered servers are utilized to their fullest potential and will also use your older servers as effectively as they can be used.

Vogels is right. Scalability is difficult. Scalability is important. My suggestion? Throw a grid at it!

Technorati tags: scalability, grid computing

MSDN Webcast: Object Oriented Programming for Grid

If you've seen one of my (or Kim's) presentations or attended a Digipede webinar, you've had a brief introduction to Object Oriented Programming for Grid.

I've been asked by Anand Iyer, a Microsoft Developer Community Champion, to give a couple of webcasts this month to go in depth on this technology. I'm thrilled at the opportunity.

We've outlined two courses; the first will be an introduction to Object Oriented Programming for Grid, along with general information about grid computing. In the second, we'll dive in deeper and show practical applications of grid: using a grid behind a forms app, behind an Excel spreadsheet, and behind a web service.

Sound interesting? The first is a week from tomorrow. Tuesday, April 11th, 11:00 PDT (that's 1900 GMT for you internationalites).

Register here.

The second one is April 25th at 11:00 PDT, and the registration is here.

Technorati tags: grid_computing, digipede, msdn, OOP-G