Concurrency: It's the New New Thing, Elvis

T

he last week has been way too busy for me to get any blogging done, which is unfortunate: I've had lots of things I've wanted to say about lots of things.

I've been paying a lot of attention to what people have been saying about concurrency recently. Kim has had a great series of posts (look here, and at the preceding few) on concurrency. Savas had a post on PLinq. My new new favorite blog is Larry O'Brien's Knowing.net. He's also had some great concurrency posts recently, as well as a couple of articles up at DevX. He pointed me to Michael Süß's Thinking Parallel blog, which is jampacked with highly detailed info about concurrency.

In a related note, I spent yesterday at Microsoft, attending yet another briefing about Compute Cluster Server. Jeff Kahn and Bob Withers teamed up for the presentation, and it was by far the best CCS presentation I've seen yet. And yet...

...I still left feeling that, by making clusters so much easier to install and manage, Microsoft has only solved half of the problem. Or less than half.

I mean, sure: installation and administration is a big deal. Some would argue that it isn't hard with current technologies, but some would argue that it is and that needs to be fixed. Clearly, Microsoft falls in to that second camp, and they've done a ton of work to make it easy.

I'd argue: it isn't just about installation and administration. It's about the apps. How can you write applications to run on that cluster? How can you take clusters from the realm of the technical and scientific to the realm of the enterprise?

How do you make 16, 32, 64, 1024 processors available to regular developers? Not the world's greatest experts, but for Elvis and Mort as well as Einstein.

That's what needs to happen. Einstein can already write distributed software (and he already does). But for every Einstein out there, there are 10 Elvises. And for every Elvis, there are 20 Morts. Give them the ability to write software to run across the cluster, and you've got yourself a winner. (And in case you haven't seen the blatant self-promotion coming a mile away, here you go).

Sorry to only post once this week. We're due to release in a couple of weeks, and I've been slammed. This is only made worse because I'm taking the next couple of days off: I'm heading out to Knoxville to see my beloved Golden Bears take on the Volunteers in one of the most hostile environments in the country.

I'll return after Labor Day.

Technorati tags: concurrency, ccs

And writing software, too...

T

raditionally summer is a slow time in business. Kids are out of school so families vacation. Meetings are difficult to schedule so things get delayed. People stretch weekends into long weekends to accomodate camping trips and visits to relatives and the like.

But this has been a busy season around here. Cindy and I haven't taken any vacation yet; I've been in the office most of the time (with some work travel thrown in), and we've had an exciting summer. Digipede has signed new customers, has a ton of prospects, and we have several burgeoning partnerships.

But, most exciting: we're getting ready to release the Digipede Network 1.3 next month. The development for this release has gone very well (mostly due to Rob, his adoption of some agile development techniques, and his devotion to automated builds and tests). We're on schedule for a September release.

The feature set for v1.3 is exciting. First of all, we will release our Microsoft Compute Cluster Server integration piece (just in time for CCS to start shipping). For us, this is a big deal. As Microsoft enters the HPC world, we're trying to make sure that world is accessible to everyone who's doing .NET development and needs more speed or scalability. We'll have both integration with the scheduler and improved integration with 64-bit versions of Windows. We've already got proofs of concept running internally and at Microsoft, and I can't wait to see these in the field.

We're also adding a bunch of functionality that developers will love--especially software vendors that are grid-enabling their applications using the Digipede Network. We've opened up the API even more, giving more control behind the scenes. We're also allowing different scheduling algorithms (even for pools on the same Network).

Lastly, we're adding functionality in the Digipede Workbench that will make it easier for users to see everything that's happening on the Digipede Network--from within Workbench.

Besides the CCS improvements, I think my favorite thing about this release is that many of the features did not originate here at Digipede Headquarters. We've got several features that came directly from suggestions from users, and we've got several that came from discussions with partners. One thing that 16 years of software development has taught me: as much as developers think we know about software, it is the users who are the real experts.

Our users (and partners) have spoken, and we're listening.

Technorati tags: ccs, digipede

A Tale of Three CEOs

B

ack in May, I wrote a post that summarized a CRN Magazine comparison of the usability of the Digipede Network and Sun Grid (you may recall, CRN found that the "difference in ease of use in Sun Grid vs. Digipede was enormous."). I jokingly titled that post "John vs. Jonathan? Digipede outshines Sun," setting up a fictional rivalry between the two blogging CEOs (Digipede's John Powers and Sun's Jonathan Schwarz).

Apparently I'm not the only person who wants to compare John to a Sun CEO.

This week Grid Today published a story about the Digipede Network's one-year-anniversary. From the column that was published alongside the article, Grid Today's Derrick Harris must have had a great time interviewing our outspoken leader.

Titled "Digipede's Powers: The Next McNealy?", the column details what an experience it is to sit in a room with John for an hour and talk about his favorite subject--distributed computing. John is smart, he's opinionated, and he's talkative: combine those three, let him have the floor, and you're in for a ride. He'll give anyone an earful. Derrick recognized that trait and noted the similarities with Sun's old CEO, Scott McNealy (with whom I once sat at a wedding, by the way).

Is the comparison apt? I think so.

You can say a lot of things about Scott McNealy, but he took Sun on an amazing ride. And as Derrick (and a lot of other journalists) knows--he was always good for a soundbite. That's definitely a trait that Scott and John have in common. Why do they both exhibit this trait? Is it just a huge ego? No. It's because they both believe so strongly in what they do.

And that's what I like about working with John (which I've done, by the way, across three different companies for the last 16 years): he's a visionary. He has an incredible amount of conviction. And yes, Derrick, he's not afraid to speak his mind.

Technorati tags: mcnealy, digipede, powers, schwartz, sun

Two birds with one stone

L

ooking for a place where you can hear about the latest, greatest, and upcoming tools from Microsoft? Have you been wishing you could meet me face to face?

Great news! I'll be at the MSDN events today (in San Jose) and Thursday (in San Francisco). The topic for the event is "Get Connected with the .NET Framework 2.0 and Visual Studio® 2005 (and Beyond)." I'll be sitting at the Digipede table extolling the virtues of distributed computing on the .NET platform. Of course, I'll have a few Developer Editions of the Digipede Network to hand out.

Today's event(register here):
1:00 - 5:00
AMC Saratoga 14 Theater
700 El Paseo De Saratoga
San Jose California 95130

Thursday's event (register here):
1:00 - 5:00
Loews Metreon
101 Fourth Street
San Francisco California 94103

Technorati tags: msdn, digipede

Nouning: it's the new "to verb"

G

etting to attend lots of Microsoft events means getting a regular peek into the way English is evolving in Redmond. Sometimes, I hear distinctive words or phrases -- for instance, at the Worldwide Partner Conference this year, I realized that "at the end of the day" (which I thought had been drifting into the sunset of business clichés in America) is still alive and well with the Redmondians. On another visit to Microsoft HQ, a friend said "I bet you a hundred dollars that in your next meeting someone uses the word scenario." Luckily for me, I didn't take the bet.

The newest up-and-comer for annoying Redmond verbal ticks is "nouning" verbs. Don't confuse this with the "verbing" that had been going on for the last decade or so ("solution a problem," "access an account," "network the office"). No, this is the reverse. I was in a meeting recently where we were discussing "yearly spend" and were told "this is my ask."

I guess, at the end of the day, I'd rather hear a tired cliché than blatant misuse of a part-of-speech.

[Note: And, with that, I'll return to your regularly scheduled program--distributed computing]

Technorati tags: english

220, 221. Whatever It Takes

T

oday is the two hundred twenty-first day of 2006. For anyone who has ever clicked on the "pushup-o-meter" on the sidebar of my blog, you know that means I did 221 pushups and crunches today as part of my New Year's Resolution. The total so far this year is nearly 25,000 (on my way to 66,795).

My colleagues John, Robert, and Nathan are all still on board, as well—I'm not sure about Kim, but I do know she wore her skinny pants to Fresno today!

So, nearly 2/3 of the way through the year, am I happy with the resolution? I love it. I'd recommend it to everyone. I feel great, and I think all of us are looking better than we did eight months ago. I've also started riding my bike the 4 miles to work whenever possible, and that feels great, too.

It's simple. One pushup and one crunch on January 1st. Add one each day of the year. On December 31st, do 365 and be the envy of all of your friends. There are no other rules—split them into as many sets as you like. Right now, I'm doing 4 sets of pushups and 3 sets of crunches (and I do a variety of types of crunches). I've had friends chide me on my pushup form, and I've been working on it, but I'm not even a stickler about that: if you're getting down on the floor and doing 200 pushups, you can do them however you please!

As I said in January, I'm learning what I can accomplish by dedication, tenacity, and an incremental increase in effort. As a member of a startup, those are important lessons. My colleagues and I are all dedicated, we're all tenacious, and we're each putting in tons and tons of effort. I believe it shows in our product and in our company--and it shows when someone walks in the conference room and finds a bunch of people doing pushups!

Technorati tags: fitness, pushups

Grid Computing? In the Central Valley?

Y

ou know you're big when you've made it to Fresno! Well, Digipede is hitting the big time.

Next Wednesday night, August 9th, our resident Evangelista, Kim Greenlee, will be heading south on highway 99 to present to the Central California .Net User Group. Kim is a very experienced developer (despite her "marketing" position, she's taken on several of our most difficult technical integration issues, including many ways to make Excel interact with the grid).

Kim's talk is going to be great. The title is "From Threads to Grid - Application Scalability and Performance for the 21st Century," and she's going to delve into the various ways to add scalability to your app: concurrency through multi-threading, grid, etc. As I said, Kim's an experience developer (writing software for 19 years), so this will be a very technical, very informative talk.

If you're in the valley, don't miss it! RSVP at rsvp@centralcaldotnet.com.

Technorati tags: grid computing, digipede

Blurring the lines of HPC

O

  ver at scalability.org --one of my favorite reads, despite (or is it because of?) my occasional disagreements with Joe--you'll find an interesting note about using FPGAs for non-scientific applications: specifically, business applications.

However, his post isn't really so much about FPGAs as it is about the blurred lines of HPC these days. He talks about a couple of pieces of hardware out there and wonders how they may be applied to domains outside of HPC. Joe sums it up thusly: "HPC is not the only thing that needs acceleration."

This is such an important concept, and it's one that many people need to realize.

The computing universe is changing. It was once separated into many fiefdoms, with nearly no overlap: Business/back office Computing. Technical/HPC computing. Supercomputing. Desktop computing. The hardware, the OSs, and the software for each were unique to that fiefdom.

But ever-increasing commodity hardware speeds, ever-improving operating systems, and ever-more-connected systems are blurring the lines. Many of the Top 500 are no longer super computers--they're Linux clusters. One is even a Windows cluster.

Large businesses are routinely doing analysis that would make an HPC cluster wipe its brow with sweat. Some of the calculations that companies are doing as part of routine business intelligence wouldn't have even been possible a few years ago.

Even some technical computing is now being done on commodity Windows systems: the Army Corps of Engineers is doing some of their weather simulation on Windows PCs.

All of this line-blurring is why there is such a growing market for distributed computing. The hardware and OS have the ability. The software is coming along, too, to enable 'regular companies' to do the kind of processing that, until very recently, would have required 'HPC.' Joe says, in reference to FPGAs being used as Accelerator Processor Units in business computing, "I think the market for these might just be a whole lot larger than the “small” $9 billion that is HPC these days."

I think he's right. And I think his statement can be expanded: the market for general distributed computing is much, much larger than the HPC market is today.

This is exactly why I think Microsoft will succeed with their cluster operating system (and why my company is succeeding with our distributed computing product). We are finding customers who never dreamed about putting in a traditional, HPC cluster. Some are small customers who can't afford dedicated hardware or training on a new OS. Some are large customers who are simply scaling their SOA (and want to continue using Microsoft's development tools). They're not HPC users. This isn't HPC. This is, as Joe puts it, "Accelerated Computing."

Technorati tags: distributed computing, hpc

Linux's doors blown off by Windows

S

everal times in the past, I've blogged about my annoyance at some people's inability to take the .NET platform seriously for enterprise computing. Whether it's competitors, VCs, potential customers, or just blogs I read, there is an attitude out there among some people that "real computing doesn't happen on Windows."

Those people are still living in the 20th century (ok, maybe early 21st). Microsoft has had made great strides at every level in their stack: OS, web server, database, and with .NET.

eWeek's July 10th article ought to change a few minds. (Yeah, I know, I should have written about this earlier. As Albert Ganz said in 48 Hours: "I've been busy!")

As part of eWeek's "eWeek Labs Bakeoff" series, they did "Open Source Versus .Net Stacks." Their definition of a stack: OS, web server, a database, and a scripting or development language. There are certainly those out there who will scream "LAMP!" and stop reading. There are others who will show their Linux/J2EE tattoos and head for the door. But for those enough brave enough...

As it turns out, the .NET stack did well. Very well. As in, it beat the LAMP and Linux/J2EE stack in nearly every category they measured. In transactions per second, average throughput, average page download time, and average document download time, the .NET stack beat every combination of Linux-based stacks. The only category where any Linux stack beat .NET was in average hits per second, where Linux/J2EE had about a 20% advantage over .NET (a paltry difference, seeing as the Windows solutions were often several times or even an order of magnitude better than the Linux solutions in the other categories).

Now, eWeek did something else very interesting. They did a bunch of mix-and-match stacks as well: WAMP, JBOSS running on both Linux and Windows, Python on both OSs, etc. And they found that in almost every case in almost every category, the solution running on Windows was better.

Microsoft's .Net stack performed very well in our tests, clearly showing the benefits of the tight integration among each of the stack components.

Bear in mind, they didn't do tons of tuning. As Steven Vaughan-Nichols points out over on Linux Watch, everyone knows that any stack needs to be tuned to work well, and he thinks that all stacks involved would benefit from some expert tuning. But he goes on...

You know what, though? I think the tests were perfectly fair.

Many, perhaps most, IT departments really don't know how to get the most from their software stacks. Oh, they always claim that they know Windows, Linux, or whatever best, and that's why they can't switch to something new. But, when you actually look at what the staff does -- as I have -- you'll find that their expertise is really little more than knowing how to use the most popular, automated scripts.

For all practical purposes, many production systems do end up running on their default settings.

He also rightly points out that the most interesting thing was how well the open-source tools worked on top of Windows--those produced some of the most impressive numbers. And, of course, that all of that Windows stuff costs money.

Bear in mind, I'm not trying to say that everyone should be working in Windows, and I'm not trying to tell everyone to drop Linux like it's hot.

But for those of you who think that the Microsoft stack just can't compete with the Linux/Java/Open Source solutions out there: Welcome to 2006.

Technorati tags: linux, windows, .net, java, lamp, wamp, jboss, j2ee