The more things change, the more they, uh, you know...

T

his quote came across my news alerts today:

Fewer than one in 10 businesses is planning to adopt grid computing because of concerns about cost, complexity and security, according to a survey of IT managers.

It was reported here in Silicon.com, quoting a study done by Vanson Bourne.

The article paraphrases Peter Critchely of Morse, who commissioned the study, saying, "...grid computing is being held back by confusion and myths surrounding the technology."

I was all fired up to write a post about how needless complexity is what has been holding back grid computing for quite a while...and then I remembered that I already wrote that post. Last September. I wrote about how, at Digipede, we felt that addressing complexity issues was our top priority.

We concentrated on ease-of-use in every phase of the product--from how it would be sold, how it would be installed, how developers can work with it, and how it would work for people who have never written a line of code in their lives.

In the intervening 9 months, we've been able to show that grid computing doesn't have to be difficult. There's the bazillion dollar investment fund that ported their trading analysis software faster than their lawyers could examine a license agreement (case study here). There's the media company that took a CPU-starved, multi-threaded application that was threatening to choke their server, and in just a few days had it running on 20 machines on their network (case study here). There are companies scaling their web applications. There are biologists doing complex genetic analysis. There's even weather simulation. All of it happening by doing "grid computing."

And how many clients have we visited? Zero. Nada. Zilch. Oh, sure, Rob and John went to Universal Studios with one client, but that was really for the Mummy ride, not to discuss intricacies of grid computing.

So how have we been able to help our customers if we aren't holding their hands? Simple: our product is radically easier to BILU.

Today, we made it even easier to try out. Today we announced our free Digipede Network Developer Edition. What is the developer edition? It's a fully-functional, full-featured version of our product. It lets any developer working on a Microsoft platform check out the Digipede Network up close and personal. Run our samples. Run your own code. Play with it. We want more developers to experience what our customers already have: Grid computing doesn't have to be hard.

To download the Digipede Network Developer Edition, go here.

Technorati tags: grid computing, digipede

Vogels and Gray

I

 f you're interested in distributed computing (and if you're reading this, you probably are), I sure hope you're also reading Werner Vogel's All Things Distributed. Last week, he mentioned an interview that ACM Queue magazine had published. Jim Gray of Microsoft Research sat down with Werner (CTO of Amazon) and did a terrific interview. Hat tip to ACM for giving them the space to have a great conversation--it runs about 8 pages long.

Reading these two discussing distributed computing is a little like listening in on Bohr and Planck talk atomic theory. They really are two of the world's preeminent authorities in this territory.

The topic of the interview is Amazon's technology, so there is a ton interesting history. Among the nuggets you'll find is that Amazon was a single, monolithic application until 2001. At that time, they changed to a service-oriented architecture (and, as Werner points out, that wasn't a buzzword back then--it's what they had to do to scale).

I won't try to summarize the whole article (you should go read it). It's a great exposition on the practical application of SOA. The relevant thing for this blog--it's distributed computing that makes the whole thing possible.

Technorati tags: distributed computing, amazon

Well, at least they're paying you to go through it...

A

  ccording to this recent review, writing code to run on Sun Grid is a serious pain in the astral plane ("analysis finds hidden costs in Sun platform" and "Sun Grid requires extensive man-hours and code rewriting to run custom applications").

Apparently, Sun is feeling the heat. As I saw in Greg Nawrocki's post today (and other places, including Slashdot but not Techmeme, interestingly enough), Sun is offering $100,000 to anyone who can figure out how to do it as a prize in a contest.

Have fun with that!

Technorati tags: sun grid

Scaling A Web Service--Webcast Tomorrow

O

 ne of the most exciting ways that people are using the Digipede Network is to scale web services. I know it sounds pretty pedestrian; so why is it exciting?

First, people are using web services to do some exciting things these days. The increasing adoption of Service Oriented Architecture, along with the capability of providing useful tools across the internet (think Web 2.0), mean that people are putting more and more cool applications behind web services.

But second, scaling a web service on a grid is the kind of thing that doesn't involve any re-architecture of an application. See, when many people think about the concept of grid computing, they think, "Well, that sounds great for a very special type of application, but it wouldn't work for me"--especially those who haven't seen the light of Object Oriented Programming for Grid.

However, scaling a web service doesn't mean you have to re-architect your application. There's no need to write any "parallelization" code. It just means running your web service on the grid to provide scalability. You can think of it as a CPU Load Balancer. It means that when a user hits your compute-intensive web service, it gets processed by a machine that has CPU capacity ready.

To show how easy it is to wrap a web service in some grid code and have instant scalability, I'm giving a webcast tomorrow morning (10:00AM PDT). I'll give an overview of the Digipede Network, then take a web service and grid-enable it. The whole thing will only take 30 minutes. There's only room for 15 people in the webcast; register here to secure a spot!

No parallelization required.

Technorati tags: grid computing, digipede, web services, soa, scalability

Double Your Pleasure, Double Your Fun

A

 t the risk of linking to Nick Carr once too often, I'm going to offer some comments on his latest incisive post.

Microsoft has an opportunity to integrate the two OSes in a way that can put Google at a significant disadvantage. To put it another way, it has an opportunity to manage customers' transition from the computer OS to the Web OS in a way that furthers its own interests - and damages Google's.

An excellent point, Nick. When I heard about Microsoft's increased capital expenditures, the idea of offering some easy transition from desktop to web OS hadn't really occurred to me. It's a fascinating idea, though, and one that their .NET development tools will help make even easier.

The funny thing is that I did think that perhaps Google was going to open up their server farms to outside software. Last fall, when Google announced their partnership with Sun, I posited that with Google's distributed OS (and ginormous server farm) and Sun's utility computing experience, they may be creating the world's largest grid-for-hire; I noted that one inconvenience was "you're going to have to rewrite your code to run on a different OS."

In an earlier post, I noted that "For an enterprise distributed computing system, enterprises want an OS that is deployed within their enterprise. They want an OS that they develop on all the time."

As Carr points out, Microsoft already has that advantage. They've also got 20 years of putting together world class development tools for that OS.

Carr asks the question

Google could, of course, try to counter Microsoft's advantage by offering its own computer OS - a version of Linux, for instance - but it's hard to imagine such a move succeeding. Would a critical mass of users really make the leap?

Not only that, but does Google want to release GLinux? My comments from last September:

It would surprise me greatly to find out that Google has any interest in all at making an all-purpose operating system. One of the hardest things about making any public OS is the hundreds of thousands of drivers that have to be written to handle peripherals; does Google want to get in the business of making sure every printer on earth works with their OS?

If Nick is right, and Microsoft will indeed be able to bridge "the two OSs," it will be a fascinating battle. Google may be forced to release an OS just to compete. And if that happens, sparks are really gonna fly.

Technorati tags: grid computing, utility computing, microsoft, google

Are the server wars over?

A

 pparently my comment notification system isn't working right--or I just received a comment on a two-month-old post. The misconceptions in the comments were great enough that I thought it deserved its own post.

I had been engaged in a bit of repartee with Nick Carr (one of the best in the blogosphere, IMHO, so click on an ad next time you're on his site) about whether the server market is doomed (one of the few subjects in which I disagree with Mr. Carr). My post is here ( Because 1.21 gigaflops just aren't 1.21 gigawatts).

A reader--Bert Armijo of HotCluster left the following comment:

I'm afraid you're using the exceptions to build your rule.

90% of all compute cycles are exactly the same. If you doubt that, take a quick look at the server marketshare data. The processor, memory and I/O architecture battle has been settled. So, as long as you're running typical business or web applications and not HPC, computing resources can be considered a commodity.

How to deliver that resource is decided. It's TCP/IP over the internet.

What's been missing is a standardized way to organize the usage of remote cycles to power a distributed application. This challenge has also been solved and now we get to see how the market reacts.

As I said in my follow-on comment: There's a standard processor? Is there a standard OS? Are there standard "typical business applications?"

At Mr. Armijo's suggestion, I decided to take a quick look at the server marketshare data. Although he never said it explicitly, I can only assume that he means that the world has standardized on Linux. However, according to this article on LinuxInsider, there is no clear leader in the server market.

In terms of revenue, Windows has the largest share of the market at somewhere north of $17.7B in 2005. Second place? UNIX, of course, at about $17.5B. Linux systems come in third, with about $5B in revenue in 2005--the first time Linux systems have finished that high. Linux is growing quickly, of course, experiencing 20% revenue growth.

Surprising? A lot of people would think so--although not the server manufacturers themselves. I met with a major manufacturer recently who ships UNIX, Linux, and Windows servers, and over 40% of their servers (in terms of units) ship with Windows on them.

My point here? Simply that there is no clear market leader. UNIX was the server leader for more than a decade, and it is slowly slipping out of site (the sad news about SGI this week was one more indicator). Windows has taken the lion's share of the UNIX slice of the pie, but Linux is growing very quickly (and will continue to grow).

Mr. Armijo also says that the processor battle has been settled. Has it? AMD has been enormously successful in the 64-bit arena--so much so that Intel has rethought its entire strategy for the 64-bit platform.

So to say there's a standard is to willingly put blinders on. There are multiple operating systems out there. There are multiple hardware platforms out there. And many are thriving. Many will continue to thrive.

Mr. Armijo didn't explicitly call out any standard operating system, and it's possible that I've mistaken his point. But all software doesn't run on all operating systems, and that's part of my point when I say "compute cycles aren't all alike." Software needs an OS. Those "typical business applications" that he talks about run on different operating systems, and many are dependent on a single OS.

To repeat myself: I'm a believer in utility computing. I think it will have an ever-increasing role in business. But it's not a panacea, and it's not easy. Using utility computing is still a difficult, nasty business (see CRN's review of Sun Grid).

Moreover, there are the concerns I raised in my previous post:

Not all compute problems can be "shipped out" easily. There are huge data concerns. First of all, there are the ubiquitous privacy and security issues: some data people just don't want leaving their building.

Beyond that, though, there's the issue of data size and compute-to-byte ratio. If I need to do a quick search of a huge dataset I just collected from my genome lab or my jet design windtunnel, it may not make sense to move that to a "computing powerplant." Heck, I may be collecting tons of data in real time, and I need to analyze it in real time. I need my computes where my data is. As Jim Gray says, "Put the computation near the data." If your data are with you, that means your computes are with you as well.

For the last time: I'm not arguing against utility computing. But if you think that it will apply to every computing problem, that enterprises are going to get all of their CPU cycles over the net and no one will buy servers anymore, you're just not looking at the real world.

(I also used some numbers directly from this IDC press release, which was the source for the LinuxInsider article).

Technorati tags: utility computing, linux, unix, windows

Grid: No Simple Answers? Or Simpler Than You Think?

D

  riving around Oakland this weekend I saw thousands of Mexican flags--the confluence of Cinco de Mayo the same week as the May Day immigrant protests led to a huge community celebration. I was at a gas station and chanced into a conversation with two Mexican immigrants--one legal, and one illegal. Because I have a passable knowledge of Spanish, they asked what I thought about the situation. My answer: it's a tough nut to crack. There simply are no easy answers.

When it comes to grid computing, people want easy answers, too. It's funny how many very educated people (like, PhD kind of people) want to know if we've found a way to take a single-threaded, monolithic piece of code and run it in parallel on many machines to make it go faster. The answer to that is, of course: no. We're not magicians, and anyone who can tell you they can do that has probably also spent time in a sphere full of water in New York.

So distributed computing isn't magic. That said, it also shouldn't be as hard as people have made it out to be.

There's been an acceptance in the industry that distributed computing is hard (and, to be certain, some aspects of it are very hard). But that has led most industry players to allow all of it to be hard. The prevailing attitude seems to be: "It's difficult stuff; you want to learn how to do it? Install the right flavor of the right version of the OS, learn Perl, spend weeks poring over the message boards, or hire some expensive consultants to do all of that for you."

But it doesn't have to be that hard. You don't need consultants to install most software--why should grid software be any different? Most systems don't prevail upon their users to learn scripting languages in order to take advantage of them--after all, most users aren't programmers. So why do all traditional grid systems rely on scripts?

This isn't to say that people who use those systems don't understand the concepts of ease-of-use. It's just not a feature they've been thinking about. I had a meeting last year with some PhD students who do work on the GLOBUS Toolkit. When they saw the Digipede Workbench (the tool that lets you design and run jobs without having to write scripts), they didn't snicker at a silly Windows tool. They said, "I wish we had one of those!"

That, incidentally, is the tool that the reviewers from CRN magazine used when they compared the Digipede Network to Sun Grid. Of course, their conclusion (as I wrote last Friday): "The difference in ease of use in Sun Grid vs. Digipede was enormous."

We used the same ideas when we designed our critically acclaimed development tools: distributed computing doesn't have to be hard.

So can grid computing make all of your software magically run faster? Of course not.

Does grid computing have to be very difficult to implement? I say, the answer is the same: Of course not.

There may not be simple answers to every question in life, but maybe there are some simple answers in grid after all...

Technorati tags: grid computing, digipede, sungrid, sun

John vs. Jonathan? Digipede Outshines Sun

CRN magazine ("Vital Information for VARs and Technology Integrators") released an article today on Sun Grid (Sun's CEO is Jonathan Schwartz) and the Digipede Network (Digipede's CEO is John Powers): Review: Gridlock Alert For Sun Grid?

The article's subhead says it all:

Analysis finds hidden costs in Sun platform, while competing Digipede offering shines

Now, I'm not sure that the Digipede Network is really a competing offering--after all, Sun Grid lets you rent distributed computing on Solaris boxes by the CPU hour, while the Digipede Network lets you use your own Windows machines (without any hourly charges).

The reviewers found was that Sun Grid was extremely difficult to use. "In the end, the Test Center was unable to complete the project as originally planned, even with the help of Sun engineers." Ouch.

How about doing the same job using the Digipede Network? "The difference in ease of use in Sun Grid vs. Digipede was enormous."

How so? Well, they didn't have to rewrite their application to use NetBeans--the Digipede Network is just as comfortable distributing command-line executables as it is distributing .NET objects--they just used their existing app. They used the Digipede Workbench Job Wizard to define their job ("developers need not learn how to build scripts"). And the result:

Test Center engineers completed the Digipede job in less than one hour vs. five hours for Sun Grid.

Mind you, the five hours they mention aren't the $1 per hour kind: they're talking about human effort time. Expensive time. User time. In less than an hour, they had the Digipede Network up and running their jobs: no recompiling, no relinking, no rearchitecting, and no strange scripting.

One side note: while we love the Microsoft development tools, we know not everyone uses them. In this article, they mention that they built their application using the DOS GNU compiler. We have customers using development platforms as diverse as PHP and Python, as well as many different flavors of Microsoft platforms.

Technorati tags: grid computing, digipede, sun, sungrid

Thanks, Profes...I mean, John!

I had a great meeting yesterday with John Ousterhout of Electric Cloud.

John used to be a professor of Computer Science at Berkeley, and he's one of the reasons I majored in CS. He was really the ideal professor--the kind of guy who knows every single student's name (even in a class with close to 200 people in it), who blends education with humor (I remember the first day of class, he told us that his name is pronounced "OH-stir-howt," but he responds to anything from "Oosterhoot" to "Easterhat"), and is active in cutting edge technology.

In the early 90s when I worked at Quantum Consulting under Margo Seltzer, I even used his Tcl/Tk toolkit when we ported one of our Windows products to run under Solaris.

Professor Ousterhout moved on from Berkeley to Sun as a Distinguished Engineer (and, I guess, stopped being called "Professor"). After Sun, he started Scriptics, which was later purchased by Interwoven. In short, he's had a stellar career thus far. His latest venture is Electric Cloud.

Electric Cloud is a pioneer in distributed builds--they have great technology and an impressive client list.

I really enjoyed catching up with John, and I very much appreciated the hour we spent together. He is quite an incisive guy: within minutes of sitting down, he was offering fantastic advice about Digipede. We didn't talk so much about the technology (although, as someone who has been working with distributed computing for decades, I'm sure he'd be a great technical resource), but about the business.

Clearly, he has a head for business as much as he as a head for software. I valued his opinion because he's done something extremely difficult to do: taken a development tool (a decidedly unglamorous product when compared with the AJAXy/SocialNetworky/Web2.0ish companies getting funded these days), gotten it funded, and brought it to market.

We discussed pricing strategies a fair bit. None of his ideas were brand new, but it is always great to get an outsider's perspective.

One thing they concentrate on at Electric Cloud is the ability to determine who their customers are in three questions: What platform do you develop on? How many developers do you have? How long do your builds take? The answers to those three questions determine whether a company is a prospect.

I can't boil my customers down to three questions--but I'd really like to be able to. This is a great exercise, and it's one we need to undertake at Digipede. It's not just a useful tool for a junior salesperson. It's also great for us, as a company, to be able to define our customers that way. Investors need to know who the customers will be. We need to be able to refine our marketing to hit the right people. We are sure we've got a huge potential audience for our products--unfortunately, it's amorphous and hard to define. My hour with John made it clear that we need to define it clearly, even if that means shrinking it a bit.

Of course, when our customer base is doing things as different as risk management, Monte Carlo simulation, hurricane simulation, Bayesian analysis of text messages, map generation, gathering web content, PDF generation and genetic algorithms, it's a tough nut to crack.

Technorati tags: Electric Cloud, digipede, John Ousterhout