What’s Really ‘Eating the World’?

George Rebane

Web browser pioneer Marc Andreesen writes ‘Why Software is Eating the World’ in the 20aug11 WSJ.  It is a well written compendium of the commercial sectors that have grown in recent years, and promise to really grow more because of cheap and ubiquitous computing power.  Andreesen, whom I much admire, outlines how such computing has literally transformed all those sectors – manufacturing, medicine, merchandizing, communications, finance, transportation, … - and is now creating new enterprise sectors yet undreamed.  But he makes a fundamental error in his main premise – that it is “software companies” that are ascendant in this transformation.  They are not.  Please let me explain.

Today, most information technology (IT) based businesses develop, market, maintain, and consume software.  But the software is just the medium through which the plethora of unique and powerful functions required of the business are first computed and then implemented by humans or other energy/mass moving machines.

Andreesen goes on to a long list of companies populating the business sectors that he identifies as really being “software companies”.  He could just as well have made the argument that ‘processors are eating the world’, or that ‘silicon is eating the world’.  In a former day he could have said that ‘metal machining is eating the world’, or ‘printed paper is eating the world’.  Of course, none of these have eaten the world nor will they do so in the future.

Each of these world eaters did nothing more or less than serve as the medium for products that required more than, say, machinists or printers to create and purpose them for use in a society.  The creators and developers, who saw an existing need that could be filled better by a configuration of newly bent and/or machined pieces of metal, it is they who ate the world and are still doing so.  It is they who summoned the machinists to make the stuff, and printers to print the information that changed the world in their day.

Andreesen attributes to computer programmers all the wondrous systems that today make modern life the affordable convenience that it is, and the more so what it promises to be tomorrow.  Nothing could be further from the truth.  Today computer programmers provide the important final step to implement an algorithm or to generate a functional module that fits into a more comprehensive system design, whether that be interpreting geophone signals to map out possible oil fields or managing inventory in a robotized warehouse.

Just as machinists and book binders were expert in their stage of producing something conceived and designed elsewhere, so are computer programmers.  But Andreesen does point out what I have here reiterated, and that is the importance of students learning programming and being able to program in at least one language that is popular enough to warrant its own development environment.  That is the job insurance or minimum skill set required to make one employable in uncountable fields because it subsumes the larger area of computer literacy.

So who are the real world eaters today, and those who will be here at least until the Singularity?  These are the systems engineers and scientists with broad sets of skills acquired over years of formal schooling and concurrent hands-on work – these guys and gals always have one foot in industry or entrepreneurial activity, and the other foot in academe where they are either taking courses or teaching them.  Their systems skills include the most arcane of technical fields imaginable such as applied mathematics, algorithmics, optimization, chaos, machine intelligence, pattern recognition, signal processing, control theory, estimation theory, statistics, stochastic processes, electronics, finance, … .  They must be equally at home in trans-philosophic areas such as causality and cosmology, and in the fundamental sciences of physics, chemistry, biology, nano-materials, … .  In short, the toolset of systems practitioners must be large and growing, for the problems they are asked to solve are never the same.

These people hold degrees such as in computer science, control engineering, bio-cybernetics, and even (though more rarely) in such areas as geophysics, materials science, neurobiology.  In assembling these skill areas in today’s world, such individuals are literally able to think thoughts that are inconceivable for the overwhelming majority of people now alive.  Whenever you look into a corporation or institution that is producing something leading edge that is making the world safer, cheaper, healthier, more nutritious, and entertaining than it was yesterday, you will see one or more systems people at the creative end.

In industry it is only the systems engineer who can never turn down a project by mumbling something like, ‘That’s outside my field.’  And the same goes for systems scientists in institutional settings.  Both professions are supposed to have the skills to structure any problem for solution, which also includes identifying and recruiting the more narrow technical specialists (e.g. the manufacture of high-temperature carbon nano-tubes) needed to fill the design team. 

And then somewhere down the line here we begin thinking about software programmers and software to implement the processing parts of the system in a selected computing environment.  One systems engineer may productively ‘drive’ ten computer programmers who actually program the designed algorithms that make up the ‘world eating software’.  It doesn’t work the other way around.

The lay reader here may now dredge up what appear to be counter examples, people who definitely were not systems educated, but did come up with an idea for a computer game or even the spreadsheet.  They went straight ahead and applied (or hired) their computer programming skills to implement the first software versions, attract venture capital, put a real development team in place,  hire or grow into management, and go to glory.  While commercial success is possible through such avenues, the development of a Boeing 707, laser printer, a world class search engine, or a functional-MRI is not.

Comments

[Russ Steele]

I agree, until the systems engineers do their job, the programers do not have much to work on. I will go back one step even farther, until some one comes up with the idea for a product, feature or function, the systems engineers do not have too much to do either. That said, in many cases it is a systems engineer that connects the dot and comes up with a new idea, as they have a larger world view than your typical programer, who is highly skilled at crafting computer code, but is not skilled at thinking about the larger picture.

In my experience, systems engineers can also have unique skill sets that are not fully transportable. Having developed products in both aerospace and automotive sector, trying to bring aerospace systems skills to an entrenched auto industry was a real challenge. Getting metal benders to think about adding computers, software and networks to cars and trucks was a real challenge. Especially, getting the metal benders to think in systems terms, not only in terms of the vehicle, but the whole world that the vehicles will be operating in. I once entertained writing a book about this systems engineering challenge.

[Douglas Keachie]

"These people hold degrees such as in computer science, control engineering, bio-cybernetics, and even (though more rarely) in such areas as geophysics, materials science, neurobiology."

You left out one:

"operations research" "systems engineer" 959,000 Google hits

[Douglas Keachie]

Of course, in this case, the systems engineer part of the C.V. is just a footnote early in her career.

http://www.ieor.berkeley.edu/~rrighter/

[George Rebane]

Actually DougK, I left out a bunch of 'tools' as I tried to give a flavor to what a systems professional has to know.

[Douglas Keachie]

If I were still teaching computer programming in a high school, I would love to have you come in about April May to talk about what you do, as by then the kids would be somewhat able to appreciate it.

[John S]

All well and good. But I think you guys might want to explore how the regulatory environment fits into this. You can have all of the wonderful ideas in the world and the capacity to build it, but if you can't get over the regulatory hump you're sunk before you even start. In the US, "regulations" are eating us up, and THEY dictate where we are or will be headed.

[Douglas Keachie]

Here's one solution, John S.

Require that a staff of trained USA citizen operators be at the beck and call of any business person attempting to fill out any governmental form, and that the wait time is never longer than 3 minutes. If the trained operator and the 2 supervisors above him or her, are unable to find the answers and fill out the form inside of two hours, the business person is exempt from that regulation for 1 year.

The business person must be the highest ranking staff member of the business, the owner is preferred, so this would be only applicable to small businesses.

Business person must be able to email all relevant information in a timely manner, and a video link is preferred.

[George Rebane]

Excellent point JohnS, and a good stake in the ground DougK for a remedy. But the long term solution is a review and drastic reduction of regulations starting from a zero-base approach, and requiring every standing regulation to have a built-in sunset and/or review and renewal.

My approach when serving as CEO or project manager was to always have appropriate counsel review system designs starting with the requirements specification. The added fron-end costs saved much bigger sums during development and fielding of systems. Nevertheless, it was still a component of friction in doing business.

[stevenfrisch]

Ironically, we may have at least a measure of agreement here...I support automatic sunset of regulations, based on a clear set of performance metrics and an evaluation of their effectiveness meeting their objectives.

Every project we manage starts with a 'life cycle analysis' to determine system design needs including regulatory constraints or requirements. Planning in advance for regulatory requirements almost always ensures approval at lower costs and 'no surprises' in the process.

[JohnS]

Ok, some admit it, most don't; what our governments do is create an environment through regulations whereby business can exist and hopefully thrive. Long ago much got done because there was not a laborious regulatory environment. Some think that was a very bad thing, but we got a lot of infrastructure and millions of jobs. Today (with the evolution of a much more strict regulatory environment) we find ourselves incapable of making anything happen. We have advanced, no question. There has been a lot of good things brought about by advancing regulations. But I have to ask, have we gone too far? Have we created a system that is too dificult to get through?
Certainly when there was plenty of money flowing we could do it, now I'm not sure. There isn't enough money flowing to enable us to get over the regulatory bar. Unless we see this fact, we will not be able to make anything happen. Just imagine what it would do (for jobs) if we suddenly said ok no more EIRs need to be done. Now before you let out a big scream Steve F and E, I only use that one as an example.

[Keach]

No more EIR's for the coal and gas extraction back on the East Coast. Let them frack up the water supplies to their hearts content, and take off coal weight willy nilly, with Hiroshima quantity explosives on an monthly basis. None of these events could possibly have anything to do with the earthquake.

[Larry Wirth]

Keach, for once you're correct- what's gotten into you?

[Keach]

Actually, the quake was caused by the Founding Fathers rolling over in their graves...

[Greg Goodknight]

Russ and George, you're both about 25 years behind the times regarding divisions of labor in software and systems engineering. The top down approach isn't what it used to be. I remember when Hughes Malibu was Mt. Olympus, but that was a previous time.

For most of the industry "programmers" doing menial implementations or fixes are mostly in the middle and end life cycle tasks, or just turning the crank to customize web commerce applications, and software engineers are the ones determining the algorithms and data structures needed, writing the specs and driving the development. The intelligence and expertise is much more distributed than it used to be. Top down was slow and ponderous, not to mention often divorced from reality.

[Greg Goodknight]

George, you wrote "Andreesen attributes to computer programmers all the wondrous systems that today make modern life the affordable convenience that it is, and the more so what it promises to be tomorrow."

The word "programmer" isn't in the story. It is software, computer and networking engineers that are driving the innovation, and software engineering has been driving more technology into systems engineering than the other way around.

[Douglas Keachie]

Tee Hee, another Greg-o-centric Labeling Dispute.

[Greg Goodknight]

No, but there is yet another superficial Keachie remark devoid of content.

George and Russ are correctly describing the aerospace view of software leading even into the 90's but it was already changed in the commercial workd. Indeed, even in the mid '80's it was not uncommon for software developers to not have any significant background in math, science or engineering and would use ad hoc design methods to implement what they were told to implement, but that time is long past. "Programming" is a fairly low value occupation now, and has been for years.

[Douglas Keachie]

HB1 visas have nothing to do with that, or outsourcing?

[Greg Goodknight]

"HB1" visas? LOL.

[Douglas Keachie]

"It is software, computer and networking engineers that are driving the innovation"

GregGotta have that engineering and math in there somewhere for it to amount to a hill of beans, doesn't it?

So Greg, the colleges now all concentrate on teaching these topics, and never bother to teach programming any more? And the old fuddy duddy systems analysts just don't have the proper cahones to get the job done anymore? Is that what you are saying? The departments of Engineering and Math have taken over the leadership in innovation from the departments of computer science? How interesting... How about a URL or two or three where we can check these developments out?

[Greg Goodknight]

Keach, don't use big words you don't understand.

[Douglas Keachie]

Just when did you think the combination of EE and computer science took place at Berkeley? 5 years ago, 20, 30, 40? How about the dept at Stanford, when? Now Cornell is still just plain Computer Science, but they were home to it as a matter of prominent historical fact.

The first computer at Berkeley for purposes of curriculum showed up in the basement of Campbell Hall, which was then the math and astronomy dept. It was there by 1965. I know, because I used the punch card facilities associated with it, in the trailers in the parking lot. Of course the first nuclear reactor went into EE over on the north east corner of campus, quickly followed by one in Etchevery, the Engineering building. Physics and math didn't get one, AFAICR.

[Greg Goodknight]

Keach, it's clear you don't understand what George wrote, what Andreesen wrote or what I wrote. There is not enough in your posts that is correct for it to even just be wrong.

Please, put all your straw men in one place and set them ablaze.

[Douglas Keachie]

"Etcheverry"

In the UC EECS course list for the CS side, we find the word programming used well over 50 times.

http://sis.berkeley.edu/catalog/gcc_list_crse_req?p_dept_name=Computer+Science&p_dept_cd=COMPSCI&p_path=l

[Greg Goodknight]

You'll also find the word "writing" in searches of departments of English Literature.

[Douglas Keachie]

Back to the original proposition:

"For most of the industry "programmers" doing menial implementations or fixes are mostly in the middle and end life cycle tasks, or just turning the crank to customize web commerce applications, and software engineers are the ones determining the algorithms and data structures needed, writing the specs and driving the development."

And software engineers are not functionally equivalent to systems analysts of yore?

Your "beginning of life cycle tasks," all the programming then is done by the systems engineers? Or by the programmers?

Do you have committees of software engineers working together for a better world? Unlike the "lone wolf" systems analyst?

[Douglas Keachie]

"You'll also find the word "writing" in searches of departments of English Literature."

Last time I looked Berkeley EECS wasn't in the business of producing menial laborers.

[Greg Goodknight]

You should find an "HB1" visa holder to help you out, Keach.

[Greg Goodknight]

Here's some help, Keach. You wrote:
"Last time I looked Berkeley EECS wasn't in the business of producing menial laborers."

But it was George's thesis that they can't do anything but code up what the "system engineers" tell them to code up. What you have done, by successive construction of poorly conceived straw man arguments (i.e. visits to Keachiespace) is bend what I wrote into a faithful representation of what George actually wrote. I suggest you take that up with him.

[Douglas Keachie]

Maybe it's time you should both read, Merrill Chapman's "In Search of Stupidity" and grow some humility. Here's from the forward, by a guy who's not ashamed to admit, "he's a programmer."

**********************************************************************************

If you ask me, and I’m biased, no software company can succeed
unless there’s a programmer at the helm. So far the evidence backs me
up. But many of these boneheaded mistakes come from the programmers
themselves. Netscape’s monumental decision to rewrite its browser
instead of improving the old code base cost the company several years of
Internet time, during which its market share went from around 90 percent
to about 4 percent, and this was the programmers’ idea. Of course,
the nontechnical and inexperienced management of that company had
no idea why this was a bad idea. There are still scads of programmers
who defend Netscape’s ground-up rewrite: “The old code really sucked,
Joel!” Yeah, uh-huh. Such programmers should be admired for their
love of clean code, but they shouldn’t be allowed within 100 feet of any
business decisions, because it’s obvious that clean code is more important
to them than shipping, uh, software.
So I’ll concede to Rick a bit and say that if you want to be successful
in the software business, you have to have a management team that
thoroughly understands and loves programming, but they have to
understand and love business, too. Finding a leader with strong aptitude
in both dimensions is difficult, but it’s the only way to avoid making one
of those fatal mistakes that Rick catalogs lovingly in this book. So read
the book, chuckle a bit, and if there’s a stupid head running your company,
get your resume in shape, and start looking for a house in
Redmond.
Joel Spolsky
http://www.joelsonsoftware.com
http://www.fogcreek.com

[Greg Goodknight]

Keach, I accept your forfeit.

[Douglas Keachie]

I call them HB1's because they are what bombed the Silly Valley American citizens' economy. I guess you missed that reference too. More poetic license put more spice in your writings. Try it sometime.

[Greg Goodknight]

It's H-1B, Keach.

[Douglas Keachie]

I know that, Greg, sheesh, are puns beyond your intelligence spheres?

[Douglas Keachie]

@Greg 4:09,

"Well played, Naomi Price."

[Greg Goodknight]

Keach, no one is laughing with you.

[Douglas Keachie]

And your allies are innumerable.

[Douglas Keachie]

BTW, KVIE, excellent NOVA on Fractals and Mandelbrot, happening now 8 to 9 pm, Friday evening.

[Greg Goodknight]

Keachie, ADD can be treated. Talk to your doctor.

[Douglas Keachie]

Weightwatchers has a plan for overbloated egos that might meet your needs.

[Greg Goodknight]

Keachie, not a single one of your responses was both (1) on topic, and (2) not based on a misrepresentation of my words. This thread contains, in a nutshell, why I think you're the poster child for what is wrong with the teaching profession in the US in general, and California in particular.

Argumentum ad ignorantiam is not what most of us send our kids to school to be taught.

[Douglas Keachie]

You have still failed to give a proper demonstration of current systems design that no longer follows what George outlined.

This whole string of commentary was designed to elicit from you a few examples demonstrating, and here I quote you exactly, copy and paste from above:

"and software engineers are the ones determining the algorithms and data structures needed, writing the specs and driving the development. The intelligence and expertise is much more distributed than it used to be. Top down was slow and ponderous, not to mention often divorced from reality."

Evidence?

[Brad Croul]

GeorgeR, your 22Aug11:03AM post got me wondering "what if".

What if, all regulations and laws were eliminated for, say, one year? Would people just go crazy and do all the things the laws and regulations are trying to control or prohibit?

Would contractors start using only one sack of cement, instead of five (or whatever) sacks in a concrete mix in high rise projects in earthquake prone areas? Would people just start shooting each other? Would folks get even more liquored up while dining out and rob a bank on the way home?

I'll admit I have always wanted to see how fast my car could go.

Would civilization cease to exit? Would our insurance rates go up?

[Greg Goodknight]

"Tee Hee, another Greg-o-centric Labeling Dispute" is about 60 IQ points below "This whole string of commentary was designed to elicit from you a few examples" and it took a good shaming for you to come up with the latest. However, even had you started with a rational adult focus you'd have ended up losing it because your desire is for an argument, not a discussion.

I gave as much evidence as George and Russ gave, but if third party opinions are to be highly valued, I'd suggest the WSJ piece George was criticizing and the quote from Joel Spolsky you dredged up qualify, and Andreesen managed to make billions for investors with a software, not systems, engineering focus.

[George Rebane]

BradC - in a dominant or monoculture society the lifting of all regs and laws would have a markedly smaller impact on civilization than in a multicultural society.

Re programmers vs systems engineers - what Andreesen "managed to make" with his software project is an anecdotal incident to the generalization that he proposes in his WSJ article. It is the generalization that I address in this commentary.

[Greg Goodknight]

George, your generalization would be overstating the case even for aerospace engineering circa 1990, and completely misses current reality.

If anyone waited until "systems engineering" completely specified the deliverables, there would be 10 competitors firmly entrenched in the targeted market before the prototype would be ready for testing. Software engineering is not just programming, and never was just programming.

[George Rebane]

GregG, I never stated that the systems development cycle must still proceed in distinct sequential steps, or that systems scientists/engineers don't code. In fact, since about 1970 I can't recall any systems professional who didn't write code and was not proficient in at least one development environment.

However, I do make the easily verifiable point that the overwhelming number of strictly software engineers (vs systems engineers) are woefully lacking in the cited toolsets. Therefore as such, they are not the ones that have spearheaded breakthrough systems. In the final view, a team effort must obtain, but my stated asymmetry holds.

BTW, 'complete specification' of deliverables is still done in both government and commercial systems procurement. As development engineers, we also recall that procuring customers must carefully be educated in the maxim 'Performance, cost, schedule - pick any two.' Where the development cycle is most often parallelized is in in-house developments of primarily 'me too' systems of (almost) known functionality and market application. (Not a good work environment for systems types.)

Finally, proprietary new-capability systems, the development of which is not known to the outside world, are a crapshoot with the investor(s) making the call on performance, cost, and schedule risks. Nevertheless, all three risk categories are best managed through a rigorous development process (that often includes rapid prototyping of risky modules) in which you plan your work before working your plan.

PS. None of this has anything to do with programming the next hot twitch game where, more often than not, the game designer is also the production programmer.

[Greg Goodknight]

George, the Andreesen piece was not about "programming the next hot twitch game", your "strictly software engineers" don't exist, and "systems engineering" isn't what has been driving software based engines into key elements of global commerce.

[George Rebane]

GregG, I guess that we'll have to leave our diametrically opposite views of what's happening out there for the readers to sort out.

I am still very heavily involved in entrepreneurial systems development with a large software component, and I have yet to see one of these production software engineers (cum systems engineer). Since we will soon be adding to staff, please have some of these marvels send me their resumes.

BTW, not sure where you deduced my understanding of Andreesen's article being about programming twitch games. Maybe that's an indicator of our disconnect.

[Greg Goodknight]

George, I have no idea what you mean by a "production software engineer". You do seem to have a caricature in mind that is variously that, a "strictly software engineer" and a "computer programmer", i.e. a very narrow occupation unable to see the world beyond the last line of code.

Show me the website describing the jobs and I'll see who I can send your way.

[George Rebane]

GregG, actually the problem is worse than that. My "caricatures" include a definite delineation between even 'production computer programmer' (who indeed are "unable to see the world beyond the last line of code", and thankfully so), and 'software engineer', let alone piling on 'systems engineer' and 'systems scientist'. And all of these categories have their own distinct professional literature. In the old days each of these fields had a limited 'knowledge domain' so that quite often overlap was possible. But things have changed markedly in the last thirty years. To claim a competitive and productive foot in more than one of these today would require mastering a prodigious curriculum that expands and obsoletes at least yearly. And a few actually achieve that, but it is not the norm.

In any case, I appreciate your offer to help and will stay in touch.

[Greg Goodknight]

George, it seems possible that only a systems scientist can hope to truly understand how valuable a systems scientist can be, and how hapless any development can be expected to be without an accomplished systems scientist to tell everyone what to do, but I doubt it.

Now, as always, any given curriculum only provides a few tools. The "software companies" of which Andreesen speaks are really are "eating the world", and they are doing so because of the competitive advantages disruptive technologies. Andreesen had it right.