Finding Wisdom — Ralph Ellison


“I am an invisible man. No, I am not a spook like those who haunted Edgar Allen Poe; nor am I one of your Hollywood-movie extoplasms. I am a man of substance, of flesh and bone, fiber and liquids—and I might even be said to possess a mind. I am invisible, understand, simply because people refuse to see me…” — the nameless protagonist in Ellison’s novel Invisible Man

The Time Magazine essayist Roger Rosenblatt said that “Ralph Ellison taught me what it is to be an American”” and upon reading the book for the first time in my twenties as a young Navy officer I came to the same conclusion for myself.  From that first paragraph with its initial line that grabs you by the collar, the story’s narrator takes you for the ride of your life, opening your eyes to those things hiding in plain sight, revealing uncomfortable truths that the cowardly and dull among our fellow citizens refused–and continue to refuse–to see.

Ralph Waldo Ellison was born in Oklahoma in 1914 and it is through his experience in a pioneer state that had no history of slavery–a large part of what had been known as Indian Territory just eight years earlier–where though he grew up the “poorest among the poor,” he was given access to interact with white people and attend a good school; opportunities not even open to African Americans in the northern states of the time.  It is through his experiences in this western part of the American Midwest that he learned to see the interplay and interconnections of white and black culture, though strictures still existed.  He father sold ice and coal but died in an accident when Ellison was a child.  His raising was left to his mother, Ida, who was an activist and was arrested several times for violating segregation laws.

Young Ellison was a talented young man and saw many mentors–both black and white–during his developing years.  Among these was Ludwig Hebestreit, the conductor of the Oklahoma City Orchestra, who saw great promise in the young musician.  Ellison was thus accepted to Tuskegee Institute in Alabama on an Oklahoma state scholarship to study music.  Wishing to play jazz trumpet, he faced opposition by the more conservative-minded faculty who judged the music base and reflecting poorly on the “race.”  At this point stories diverge.  What is clear is that Ellison traveled to New York either to find summer employment with the intent of returning to Tuskegee, or to pursue a different career in the visual arts (photography) or sculpture.  In either case the artifacts from these interests show a man of multiple and considerable talents.

While in New York he came across Langston Hughes, Richard Wright, and other influential members of the “Harlem Renaissance” and switched his energies to writing for the Federal Writers’ Project of the Works Progress Administration (WPA).  There he worked in the Living Lore Unit of the project where he gathered materials of–and was influenced by–black folklore and culture.  Until he joined the Navy during the Second World War he contributed essays and stories to various publications, eventually becoming editor of The Negro Quarterly.  Unpublished stories from this period prior to the publication of Invisible Man, posthumously published in 1996 under the title Flying Home and Other Stories show the development of a unique and powerful voice about to enter American letters, .

Invisible Man is a fully modern novel and Ellison’s influences–Hemingway, W.E.B. DuBois, T.S. Elliott, Joyce, Richard Wright, and Cervantes–are apparent both in his ability to incorporate their literary devices and to transcend them.  His ability to move the novel far beyond its time and methods is what makes the work as readable and understandable today, over 62 years since its publication.  At the heart of Invisible Man is the desire of the individual to overcome not only the strictures that society in its various incarnations has imposed and wishes to impose on on him, but also his struggle to overcome his own base desires and limitations.  Much has been made of this last point, some literary critics going so far as to have Ellison hearken back to the American Transcendentalists.  But I find this contention too simplistic and–frankly–ridiculous.  This judgment does the work much disservice and ignores its modernism.

Ellison himself called his work “a novel about innocence and human error, a struggle through illusion to reality.”  The protagonist writes his story from an underground room that is illuminated by 1,369 light bulbs, the power for which are stolen from the Monopolated Light & Power Company.  He recounts his misadventures, from growing up in the south with a talent for public speaking, which is used by his white benefactors for their own amusement, forcing him to agree to fight in a “battle royal” in the ring of blindfolded black men.  Nonetheless, he secures a scholarship to attend Tuskegee.  While there he helps to make ends meet by working as a driver for one of the college’s white benefactors.  While driving in the country the benefactor becomes transfixed by the intimation of incest in one of the local black families.  The narrator soon finds himself in trouble with the college for “encouraging” the white man’s mistaken impression of black culture and is expelled.  He is told, however, that the college will write letters of recommendation for the young man to the benefactors of the college in New York City.  When he arrives there he finds that rather than recommendations, the letters describe the young man as unreliable and untrustworthy.  The son of one of the benefactors, feeling the man’s injustice, helps him secure a low paying job at Liberty Paints where their claim to fame is “optic white.”  He works for the senior mixer who makes the paint and is also black.  Suspecting, however, that the narrator is engaged in union activities the older man accosts him and the two men fight as the mixer containing the paint explodes.  The narrator, awaking in the company hospital, finds himself unable to speak and that he has lost his memory.  The hospital uses the opportunity of the appearance of an anonymous black patient to conduct experimental shock experiments on him.  Soon he regains his memory and leaves the hospital, albeit in poor condition from his mistreatment.  He collapses on the street and is taken in by a kindly black woman in Harlem by the name of Mary.  There he is nurtured back to health and black Harlem society.  While walking down the street he witnesses an eviction of an old black couple and speak eloquently in public in their defense.  This talent for speaking is noticed by the Brotherhood, an integrated organization to help the politically and socially oppressed.  He is recruited by them and given a new place to live and new clothes.  He is trained in rhetoric by the Brotherhood and used by them to advance their causes until he is accused of advancing his own fame at the expense of the organization, which causes him to be censured.  He is reassigned to the woman’s rights cause where he is seduced by a white woman who fantasizes about being raped by a black man.  The narrator’s best friend in the Brotherhood, Tod Clinton, another black man, leaves the organization, as do many other black members who feel the organization is using them as tools.  Increasingly Harlem is being influenced by Ras the Exhorter, who is a black nationalist and separatist who feels that the narrator and other blacks are betraying their best interests.  Soon the narrator sees his best friend, Tod Clinton, on a sidewalk in Harlem selling “Black Sambo” dolls.  Police stop Clinton for a license and when he attempts to flee he is shot and killed on the street.  The narrator organizes a funeral for his friend and speaks out in his defense against the police.  Despite this show of community solidarity his actions fail to serve the interests of any of the powers in Harlem.  He now finds himself isolated, pursued both by the now largely white Brotherhood, who consider his actions selfish and self-serving, and by Ras and his separatist followers, who consider him to be a traitor to his race.  The racial tension caused by the funeral and continued police brutality breaks out in a race riot.  The narrator finds himself pursued on the street by the police, who believe that he is a looter.  He falls down an open manhole and the police close it up on him.  From that point he vows to remain invisible to society and to live underground.

By turns tragic, horrifying, and hilarious, Invisible Man is a modern picaresque novel in the tradition of Don Quixote, told in prose by an exponent of the jazz form.  The narrator leads us along the path of the hero and, though African American, transcends his race to reveal his humanity in all of its fragile forms–bravery, selflessness, foolishness, stupidity, naivete, kindness, solipsism, lust, hope, and fear.  In the end he is bathed in light, though existing under the surface of the world.  As a result, he is anything but a character seeking transcendental enlightenment, which is illusion.  He is, instead, a character who has found the ability to see things as they are, including those uncomfortable truths about himself.

Ellison and his protagonist are fully modern in their views.  Ellison’s character is led down blind allies both through his own guilelessness, and the sometimes misguided and other times malicious intent of others.  Rather than a victim, in the tradition of the writing of Richard Wright, Ellison’s character overcomes the vicissitudes imposed on him by accepting what he is and what he can be.  We see that white society and black society in America are engaged in a dance, sometimes violent and sometimes in opposition, oftentimes spawning fear, that inevitably draws them closer together.  In this way the story is not so different from the struggle of others, each wave of immigrants and other traditionally disenfranchised groups working against the limitations placed on them by the powerful.  Each is rejected, abused, and manipulated.  In the end, though, each strives toward the ideal of freedom, not just for themselves, but for everyone.  To do that requires the clear eye of critical thinking and the ability to live life in reality, bathed in the unforgiving clarity of light.


Saturday Music Interlude — Ruthie Foster singing the blues

As a relatively young nation (still) the United States has few forms of music that it can claim as its own.  American folk, bluegrass, and country have their roots in Scots-Celtic and British folk forms of expression.  Many of the songs currently performed today even reprise traditional themes and melodies, but graft onto them American concerns and limitations for a rich fusion of the traditional and modern.

Two forms of music, however, that are uniquely American are blues and jazz, which eventually gave rise to Rhythm & Blues and early Rock & Roll.  The blues are the folk music of a people enslaved, given the hope of freedom, enslaved for all intents again, and–as the country has progressed–achieving full citizenship and freedom in law if not fully in practice.

Jazz, of course, which is based on the blues, is America’s classical music.  Despite attempts to straight-jacket it, as European classical music has been straight-jacketed–where variation from an accepted form based on the tastes of a privileged economic elite is the rule–jazz continues to develop and improvise.  This is to be somewhat expected.

The various forms of European classical music was financed and supported by royalty and robber barons–and continues to be financed by an economic elite which tends to expect uniformity.  The music, while among the greatest forms of human musical expression, has had over the years been allowed only so much freedom within the established boundaries of approval by a ruling class.  The genius found within it is to hear the rebellion under the surface, borrowing from folk forms where it can be masked from disapproving ears.  The subversive music from Mozart’s Barber of Seville, among others, comes to mind.

Jazz, however, is based on a democratic ideal–that the players working together, each given improvisational freedom within a structure, will create something new–a synthesis of old and new that drives the music forward.  Segregation allowed African Americans to freely express themselves and to do so in ways that ran under the surface of society.  The brilliance of the musical expression was soon realized and the mainstream of American society adopted many of its forms of expression and the lifestyle that often accompanied jazz and blues life.

Thus the core belief in both jazz and blues is progression–driving things forward, to a better day; not as individuals who work against each other and who strive against the success of the other–which would undermine and destroy the composition and the music–but together.  Only then can the music succeed.  Thus, while jazz is the music that speaks of the ideal of democratic society, blues speaks the story of the individual in society which can be cruel and unforgiving without love, compassion, decency, forgiveness, and more than a little bit of luck.

Ruthie Foster is an effective purveyor of the blues.  She started singing in her church choir and, leaving her rural town, continued to perform while on active duty in the U.S. Navy Band.  Since leaving the Navy she has taken the blues community by storm, winning multiple awards since her first release in 1997.

One can hear her background in her songwriting and singing.  On her newest album, Promise of a Brand New Day, the song “Let Me Know” contains the familiar call-and-response structure, though a chorus never enters into the song, the instruments providing an effective substitute for the anticipation of the response to the powerful instrument of her voice.  Here she is singing some selections from her new album.

My Generation — Baby Boom Economics, Demographics, and Technological Stagnation

“You promised me Mars colonies, instead I got Facebook.” — MIT Technology Review cover over photo of Buzz Aldrin

“As a boy I was promised flying cars, instead I got 140 characters.”  — attributed to Marc Maron and others

I have been in a series of meetings over the last couple of weeks with colleagues describing the state of the technology industry and the markets it serves.  What seems to be a generally held view is that both the industry and the markets for software and technology are experiencing a hardening of the arteries and a resistance to change not seen since the first waves of digitization in the 1980s.

It is not as if this observation has not been noted by others.  Tyler Cowen at George Mason University noted the trend of technological stagnation in the eBook The Great Stagnation: How America Ate All the Low-Hanging Fruit of Modern History, Got Sick, and Will(Eventually) Feel BetterCowen’s thesis is not only to point out that innovation has slowed since the late 19th century, but that it has slowed a lot, where we have been slow to exploit “low-hanging fruit.”  I have to say that I am not entirely convinced by some of the data, which is anything but reliable in demonstrating causation in the long term trends.  Still, his observations of technological stagnation seem to be on the mark.  His concern, of course, is also directed to technology’s affect on employment, pointing out that, while making some individuals very rich, the effect of recent technological innovation doesn’t result in much employment

Cowen published his work in 2011, when the country was still in the early grip of the slow recovery from the Great Recession, and many seized on Cowen’s thesis as an opportunity for excuse-mongering and looking for deeper causes than the most obvious ones: government shutdowns, wage freezes, reductions in government R&D that is essential to private sector risk handling, and an austerian fiscal policy (with sequestration) in the face of weak demand created by the loss of $8 trillion in housing wealth that translated into a consumption gap of $1.2 trillion in 2014 dollars

Among the excuses that were manufactured is the meme that is still making the rounds about jobs mismatch due to a skills gap.  But, as economist Dean Baker has pointed out again and again, basic economics dictates that the scarcity of a skill manifests itself in higher wages and salaries–a reality not supported by the data for any major job categories.  Unemployment stood at 4.4 percent in May 2007 prior to the Great Recession.  The previous low between recession and expansion was the 3.9 percent rate in December 2000, yet we are to believe that suddenly in the 4 years since the start of one of the largest bubble crashes and resulting economic and financial crisis, that people no longer have the skills need to be employed (or suddenly are more lazy or shiftless).  The data do not cohere.

In my own industry and specialty there are niches for skills that are hard to come by and these people are paid handsomely, but the pressure among government contracting officers across the board has been to drive salaries down–a general trend seen across the country and pushed by a small economic elite and therein, I think lies the answer more than some long-term trend tying patents to “innovation.”  The effect of this downward push is to deny the federal government–the people’s government–from being able to access the high skills personnel needed to make it both more effective and responsive.  Combined with austerity policies there is a race to the bottom in terms of both skills and compensation.

What we are viewing, I think, that is behind our current technological stagnation is a reaction to the hits in housing wealth, in real wealth and savings, in employment, and in the downward pressure on compensation.  Absent active government fiscal policy as the backstop of last resort, there are no other places to make up for $1.2 trillion in lost consumption.  Combine this with the excesses of the patent and IP systems that create monopolies and stifle competition, particularly under the Copyright Term Extension Act and the recent Leahy-Smith America Invents Act.  Both of these acts have combined to undermine the position of small inventors and companies, encouraging the need for large budgets to anticipate patent and IP infringement litigation, and raising the barriers to entry for new technological improvements.

No doubt exacerbating this condition is the Baby Boom.  Since university economists don’t seem to mind horning in on my specialty (as noted in a recent post commenting on the unreliability of data mining by econometrics),  I don’t mind commenting on theirs–and what has always surprised me is how Baby Boom Economics never seems to play a role in understanding trends, nor as predictors of future developments in macroeconomic modeling.  Wages and salaries, even given Cowen’s low-hanging fruit, have not kept pace with productivity gains (which probably explains a lot of wealth concentration) since the late 1970s–a time that coincides with the Baby Boomers entering the workforce in droves.  A large part of this condition has been a direct consequence of government policies–through so-called ‘free trade” agreements–that have exposed U.S. workers in industrial and mid-level jobs to international competition from low-paying economies.

The Baby Boom, given an underperforming economy, saw not only their wages and salaries lag, but also saw their wealth and savings disappear with the Great Recession, when corporate mergers and acquisitions weren’t stealing their negotiated defined benefit plans, which they received in lieu of increases in compensation.  This has created a large contingent of surplus labor.  The size of the long-term unemployed, though falling, is still large compared to historical averages, is indicative of this condition.

With attempts to privatize Social Security and Medicare, workers now find themselves squeezed and under a great deal of economic anxiety.  On the ground I see this anxiety even at the senior executive level.  The workforce is increasingly getting older as people hang on for a few more years, perpetuating older ways of doing things. Even when there is a changeover, oftentimes the substitute manager did not receive the amount of mentoring and professional development expected in more functional times.  In both cases people are risk-averse, feeling that there is less room for error than there was in the past.

This does not an innovative economic environment make.

People who I had known as risk takers in their earlier years now favor the status quo and a quiet glide path to a secure post-employment life.  Politics and voting behavior also follows this culture of lowered expectations, which further perpetuates the race to the bottom.  In high tech this condition favors the perpetuation of older technologies, at least until economics dictates a change.

But it is in this last observation that there is hope for an answer, which does confirm that this is but a temporary condition.  For under the radar there are economies upon economies in computing power and the ability to handle larger amounts of data with exponential improvements in handling complexity.  Collaboration of small inventors and companies in developing synergy between compatible technologies can overcome the tyranny of the large monopolies, though the costs and risks are high.

As the established technologies continue to support the status quo–and postpone needed overhauls of code mostly written 10 to 20 years ago (which is equivalent to 20 to 40 software generations) their task, despite the immense amount of talent and money, is comparable to a Great Leap Forward–and those of you who are historically literate know how those efforts turned out.  Some will survive but there will be monumental–and surprising–falls from grace.

Thus the technology industry in many of its more sedentary niches are due for a great deal of disruption.  The key for small entrepreneurial companies and thought leaders is to be there before the tipping point.  But keep working the politics too.

Saturday Music Interlude — Phox performing “Slow Motion” and “Blue and White”

Phox is a sextet doing alternative folk and indie music ostensibly out of Madison, Wisconsin after returning to their home state after various individual career failures and wrong turns.  According to their own copy, they originated as friends in high school from the hamlet of Baraboo, WI, “a place where kids often drink poisoned groundwater and become endowed mutants.”  They hit the musical circuit last year and became overnight successes after having hit the woodshed for two or so years prior.  They caught everyone’s attention at SXSW, the iTunes Festival, and Lollapalooza.  Their debut self-titled album was released June 24th on Partisan Records.  Here is a video of them while they were in the beginning of their rise:


Why Can’t We Be Friends — The Causes of War

Paul Krugman published an interesting opinion piece in the Sunday New York Times entitled “Why We Fight Wars” in which he attempts to understand why developed, relatively affluent countries still decide to wage war, despite the questionable economic costs.  Many websites seconded his observations, particularly those that view social systems and people as primarily rational economic beings.  I think the problem with Mr. Krugman’s opinion–and there is no doubt that he is a brilliant economist and observer of our present state of affairs with a Nobel to his name no less–is that he doesn’t fully comprehend that while the economic balance sheet may argue against warfare, there are other societal issues that lead a nation to war.

Warfare, its causes, and the manner to conduct it was part of my profession for most of my life.  My education was dedicated not only to my academic development but also to its utility in understanding the nature of civilization’s second oldest profession–and how to make what we do in waging war–at the tactical, operational, strategic level–that much more effective.  In the advanced countries we attempt to “civilize” warfare, though were it to be waged in its total state today, it would constitute post-industrial, technological mass murder and violence on a scale never seen before.  This knowledge, which is even recognized by peripheral “Third World” nations and paramilitary organizations, actually make such a scenario both unthinkable and unlikely.  It is most likely this knowledge that keeps Russian ambitions limited to insurgents, proxies, Fifth Columnists, and rigged referendums in its current war of conquest against Ukraine.

Within the civilized view of war, it begins with Clausewitz’s famous dictum: “War is the attainment of political ends through violent means.”  Viewing war as such we have established laws in its conduct.  The use of certain weapons–chemical and biological agents for instance–are considered illegal and their use a war crime; a prohibition honored throughout World War II, Korea, Vietnam, and most other major conflicts.  Combatants are to identify themselves and, when they surrender, are to be accorded humane treatment–a rule that has held up pretty effectively with notable exceptions recorded by Showa Japan, North Korea, and North Vietnam and–tragically and recently–by the United States in its conduct in the War on Terror.  War is not to be purposely waged on non-combatants and collective punishment as reprisals for resistance are prohibited.  There are also others that apply, such as Red Cross and medical persons being protected from attack.  In the U.S. military, the conduct of personnel at war are also restricted by the rules of engagement.  But in all cases the general law of warfare dictates that only the necessary amount of force to achieve the desired political ends is to be exercised–the concept of proportionality applied to a bloody business.

Such political ends typically reflect a society’s perception of its threats, needs, and grievances.  Japan’s perception that the United States and Western Europe was denying it resources and needed its own colonial possessions is often cited as the cause of its expansion and militarism under Showa rule.  Germany’s economic dislocations and humiliation under the Allies is often blamed for the rise of Hitler, rabid nationalism, and expansionism.  In both cases it seemed that at the societal level both nations possessed the characteristics on the eve of war that is typically seen in psychotic individuals.  Other times these characteristics seemed to behave like a disease, infecting other societies and countries in proximity with what can only be described as sociopathic memes–a type of mass hysteria.  How else to explain the scores of individuals with upraised hands in fealty to obviously cruel and inhumane political movements across the mess of human history–or the systematic collection and mass murder of Jews, Gypsies, Intellectuals, and other “undesirables”: not just in Germany but wherever the influence of this meme spread across Europe, Africa, and Asia?

Nations can also fool themselves in learning the wrong lessons from history.  Our own self-image of righting the wrongs of the Old World go back to our anti-colonial roots and the perceptions of our immigrant ancestors who were either rejected by or rejected that world.  Along these lines, the example of Munich in the 20th century has been much misused as a pretext for wars that have ended disastrously or created disastrous blowback resulting from the fog of war simply because the individuals assessing the strategic situation told themselves convenient stories gleaned from an inapplicable past and ignored the reality on the ground.  We have seen this in my lifetime in Vietnam, Iraq (twice), and Afghanistan.

For all of the attempts to “civilize” warfare and lend it rationality, there comes a time when its successful prosecution requires the rejection of rationality.  This is why soldiers and combatant personnel use euphemisms to dehumanize their enemy: it is easier to obliterate a person who is no longer seen as human.  Correspondingly the public is inflamed, the press becomes a tool of the war party, and dissent is viewed with suspicion and penalized.  This is why warfare cannot be interpreted as an extension of neo-classical or–any–economics.  There are no rational actors; at least, not as it is presently conducted by modern nation-states no matter their level of economic development or the maturity of their political systems.  War is unhinged–part of the savagery found in all of us from our primate pasts.

One of my most effective professors when I was seeking my Masters in Military Arts and Sciences was the brilliant historian Dr. Roger J. Spiller–a protégé’ of T. Harry Williams.  “We are always learning,” he would say in repeating a familiar refrain in the military profession, “the lessons from the last war.”  For students at the Army Command and General Staff College it was the critique that doctrine (and therefore the organization and construction of the force) was based on the 1967 Arab-Israeli War; probably the only analogue that could be used in Iraq and–unfortunately for Russia–if they decide to turn their armor on Ukraine or any Article V NATO countries.

Aside from these few exceptions, however, the American way of total warfare that we learned first in our own Civil War and then perfected on the battlefields of Europe and Asia–and our success in its use–has rendered it largely obsolete.  It has been obsolete for quite some time because warfare has changed; its practitioners have evolved.  It has changed because its present incarnation is being prosecuted by people and groups that have no significant power and so use violence to destroy power.  This is the purpose of the terrorist.  Even the strength of this new form of warfare–Low Intensity Conflict–is transient–evident only in tactical situations.  What it cannot do is establish legitimacy or power.  Thus, meeting violence with violence only exacerbates the situation in these cases because power is further eroded and–along with it–legitimacy.  We see the results of the vacuum caused by this inability to adjust to the new warfare in the political instability in both Iraq and Afghanistan–and the rise of ISIS.

While I would argue that the use of economic balance sheets are not what we need in assessing the needs to ensure global stability and peace, we do require a new theory of war that infuses greater rationality into the equation.  Clausewitz–and his intellectual successor Antoine-Henri Jomini–in looking at the guerilla warfare in Spain against French rule, simply admonishes war’s practitioners not to go there.  It is not until T. E. Lawrence and Mao Zedong that we have a modern theory to address this new, societal form of “revolutionary” warfare and then only from the perspective of the revolutionary that wishes to establish neo-colonial, authoritarian, or totalitarian regimes.

Thus, we possess the old templates and they no longer work.  With the specter of nuclear weapons still held over the head of humanity we can ill afford to view every situation as a nail, needing a hammer.  We must, I think, follow the lead as advocated by Hannah Arendt, who distinguished the differences between power, strength, force, violence, and authority.  There is, as John Dewey observed, a connection in consequences between means and ends.  The modern form of violence through terrorism or paramilitary revolution has all too often, in the last quarter of the 20th century and the first decades of the 21st century, led to new oppression and totalitarianism.  This has probably been inevitable given the indiscriminate brutality of the struggles.  Diplomacy backed by credible power and sufficient military force to counteract such violence is the new necessary face of achieving stability.  Contrary to the assertions of neo-cons at the time, the very thing we needed in the wake of 9-11 was an effective police action in lieu of chaotic regional warfare.

Interestingly, the insight between means and ends in warfare was perceived early by George Washington when he won his struggle over the conduct of the war against the methods of Nathaniel Greene.  Greene’s irregular methods of warfare were designed to win the war but to unmake a nation, while Washington’s method–the existence of the disciplined continental army as the strategic center of the revolution–was designed to make a nation once the war was over.  Unfortunately for the French and the Russians, there was no George Washington to see this important distinction in their own revolutions.

So too in the 21st century is this connection between means and ends in the handling of conflict–and terrorism–important.  The years since the fall of the Soviet Union seem to have turned the clock back to 1914 for the pressures and conflicts that were held in check by a bi-polar world: the Balkans, the Middle East, Eastern Europe all have been engulfed in conflict.  The tragedy that can result given the new technologies and approaches for inflicting violence and chaos on civilization require that we not apply 1914 methods in meeting them.

Sixteen Tons — Data Mining, Big Data, and the Asymmetry of variables and observations

Last Thursday I came upon what I can only interpret as an ironic comment at Mark Thoma’s Economist’s View blog entitled “Data Mining Can be a Productive Activity.”  I went to the link and it went to a VOX article by Castle and Hendry entitled “Data Mining with more variables than observations.”  All I could think after the opening line:  “A typical Oxford University econometrics exam question might take the form: ‘Data mining is bad, so mining with more candidate variables than observations must be pernicious. Discuss.'” was: are these people serious?

Data mining is a general term in high tech and not a specific approach to finding patterns and trends in large elements of data.  The authors–and I’m guessing that they are not alone in the econometrics profession–seem to be addressing a “Just Say No” approach to performing what for most of us who deal in statistical analysis and modeling of large datasets do every day, largely based on the fact that it involves these scary things called computers that run this mysterious thing behind the scenes called “code.”  Who knows what horrors may await us as we mistakenly draw causations from correlations by anything more than the use of Access or Excel spreadsheets?  It seems that Oxford dons need to get out more.

The use of microeconomic data mining has been in general use for quite some time in many businesses and business disciplines with a great deal of confidence and success (too much success in the medical insurance, financial services, and social networking fields to raise legal and ethical objections).  So the assertion that seems to be based on those of a single group of econometricians does seem to be odd.  In the end it seems to be a setup for a proprietary set of calculations placed within an Excel spreadsheet given the name “Autometrics.”  This largely argues for the proper approach to the organization of data rather than a criticism of data mining in general.

The discriminators among data mining and data mining-like technologies involve purpose, cost, ease of use, scalability, and sustainability.  New technologies are arising every year that allow for increased speed, more efficient grouping, and compression to allow organizations to handle what previously was thought to be “big data.”  Thus the concept of data mining and big data is a shifting one as our technologies drive toward greater capability in integrating and interpreting large datasets.  The authors cite the techniques of taking large data to prove anthropomorphic global warming as one of the success stories of large scale modeling based on large data.  Implicit in acknowledging this is that not every variable needs to be included in a model–only the relevant variables that drive and explain the results being produced.  There is no doubt that reification of statistical results is a common fallacy, but people had been doing that long before the development of silicon-based artificial intelligence.

There is no doubt that someday we will reach the limit of computational capabilities.  But for someone who lived through the nonexistent “crisis” of limited memory in the early ’90s followed not too after by the bogus Y2K “bug,” I am not quite ready to throw in the towel on the ability of data mining and modeling to effectively provide the tools for the more general discipline of econometrics.  We are only beginning to crack heuristic models in approaching big data and on the cusp of strong AI.


Monday Night Music Interlude — Shannon McNally on “One on One”

A lot of blogging to catch up on as I return from yet another extended trip

The classic country lilt is somewhat misleading for this Long Island native.  According to the site Allmusic, she was greatly influenced by folk-blues from her parents’ record collection.  I first heard about (and heard) her at her electric live performance at the 2007 New Orleans Jazzfest, where her rendition of “Sweet Forgiveness” (which can be heard on the critically acclaimed album North American Ghost Music) set the crowd on fire.  Please enjoy.