The Big (D) — Ways of Looking at Big Data

Recently I have been involved in several efforts regarding what is often referred to as Big Data, but of a particular kind.  Oftentimes the term, which was first defined by Doug Laney now at Gartner, is seen as utilizing data in order to monetize consumer information that is being collected to allow business to focus advertising, marketing, and product development.  More generally, however, big data (as defined by Laney) distinguishes itself from normal relational database management by its volume, variety, velocity, variability, and complexity.  The Wikipedia definition is slightly different with the additional attribute of veracity.

Recently in a blog post at AITS.org I talked about the role of normalization of data which prompted some discussion via e-mail.  I have identified under the rubric of an Integrated Digital Environment (IDE) the need to find the lexicon of big data based on its purpose in order to solve the issues inherent in it, but in particular of variability and complexity.  The questions raised were whether the IDE concept specified the solutions thereby limiting its sustainability as a concept.  It was certainly not my intention to specify the method of normalization and integration.  I proposed what current technology allows and left open other methods.  I am keen to hear any ideas on alternatives that will push the technology further.  But my concepts of how best to approach the issue of Big Data certainly is influenced by the data I encounter.

For example, the world I inhabit is focused on project management.  Traditional “tools” have focused on very small pieces of the overall data being collected in order to derive importance regarding performance, progress, and effectiveness.  The ability to normalize the same ‘kind’ of data coming from different proprietary solutions improves the ability of the organization to optimize the use of data and turn it into information.  Breaking down proprietary barriers reduces the need to make investments in multiple information solutions, along with the overhead associated with them.  But in order to achieve this ‘agnosticism’ some approaches are more effective than others.

Information when it comes to big data is usually of two types: information derived from the patterns from the various types of data, and the information content inherent in the various types of data.

The first does not necessarily require normalization since it simply requires access to data using standard interfaces designed to scale.  The latter, which is the data I most often deal with, can also be accessed by interfaces, but then the value of the information must be derived either during or just after retrieval.  That is, the interface must be made ‘smart’ or the raw data, once delivered, must be properly identified and processed in order to derive its significance.  This is easy when technology solutions play in their own sandbox and can easily interpret their specialized terminology.  Not so good for the agnostic, non-proprietary ideal.  The second method prescribes a neutral schema that allows for all proprietary terminology to be eschewed in favor of clarity so that we can easily equate apples with apples.  Some processing must be made during retrieval, but not to the extent of direct access of data.

In the real world, however, this is not an either-or proposition.  Data can be very discrete, even when it is part of a “big data” environment.  For example, time-phased systems that are used to manage projects use different terminologies for the same normalized concept represented by the data.  Oftentimes direct access to some of these elements, because of the complexity of the algorithms that derive data values, is impossible.  In those cases, then, a schema resolves any variation in properly representing the interrelationships between the elements after retrieval and integration, especially to ensure accurate representation of time-phasing.  Other data is more ‘flat’ and, by itself, can inform and append other more complex elements of data without requiring extensive processing.  In those cases standard interfaces that are scaled to the task at hand will work just fine.

Thus, we must utilize different methodologies based on the characteristics of the data. One size does not fit all.  Instead, a flexible solution that utilizes all of the methods at hand: schemas, normalization of data after retrieval, use of direct access to tables (such as OLE DB), programming interfaces (APIs), and calling web services provides what we need in approaching big data.  Some of these are application neutral.  Some utilize the inherent proprietary interfaces in the data itself to take it outside of its sandbox.  Combined with a flexible user interface, the new paradigm is moving away from best-of-breed or stovepiped “tools” that perform only one primary function in interpreting a restricted set of information.

Let’s Put An Axe to the Axis — Pearl Harbor Post-Mortems

Holiday, work, travel, and a recent bug have conspired to keep my blogging output relatively low.  Thus, some of my friends and colleagues were surprised when December 7th passed without some mention of that event from me.

There is a large cottage industry on the Web touting conspiracy theories about Roosevelt who, it is said, either provoked or engineered the Japanese attack in order to find a pretext to enter the war against the Axis powers.  As a student of history I have always been fascinated by this particular conspiracy theory because of its obvious inconsistencies and non-sequiturs.

By December 1941 Nazi Germany had proven itself the epitome of totalitarian brutality, barbarism, and nationalism run amok.  The same could be said for the behavior of pre-defeat Showa Japan, especially in China and Korea.  The expansionist warring of both Germany and Japan not only posed an existential threat to the ideals of freedom and self-government, but to civilization itself; to any concept of civilization based on law as opposed to predation.  They were also existential threats to the United States–or, at least, a United States based on constitutional government and democratic processes.  President Roosevelt made it no secret that he viewed the world in exactly this way.  His actions were provocative to all of the Axis governments.  Short of capitulation this was both the right and only reasonable position to take.  Fascism and Showa ideology were hostile to American interests and institutions, and those of its allies.

A sizable portion of the U.S. electorate did not wish to get involved in the war.  In this way Roosevelt was ahead of the American people.  The Selective Service Act was extended that October by a single vote in the U.S. House of Representatives.  But Roosevelt did everything he could short of war to support China and the United Kingdom, as well as Russia once Hitler double-crossed Stalin on their non-aggression pact, while at the same time preparing the United States for war.  These actions were taken publicly and garnered a lot of attention and criticism from isolationists and many prominent individuals–especially on the right–who outwardly supported Germany and fascism, at the time.

Many on what is considered the American right–though it often appears in the literature of the New Left–seem to adhere most to the theory that Roosevelt provoked or engineered the attack to sway the American people to support the war.  Thus, in this thinking, the American Servicemen who gave their lives at Pearl Harbor were martyrs to Roosevelt’s intention.  This ignores so much history that one has to live in an alternative universe or imagine a conspiracy so wide as to include individuals of differing walks of life and political allegiances over several decades.  It also ignores the actions of Japan.

By the summer of 1941 the United States and the Axis powers were on a collision course.  Japan’s war of aggression continued to expand as they sought to form puppet governments under their economic and political suzerainty in order to create what they called a Greater East Asia Co-Prosperity Sphere.  Under this regime, the self-described Japanese Asian master race would reign over all other Pacific peoples.  In the way of this ambition stood the United Kingdom and its Commonwealth, France, China, Russia, and the United States.

In particular, the United States considered the Pacific Ocean an “American lake.”  Governments across Asia considered (and still consider) the United States Navy the honest broker in ensuring that international waters and trade went unmolested.  Hawaii, Guam, and the Philippines were United States territory.  (The first, of course, is now a state and Guam is still a possession of the United States.  The Philippines was given its independence after the war).  The United States, especially due to the sizable U.S. west coast population of Chinese descent, had developed a close relationship with China, especially supporting the efforts of Sun Yat-sen and his (somewhat problematic) protégés in their efforts to create a Chinese constitutional republic.

By November 1941 it appeared to the opposing sides in the Pacific that war was inevitable.  The question was: when?  When Japan invaded French Indochina, the United States froze all Japanese assets in the United States.  Scrap metal, as well as oil and gasoline products were also embargoed, and the Panama Canal was closed to Japanese shipping.  These economic actions had a direct adverse impact on the Japanese economy and its ability to sustain the war.  Instead of seeking peace and pulling back from its imperial ambitions, the Japanese doubled-down and adjusted their strategy to take out the ability of the United States to undermine their ambitions and to seize these resources by expanding their zone of control.

Thus stood the two sides when Japanese emissaries came to the United States to negotiate an alternative to war.  What the Japanese did not know was that the U.S. was monitoring its cryptographic traffic and had broken its highest codes under the codename MAGIC.  What the United States did not know, because the MAGIC communications did not provide such information, was the location of a portion of the Japanese fleet.  The most likely strike was expected in the Philippines, but Pearl Harbor, Guam, Wake, Midway and the Aleutians were also identified as possible targets.

The United States’ possessions and interests in the Pacific were quite extensive.  Much has been made of the decision to forward deploy the fleet to Pearl Harbor, but given the distances involved it made much more sense to place them in Hawaii, where they could be called up when needed, while at the same time being out of harm’s way near much of the action in Southeast Asia.  One of the worst things a country can do in war and diplomacy is underestimate one’s opponent.  It was felt that the Japanese could not project power and any long-range strike, even if carried out, could not be sustained.  But forward deploying the fleet, even with the losses on December 7th, proved prescient.  Despite the loss of life and materiel, most of the ships sustaining damage–including most of the battleships–were repaired or refloated, and entered into combat.

Anyone who has served in the sea services in the Pacific can understand the scale of that ocean.  It contains just over half of all of the earth’s oceanic water.  It extends 9600 miles north to south and over 12,300 miles from east to west.  Even when I served in the U.S. Navy back in the 1980s and 1990s I engaged in operations, now declassified, in which we were able to evade submarine, electronic, and satellite detection by assets of the old Soviet Union and pop up in unexpected places–allowing detection (in order to send an unequivocal message) when we chose.  It is not inconceivable, given the technology of 1941, that the Japanese fleet would have traveled within striking distance of Pearl Harbor without detection, and without need for some elaborate and tone-deaf conspiracy theory.

There were eight investigations to determine the root cause of the failure to determine why the Japanese attack was not anticipated where it occurred.  Many of the documents and findings from these investigations remained classified until the 1980s.  Generally the investigations, especially the U.S. Navy Court of Inquiry that was held from July to October 1944, found that Admiral Harold Stark, the Chief of Naval Operations at the time of the Pearl Harbor attack, bore the brunt of the blame.  Later more politicized investigations held by Congress found a number of scapegoats, mainly Admiral Husband Kimmel and Army General Walter Short, the area commanders at the time of the attack.

The following are excerpts from the summary of the investigation conducted by Admiral Kent Hewitt in from 14 May to 11 July 1945.  The exhibits referred to in the text can be found on-line here.  Captain Layton in the text refers to the future Admiral, Edwin T. Layton, who was Kimmel’s intelligence officer.  The Pacific theater commanders had been warned of impending war on November 27, 1941.  The inquiry focuses on the speculation of where the possible attack would occur and the countermeasures that were undertaken at the time.

Admiral McMorris stated that he did not know whether Washington kept CincPac fully informed but, he said, the information which was received was taken at its face value (page 899).

The “war warning,” it will be recalled, had been sent to Admiral Kimmel on November 27th. On November 28th, the Army dispatch had been repeated to Admiral Kimmel, advising, among other things, that Japanese future action was unpredictable but that hostile action was possible at any moment. Also on November 28th, there had been sent to Admiral Kimmel a copy of CinCAF’s dispatch advising of the “winds code” to be used if diplomatic relations were on the verge of being severed. And, on the same day, he had been in communication with OPNAV concerning his plan for the reinforcement of Midway and Wake, and, in that connection there had been mention made of the shortage of antiaircraft guns.

On November 30th, Admiral Kimmel sent a dispatch (Exhibit 77) urgently recommending the shipment of 37 mm. anti‑aircraft guns and ammunition for familiarization and training.

On November 30, 1941, OPNAV sent a dispatch to CincAF for action and to CinCPac for information (Exhibit 76), which advised in part:

“Indications that Japan about to attack points on KRA by overseas expedition X. Desire you cover by air the line Manila Camranh Bay on three days commenc­ing upon receipt of this dispatch X.

A second similar dispatch was also sent on the same day (Exhibit 77) requesting a daily report from CincAF, even if there were no contacts and the information were all negative….Also on November 30th Admiral McMorris prepared, at the direction of Admiral Kimmel, a memorandum setting forth the steps which he recommended to be taken in the event of American‑Japanese war within the ensuing twenty‑four hours (Exhibit 69A). This was revised on December 5th and set forth the steps to be taken in the event of war within forty‑eight hours (Exhibit 69B).

Vice Admiral McMorris testified that during the first week of December, 1941, he, and he was sure Admiral Kimmel, had in mind constantly the “war warning,” the fact that the Japanese forces were, according to Intelligence, on the move, the fact that the Japanese were destroying codes, and that the Japanese in the past had attacked without declaration of war (p. 328).

He stated further that during that time he was also considering the tasks set forth in Phase IA of the Pacific Fleet Plan, and that daily or on alternate days he furnished Admiral Kimmel with an informal memorandum as to the action that should be taken by important elements of the Fleet if war were initiated within twenty‑four hours. He stated that typical of such memoranda were Naval Court of Inquiry Exhibits 69A and 69B, which were dated 30 November and 5 December 1941, respectively (p. 328‑329).

With reference to the Phase IA task of maintaining air patrols against enemy forces in the approaches to Oahu and the fact that no provision was made for carrying out that task, Vice Admiral McMorris testified that submarines were considered to be the greatest element of danger. He said that anti‑submarine patrols had been placed in effect (p. 339).

At about this time, it will be recalled, Admiral Kimmel also received information concerning the estimated position of the Japanese Fleet. As Captain Layton expressed it:

Captain Layton testified that in accordance with the request of Admiral Kim­mel, he prepared a memorandum for the Admiral, dealing with the location of the Japanese Fleet. This was prepared, according to Layton, on the evening of December 1st and was submitted by him to Admiral Kimmel on 2 December 1941. The original memorandum bears certain notations in red pencil which, Layton testified, were inserted by him on December 2nd prior to submission of the memorandum to Admiral Kimmel and which reflected the later information received after Preparation of the memorandum on the night of December 1st‑2nd. It also bears certain lead pencil notations which Layton identified as the hand­writing of Admiral Kimmel. This memorandum, according to Layton, sum­marized his best estimate of the location of the Japanese Fleet, based on all information available to him and to Admiral Kimmel up to and including 1 December 1941.

Layton’s estimate stated that from the best available information, units of the Orange (Japanese) fleet were “thought” to be located as listed in the memorandum. In the Kure‑Sasebo area he listed the Commander in Chief of the Combined Fleet and Commander in Chief, First Fleet, with six battleships, “(f)” and other units. He listed the Commander in Chief, Third Fleet, at Nagara initially and then corrected it in red to indicate that it was at Takao. Also in the Kure‑Sasebo area he located Cruiser Division 8.

In the Shanghai area, Layton’s estimate located the Commander in Chief, China Fleet, the Shanghai Base Force, and an air group.

In the Bako‑Takao area Layton listed Third Fleet submarine squadrons and various destroyers and the commander of the Combined Air Force with numerous air groups and the KASUGA MARU (thought to be a converted carrier with 36 planes). He estimated that the Commander in Chief, Second Fleet, had been en route to Takao (this he corrected in red pencil to indicate that he was at Takao) with a cruiser division, destroyers, and with “Cardiv 4‑two CV and four DD; Cardiv 3‑two CV and 3 DD; Batdiv 3 less HARUNA‑3 BB (maybe 2 BB) and, he added in red pencil, certain cruisers and Destroyer Division 2.

In the Hainan‑Canton area, Layton located the Commander in Chief of the South China Fleet and various cruisers and destroyers and transports. In the French Indo‑China Area, he located the Commander in Chief of an Expedi­tionary Fleet with various ships, including 21 transports and some base forces, among others. In the Mandates area, he located at Palao an air group and base force; at Truk, the Commander in Chief of the Fourth Fleet with cruisers and destroyers, and a base force and an air group At Saipan he located the Com­mander in Chief of the Submarine Force with possibly submarines and various air groups and a base force. In the Marshalls area, he located various air groups and the carrier “KORYU” plus plane guards,” and several submarine squadrons and base force (Hew. Ex. 23).

Layton’s memorandum did not make any reference to the location of Carrier Di­visions 1 and 2 of the Japanese Fleet (which in fact were en route to attack Pearl Harbor). According to Layton, on 2 December 1941, during his conference with Admiral Kimmel, the Admiral noticed and commented on the absence of informa­tion concerning Japanese Carrier Divisions 1 and 3. In his testimony, he described the conversation on this point as follows:

“Mr. SONNETT: Will you state the substance of what he said and what you said, as best you recall it?

“Captain LAYTON: As best I recall it, Admiral Kimmel said, ‘What! You don’t know where Carrier Division 1 and Carrier Division 3 are?’ and I replied, ‘No, sir, I do not. I think they are in home waters, but I do not know where they are. The rest of those units, 1 feel pretty confident of their location.’ Then Admiral Kimmel looked at me, as sometimes he would, with somewhat a stern countenance and yet partially with a twinkle in his eye and said, ‘Do you mean to say that they could be rounding Diamond Head and you wouldn’t know it?’ or words to that effect. My reply was that, ‘I hope they would be sighted before now,’ or words to that effect . . . . (P. 213‑213)

“Mr. SONNETT: Your testimony, Captain, was not quite clear to me, arising out of your description of Admiral Kimmel’s twinkle in his eye when he spoke. What I am trying to get at is this: Was the discussion about the absence of information concerning Cardivs 1 and 3 a serious jocular one?

“Captain LAYTON: His question was absolutely serious, but when he said, ‘Where are Cardivs 1 and ,3?’ and I said, ‘I do not know precisely, but if I must estimate, I would say that they are probably in the Kure area since we haven’t heard from them in a long time and they may be refitting as they finished operations only a month and a half ago,’ and it was then when he, with a twinkle in his eye, said, ‘Do you mean to say they could be rounding Diamond Head? or words to that effect. In other words, he was impressing me on my complete ignorance as to their exact location.

“Mr. SONNETT: He was conscious, therefore, of your lack of information about those carriers?

“Captain LAYTON. This incident has been impressed on my mind. I do not say that I quote him exactly, but I do know that he made such a statement to me in the way to point out to me that I should know where they are but hadn’t so indicated their location” (P. 255‑266).

On December 1, 1941, he submitted to Admiral Kimmel, on request, his esti­mate of the locations of all major units of the Japanese Navy (page 913). After this was typed, more recent information caused it to be changed, in red; it showed available in the Empire—4 aircraft carriers, 6 battleships, with a question mark after them, 4 heavy cruisers, with a question mark after them, and 12 destroyers—available for use in the home area. This was a portion of the entire Japanese Navy, the majority of which was shown as disposed to the south and implicated in the impending moves, from their sources of information. The witness referred to his translation of a book (“a novel published in Tokyo to inflame public opinion toward larger armament money”—page 911), which stated that it would be very dangerous for Japan to launch a surprise attack on Pearl Harbor using carriers, battleships, and cruisers; with Japan staking its existence on the move to the south it could not afford to gamble its defenses by sending some of this force on a raid which would denude the Empire of vessels. That was generally his reasoning (page 913);

Captain Layton further testified that there were delivered to him, for presenta­tion to Admiral Kimmel, daily communication intelligence summaries, during the period of time subsequent to the sending and receipt of the dispatches of 26 November. (Page 192). Captain Layton identified such summaries for the period 14 October to 14 December 1941, (Exhibit 22), and pointed out that the initials appearing in the lower right hand corner of these documents reading HEK, appeared on the original and were the initials of Admiral Kimmel (Page 193‑194)….

during this period, as previously noted, included the fortnightly summary of current national situations, prepared by ONI and issued on December 1, 1941 (Exhibit 57). In addition to the military and naval information furnished, this stated, concerning the Japanese diplomatic situation, that unless the Japanese requested a continuance of the conversations, the Japanese‑American negotia­tions would have virtually broken down; the Japanese government and press were proclaiming loudly that the nation must carry on resolutely the work of building the greater East Asia sphere; the press was also criticising Thailand severely; strong indications pointed to an early Japanese advance against Thailand; and, relations between Japan and Russia remained strained.

On December 2, 1941, Admiral Kimmel wrote to Admiral Stark (Exhibit 50) and advised that Admiral Halsey’s advance in the ENTERPRISE to Wake had been covered by two patrol squadrons operating from Johnston, Midway, and Wake, and that upon completion of the movement, Admiral Kimmel planned to return one squadron to Pearl Harbor and leave the other at Midway and awaiting developments. The letter discussed the difficulty of supply and defense of the out­lying islands. The letter also stated that consideration was being given to the dispatches concerning the use of Army personnel in outlying islands; and, that Admiral Kimmel had frequently called Admiral Stark’s attention to the inade­quacy of Army anti‑aircraft defense in the Pearl Harbor area, with particular reference to the shortage of anti‑aircraft guns. So far, he said, very little had been done to improve this situation. It was pointed out that because of the Army’s lack of equipment, Admiral Kimmel was unable to understand the dis­patches directing that the Army be utilized in the defense of the outlying islands.

A postscript to this letter stated, “You will note that I have issued orders to the Pacific Fleet to depth bomb all submarine contacts in the Oahu operating area.” It will be recalled that Admiral Stark testified that he took no exception to this (page 153)….

On December 3, 1941, OPNAV sent two dispatches advising of Japanese instructions to destroy codes as follows:

1.  A dispatch from OPNAV to CincAF, ComSIXTEEN for action, and to CinCPac and ComFOURTEEN for information which advised that Tokyo ordered London, Hongkong, Singapore, and Manila to destroy Purple machine and the Batavia machine already had been sent to Tokyo; Washington also had been directed to destroy the Purple and all but one copy of other systems, and all secret documents; also, that the British Admiralty had reported that the Embassy at London had complied (Exhibit 66).

Captain Safford referred to OpNav secret dispatch 031855 which he said he prepared on December 3, and to a similar dispatch released by Admiral Wilkinson. He then said:

“Before drafting my message, I called Commander McCollum on the telephone and asked him, “Are you people in Naval Intelligence doing anything to get a warning out to the Pacific Fleet?’ McCollum emphasized both “we’s’. McCollum replied, ‘We are doing everything we can to get the news out to the Fleet.’ In sending this information, I was overstepping the bounds as established by approved war plans and joint agreement between Naval Communications and Naval Intelligence, but I did it because I thought McCollum had been unable to get his message released. OpNav 031855 was addressed to CinCAF and Com 16 for action, but was routed to CinCPac and Com 14 for information. It was written in highly technical language and only one officer present at Pearl Harbor, the late Lieutenant H. M. Coleman, U. S. N., on CinCPac’s Staff, could have explained its significance.” (p.359‑360)

Captain Safford said that the unit in the Fourteenth Naval District did not have any material from which they could have gained this information through their own efforts. (p. 360)

2.  A dispatch from OPNAV to CincAF, CincPac, ComFOURTEEN, ComSIXTEEN, for action:

“Highly reliable information has been received that categoric and urgent instructions were sent yesterday to Japanese diplomatic and consular posts at Hong Kong X Singapore X Batavia X Manila X Washington and London to destroy most of their codes and ciphers at once and to burn all other important and confidential and secret documents” (Exhibit 20)

Admiral Pye said that he saw the December 3 dispatch concerning the destruction of codes and discussed it with the Commander in Chief, that it seemed perfectly evident that such action could precede war by many days and it did not indicate immediate action. They were unaware of the source of this information and as it had appeared in the newspapers it probably did not mean as much as it would have meant had they known the source. Admiral Pye felt that at Oahu they were pretty much operating in the dark so far as the international situation was concerned. (p. 157)

Exhibit 11 is the December 3 dispatch by CNO to CinCAF, CinCPac and others.

Concerning the code destruction messages, Admiral Turner said that it was impossible for him to understand how anyone could believe that because of the messages the war was coming in the Western Pacific and would not involve Oahu. He pointed out that both Washington and Manila had been included in the dispatch.

Admiral McMorris said that the December 3rd dispatch concerning codes was the best indication, in his opinion, that the United States would be involved in war with Japan. He did not recall Admiral Kimmel’s view. (p. 247)

Admiral Anderson said that he had not precise knowledge but he thought that there had been a proper dissemination of information among the officers of the higher command. He saw the Commander in Chief at least weekly. Admiral Anderson did not think that he had seen nor had he been told about the October 16 dispatch. (p. 392). Admiral Anderson could not recall the November 24 dispatch or the November 30 dispatch to CinCAF with copy to CinCPac for information. He did remember having seen the war warning and believed that he had seen the December 3 dispatch.

As a matter of interest, it may be noted that a sentence appearing at the end of this copy of the dispatch, which was released by T. S. Wilkinson, was stricken out in pencil. That sentence read: “From foregoing infer that Orange plans early action in Southeast Asia.”

On December 4 1941, OPNAV sent a dispatch (Exhibit 21) to NavStaGuam for action, and to CincAF; CincPac; ComFOURTEEN, and ComSIXTEEN for information stating:

“Guam destroy all secret and confidential publications and other classified mat matter except that essential for current purposes and special intelligence retaining minimum cryptographic channels necessary for essential communications with CincAF, CincPac, ComFOURTEEN, ComSIXTEEN, and OpNav X Be prepared to destroy instantly in event of emergency all classified matter you retain X Report crypto channels retained”

On December 6, 1941, a dispatch (Exhibit 22) bearing time‑date stamp 061743 was sent by OPNAV to CinCPac for action, and to CincAF for information. This stated:

“In view of the international situation and the exposed position of our outlying Pacific islands, you may authorize the destruction by them of secret and confidential documents now or under later conditions of greater emergency X Means of communication to support our current operations and special intelligence should of course be maintained until the last moment”

Commander Kramer drafted Exhibit 66, the December 3rd dispatch concerning Japanese destruction of the “purple” machine. This was sent on the “Kopek” Channel, which was a channel for technical traffic between the Navy Department; Pearl Harbor and the Asiatic Station (page 971). It indicated, he said, a break in diplomatic relations. Exhibit 20, sent out by OPNAV on the same day as Exhibit 66, was the interpretive dispatch of Exhibit 66 (page 960).

Admiral Noyes said that Exhibit 21 (dispatch to Guam of December 4th, directing the destruction of codes) was prepared by him and motivated by the growing feeling that war in the Pacific was imminent (page 1031). It was released by Admiral Ingersoll.

Admiral Noyes said that Exhibit 22 (authorizing destruction of codes on outlying islands, dated December 6, 1941) was prepared by him and treated as priority dispatch, despite the lack of priority shown on its face (pages 1040, 1042).

Admiral Stark testified that the code destruction message was sent to Guam because he felt that Guam was in the most danger; he did not similarly advise ComFOURTEEN because he did not think Hawaii was in as much danger as was Guam. (page 69). At this time, Admiral Stark testified, he believed that war was imminent. Of particular significance he felt, was the information relating the destruction of codes (pages 165‑6). We also knew that the Japanese consuls were advising the evacuation of Japanese nationals from Malay, the Philippines, Hawaii, the United States, etc. (page 157). Admiral Ingersoll stated that the dispatch concerning Japanese destruction of codes strengthened the “war warning” (page 835).

Admiral Turner said that on Friday, December 5, 1941, there was a discussion between Admiral Stark, Admiral Ingersoll and himself concerning the general situation, and they all felt that all necessary orders had been issued to all echelons of command preparatory to war and that nothing further was necessary (page 1006). They did send some other messages about destruction of codes, both Japanese and our own (page 1007).

Admiral Kimmel testified that he regarded the dispatches concerning Japanese destruction of codes as indicating that the Japanese were going to take steps to prevent the seizure of their codes upon the breaking off of diplomatic negotiations, and regarded the dispatch directing Guam to destroy classified matter as a general precautionary measure (page 327). He “presumed” that he received the December 6th dispatch prior to the attack (page 327).

Admiral Smith testified that he saw the dispatches relating to the destruction of codes but that this meant little to him as CincPac was prepared to destroy codes (page 533). At this time, he said, war was inevitable (page 534).

About December 3rd, Admiral Smith said, after receipt of the dispatch of that date, and information from the Asiatic Fleet to the effect that heavy Japanese movements were on the way to the Southward, he believed that the Japanese were going to attack Malay Peninsula and possibly the Philippines; he thinks that the reaction of others at Admiral Kimmel’s headquarters was the same.

Admiral Pye testified that he saw Exhibit 20 (Japanese destroying codes) on December 4th, but that this information was published in the newspapers (page 427). Admiral Pye testified that he had not seen Exhibits 21 and 22 (Page 428).

Commander Rochefort, who was in charge of combat intelligence of the Four­teenth Naval District, stated that during this period the Japanese Consulate was burning or destroying various papers (page 474).

Captain Layton said that the messages concerning Japanese code destruction meant to him only that the Japanese, were destroying a cipher machine; he knew “purple” designated it as a diplomatic code (pages 904‑5) and that the “purple” cipher was A high class cipher (page 908).

On December 5, 1941, he said, they received word from the Naval Observer at Wellington that the Japanese were destroying codes. This was given to CincPac and was considered along with other information received at that time (page 906). At that time, they received messages from the British and from Washington Stat­ing that highly secret and reliable information indicated a Japanese attack on the Kra Peninsula; this seemed to dovetail with the other information which they had (page 906).

There was a discussion concerning the significance of the code destruction mes­sages; they seemed to indicate to Layton that Japan was preparing for all even­tualities. He presumed that when it was discussed by Admiral Kimmel with the War Plans Officer and others, it was a matter of discussion (page 906).

Admiral McMorris said that he felt that Exhibit 20 (Japanese destroying codes) indicated strongly that there would be war with Japan. He did not recall whether this opinion was prevalent on CincPac’s staff, but believed it was thought by CincPac that war was then extremely possible (page 895).

It may be noted that although Admiral Kimmel stated that he had kept Gen­eral Short informed, he did not personally direct that General Short be given the dispatches concerning codes destruction (Exhibits 20, 21, 22) and did not know whether they were given to him (page 327).

The only action that Admiral Bloch recalled as a result of the December 3 dispatch concerning the destruction of codes was the security measures already prescribed, the additional inshore patrol in Honolulu, a warning that was given to the Destroyer captains, and his belief the Army was on a full alert. (p. 18)

Admiral Bloch did not believe that the Army had been informed of these warning (code) messages. The messages, he said, ‘were secret and they had been admonished to keep them secret to prevent alarming people, and one thing and another, Admiral Bloch’s War Plans Officer would know about them. Admiral Bloch said that since General Short and he saw one another very frequently and Admiral Kimmel and he saw one another practically every day, it is hard to believe that anything of importance could take place or that anything could be received of even small importance that was not discussed because it is only fair to assume that they discussed everything. (p 18)

General Short, he said, had an Army Colonel as a liaison officer in Admiral Bloch’s office and Admiral Bloch had in the General’s office a Lieutenant of the Naval Reserve as a liaison officer and these men were supposed to be kept informed. Admiral Bloch’s liaison officer did not know of the dispatches that had been received because Admiral Bloch did not consider that it was proper to tell him; he was quite inexperienced. Nor was Admiral Bloch sure that the Army’s liaison officer knew of the messages, but he said they were in close touch through these liaison officers and felt that they knew what was going on. He said that it was indicated that there were some things which they were not correctly informed about. (p. 18)

Neither General Short nor his Chief of Staff recalled having seen these messages prior to the attack (pages 255, 486).”

As with all information, intelligence is only as good the context and significance that can be discerned from it.  Once processed it can then inform decision-makers of the actions that must be taken.  In 1985, after most of the MAGIC and other documents had been declassified, including many of those from the various investigations, Admiral Layton broke his silence in the book And I Was There: Pearl Harbor and Midway–Breaking the Secrets.  In the book Layton, breaking his silence of over 40 years, detailed the infighting and professional jealousy between the Washington and fleet commands within the Navy which caused important intelligence to be withheld and misrouted that not only contributed to the lack of readiness on December 7th at Pearl Harbor, but had almost led to a similar disaster at Midway had better minds not prevailed.

The title of the book was derived from a heated exchange between Layton and Admiral R. K. Turner that almost led to fisticuffs.  According to Layton, who had served the war as the fleet intelligence officer for Admiral Chester Nimitz, the day of the Japanese surrender Turner had entered the wardroom of the USS Missouri and was jubilant that another inquiry had placed the blame for the size of the disaster at Pearl Harbor on its area commander, Admiral Kimmel.  According to Layton:

“Turner continued to hold forth.  Time and again he said, ‘Kimmel was given all that information and didn’t do anything about it.’  I sat there stunned. I knew that what he was saying was not only untrue, but a monstrous slur on my former commander in chief….Turner repeated his rant: “They should hang him higher than a kite!”  And I, boiling with indignation, had to correct him.  ‘Admiral,’ I said, ‘I’m sorry.  But Kimmel did not have that information.  You say that he did.  I know that he did not, and I was there.”

Turner had good reason to celebrate the placing of blame elsewhere since, as the head of war plans for the Navy, it was his job to keep the fleets informed of the latest intelligence.  According to Layton it is not Kimmel or any subordinates in the fleets that deserved blame.  As CNO Admiral Stark certainly was responsible for ensuring that his fleet was informed.  In his own role as part of the triumvirate of Naval Operations, Admiral Turner, who monopolized intelligence for his own purposes, bore as much responsibility as the CNO for the disaster.  Layton wrote that had he had the same intelligence that had been selectively routed elsewhere within the Navy, that he would have had sufficient warning to provide to Kimmel to minimize, and perhaps thwart, the Japanese attack.  He also wrote that, having learned the lessons from Pearl Harbor, the commanders and officers in the fleet learned to quickly adjust to prevent a similar disaster from occurring at Midway, but just barely.

It must be said that Turner, though loathed by many men and officers around him for his temper and alleged bouts with the bottle, distinguished himself during the war in charge of amphibious operations.  Reality rarely makes for black-and-white characters.  Admiral Stark took the initial blame, as he must have.  Admiral Turner dodged a bullet.

But Pearl Harbor was an extraordinary event.  No one possesses clairvoyance.  The audacity, bravery, preparation, and dedication of the Japanese fleet was as much responsible for the effectiveness of the attack as any failure on the part of U.S. Navy intelligence or its commanders.  The standard for placing blame under a court of inquiry or JAG investigation is to determine whether gross negligence or incompetence was a contributing factor.  Looking back in history through the various reports, most of what I see is the fog of war.  That and organizational confusion regarding the use and distribution of intelligence, and internecine rivalry.  The secrecy of MAGIC was of utmost concern to Naval Operations in Washington.  Excluding Kimmel due to rivalries among senior command certainly contributed to the tragedy.

Even a man as otherwise rational as Layton finds regret in not encouraging Admiral Kimmel to read the translation of a Japanese book written in 1933 that plays out the scenario of an attack on Pearl Harbor.  Had Kimmel found the time to do so, would he have acted differently on December 7th without the intelligence that Turner possessed?  I doubt it.  But there lies the basis of valid criticism that also reveals the defect in Kimmel’s defense (and I started out doing the research for this post wanting to absolve him).

While it is true that Kimmel believed that the missing carriers were reported as heading to Malaysia or the Philippines, thousands of miles away, that was only supposition and he expressed that to Layton.  Knowing war was imminent what would a circumspect commander do?  Kimmel’s own defense was that he relied on Navy Operations to share all essential intelligence derived from MAGIC.  But this was 1941 where international and naval communications were relatively slow.  He was in Pearl Harbor and MAGIC was in Washington, where raw data went through a series of decryptions and processing before it became intelligence that then had to be properly interpreted and routed.  His argument assumes a level of inerrancy in intelligence sharing unknown in military history before or since, especially considering the pace of events from November 27th, when he received the war memo, to December 7th.

I would say that Kimmel should have remained on a readiness footing that Sunday.  He should have continued to fly missions to the north as identified by the Orange war games that identified that position as the most likely place from which the Japanese would launch an attack, even if he thought his aircraft levels were too low to be completely effective.  (Forces are always deficient.  Not using what one has is inexcusable).  And he should have continued to keep a portion of the surface fleet underway along with his carriers.  Then and only then could he argue that he had done everything that could be possibly done absent the latest intelligence.  The overall impact of the attack would certainly have been lessened and it is possible that the Japanese fleet could have been directly engaged.  Instead, he abrogated his responsibility by relying on Washington to give him sufficient advance warning–4,828 miles away.  That is at least simple, and perhaps, gross negligence.  While not a proximate cause, it is a contributing cause to the severity of the attack and the inability for his forces to engage the enemy.

But Navy Operations also bears a great deal of responsibility.  Kimmel was left on his own, and neither Stark nor Turner displayed any urgency regarding the readiness of their two main areas of operation: Pearl or the Philippines.  Given the information they had, if they could not share information due to the sensitivity of MAGIC, then they should have taken it upon themselves to order measures without revealing the particular reason for it.  The post-World War attempt by some officers and their political allies to scapegoat others, including General Marshall and the President, ring hollow when one sees that the Navy had all of the information it needed prior to December 7 to at least defend from, if not counter, an attack.

To the U. S. Navy Pearl Harbor is a cautionary tale.  Since that time Command at any level has been imbued with a high degree of accountability and responsibility.  As a result, a great deal of emphasis has been placed on readiness at all times–even during peacetime.  During the war the Navy developed new methods of leadership and reformed itself to be more inclusive in its officer corps.  For all that, the Navy still experiences many of the internal conflicts and rivalries of those days.  It’s inability to balance the need for security and control while taking advantage of new information technologies is its biggest challenge today.  In that way the lessons of Pearl Harbor still need to be learned.

No Bucks, No Buck Rogers — Project Work Authorizations, Change Control, and Cash Flow

As I’ve written here most recently, the most significant proposal coming out of the Integrated Program Management Conference (IPMC) this year was the comprehensive manner of integrating all essential elements of a project, presented by Glen Alleman et al.  In their presentation, Alleman, Coonce, and Price, present a process flow (which, in my estimation, should be mirrored in data and information flow) in which program artifacts were imbued with measures of effectiveness, measures of performance, and measures of progress, to achieve an organic integration of all parts of the project that allow the project team to make a valid assessment of achievement against the plan, informed by risk and opportunity.  (Emphasis my own).  The three-legged stool of cost, schedule, and technical performance are thereby integrated properly at the appropriate level of the project structure, and done in such a way as to overcome the rigidity and fallacy of the single point estimate.

But, as is always the case with elegant models, while they replicate a sufficient portion of reality to allow us to make our assessments using statistical methods, there are other elements that we have purposely left out because our present models do not incorporate them into the normal and normative process.  They are considered situational, and so lie just outside of the process flow, though they insert themselves when necessary–and much more frequently than desired.  I am referring to the availability of money and resources, and the manner in which they affect the project: through work authorizations (WADs) and baseline change requests (BCRs).

I have seen situations where fully 90% of the effort in project management is devoted to manage and adjust the plan based on baseline changes.  This is particularly the case where estimates are poorly developed due to the excuse of uncertainty.  Of course there is uncertainty–that’s the purpose of developing a plan.  The issue isn’t the presence of risk (and opportunity) but that our risks are educated ones, that is, informed by familiarity with similar efforts, engineering assessment, core competency, and other empirical factors.  This is where the most radical elements of the Agile Cult gets it wrong–in focusing on risk and assuming that the only way to realize opportunity is to forgo the empirical process.  This is not only a misreading of risk and opportunity assessment in project management, it is a sort of neo-Luddite position regarding scientific management.

The environment in which a project operates undergoes change.  The framing assumptions of the project determine the expectations of scope, cost, and what defines success.  The concept of framing assumptions was fully developed in a RAND study that I covered in a previous blog post.  Most often, but not always, the change in framing assumptions is reflected in the WAD and BCR process, most often in the latter.  Thus, we have a means of determining and taking account of changes in framing assumptions.  This is in the normal process of project management, as opposed to the more obvious examples of a complete replan or over target baseline (OTB).

So where do we track WADs and BCRs in our processes that will provide us sufficient indicators in our measures of effectiveness, performance, and progress that our resources (both size and type) many not be sufficient or that these changes are sufficient enough that our framing assumptions have changed?  I would argue that the linkage for resources must also be made through the Integrated Master Plan (IMP) and reflect in the IMS, cross-referenced to the PMB.  Technology can provide the remainder of the ability to integrate these elements and provide the process flow necessary to provide early warning.  This integration goes beyond the traditional focus on cost and schedule (and the newly reintroduced emphasis on technical achievement).  It involves integration with resource management systems (personnel, skillset assignments, etc.) as well as financial management systems to determine the availability of money (both its sufficiency and “color”*) being applied to the right place at the right time.

Integrating these elements together then allows for more sophisticated methods of determining project success through the introduction of metrics that provide correlations between the elements.  It also answers, absent politics, the optimum level of both analysis and reporting.

*The “color” of money applies mostly to public investments in which monies appropriated are designed by their purpose:  operations, maintenance, engineering, R&D, etc.

Note: This post was modified to add a point of clarification in applying WADs and BCRs to the PMB.

What did I miss over the holiday? — Dave Gordon at AITS.org

Great post by Dave Gordon and a valid point for anyone undertaking development:  determine what “done” looks like.  Understand that people and systems are not perfect, that version 1.0 doesn’t need to do everything.  Here is Dave for the rest:

“Perfectionists are sadomasochists. They are masochists, because they rarely reach perfection, and can’t maintain it for more than an instant when they do. So they are continually frustrated, in addition to being obsessed. And they are sadists, because they drive everyone around them to pursue the same silly goals that they obsess over….” Read more.

Over at AITS.org — The Need for an Integrated Digital Environment Strategy in Project Management

To be an effective project manager, one must possess a number of skills in order to successfully guide the project to completion. This includes having a working knowledge of the information coming from multiple sources and the ability to make sense of that information in a cohesive manner. This is so that, when brought together, it provides an accurate picture of where the project has been, where it is in its present state, and what actions must be taken to keep it (or bring it back) on track….Read More

Holiday Music Interlude — Trigger Hippy performing “Who Will Wear The Crown”

Trigger Hippy is the roots supergroup out of Nashville. Coming out of the AmericanaFest they were crowned as the Best Supergroup by Rolling Stone in their “20 Best Things We Saw at AmericanaFest”.  The group is made up of singers Jackie Greene and Joan Osborne, Black Crowes drummer Steve Gorman, bassist Nick Govrik, and guitarist Tom Bukovac.  Most supergroup efforts are hit-or-miss affairs, especially given the conflicting egos and visions of musicians otherwise used to performing with an established set of bandmates or acting as a leader in their own right.  This is not the case here.  These talented individuals have a common feel and passion for the music and it can be heard in their playing.  Their album was just released this past September and has earned raves from critics.  Their musical influences are from the blues, rhythm & blues, soul, folk, and country-rock.  Here they are performing at the XPoNential Music Festival 2014 this past September in Camden, NJ.  WXPN, which is the sponsor of XPoNential is the public radio station of the University of Pennsylvania.  They can be found at XPN.org, and for those who haven’t given them a listen.  If you do I think you will agree that they are also one of the best radio stations in the country.