High Priests, Quakers, Karl Marx, Adam Smith and Gemini

Following an exchange with some BI folks on Twitter, in addition to the various articles on spreadmarts and compliance issues with Excel usage by those pesky ‘users’ there is definitely an element of the old ‘high priest’ model in all of this. However, it is most likely economics that calls the tune.

In essence, when something new is introduced, whether that be technology, or religion, there is always a period of control by the early adopters (and indeed charlatans), who are naturally keen to implement a hierarchy to benefit themselves.  They become the required intermediaries in order for the masses to get what they want.  In the IT world, I don’t necessary claim it’s full of charlatans, or the hierarchy is there for job security, but the traditional model is what it is.

Quakers, for example, broke away from this and decided to cut out the middle men. 

In their case, it was more a theological gap, but with BI, the gap is the one between supply and demand.

In real world BI today, this is reflected by the use of familiar and available tools, primarily Excel, to bridge requirement/delivery gaps.   To an extent, Microsoft have historically recognized this and provided plenty of rope for users to hang themselves with.

To backtract a little and revisit what I consider to be BI – some folks associate BI solely with the more complex (and/or esoteric) analysis found with data mining and heavy statistics on massive data sets, for example the (sadly untrue) beer and nappies (diapers) story. I am a little more catholic than that. 

To lift an idea from Michael Gentle, here’s Karl Marx on complex BI:

“A commodity appears at first sight an extremely obvious, trivial thing. But its analysis brings out that it is a very strange thing, abounding in metaphysical subtleties and theological niceties.”

BI is all that, but covers much more of the mundane as well.  If you cannot easily find out how many of a certain widget you sell in the week before a holiday, or even how much of a certain widget you sell in a given geographic area, then when you eventually get your hands on this information, it’s business intelligence, or decision support, depending on your vintage.

Marx again:

“From each, according to his ability; to each, according to his need”

Therein lies the rub.

The maths just doesnt add up in the arms race of IT vs information workers. IT’s ability is completely swamped by the needs of users.  Nothing new here, I would contend that this is accepted wisdom.

To give you an example.  Of 50 finance users, every week, 5 will require new custom information.  Sounds reasonable.  However, I am talking about a real world example here.  So, each piece of new information information could come come directly, or from a combination of 20 different systems.  In addition, some systems are not in-house, so a short specification has to be written, sent to the provider, a quote comes back, is examined, eventually approved and implemented. So we are talking about 250 requests per year, but as you can see, these requests can be pretty costly and/or difficult to fulfill. <potential_product_plug_warning>I know about this stuff, as our product is often used in a guerilla-style way to sidestep these issues</potential_product_plug_warning>.

The planned economy just don’t work here my friends, we’re going to have to consult Adam Smith.

Information needs come from many sources, many levels of seniority and also have different profiles in terms of how time-critical they are.  The internal market for information, when in a planned economy scenario (i.e. IT-centric) is often dictated by how important the information requestor is and how easy the request is to fulfill – to meet internal SLAs and so forth.

“The monopolists, by keeping the market constantly understocked, by never fully supplying the effectual demand, sell their commodities much above the natural price.”

When the market is opened up to users, who generally know their data and exactly what to do with it – look – there’s the benevolent figure of Adam Smith again!

“The greatest improvement in the productive powers of labour, and the greater part of the skill, dexterity and judgement with which it is any where directed, or applied, seem to have been the effects of the division of labour”

And the users rejoice (whilst mangling a Smith quote):

“The natural effort of every individual to do their own analysis  … is so powerful, that it is alone, and without any assistance, not only capable of carrying on the society to wealth and prosperity, but of surmounting a hundred impertinent obstructions with which the folly of IT too often encumbers its operations.

However, much as users would rejoice in having these freedoms, there is a downside.

“The property which every man has in his own labour; as it is the original foundation of all other property, so it is the most sacred and inviolable… To hinder him from employing this strength and dexterity in what manner he thinks proper without injury to his neighbour is a plain violation of this most sacred property.”

The downside is the small matter of the injury to his neighbour, which in this case is probably a whole street of neighbours – Database Team(s), Apps Team(s), Data Quality folks, Internal Audit/Compliance etc etc.

So how do you resolve the conundrum of creativity/productivity vs control/compliance?

Microsoft’s stab at this is Gemini, using an analogy of the twin aspects of IT control, with ETL processes feeding into SQL/DatAllegro/Analysis Services and the user empowerment of the ubiquitous, familiar Excel client plus the “social” aspect of sharing their creative works through SharePoint. 

I have only limited information on Gemini at the moment, so my summary is probably sketchy, although I hope, still accurate.

The hope is, as I see it, to prevent the users from poisoning the well, but still allowing them to drink deeply with the paternalistic arms of IT around them.  Kind of like socialism, rather than rabid, laissez-faire capitalism.

 

Hat tips & further reading

Sean O’Grady (Control vs Creativity)

Michael Gentle (Good guy/Bad guy)

Nigel Pendse (Gemini Comment & Review)

Brent Ozar

Doug Henschen

Watch MS BI Conference Keynotes (Mosha Pasumansky)

Andrew Brust