Think of all the tables that exist in your ERP system, now think about all of the columns in those tables, now think about all of the other systems that may contain information you want to report on.  The average ERP system contains hundreds of thousands of columns on its own, let alone all of your other key business systems.  That’s a lot of data to work with.  The majority of these columns contain data that has no business value.

Now think about core business issues you are trying to solve to and the reports that you would use to gain visibility into these issues.  How many data elements are in those reports? 10? 20? 100?  Even if it’s 1000 that still represents a small percentage of the total number of data elements available.  If data has no business value, then why go through the effort of extracting, translating, mapping, loading, and validating data(to name only a few tasks), that no one will ever look it?

Yet many business intelligence initiatives set out to do just that.  Imagine the impact on your implementation timeline if you reduce the amount of data by a factor of 100 or 1000?  Imagine the impact if the project is focused on the business issues and visibility into the data required to help solve those business issues, rather than trying to provide reporting on every data element in the system.  Not only will the implementation timelines be drastically reduced, but the end solution will have greater business value and will get into your users hands in a fraction of the time.

Companies need to start asking their BI project teams to focus on the business issues and the end solution, not on the vast sea of meaningless data that will never impact the bottom line.  What is your BI team focused on?

This development clearly reflects trends I am seeing in the marketplace.  Traditionally, BI and analytics focused on marketing, customer and finance applications.  In recent years however, I have seen the interest in supply chain BI analytic solutions explode.  I think this is driven by the increased importance of supply chain in many companies; the analytic orientation of supply chain professionals; most importantly the cross functional nature of supply chain, and that  solutions are now available commercially to address SCM analytic needs.

Most supply chain professionals lack the information they need to do their jobs.  Why?  Primarily because the cross functional orientation of supply chain requires that companies assemble data from different systems.  For example, if one wants to measure supplier performance, one needs access to cost data, on-time delivery data, and quality data — at a minimum.  These all come from different systems and require a BI solution to assemble.  Same is true if you want to analyze cost-to-serve or customer profitability.

Let me know if your company is beginning to use SCM BI solutions.  What application areas are driving the greatest benefits?  Are you able to find or develop the analytic applications or solutions that you need?

The confusion about what MapReduce is and isn’t about actually goes back a ways.

The arguments within computerdom tend to generally be fairly religious in tone. That’s because people really like computing and tend to be passionate about it, and when people think they’ve something new, their ego interest is in showing how new and innovative it is, and the ego-interest of the creators of prior-generation innovations is in casting it as old news.

I am biased, but not working in scalable computing at the moment, so I’ve had a little bit of time to develop some perspective here I’d like to share.

Scalable computing via parallel processing is big business these days. There’s heavy data lifting at the transaction processing level, at the data analysis level, for web search indexing, for enterprise search indexing, scientific computing, data mining for target marketing, affinity advertising. There are many places where data processing of really large amounts of data has commercial benefits, and that means parallel processing adds lots of value.

At the smaller end of the parallel processing scale there’s actually a crisis of sorts – we need parallelism to use up all these multi-core CPUs that the chip makers can now create easily, yet there is no mainstream accepted programming idiom that creates the multi-threaded programs yet is easy to use.

MapReduce was designed initially for the other end of the spectrum – huge data, Internet-scale data. Much bigger than enterprise data for even the largest enterprise. This is where clearly you want to use lots of computers all at once. Advocates will tell you it’s also great for that downward scale to multi-core CPU thing – and I concur. But that wasn’t the first intention for it.

The other big scalable computing systems of commercial import are the relational database engines, and the ETL tools that transform data, and sometimes also support data mining. In common to both of these: commercial data, commercial data types, lots of structured data in the mix, of course with gradually improving capabilities to handle unstructured data. Databases are of course data storage tools, where ETL systems are algorithmic systems. For example, in a database when you talk about how to break up data into partitions you are talking about how it is stored. In the ETL world, partitioning data is about how it is moving, and incidently it can be stored that way also. This latter is very close to the way MapReduce folks talk about data even though I don’t think ETL folks and MapReduce folks drink in the same bars.

What all three of these kinds of systems, MapReduce included, have in common is that they are all about scalable Flow-based programming (FBP for short) to some degree. Not everyone agrees on this term flow-based programming, preferring dataflow, or streaming, or flow-graph or some other term. RDBMS systems have their query graphs at their query processing cores – they transform SQL into these. A query graph is a FBP. ETL tools let you draw these graphs (and script them also sometimes). They allow the graphs to have multiple outputs, unlike a query graph, but otherwise are fairly similar. If you use the Apache Pig (so NOT a commericial software system!) system to create MapReduce programs, it cascades the map-reduce operations together into a flow graph. This was created because so many map-reduce applications found themselves wanting to connect one mapreduce back to back into another that someone wisely observed that this flow composition was FBP and that was really part of the idiom.

The primary source of parallelism in all the above is called partitioning. Basically it is the principle that if you have a large piece of data, or large collection of data items, you can divide-and-conquer it, and then put the pieces back together (if necessary). The divided processing is the “map” part, the putting back together is the “reduce” part. You might map and map and map and then reduce, or alternate. Do this kind of thing over and over as you process the data, flowing the output of one such operation to the input of another.

So here’s one major point of total agreement: There’s a agreement across many different areas of computerdom that flow-based programming is a very good way to go if you want parallelism. I think people would also say the idiom is pretty easy to learn, adequately expressive for many problems, and so forth.

That’s very important common ground. If you are a software developer and can say you’ve done object-oriented programming and you’ve used threads or actor libraries or some such, you may feel relatively accomplished at your trade. If you are not directly familiar with this FBP idiom it is something you should learn about, because you will likely find it very valuable, and it will change the way you think about solving problems.

FBP principles are where the similarities end among these kinds of data processing systems though. There are huge differences about what these systems are designed to be used for.

Now, advocates of each seem to have some “hammer vs. nail” mentality – By that I mean if you have a big enough hammer, *everything* starts to look like a nail. Example: extending an RDBMS to do what it does, but also handle spatial data, or blobs of image data, or real-time streaming input data, or transactions from a queue, or sensor data feeds. You pick it. Extending a RDBMS hammer to hit these kinds of nails, this is a very attractive option to those familiar with RDBMS technology, and particularly if they have access to the source code for one. Same for ETL tools. I was chief architect for a major commercial ETL processing engine, and we tried to extend it in all kinds of stretch directions. MapReduce advocates would look at these same problems and recognize the maps and reduces that are being cascaded to form the solutions to them.

So, what separates these technologies?

I claim the first thing is maximum scale. But what is a big enough scale to start telling these apart?

Ok, here’s a problem suitable only for MapReduce today: Let’s play where’s Waldo and find every video frame containing my face, in every video CCTV surveilance capture everywhere on earth for the last year. You have 60 seconds to find the answer.

That is the kind of problem map-reduce can scale up to. The data is petabytes or exabytes of data. Now, the actual where’s Waldo search might be very cleverly indexed, so I probably do not have to stream all these exabytes of video frame by frame to look for my face. But at some point every frame was processed to create whatever index answers this problem. That processing scale is truly vast.

None of the other technologies were built with anything like this scale in mind. Yes there are FBP principles of operation which are in common. The distinction is what frameworks have been built to actually do.

What about ETL or RDBMS technology? What’s suited to them that crisply distinguishes them from MapReduce frameworks. Answer: processing structured data records repeatedly. I’ll characterize what I mean by this, but first let me talk about some of the design limitations of RDBMS and ETL technologies.

First the problems are ones at more traditional enterprise scale. These days that’s multiple terabytes of information, maybe even 100 terabytes now adays. One assumption common to these systems is that the whole computing cluster across which the work is spread is assumed to be up and working. Failure of any part of the cluster is expected to stop the processing. There are various checkpoint/recovery schemes which are pretty much the cutting edge of these systems these days. These always assume a failure is a relatively rare event. Most executions will complete without failure.

There’s a thing I call a time/complexity trade off at work here. Which would you rather have? A system that is slower, and more complex, but which takes twice as long to compute something thereby doubling the likelyhood of encountering a failure during a run? Or would you prefer a faster system without the complexity? Computers are fairly reliable these days, so at the scale of a cluster of computers small enough where I can reasonably expect them all to all be running properly for hours at a time… Well I know my own preference here and that of many “enterprise” customers matches my own – simple wins.

MapReduce makes no such assumptions about failures. The implemented scale is assumed to include computers that will fail during the execution of a flow, and the computation must continue and mask the failure.

So here’s a real and concrete question: Does your processing problem have the vastness to it that means the compute cluster will have failures in it most of the time, or is the processing scale such that the cluster is very unlikely to have a failure in it that interrupts processing? If the former, MapReduce is the only viable technology today. If the latter, there are still reasons you might want to use a MapReduce-like means.

A big difference between MapReduce systems I’ve looked at and the enterprise RDBMS and ETL world is efficiency of data access vs. complexity.

RDBMS and ETL are optimized for handling business-oriented data. Quite literally, I worked on engineering an ETL engine so that when a processing component accessed an integer from a data record, the C++-based implementation would generate code that would compile into a base+offset-indirection – really our goal, and we got very close, was for this to be a single CPU instruction.  The assumption here: you are accessing billions of rows of well behaved small data that has been imported into a well behaved representation, bringing it in once from whatever it was before you started processing it. By the way, getting this super low level efficiency introduces lots of complexity. Many would rightfully say this was “wicked over-engineered” – mea culpa. Databases go even further and try to apply cost-based dynamic programming optimizers, move things around in the flows to reduce work done. Very cool stuff. Actually one of the first applications of ETL tools was actually a backlash to the unpredictability of SQL database optimizers. People wanted to draw a flow and have it run that way dammit! That was 1995. The optimizers have improved, but ETL is important for other more fundamental capabilities like superior expressive power. ETL tools can produce multiple outputs from a single input in one pass. SQL is a query language and is incapable of expressing this directly even though the underlying engines could easily implement this.

Anyway, all this low-level data access optimization and high-level flow-graph optimization, this kind of thing is just not considered particularly important by the MapReduce advocates. You process data in the form it was found, which is assumed to be an HTML document, XML data file, or perhaps JSON. Perhaps other forms that you happen to have some code to handle. Really it’s a text-intensive world. These formats are all very verbose textual formats full of tag syntax. Verbose by comparison with carefully layed out binary data records is what I mean.  It takes thousands, of machine instructions to extract a piece of data like an integer from such representations because you have to find it first (parse and/or search for tag), then convert text to integer.

But there is absolutely nothing wrong with that overhead. Because accessing small numeric fields is just not what the MapReduce world is about. Consider this: in RDBMS and ETL worlds, if you access a string, there’s low overhead to get to the start of the string, and then you’re processing the string data using the usual libraries available in the programming language of your implementation.

In MapReduce, there’s high overhead to get to the start of the string you care about, but then you’re processing the string data using the usual libraries available in the programming language of your implementation.

Get my point? The only place where the RDBMS/ETL world has advantage is if the strings are really small and must be repeatedly accessed. In many many very interesting problems, the strings are not at all small, and many problems are “one-pass” over the data by each algorithm.

So the conclusion should not be surprising: it’s “horses for courses” – none of these technologies is superior to the other. They all share flow-based programming principles – they jury is back, flow-based programming is a GOOD IDEA whose time has come and it is being used all over the place. Beyond that, what kind of processing are you doing? What kind of data is at its core, and at what scale?

Presentation after presentation by IT professionals took one of two angles:

1. We built a huge, super fast data warehouse and let me tell you how we did it; or
2. We built this home-grown system to do BI, let me tell you exactly how.

In the case of the big data warehouses, there were some very cool approaches with some real innovation.

In the case of the home grown systems…help! One VP of IT was proud that he built an operational data store from the ground up. Another was proud that he put a tremendous amount of business process in place, but lets users pull data down into spreadsheets and do what they like with it – strict process with loose data governance. Another said it all starts with the data: identify all the data, get it in a data warehouse, then figure out what to do with it.

Doesn’t the IT organization have more value-adding contributions to their company than simply building systems from scratch that are commercially available today?  Are these really examples of innovation in BI?

- How about CIOs talking about how they drove business benefits for their companies?
- How about VP, IT architecture talking about having business needs drive analytics that drive data that drives the solution?
- How about Directors of BI talking about how they leverage open source or SaaS to address a certain set of specific requirements from their user community?

Let’s hope we’re not hearing the same stories of “BI success” three years from now.

Is there any innovation around BI going on out there by IT organizations? Are they simply reluctant to talk about it? What do you think? What are your stories?

a) Privately held and therefore hard to gauge in terms of “how they are doing”, and

b) Are still too young in their evolution…meaning that even if they were growing at a healthy clip in the down economy (ie: double-digit growth), it is hard to tell if “they are doing well”.

Also, what metrics would one look at to assess growth for these firms?  What might be a meaningful metric to measure some of these firms may be far less meaningful for the others.

Measurements aside, here are some thoughts about why the SaaS Business Intelligence segment should be doing well:

• It’s faster to deploy vs. on-premise, therefore, can provide faster business benefit
• It offers a better economic model vs. on-premise….lower TCO
• Customers can “stretch” their existing budgets more (the SaaS model usually means little-to-no capital expenditure) and still get a solution
• An already over burdened IT staff can off load much of the low-value work associated with hardware provisioning, software provisioning, software configuration, database tuning, backups, etc. to the SaaS provider…therefore freeing them to focus on higher-value tasks to support the business
• Some SaaS Business Intelligence offerings are geared to specific functional areas and business problems and can therefore drive some tangible business benefit (ie: Inventory Cost Reduction, Increased Customer Retention, Lower Transportation Costs, Improve Gross Margins, etc.).

These points all make sense to me.  If you were to say to a CFO…”you can have a BI solution up and running in 8-10 weeks, with little-to-no upfront investment, minimal disruption to your IT staff, and it would be well aligned to your strategic business initiatives…” why wouldn’t he/she not want to give it a try?

Here are some reasons why many organizations are still not making the leap to SaaS BI:

• The SaaS model is disruptive to many of the traditional ways of procuring and deploying software….it’s different, therefore, it requires change and a lot of people don’t like change
• The “trusted advisors” of many organizations are not recommending SaaS solutions yet because they don’t fully understand the benefits and/or they have a vested interest in doing things the old way (meaning they make a lot of money from that)
• Control…the nature of SaaS is that if you are going to use a SaaS solution, a lot of things are no longer going to be in your control
• Trust…you need to take a leap of faith in your provider to trust that your data will be secure, your solution will be available/responsive and recoverable should there be a “disaster”
• Misplaced Fear…Many IT organizations are resistant to the SaaS model because they perceive it will impact job security.

But times are different now (didn’t General Motors just go bankrupt?) and doing things the old way may not always be working anymore.  Can we afford not to look at new and different ways of doing things?

COMMENTS (archived)

I don’t think anyone can afford not to look at new and different ways of doing things. SaaS is a good example. Although they’re not suitable for all organizations, they are for many and usually cost a fraction of large scale server/client solutions. Deployment is usually a child’s play, so IT resources can be spared for something else. That said, business intelligence software or SaaS is something that should not be cut in order to save money. Actually, BI software helps save money, among other things, when used wisely. A better solution could be to find a cheaper product, maybe, but getting rid of a BI solution isn’t quite a wise decision.

————————–

I believe Travis is spot on in his comments. Properly designed and deployed BI can provide organizations with enormous leverage and business benefits that far outweigh the costs. One big challenge is transitioning the discussion from a cost-basis to a value-basis. Yes I know this sounds a bit like “motherhood and apple pie” but this really is the dialogue that technology decision makers and leaders must engage in.

————————–
There needs to be some context set around the economics of any new innovative solution, like SaaS BI. Engaging in a total cost-of-ownership comparison of a build it yourself with traditional BI tools, versus say using a SaaS model, is interesting but only half of the story. Folks need to dig in with as much vigor to determine which solution will do a better job driving the value or “Return” in the ROI equation. There are differences and fundamental questions to be asked:

1. Which model allows more universal access to the solution by the community that counts — the internal business users (and perhaps even external supply-chain partners and custoemrs if appropriate) to help drive informed decision making?

2. Which approach does a better job leveraging functional best practices and domain expertise and who does a better job exposing these concepts into specific reports and analytics?

3. Which appoach is more of a business relevant solution solving a specific business challenge and driving specific initiatives?

Too often the question asked in traditioanl deployments with tarditional tools is “What does the tool do and what is it good at?. The common response is “That really depends what you want it to do.” I would sugegst this becomes a circular dialogue chewing up precious cycles and proloning already long production deployment times.

So Travis I agree folks need to be specific, ask the detailed business questions, and make certain that whatever approach is taken really solves the domain specific business challenge.

Borrowing from the dated, but still entertaining movie, Blazing Saddles: “We don’t need no stinking general reporting tools.”