SOA, Data Warehouses and Modeling

For the past few years Service Oriented Architecture (SOA) and Data Warehouses have not been discussed within the same context. It is as if the two are mutually exclusive or at least the analysts and vendors want you to perceive they are. This never really did make sense to me. I am not an analyst so I really did not pay too much attention to this until recently when I had to address questions in a discussion about SOA and data warehousing.

I have been involved in the data warehousing, data mining, business intelligence domain for the past ten years so the problems and technologies in that domain are familiar to me. I have not been involved in a SOA solution yet. As I did my research into SOA I discovered that very similar problems exist. The deeper my research went into SOA the more commonalities I saw between these two domains.

I just finished reading "Whipping Data Into Shape" published in the 02/06/2006 issue of InfoWorld. The premise of this article is that solving a SOA problem is very similar to solving a data warehouse problem. IBM, Informatica and Oracle have even gravitated towards what they call an operational data warehouse type of concept and architecture for their SOA solutions. This surprised me but makes sense. What is old is new again just reborn with a new label and a few twists.

Defining a data architecture is coming to the forefront again in the SOA problem space due to the issues encountered over the past few years. The InfoWorld article talks about the concept of a Master Data Management Architecture and provides a nice graphic 50,000 foot view of it. It makes sense on paper.

The primary problem is the metadata which leads me into my other topic which appears to be the forgotten art of modeling. More specifically information and data modeling. In my experience, the past six years has been a form of backlash against modeling applications, systems and data. I think this coincided with the dot com boom and the need to get web sites, services and applications up and running as quickly as possible. The agile technique movement appear to address this need.

Well in a SOA or data warehouse agile and fast architecture and design decisions will get you into 'deep kimchi' really fast. I guess that is what has been happening in the SOA industry lately and a regrouping or rethinking of these rapid approaches is coming to the fore front again. This is a good thing. In the past few months I now hear many technical and management discussions centering on the requirements, modeling, use-cases, business-rules, 'by design', doing it right, etc. with respect to SOA today. Well, if you were involved with the last push for data warehousing and business intelligence in the late 1990s then this is just a repeat of what was done in the past and appeared to work. It did for me since the organization I work for has been managing a data warehouse for the past nine years quite successfully.

The twist on the massive amounts of data and services that has to be managed today is getting a handle of the context and semantics of it all. From an enterprise perspective, you are dealing with applications, files of all types, databases, web sites, and information all over the organization. What does it all mean from a SOA perspective? This is where a metadata repository and good models (data, information, semantics) are critical for success. The solutions and tools for this requirement are not yet built. At least from what I have seen they are not. What is required is some type of enterprise level content management, metadata, repository, and modeling facility. I am not sure who is going to solve this problem but I think platforms like EMC Documentum are headed in the right direction. Whether or not they solve the problem is to be seen in the coming years.

To keep it all simple, from the modeling perspective, I think getting back to fundamentals and building the models is key to success. If you can't define the problem you are trying to solve then how can you measure the level of success you have? I am glad that the old problem solving 101 still applies to today's much more complex world. I have yet to see my problem solving skills fail me yet. I follow the KIS (keep it simple) or KISS (I won't spell out the acronym here) principle. If you can make the complex appear simple, you have a better chance of getting more people involving in understanding the problem and thus ultimately help in being part of the solution.