Unstructured Information Management

Unstructured information makes up most of the information content on the internet today. Estimates are as high as 90% of available information on the internet is unstructured. So with all the databases, portals, websites, repositories, hard drives and trillions of files that exist today how do you harness this information? This is where the field of Information Management has the technical challenge to turn all this information into useful information and knowledge. This is entirely conceiveable given sufficient time, computing power and storage. The challenge is making this happen in near realtime.

To meet this challenge, DARPA has funded IBM Research in 2005 to create UIMA which stands for the Unstructured Information Management Architecture. It is an open, industrial-strength, scaleable and extensible platform for creating, integrating and deploying unstructured information management solutions from combinations of semantic analysis and search components. IBM makes UIMA available as a free SDK (alpha), and makes the core Java framework available as open source software (UIMA at SourceForge) to provide a common foundation for industry and academia to collaborate and accelerate the world-wide development of technologies critical for discovering the vital knowledge present in the fastest growing sources of information today. IBM developerWorks has a tutorial for using the UIMA SDK with Eclipse.

Since IBM released UIMA as open source in early 2006, it has been widely adopted. Open source projects such as GATE, OntoText, and many other have been utilizing UIMA as the framework for unstructured information management research. As research into managing and harnessing unstructured information grows, there will be more available solutions to solve these problems.

The Expanding Google Earth

If you have not used Google Earth lately then you are missing out on one of the killer applications that merges the web with rich native applications. The 3-D Warehouse of Google Earth plugins has been growing steadily after Google released Sketchup, the 3-D modeling aplication. Sketchup comes in a free version and a professional version. The nice thing about this approach is that you get to use a fully functioning program that scales to the professional level if you need that capability.

"The growing world of Google Earth" provides some insight into just how Google Earth has been evolving. I use Google Earth quite frequently as the means to find places and to explore places that I have never been. You can spend hours doing this and is quite entertaining.

The real power of Google Earth is the API provided by Google that allows you to build upon Google Earth's capabilities by integrating it into your custom applications. Additionally an SDK is provided to allow even more customization. Google Earth is becoming a platform for creating new applications or mash ups.

The imagery database is constantly being updated and I can see the day when the 3-D models for just about everything is integrated and available. I can think of several applications of integrating Google Earth technology and designing 3-D programs for everyday use. For now, we all get to watch this killer application evolve slowly which is ok with me.

Open Source To Rescue OCR

This blog is not entirely about open source intentionally. It just happens that open source technology is where all the action and innovation appears to be happening in software today. I just bumped upon an article about OCR that interested me. "Google releases open source OCR tool". If you thought at OCR was a stagnant technology then you just joined the crowd and Google's viewpoint.

What is interesting is the fact that it took an open source mentality, a stagnant technology (OCR), and a few bright Google engineers to reinvigorate the OCR technology space. The new open source product called, Tesseract, is released under an Apache Software License. Google hopes that the open source community will create solutions and new products for the OCR technology.

Prior to reading this article, I was not aware that OCR technology has literally stagnated for the last 10-years. HP and UNLV released their OCR technology into open source to re-generate interest. Well, it appears that Google does have the interest. Remember the digital library initiative that caused an uproar. Well, open source OCR looks like it is par for the course of working towards that goal.

The Next Set of Open Source Waves

I have written about my viewpoint of open source technology and how it is transforming the IT industry. Well, I am glad that this mentality or at least thought process is becoming mainstream. I really like the angle that InfoWorld has put on this in the article "Who are the Losers, Now That Open Source is Winning?".

What is even more ironic is how Microsoft has approached the Mozilla foundation to collaborate on future projects. This is just more evidence that open source is a strategic aspect of any company working in the IT industry. You can no longer ignore open source if you want to remain viable in the future.

The success of Google, Yahoo, Amazon, EBay, SalesForce.com, RedHat, JBoss... are case studies for businesses built on open source technology. Getting back to who is winning and the next set of waves, the InfoWorld article above makes a pretty good assessment of the current state of open source vs. proprietary technology in the IT field. The bottom line is for the next few years there will be peaceful coexistence. After that the two will be on a head on collision course and if you are not a player with or in open source technology, you will probably not be doing business successfully in the future.

As always in this business, time will tell and we are always just another major breakthrough away from a complete disruptive technology shift. So far this year, I think the cards are playing in favor of open source and its increasing critical importance to the IT industry.

LAMP Stack Development

Well, I have recently been pretty heavy into PHP development and must say that is quite an change compared to Java/Struts and J2EE type development. One nice thing I have found is taht Eclipse and Zend are working together to make this a smooth process with the PHPIDE. If you are a Java developer and already use Eclipse, you will feel quite at home using the Zend PHPIDE which is an Eclipse 3.2 plugin that works quite nicely even in beta form.

There are pros and cons to using the LAMP (Linux, Apache MySQL, Perl/PHP) or WAMP (Windows, Apache, MySQL, Perl/PHP) stack. I will just refer to this as the LAMP stack from now on in this posting. One benefit is that all the tools are open source. There are very few financial barriers to using these technologies. The primary con I see if that you have to ensure that you are using compatible versions of each of the components in your technology architecture. One solution is just try a VMware appliance, WAMP, or XAMMP packages pre-built for Linux or Windows. The downside to the pre-packages approach is that if you have to deploy and support a LAMP/WAMP stack, you will not have the necessary configuration skills for troubleshooting issues that may occur on your servers. I recommend that if you need to support this stuff, then learn to do it the hard way at least once so you will know the configuration and mechanics of the LAMP stack. The "Install & Configure Apache, PHP, JSP, Ruby on Rails, MySQL, PHPMyAdmin & WordPress on Windows XP/2000" and "Building a LAMP Server" HOWTOs really helped me out with this stuff among other HOWTOs I found. As usual, the HOWTO article was not bullet-proof or 100% accurate. I think I found about a 75% accuracy rate for configuring LAMP. Most issues I have found concern the PHP5 and MySQL5 issues (see this Zend post).

In other words, I discovered that MySQL 5 and PHP 5 have introduced some connectivity issues and incompatibilities with latest Apache HTTP Server. PHP5 doesn't even work with Apache 2.2 in my experience and I confirmed after doing quite a bit of research on the web. From what I have seen, the PHP community is working on supporting Apache 2.2. For MySQL5 and PHP5 you need to be using Apache HTTP Server 2.0. I learned this the hard way. Additionally, I was not aware that MySQL5 breaks quite a few PHP applications due to changes in password encoding and introduction of object-orientation in PHP5. Then there is the PHP mysqli extension that supercededs the mysql extension that is built into PHP. What a frustrating experience that is.

Anyway, what I have learned with all this experimentation with LAMP stack is that it is like all the other open source technology solutions I have been working with in the past few years. It does work with a few caveats. LAMP works as good as your understanding, skills and ability to use the web for research all the tools and components. Paraphrasing this, LAMP works if you are willing to put in the time and have the right mentality and necessary open source skills to solve problems.

Open Source Technology and National Security

I have read several articles that support the idea that the US DoD should adopt open source and open technologies more readily to match the productivity gains that many companies in the private sector have attained. The recent article, "Open Source in the national interest", provides a strong case for adopting open source in the DoD more readily. The 79 page April 2006 report mentions specifically how IBM has transformed itself by adopting open source readily.

There was a 2002 Report by Mitre that advocated the same. The 2002 article is not in as much depth as the more recent article noted above.

If Google, Yahoo, EBay, Amazon, IBM and many others are doing so well with open source technology on the large scale, then I think our government can do the same. It would just require a transformation of the government IT organizations. This is definitely more difficult than it sounds but when I see entire countries adopting open source technology (i.e. EU, Japan, Korea, China, India) I get that feeling that we might be falling behind or losing our edge by not following suit. I think it is in our national interest and so do many others in this ever changing computer science field.

Agile Web Development and the Lighter Languages

I have spent the past few months working with developing and experimenting with Java (JSP, Struts, Spring, Hibernate), Perl, PHP, ASP, Visual Basic and Ruby. Well, I will have to say there is something to be said about agile processes and open source dynamic languages. Recently I have had the opportunity to work with Java/JSP, Perl, and PHP. I am porting a PHP web application to JSP and intentionally sticking with a JSP Model 1 implementation. This is a bit of a digression in development techniques or a few steps backwards but I am testing out a theory. I can always refactor to an MVC design or other if necessary.

The theory that I am testing is: "It is possible to ignore elegance temporarily for attaining speed and practicality of getting a solution working using a 'quick and clean' process. (agility)" The old cliche in programming of doing something 'quick and dirty' to just get the job done is a concept I never really liked but have had to do occassionally in my career. The 'quick and dirty' approach almost always got me in trouble. I try to avoid doing this at all costs if possible. Based on my experiences with agile 'quick and clean' development, I feel that it is possible to attain software development speed practically in a sustained manner without sacrificing too much elegance. In other words, getting the cleanest job done in the best possible manner in the shortest period of time.

What I have found interesting in the days I've spent porting PHP to JSP is that it is relativley straight forward if you ignore your inner programming voice that says you should refactor the PHP to a more elegant Java/JSP design. In just a few days work, I have functioning webapps in JSP that rival the development speed of the looser PHP. I have to note that the PHP webapp is not using any PHP framework, design patterns (Five Common PHP Design Patterns), or other MVC pattern frameworks. This would of course complicate the design and make porting PHP to Java/JSP slower in my opinion but would require an experiment to prove it.

One other thing I must note is that having a really good IDE is critical to making this work. I am currently using Eclipse 3.1, Netbeans 5, and Zend Studio 5 in these 'quick and clean' experimental processes and this is definitely a must. I recently added Netbeans 5 to my pallete of java programming tools and it has yielded quite a productivity improvement for JSP and servlet development for me.

My next set of experiments is to get into Ruby on Rails (RoR) development which is a 'quick and clean' architecture. I have been talking about this all year however, have not made the time to start an experiment with RoR.

Archaeology and FOSS Trends

Archaeology has been a passion of mine since I was a kid. Prior to getting into computer science and music, archaeology was one of the professions I wanted to pursue. To this day, using the internet and reading as much as I can I try to stay involved in what is current in the archaeological field.

Much of the software technology used in the archeological field has been proprietary until very recently. I just bumped into an open source product called ArcheOS that is a GNU Linux distributionhas specifically configured for archaology! The 1.0 release was presented late in 2005 and has since been evolving following the open source model.

In the article, "Uncovering progress in FOSS-based archeology", it discusses the progress made to date with FOSS for the archaology. I will research more into what ArcheOS does and should have a cursory working knowledge of this stuff sometime this year if I can devote a few days to it.

I am surprised at how FOSS has been finding its way into the archeological field. I guess with continuing technology trends, open source technology appears to be invading just about every aspect of life where software is required and used. It makes logical sense that open source technology is finding its way into other fields other than business, science and technology.

Windows Genuine Advantage Spyware

I need to stand on top of my Microsoft soap box and say a few things about Windows security. I use both Linux, Windows and whatever devices make my digital life more productive and entertaining. Well, if you have not heard, just this past week Microsoft was caught and tagged as the largest spyware source to date. "Microsoft Big Brother". For roughly the past year, all modern Windows PCs have been phoning home every day. This was put into service in July 2005 as the Windows Genuine Advantage (WGA) service which was required to allow Windows patches to continue working. The true intent of WGA was never revealed by our trusted Big Brother, Microsoft.

The thing that bothers me the most about this is that Microsoft did this for security reasons and used it for some yet unknown reason (marketing) to make using Windows better for everyone. This sounds like a smokescreen to me and makes me wonder about the new integrate security features like anti-spyware Microsoft has embedded into the next version of Windows Vista.

Boy am I glad that I have options and know how to use Linux and other personal computing platforms effectively. As I look to the future, I see a less Windows dependent future for me. At least as far as home computing and entertainment computing and security are concerned. With Linux, Mac and Sony PS3 coming online soon, I think in the near future I will further diversify my personal computing technology architecture and use Windows less. Choices exist, you just have to exercise your right to choose based on good solid information.

Online Everything Is Getting Closer

I moved to using e-mail on the web in the mid 1990s with Yahoo! Mail. It was the best thing since slice bread for me. Prior to using Yahoo! Mail exclusively for my personal email, I was always on the wrong computer or at the wrong place when I wanted to access my e-mail folders containing the e-mail with information I needed where I was. This scenario could be while at home needing to access a work e-mail or while at work and having to access a home e-mail. Keeping track of the e-mail accounts and which computers could access it was just a pain.

To make a long productivity story, once I moved 100% to web e-mail, the problem just disappeared and is now a no-brainer. I do not understand the number of people I know who are still tied to thick client e-mail (MS-Outlook) today. Why? Who knows.

Well if you are still using your thick client calendar or e-mail, then you are stuck in the 1990s. With Yahoo! Calendar and most recently Google Calendar and the ease of collaboration over the web only requiring that you have access to the internet using any browser, I just don't understand the folks I hear talk about Outlook and wishing they were at home to access their personal email while away from home. Huh!?! Hello?! It's the 21st century and that is an old 1990s problem. Why are you still stuck in the 1990s?

Well, with the recent release of Google Spreadsheets and Google's acquisition of Writely, I think the days of being tied to a thick client for spreadsheets and word-processing are numbered. The only pieces missing is a nice web-based presentation application and a quick webtop database for business productivity. I suspect this will happen within the next year.

I can think of a few other web apps that would be nice replacement for their thick client equivalents, however for now I'd like to reserve some room for innovation in 2007. By the way, I just got my Google Spreadsheet account today and have already been inspired with a few ideas for its use within the first few minutes of using it. As for the "Online Everything" concept, I think this was dubbed Web 2.0 last year.

Google Web Toolkit makes AJAX Easier In Java

Today Google released the Google Web Toolkit (GWT) which is a Java framework that makes AJAX web applications easier to develop. I got my first taste of GWT today and have to say that Google has an AJAX product that makes building Java AJAX applications much easier to do. At least compared to my experiences in the past year.

A few of the features are that you don't have to write any Javascript. In comparison to other Java-based AJAX framework like DWR, you had to write some Javascript code. GWT generates all the necessary Javascript code for you! Google provides good examples for you to learn GWT.

Based on my first impression of GWT, they also provide decent documentation of GWT online. Anyway, this was a surprise that I happened to see in "Google Releases AJAX Framework".

The Modern Configuration Management System Solution

The de facto standard for source code version control is Concurrent Versions System or CVS as it is better known. Well, CVS has been around for a while and has a few flaws that most organizations just overlook. I used to work for one of those organizations that applied the "if it ain't broke, don't fix it" mentality toward source code version control. I have used PVCS, CVS, Teamsource, a few others and most recently Subversion.

CVS was good enough and it did what we required, most of the time. There were a issues with how CVS handles file renames, directory name changes, etc. which we just lived with like everyone else using CVS.

Well if you happen to use CVS today and just deal with all its binary file handling, file name change, directory handling flaws, I have some news for you. You should really consider migrating to the modern successor to CVS, Subversion. I had the opportunity to research, evaluate, test Subversion 1.3.1. I really should have done this a few years ago but now is as good a time any.

Subversion was designed to be a better CVS than CVS and I based on my experience they attained this goal. The primary reason for this is that I installed, tested and evaluated Subversion on Windows servers and it works as advertised. CVS is best installed and run on a Linux server. Subversion runs well in Linux and Windows.

Now I see why many of the Apache Software Foundation projects have switched to Subversion from CVS. Subversion handles binary files with ease and has all the nice GUI tools and Eclipse plugins that CVS has. I installed tested and really like RapidSVN, SmartSVN and TortoiseSVN. The Subclipse plugin for Eclipse 3.1 works very nicely. I did not find a good Netbeans 5.0 plugin which was my only disapppointment. Other than that, Subversion 1.3.1 is definitely ready for prime time. The Subversion website is an excellent source of information and tools available for Subversion. There are CVS to Subversion conversion tools which should make your transition smooth. I have not tried any of these repository conversion tools myself so I caveat my previous statement by saying your mileage may vary.

I found that the information in "Subversion UI Shootout" article is still relevant today. In the article "Setting up a Secure Subversion Server" which is geared towards Linux, the concepts can be applied to a Windows server deployment. For J2EE/Java EE environments, I found JavaSVN described in "Configuration Management in Java EE Applications Using Subversion" to be an excellent framework for Java server side development if you happen to need a servlet interface with your subversion repository.

For configuration management solutions today, I will be migrating everything I do related to configuration management to Subversion. It is definitely a better CVS than CVS and in my opinion, a good candidate for a modern open source configuration management system for more than just a source code version control system. I recommend that you use it for all your configuration management requirements if your organization is not adverse to open source technology. Good luck.

Google's Latest Products In April 2006

Google has been busy in their software engineering division. They recently released SketchUp which is a 3D Modeling tool. "Google Launches Free 3D Modeling Software". SketchUp allows you to create 3D models and store in Google Earth.

Google released Google Calendar in April 2006 and in the usual Google style it is dubbed a 'Beta'. Well, if you have not had a chance to try it you should. I have used many web-based calendar products like Yahoo Calendar, Lotus Domino, Mozilla Calendar, Sunbird, and even Microsoft Outlook Web Access. Well, I will have to say that Google has done a good job of entering an already crowded market and managed to make a dent and establish some traction.

Like their other recent products, Google has integrated many AJAX features which makes this product really useful. I am most impressed with the ease of use and responsiveness of Google Calendar compared to the competition. For my first set of impressions, it is as responsive as Google Maps. Which I suspect is not an easy feat. The software engineers at Google have been quite busy.

The ease of use is the most important aspect of the new Google product. Scheduling events and appointpoints is the easiest I've seen in any web calendar or desktop calendar. You just click on the date and an AJAX style window pops up allowing you enter your event. This surprised me. The ease of sharing your calendar and keeping a private calendar all in one interface is innovative. This feature is much easier to use compared to others I have seen.

The intuitiveness of Google Calendar is the most impressive. To reschedule an evevent or appointment you just drag and drop. Wow! All this from within my web browser.

For now, Google Calendar has become my favorite web-based calendar software. The integrate AJAX features and the responsiveness of the product are what impress me the most. There is still some refinements to be made like integration with hand-held devices and mobile phones. I am sure these features will eventually find their way into Google Calendar. This will be an interesting product to watch evolve.

Enterprise Portals

Portals, SOA and web services are the new buzz this spring. Well if you are involved in software engineering you are probably involved in projects that deal with content management, search, portals, SOA and some form of web services to expose all this content. So what is a portal? For the context of this posting I am using this definition of a portal. For open source java technology I try to stay focusing on JSR-168 compliant technology.

I have had the opportunity to spend some time researching portals lately and have found quite a few decent open source tools that are quite mature. eXo is a enterprise level product used by many organizations. The eXo platform is a powerful JSR 168 compliant enterprise portal built from several modules. It's based on Java Server Faces, Pico Container, JbossMX and AspectJ.

One product I like at least based on initial research is Liferay. It leverages all the best Java frameworks (Struts, Spring, Hibernate, Velocity, WSRP, MyFaces, etc..) and provides a really mature open source portal product. It has been in development for 6-years and is currently in version 4.0 release.

Over at the Apache Portals site, they are brewing Pluto which is the JSR-168 reference implementation, Jetspeed-1, Jetspeed-2, WSRP-4j and Graffito. All the Apache products are actively managed and are evolving fast.

For additional java open source portal technology, the following index contains a listing of quite a few more portal products. On the non-java front, Plone looks promising. It is built with Python. There are quite a few more open source portal products out there which are all evolving.

On the commerical side, there is Documentum which is a suite of enterprise level web applications. Documentum is large and deep. It is another JSR-168 compliant product. I mentioned it in a previous blog posting, "SOA, Data Warehouses and Modeling". Vignette is another product which I am still researching. Lotus Domino is even converging with WebSphere and becoming a player in the portal market.

So which enterprise portal products are the best? With the complexity of portal technologies that is a very difficult question to answer. I been involved with the debate that Microsoft Sharepoint is the solution. Based on my experience as a user of Sharepoint, I think it is somewhat limiting and rudimentary as a portal compared to the technologies I have seen. Everyone seems to be evolving in the direction of some type of enterprise portal. I like to stay focused on JSR-168 since it seems to have the momentum in enterprise portal technology today.

A Universal OS and new RAD Web Tool

Well, it looks like the operating system wars are coming to an end. Metaphorically speaking, this is the equivalent of the destruction of the Berlin Wall. It looks like due to the overwhelming economics of fighting open source Linux, Microsoft has decided to embrace the open source model, restructure, and support Linux with all its energy. You could see this coming with the recent reorganization announcements within Microsoft's management ranks. Recall that this type of a adaptation worked for Apple when it made a similar decision in the late 1990s for its Mac OS X project.

Economically, sustaining the battle against the open source movement was a draining the corporate giant. With the EU on the verge of fining Microsoft $2.4 million a day for violation of EU antitrust rulings, this has proven to be too much business risk to Microsoft's bottom line. Microsoft did not have any intention of complying with any of the EU's rulings which is exactly what they have done in the US in the past 10-years. Lobbying the EU just does not appear to be working as it has in the US.

In addition to the Singularity project that Microsoft R&D is working on, they have decided to join the Eclipse foundation, support the Novell Mono project, and have announced that internally they have been working on a Mac OS X like project where the Windows kernel will now be based on the Linux 2.6 kernel. The operating systems engineering group has been working on this for the past 3 years and they are nearing a point where they can unveil their progress to date to the world in true Microsoft marketing style, PowerPoint presentations. We will start getting the marketing briefings shortly as Microsoft prepares to adapt. There is a rumor that this project was initiated to counter the growing competition between Microsoft and Sony.

Sony has been collaborating with Apple, IBM, and Toshiba engineering a Linux based operating system that makes heavy use of virtualization and can seamlessly run Linux, Mac, Windows applications with ease. It is designed for the Cell chip and we should be seeing alpha versions of this new yet to be named OS in early 2007. I even saw that it will run on the Playstation 3 which makes a lot of sense.

Now you can see why a 'Universal OS' is on the horizon or at least there is OS convergence happening on all fronts. All the major players are working on some form of Linux kernel based operating system for the future.

On the software development front, the Eclipse foundation and Mono have announced a secret project that will permit developers to build Java, .NET, php, Ruby web applications using a universal web GUI/AJAX RAD plugin. Based on rumored information about early alpha versions, it's capabilities can be summed up to be equivalent of what the initial Borland Delphi 1.0 did for GUI client/server development in 1995. It changed everything as far as RAD client/server development is concerned. As a matter of fact, I saw some early hints that the IBM was purchasing Borland's IDE tools (Delphi, JBuilder) which are now on the market primarily to keep the talented Borland software engineers employed within the Eclipse Foundation. Things that make you go hmmm.

I have been waiting for a 'RAD for the web' type of tool that will give developers back the productivity levels that were attained with Delphi/Visual Basic in the 1990s. In the past few years, open source tools have come to dominate the web development space outside of .NET. With the infancy of Web 2.0, it looks like tools consolidation and a new dynamics have the prevailing winds blowing in the direction of the open source model. I personally have seen many decisions made for this movement allowing organizations to adopt open source technology where Microsoft once ruled. Look at what open source technology adoption has done for the emerging commercial space industry, "The Software of Space Exploration".

2006 is shaping itself to be a year of innovation, excitment, and surprises just as I anticipated. It is only the first of April and yet so much change is happening! I anticipate the next 9-months of this year being just as exciting.

Product Delays Becoming the Norm?

If you have been following the news these past few weeks, there have been four major delays announced. The Sony PS3, Microsoft Windows Vista, Microsoft Office 2007, Toshiba HD-DVD. Is this all a coincidence or over promising complex technology?

Well based on what I have read, Sony said the PS3 delay is due to final Blu-Ray copy protection technology specifications not being ready for a Spring 2006 delivery. Ok, I'll buy this since this can be confirmed throughout the industry. However, the product delays here are due to a specifications ratification which in essence is considered part of the software engineering project. This is a dependency risk avoidance strategy and looks to be a sound management decision. "Playstation 3 delay - a good thing?".

Then you have Toshiba delaying their HD-DVD launch to wait for the release dates of HD-DVD content in April 2006. This makes sense also. Who wants to buy a device that has no content? I would not. From a marketing perspective, this would put a dent on initial sales. From a software engineering perspective, timing the release of content to coincide with release of a product is a market risk avoidance strategy and looks to be a sound management decision.

Then you have the new bombshell announcement that the already late Windows Vista will be delayed nine more months due to quality and security programming issues. Recall that Windows Vista is already two years late. I read an article stating that a possible 60% re-programming of Windows Vista is required in order to deliver it by January 2007. "60% Of Windows Vista Code To Be Rewritten". This sounds like a rumor or FUD. If there is at least an ounce of truth to this and if you are in the software engineering field, this sounds pretty significant to me. If you have to re-write 60% of the code within the last 15% of effort, then I think there are some major design quality problems within the project. Has the complexity of the product and age of the Windows code base finally caught up to itself? Or is there some requirements creep happening internally to the project? Well according to some blogs I've read, Microsoft is even moving programming resources from XBox into the Vista project. This should be a challenge for both teams.

Adding more developers to an already late project just makes the project, well, even later. If you are familiar with Fred Brook's essays on project management, then it looks to me like Microsoft is having some serious internal project management problems. For a consumer level product which Windows Vista has become, all these delays and excuses really do not go well with the average non-technical buyer. It primarily affects the public's perception of your brand or company.

So are all these product delays now part of the norm in today's ultra fast paced high technology world? If you look at proprietary software technologies the answer is dependent on the size of the project. The larger the project, the more likely there will be delays due to the thousands of dependencies within the project. If you look at the open source universe, it seems that the open source model appears to attack size the complexity in a different manner. Since there really aren't any open source projects as large as Sony PS3, Microsoft Windows Vista or HD-DVD this is yet to be determined.

Being the optimist that I am, the jury is out on whether project delays is the norm today. I think project size, over promising, marketing hype, increasing complexity, and requirements creep are probably the real causes of delays. Each of these aspects are manageable components of the software engineering and product development life-cycle process. Of course this is much easier said than done.

Next Generation AJAX Applications

I started this blog last year inspired by my first impression of an AJAX application, Google Maps, and how I thought it would disruptively change the future of software. Well, I got my first glimpse of that promising Web 2.0 future application in the form of a beta AJAX applications called AJAXwrite 0.9 which is an MS-Word clone developed by Linspire. You have got to see this thing.

I was totally surprised at how fast AJAXwrite loads, in 6 seconds. This is due to its relatively small size of 400 kb as stated on the web site. It looks nothing like first generation AJAX applications such as Google Maps and Yahoo! Mail. These apps tend look and feel like rich browser based applications. The first generation AJAX apps have a few issues with load times in my experience. AJAXwrite looks and feels more like a traditional GUI native application that utilizes the browser engine as its platform. In my case I am using Mozilla Firefox. AJAXwrite still has some maturing to do but for a proof of concept or beta as everyone calls these today, it really shows what is possible.

Just as I predicted, AJAX is the disruptive technology that changes everything on the web. If AJAXwrite 0.9 is the trend of things to come this year, we will see some exciting new web based software that will redefine what a web application is.

I wonder what frameworks are being used to make AJAXwrite work? I need to do some research into this because for such a rich AJAX application, it sure does provide a lot of the rich client widgets and capabilities that I am accustomed to building in non-browser GUI application development.

Web Services and Small Companies

Innovation continues to take place in small companies. In today's over exposed world of instant information, web services and IT in general, it becomes difficult to sort through the background noise of what is published today to find new products and services. Especially from new or small startup companies. They just do not get the press.

I just read a really interesting article about a small company that has been able to monetize web services and do quite well at it. Google, Yahoo!, EBay, Amazon all have web services in place however are still trying to make it a profitable business. The article was published in the March 13, 2006 issue of Information Week entitled "Web Services By the Dozen".

The small company is called StrikeIron and it provides an online marketplace for web services for general use. StrikeIron acts as sort of a broker for organizations to search for web services and establish deals to consume them using StrikeIron in similar business model that makes Amazon and EBay successful. They are like this "80-pound gorilla" that dominates a field with few competitors.

In my local area, I had the opportunity recently to talk with a few small startup companies (micro companies) that are just as innovative. They are touting that web services and their agility (as in nimble) are the technologies that will allow them to distinguish their product and services to be ultra competetive and grow in the future. After reading about StrikeIron I think the creativity and innovation that will be driving the future of IT are the small companies. Google recently purchased Writely which was yet another small company providing their AJAX based web services that allows you use your browser for editing word processing documents in MS-Word, RTF, OpenOffice formats.

The only threat these micro companies have is their managing their own growth and being acquired too soon (i.e. by Google or others) while they are still half baked. We live in yet another exciting period where innovation is driving new ideas and most of these innovations are coming from the small companies.

SOA, Data Warehouses and Modeling

For the past few years Service Oriented Architecture (SOA) and Data Warehouses have not been discussed within the same context. It is as if the two are mutually exclusive or at least the analysts and vendors want you to perceive they are. This never really did make sense to me. I am not an analyst so I really did not pay too much attention to this until recently when I had to address questions in a discussion about SOA and data warehousing.

I have been involved in the data warehousing, data mining, business intelligence domain for the past ten years so the problems and technologies in that domain are familiar to me. I have not been involved in a SOA solution yet. As I did my research into SOA I discovered that very similar problems exist. The deeper my research went into SOA the more commonalities I saw between these two domains.

I just finished reading "Whipping Data Into Shape" published in the 02/06/2006 issue of InfoWorld. The premise of this article is that solving a SOA problem is very similar to solving a data warehouse problem. IBM, Informatica and Oracle have even gravitated towards what they call an operational data warehouse type of concept and architecture for their SOA solutions. This surprised me but makes sense. What is old is new again just reborn with a new label and a few twists.

Defining a data architecture is coming to the forefront again in the SOA problem space due to the issues encountered over the past few years. The InfoWorld article talks about the concept of a Master Data Management Architecture and provides a nice graphic 50,000 foot view of it. It makes sense on paper.

The primary problem is the metadata which leads me into my other topic which appears to be the forgotten art of modeling. More specifically information and data modeling. In my experience, the past six years has been a form of backlash against modeling applications, systems and data. I think this coincided with the dot com boom and the need to get web sites, services and applications up and running as quickly as possible. The agile technique movement appear to address this need.

Well in a SOA or data warehouse agile and fast architecture and design decisions will get you into 'deep kimchi' really fast. I guess that is what has been happening in the SOA industry lately and a regrouping or rethinking of these rapid approaches is coming to the fore front again. This is a good thing. In the past few months I now hear many technical and management discussions centering on the requirements, modeling, use-cases, business-rules, 'by design', doing it right, etc. with respect to SOA today. Well, if you were involved with the last push for data warehousing and business intelligence in the late 1990s then this is just a repeat of what was done in the past and appeared to work. It did for me since the organization I work for has been managing a data warehouse for the past nine years quite successfully.

The twist on the massive amounts of data and services that has to be managed today is getting a handle of the context and semantics of it all. From an enterprise perspective, you are dealing with applications, files of all types, databases, web sites, and information all over the organization. What does it all mean from a SOA perspective? This is where a metadata repository and good models (data, information, semantics) are critical for success. The solutions and tools for this requirement are not yet built. At least from what I have seen they are not. What is required is some type of enterprise level content management, metadata, repository, and modeling facility. I am not sure who is going to solve this problem but I think platforms like EMC Documentum are headed in the right direction. Whether or not they solve the problem is to be seen in the coming years.

To keep it all simple, from the modeling perspective, I think getting back to fundamentals and building the models is key to success. If you can't define the problem you are trying to solve then how can you measure the level of success you have? I am glad that the old problem solving 101 still applies to today's much more complex world. I have yet to see my problem solving skills fail me yet. I follow the KIS (keep it simple) or KISS (I won't spell out the acronym here) principle. If you can make the complex appear simple, you have a better chance of getting more people involving in understanding the problem and thus ultimately help in being part of the solution.

IBM's Quasar Impact

Earlier this year I discussed the impact I foresaw that the Cell chip was going to have on the industry. Well over at IBM the impact is huge. Just a few days ago, Fortune published an article describing the Quasar project at IBM. "IBM's Quasar: Is it the future of computing? The computer giant is betting a new chip and a reorganized corporate structure will make IBM exciting again."

The impact of what IBM learned on the Cell chip project in the last six years will be sweeping. The Quasar project is fueling an internal reorganization of IBM based on lessons learned from the Cell project and new sources of innovation within the corporate giant. The Cell concept came from Sony. It is a collaborative effort between Sony, IBM and Toshiba. It will power the soon to be released Sony Playstation 3, HDTV related consumer electroncs devices and IBM Blade Servers.

Quasar can be considered the next release of the Cell and will be a center piece of all of IBM's future hardware designs. The future information technology will be image and speech centric instead of primarily text-based as it is today. In the heavily image-speech centric processing world, the power of Cell and Quasar will be what drives the innovation from IBM's viewpoint.

Once Sony releases the Playstation 3 this year, we will have more tangible evidence of what the excitement is all about from an IBM perspective. IBM is trying to introduce the 'wow' and 'excitement' factor back into its products and with the Cell and Quasar chips it looks like the future again looks to be quite a fun place if all goes as planned.