Automation makes DWH developers / BI developers redundant?

Question: Automation makes DWH developers/BI developers redundant?

Answer:

No. I think quite a lot of tasks will be done automatically but not all. DWBI developers will use tools to automatically create most part of the deliverables (ex ETL jobs, reports, dimensional models, metadata) however some deliverables will be custom requirement that will have to be done manually or automation enhanced to deal with custom requirements. The way things will be done could be different and the profile of a DWBI developer could also change.

We already see some examples of automation. If you specify the list of source tables and target tables then a number of ETL jobs are automatically created in seconds using some of the ETL tools. Earlier ETL developer had to develop every single job even if it was a simple job. Automated ETL job creation can be very useful for one to one data migration, but as soon as a business specific logic needs to be placed in the transformation, someone needs to specify that at least once and ensure that the data transformation according to business logic is happening.

Automation is good for repetitive tasks. If something is automated then generally the process is mature. However, every day there are companies coming up with new business models new types of products and services. Mature businesses will have to keep reinventing themselves to stay alive in this competitive world. Business models are not permanent, regulations are not permanent, business environment is not going to remain same always so customization will always be required to offer something new to customers that has not been offered before. Note that business intelligence drives business forward and at the same time business intelligence systems follows changes in business. These changes will keep DWBI guys busy. DWBI guys will also upgrade themselves.

In 2008 I worked on a project where we setup BMC Remedy Analytics for a telecommunication company. I thought what BMC did was amazing. They provided around 120 BO reports and 1 BO Universe out-of-the-box as part of the BMC Remedy Analytics installation package. All you had to do was install the software and you already have 120 pre-built reports. If it were to be done manually by developers it would take minimum 6 months with a reasonable team size. These reports I believe are based on what BMC thought would be useful for all the clients who use their Remedy software for ITSM. But these reports were not enough for the client that I worked for, they required several other reports, some of these were very custom requirement that probably other clients wouldn't have. We had to develop those reports but to develop we had to understand the underlying BO Universe. Hope this example explains how the trend is going to be. Mostly automated for mature process and custom for new. By now I guess many more companies are offering such out-of-the-box solutions.