ETL developer vs Data engineer

Rephrased question : ETL developer vs Data engineer

Answer : Unfortunately there are no strict industry standards on these job titles. That is one part of it.

Before ETL tools like Datastage, Informatica, Ab Initio, etc became popular, developers were hand-coding every ETL flow. These ETL tools shortened the ETL flow development time to a great extent and allowed ETL developers to focus on business rule/logic/requirement (what to implement) than how to code it or optimize the code. There are many other benefits of using a tool but I won’t go into that in this answer.

So an ETL developer with experience in these tools without any programming (coding) experience was/is able to design and develop end to end data flows. Whenever new types of source/target data format comes up, these tools catch up but it takes time, i.e the ETL tool provider (e.g. Microsoft, IBM, etc) adds new components/ connectors within the ETL tool to be able to work with new data format. For example, let’s say xml format didn’t exist, and that ETL tool had only csv, excel, and database in-built connectors. An ETL developer would easily develop ETL flows with csv, excel, database connectors but wouldn’t be able to progress on creating a connector for xml format assuming he didn’t have programming experience and assuming that the tool doesn’t provide plug-ins (rare now). So in this situation, if the ETL developer also has programming experience and if the tools allow to write custom connectors then ETL developer can also develop the connector for xml format without having to wait for the ETL tool provider to release it. Now extend this example for any new format, any new technology, any new software that comes up. So as you can now understand once all of the connectors are available and the ETL tool is already available within the company (money spent to save time) there is less need for a coder more need for an ETL developer to implement business rules/logic. Currently from what I have been observing in the job ads when there is a need to code they are terming it as data engineer.

ETL developer vs Data engineer


A related post is here - Big data developer or BI Developer

Comments

Popular posts from this blog

IBI New Template - Excel Version

Corona Virus Analytics

At least 3 people have started IBI this year