ETL developer vs Data engineer

Rephrased question: ETL developer vs Data engineer

Answer: Unfortunately there are no strict industry standards on these job titles. That is just one part of it.

Before ETL tools such as DataStage, Informatica, Ab Initio, etc., became popular, developers were hand coding every ETL flow. These ETL tools shortened the ETL flow development time to a great extent and allowed ETL developers to focus on business rule/logic/requirement (what to implement) than how to code it or optimize the code. There are many other benefits of using a tool but I won’t go into that in this answer.

So an ETL developer with experience in these tools without any programming (coding) experience was/is able to design and develop end to end data flows. Whenever new types of source/target data format comes up, these tools catch up but it takes time, i.e., the ETL tool provider (e.g. Microsoft, IBM, etc.,) adds new components/connectors within the ETL tool to be able to work with new data format. For example, let’s say XML format didn’t exist, and that ETL tool had only csv, excel, and database in-built connectors. An ETL developer would easily develop ETL flows with csv, excel, database connectors but wouldn’t be able to progress on creating a connector for xml format assuming he didn’t have programming experience and assuming that the tool doesn’t provide plugins (rare now). So in this situation, if the ETL developer also has programming experience and if the tools allow to write custom connectors then ETL developer can also develop the connector for xml format without having to wait for the ETL tool provider to release it. Now extend this example for any new format, any new technology, and any new software that comes up. So, as you can now understand once all of the connectors are available and the ETL tool is already available within the company (money spent to save time) there is less need for a coder and more need for an ETL developer to implement business rules/logic. Currently from what I have been observing in the job ads when there is a need to code they are terming it as data engineer.

ETL developer vs Data engineer


A related post is here - Big data developer or BI Developer

Comments

  1. Data Semantics31 May 2021 at 07:53

    Your blog is very useful for those who are searching for this kind of information, this information is a solution for those who are confused about it. Thanks for sharing this blog with us.Azure Data Warehouse Solutions

    ReplyDelete
  2. Aaron jhonson3 December 2021 at 10:10

    Excellent article... Thank you for providing such valuable information; the contents are quite intriguing. I'll be waiting for the next post on Big Data Engineering Services with great excitement.
    Data Analytics Solutions

    ReplyDelete
  3. Aaron jhonson3 December 2021 at 10:17

    Very Nice, Thanks for sharing such an informative Article. I really Enjoyed. It was great reading this article. I would like to know more about Big Data Engineering Services.
    Data Analytics Solutions
    Advanced Data Analytics Services

    ReplyDelete
  4. Ranjith3 December 2021 at 10:20

    Thank you for sharing this content
    Big Data Engineering Services with great excitement.

    ReplyDelete
  5. S6 Technologies1 April 2022 at 21:43

    I am very happy to read this article. Thank you very much for writing such an interesting article on this topic. effective Managed IT Services in Houston tx Well done and always keep sharing the best knowledge with us.

    ReplyDelete

Post a Comment

Thanks for your comment. It will be posted after checks.

Popular posts from this blog

KABI - The new Agile Methodology for BI Projects - Implement BI projects quicker happily

Image
KABI is a new agile software development methodology useful for achieving quicker implementation of Business Intelligence (BI) solutions. The methods defined in KABI can also be used in full or in part for any other non BI projects that share similar characteristics as BI development projects.  The word KABI is created by combining "KA" from KA nban and "BI" from B usiness I ntelligence. KABI is a lightweight, iterative, continuous feedback based agile software development methodology that enables every team member to work to their full potential even when there are several unknowns, resource constraints, dependencies and unpredictability of work load. Thereby KABI ensures optimal team productivity throughout the project duration. At the heart of KABI is " Peer Inspiration ".  Every team member plays an important role and every team member works in an exemplary way. They do their part of the job so well that it inspires the whole team. " Mutual
Read more

BI Architect course and BI Tool question

Received two questions, 1 through LinkedIn and other through blogger. Sharing details of Q&A here for the benefit of wider audience. Question 1:  I want to start with a career in BI, and I am planning to get trained in it. The institute covers MSSQL, Power BI, MSBI, Tableau desktop 10, Informatica developer and admin, Azure data factory, SQL DBA, QLIK sense, Data warehousing. Is this all that is needed? Reply : Regarding getting into BI, the best way is to get into a team where you can learn from experienced people in the team. Regarding courses, just based on the list, that's quite a lot for a training, but it depends on what exactly is provided as part of the training. Need more details about the training. Then once the link was provided, I had a check and provided below reply. Note - Link or institute name is not shared here intentionally to avoid any confusion that I am promoting any institutes.  Reply :   I have had a look into the link, as a course for a b
Read more