Working of an ETL Tool

ETL stands for Extract, Transform and Load. ETL is one of the main components in a BI Solution backed by a Data Warehouse. ETL is also used in other projects like Data Migration and Data Integration projects. ETL flows/jobs can be built by scripting or by using ETL Tools. Most of the companies currently use one of the existing ETL tools to build ETL flows/jobs.

In short, ETL tools abstract the technical complexity and thereby enables developers to focus on ”what needs to be done” than on “how it needs to be done”, For example a developer doesn't have to bother about developing a connector to a Database, Developers design the ETL flow/Job using drag and drop, click and configure GUI of the ETL Tool and also run, test, debug, schedule the ETL Jobs using GUI of the ETL Tool.

ETL Tools provide a visual framework for ETL developers to design the ETL Jobs. It provides a level of abstraction of the code/script. There are many features that ETL tool provides but I won’t go in those details here.

ETL Tool allows the ETL Developers to focus on the Business logic/rules instead of the technical implementation. So, ETL developers focus on what to implement than on how to implement.

ETL Tool is expected to create optimized code for an ETL flow and is expected to either bring the data to the ETL server or perform operations in the Databases based on whichever is better for performance reasons. Ideally ETL developers shouldn't have to optimize the code but only configure the jobs/transformations/tool correctly such that optimized code is created for every ETL flow.

ETL Tool has pre-created connectors/drivers to almost all type of data sources, For example to connect to a DB, All that you provide is the DB server details so you as a ETL developer don’t have to know how to code to connect to a Data source.

ETL Tool has mappings for each of the commonly used functions.If you see the metadata repository of an ETL Tool you will see that there are predefined mapping for every common data transformation for every possible data source. I don’t remember exactly the mappings so I will give a simple example just to give you an idea how an ETL tool works. Lets say, Business transformation requires to find out date from timestamp, In one DB it could be date(timestamp column), in other it could be dt(timestamp column) and in another one some other convention. When you do this in an ETL Tool you just select the Date from Timestamp function from the ETL Tool and then the ETL Tool checks which type of Data Source it is and takes the corresponding mapping so that the underlying data source understands it. Now extrapolate this logic to every single transformation to every type of data source/target.

Like other tools ETL Tools are also available in both open source and proprietary flavors.
  • SSIS is a proprietary ETL Tool from Microsoft
  • Datastage - IBM
  • Ab initio - Ab initio
  • Informatica - Informatica
  • BODS - SAP
  • Popular open source ETL tools are Pentaho Data Integration and Talend


Popular posts from this blog

BI Architect course and BI Tool question

ETL developer vs Data engineer

Corona Virus Analytics