Rephrased question : ETL developer vs Data engineer Answer : Unfortunately there are no strict industry standards on these job titles. That is just one part of it. Before ETL tools such as DataStage, Informatica, Ab Initio, etc., became popular, developers were hand coding every ETL flow. These ETL tools shortened the ETL flow development time to a great extent and allowed ETL developers to focus on business rule/logic/requirement (what to implement) than how to code it or optimize the code. There are many other benefits of using a tool but I won’t go into that in this answer. So an ETL developer with experience in these tools without any programming (coding) experience was/is able to design and develop end to end data flows. Whenever new types of source/target data format comes up, these tools catch up but it takes time, i.e., the ETL tool provider (e.g. Microsoft, IBM, etc.,) adds new components/connectors within the ETL tool to be able to work with new data format. For example, let’...
KABI is a new agile software development methodology useful for achieving quicker implementation of Business Intelligence (BI) solutions. The methods defined in KABI can also be used in full or in part for any other non BI projects that share similar characteristics as BI development projects. The word KABI is created by combining "KA" from KA nban and "BI" from B usiness I ntelligence. KABI is a lightweight, iterative, continuous feedback based agile software development methodology that enables every team member to work to their full potential even when there are several unknowns, resource constraints, dependencies and unpredictability of work load. Thereby KABI ensures optimal team productivity throughout the project duration. At the heart of KABI is " Peer Inspiration ". Every team member plays an important role and every team member works in an exemplary way. They do their part of the job so well that it inspires the whole team. " Mutual...
Currently what we see is that tools and technology limitations are used as a basis for classifying data, and even worse is that the classification is in itself incorrect. The so-called big data is so wrongly named. I have already explained why the naming is incorrect in another article - Why big data is actually small? Wouldn't it be much simpler, better, more meaningful and a standardized approach to classify data into primary and secondary data instead of using misnamed, meaningless and non-standard terms such as small data and big data? Primary data is the essential or core data without which the business cannot function. For example, a purchase transaction in a retail store has to be captured and stored for billing, payment, compliance, etc. These are mandatory requirements. Business Intelligence (BI) on top of that data is not mandatory but very useful, but that is not the main purpose of storing that core data in the first place. Secondary data is all of that da...
Comments
Post a Comment
Thanks for your comment. It will be posted after checks.