Don't use front-end where it is not required

Regular manual download of data from a portal and loading it into a back-end system is like exiting an airport and then reentering the same airport to catch a connecting flight from the same terminal when you have no business outside the airport. 

I have seen people doing this, downloading data from portals and then loading into other systems manually. If there is no one looking at the data in the UI and no decision is to be made and only regular data feeds are required then don't do it via front-end (GUI), just get a data extract job created that will automatically load the back-end system. 


This is partly related to one of the previous posts - BI can take you to places

Comments

Post a Comment

Popular posts from this blog

ETL developer vs Data engineer

Image
Rephrased question : ETL developer vs Data engineer Answer : Unfortunately there are no strict industry standards on these job titles. That is one part of it. Before ETL tools like Datastage, Informatica, Ab Initio, etc became popular, developers were hand-coding every ETL flow. These ETL tools shortened the ETL flow development time to a great extent and allowed ETL developers to focus on business rule/logic/requirement (what to implement) than how to code it or optimize the code. There are many other benefits of using a tool but I won’t go into that in this answer. So an ETL developer with experience in these tools without any programming (coding) experience was/is able to design and develop end to end data flows. Whenever new types of source/target data format comes up, these tools catch up but it takes time, i.e the ETL tool provider (e.g. Microsoft, IBM, etc) adds new components/ connectors within the ETL tool to be able to work with new data format. For example, let’s say xml f
Read more

BI Architect course and BI Tool question

Received two questions, 1 through LinkedIn and other through blogger. Sharing details of Q&A here for the benefit of wider audience. Question 1. I want to start with a career in BI, and I am planning to get trained in it. The institute covers MSSQL, Power BI, MSBI, Tableau desktop 10, Informatica developer and admin, Azure data factory, SQL DBA, QLIK sense, Data warehousing. Is this all that is needed? Reply : Regarding getting into BI the best way is to get into a team where you can learn from experienced people in the team. Regarding courses just based on the list that's quite a lot for a training, but it depends on what exactly is provided as part of the training. Need more details about the training. Then once the link was provided, I had a check and provided below reply. Note - Link or institute name is not shared here intentionally to avoid any confusion that I am promoting any institutes.  Reply:  I have had a look into the link, and from the looks of i
Read more

Corona Virus Analytics

This is an interactive dashboard that can be used to find trends, patterns and eventually insights related to COVID-19 (Corona Virus) from a reliable data source. Additional notes provided below the dashboard. Direct link to the Google Data Studio Dashboard is here - https://datastudio.google.com/u/0/reporting/2e4f9914-8905-4cc9-b36a-75dc3f220438/page/7JbY Best viewed on desktop/laptop Notes : This is an interactive dashboard that can be used to find trends, patterns and eventually insights related to COVID-19 (Corona Virus) You can use the filters on the top or click on the charts to filter. Feel free to embed the dashboard in your websites (of course for non commercial use). Currently dashboard doesn’t have real-time information. There is a delay of about 24 hours as the data is mainly sourced from https://github.com/CSSEGISandData/COVID-19/tree/master/csse_covid_19_data ( 2019 Novel Coronavirus COVID-19 (2019-nCoV) Data Repository by Johns Hopkins CSSE). For some o
Read more