Navigating My Shift from Database Developer to Data Engineer: A Tale of Transformation
Audience — Database Developer, ETL developer, SQL
We all lacks consistency, and its common because if you are working and have your experience of more than 2–3 years then we all are loaded with so many office works. But this should not become a roadblock for our career. Then what is the solution?
It is very simple, plan a career that is related(at least 20–30%) to your existing job but is highly paid with future.
In our case, l researched a lot and finds out we have two best options available.
Data scientists and Data engineer.
- Data scientists — Every single organisations today or tomorrow needs Data scientists. You get a very good pay with good future options.
Prerequisites — Mathematics and statistics, Domain knowledge, Programming skill and Soft skill.
If you have not switched multiple organisations in your current company have a good domain knowledge. Then you are 40% ready. Because Mathematics, statistic is something you can practice with time. And you are already working with sql and working in same organisation then soft skill and inter-communication can be improved with time and you have good basic knowledge to start. But you have to be very sure that which domain has good future and can you handle it?
2. Data Engineer — This field has a very promising future with good pay scale.
According to StitchData, every year data engineering jobs double that of the previous year. In the USA the salary of a data engineer is $116k which is comparatively higher than other jobs. And it is welcoming in many many ways.
You lacks Domain knowledge, strong SQL skills, Good communication and consistency in jobs? No problem. In my case I have switched through many organisations and clients within small span. Because of this, I was lacking strong Domain knowledge and inter company communications. But strong points was I have good knowledge of data and have exposure to many tools and technologies.
Time to cover Previous role to Senior Data Engineer journey
My World of Database Development (MSBI and Power BI):
In my early career days, SQL was my daily language, and database schemas were my playground. As a database developer, I was immersed in managing and optimizing structured data. I started my career with Microsoft stack. Life was very simple and sorted. I was creating SSIS packages to get data from many sources to Data warehouse. Wring SQL Queries to apply business logics and creating Power BI reports.
After doing the same things for 4–5 years in many companies, I have reached to a saturation level in term of Package and knowledge. But I have few EMIs running and have family resposibilites then cannot take risk of changing job in new stack just by changing resume and finishing 2–3 Udemy courses. If I am joining any new team as a Senior on high package then will have to present a lot of skills. Then What is the solution?
I started reading blogs and watching Youtube videos but found nothing except fears and some fancy words like Pipeline, clouds, etc
The Lure of Big Data:
The swift began with only one thing in my mind. That i need to switch myself to money making technologies without disturbing existing job and money. But i was worried a lot how I am going to handle unstructured data processing, real-time analytics, and scalable data solutions.
Practical Tip: “Start by getting your hands dirty with big data tools. I began with simple Hadoop tutorials online course.”
Overcoming Challenges:
The transition was not without its hurdles. The first challenge was the reading consistency. Taking udemy course and finishing it was not working in my case.
Online tutor — I realised that I need a live guru who can teach me and removes the fear of Big data. then i joined a online tutor based course and took class for 3 months. It was not a continuous classes as office work was also increasing sometime and my focus was keep shifting. But in back of my mind I was very clear that i will take 6 months to transform my career.
My online tutor was very good with practical knowledge and he took a normal sql query and how and why we need to make it big data query.
2 month update — After having around 14 classes, I realised that I am just learning lot of new words and technologies but not gaining confidence. I learned about Hadoop, Hive, Spark, Scala, etc. This was the time when I was lossing my confidence.
4 months update — In office appraisal cycle, I got only 9% hike. I was not happy with my package. And in my study, i was keep using new technologies words again and again that after the fear, it started taking space in my mind. Now my curiosity were increasing. Thanks to my teacher as he was focusing more on practical knowledge and to company for less hike.
5 months update — After finishing small tasks with online course now i came in youtube and blog world. I purchased a new fast laptop and created a ecosystem for Big data with subscriptions on full scale. After spending full month I realised that by this approach i can not become a seniors developer. I was just working on very basics.
Real job Guy on Linkedin — Books, videos, demo projects, etc are not enough to make you senior developer in high speed and very fast changing industry. You actually need to test yourself with some industry projects. I used Linkedin and started reaching to senior data engineers , i sent out more than 500 requests and from them i personally reached out to some of them and offered money so that they can show me actual projects with day to day activities and challenges. I struggeled here for a month but it becomes the turning point for my in journey. As my basics were already ready so I was able to understand it whole very fast.
