EP 50: Evolution of Enterprise Data Engineering in Gen AI Era

Soumava Dey (1:00)

The teams right now are running two parallel worlds, a legacy ETL system that's holding the business together, and a modern cloud platform that's supposed to replace it, but hasn't quite yet.

The gap between those two worlds is where data engineering actually lives today. And today's guest has been navigating that gap across three of the most demanding domains in enterprise data.

Banking, healthcare and supply chain. Welcome to Data Science With Sam. I'm your host, Sam. Today I joined with a guest, Shashank Akinapalli. He's a senior data engineer at UNFI in Providence, Rhode Island, where he leads Cloud Data Lakehouse Initiative and real-time streaming architectures for supply chain intelligence. Shashank has 14 years of experience across IBM DataStage, Databricks, Snowflake, DBT, AWS, and Spark-based ecosystems. And he has also worked with USAA in banking and Blue Cross Shield in healthcare before UNFI. Shashank holds a master's in digital sciences from Ken State University and a B.Tech from GITM University. Shashank, welcome to the show. I'm very happy to have you here.

Shashank Akinapalli (2:26)

Thanks, Tom, and thanks for giving me this opportunity to introduce myself today.

Soumava Dey (2:33)

So let's start with the migration problem, because I think this is the reality for the majority of the enterprise data teams right now facing. It's not a green field, but the legacy modernization.

So, Shashank, you have worked with IBM DataStage at Enterprise Skill, and now lead Spark-based pipeline modernization at UNFI. What's the biggest architectural shift you make when migrating a legacy ETL system to a modern lake house, and what breaks most often during the transition that nobody wants you about? And the reason I'm asking this question is because I'm representing a vast audience of technical staff who would love to know more about that.

Shashank Akinapalli (3:19)

Yes, so definitely, Sam. So basically, you know, there is a lot of change from migrating, a lot of companies are migrating from legacy applications to more of a cloud-based architecture these days. So the main reason why companies are migrating from a legacy-based architecture or the legacy-based ETL application. So previously, we used to have IBM DataStage, or it could be Informatica. So they are more of a batch-based data processing system. So they are used primarily for integration, and they are used primarily for data ingestion into the enterprise data warehouse system. But they're usually, they are batch-oriented, where the process runs in the late night to pull the data for the current day. But now the data has changed a lot, and the volume of data has become more, very huge volume. So with the business requirements, we need to have a system in place, where we can read the data simultaneously, continuously to make a better decision. So it's not just about migrating the ETL jobs, but it's also about how to trust the data, whatever we consume, what is the trustability of the data, and how we can leverage that to make a better decision. So that was one of the reasons where companies are adopting to this Cloud-based technology because it has more robust and more dynamic pipelines, where we can get the data in real time to view the data on the dashboard sides for doing any forecast analytics, such kind of metrics in order to capture them in a real time. So that's how the companies are migrating to a Cloud-based ETL systems rather than the job-oriented or the batch system.

Soumava Dey (5:00)

Got it, got it. So if I understand correctly, so the technical migration is often the easier half. And the harder, the actual, the hard heavy lifting is reconstructing the business knowledge that was encoded in the original pipeline.

So by someone who kind of like lived years ago, right? And now somebody has to deal with that. So you kind of like, you know, kind of like going into that particular, you know, the tangent. You work across both Snowflake and Databricks in your stack.

For a data engineering team standing up a new platform, how do you decide which workloads belong where? And where does DBT fit in that picture? Because the reason I picked Snowflake and Databricks, because those are the most popular data warehouse platforms and they are becoming widely popular in every industry of articles.

Shashank Akinapalli (6:02)

Yes. So, as you said, that was a true statement, Sam. So basically, right now, in the industry, so primarily they were using Databricks or Snowflake with DBT. And the decision, you know, the decision, what are the circumstances that make the decision to choose whether Databricks or Snowflake is based upon what kind of data that we're going to be using. So, as I said, if we have a heavy data that is coming, where you need to do a lot of detailed affirmation, and where you're dealing with any semi-structured data, unstructured data, in such cases, we use Databricks, because Databricks, it is used by data engineering team, data scientists for doing a lot of Python-based code in order to interpret the data.