Incorporating all of your data from different sources is the first step in turning it into business value, according to Informatica. The company recently held a special webinar that focused on the leading trends in modern data engineering and integration.
Part of the Back to Basics: Data Integration webinar series, Makesh Renganathan, Principal Product Manager, R&D cloud, Informatica, and John O’Brien, Principal Consultant and CEO, Radiant Advisors, discuss what’s next at XOps (DataOps, MLOps, etc.), data texture effect, standalone data integration and serverless processing, and more.
Data lake building architecture is booming via Databriks and others. O’Brien explained that this next-generation technology delivers a standard open data format, supports multiple cloud computing engines, maintains consistency between users and more.
“We continue to hear from our customers about the data texture and the data network,” said Ringanthan.
He explained that the data fabric and data network democratize data and provide consistent data services across the enterprise. This improves productivity and governance.
As an industry analyst who monitors this space, O’Brien notes that he sees the same interest in the data fabric and the data network.
“As all organizations deal with more data and more integration, architecture is shifting to a data network strategy,” O’Brien said. “It’s a result of what companies are dealing with.”
A data warehouse or data lake architecture supports data catalogs and data governance. This type of architecture supports data discovery, ratios, and glossaries.
Independent data integration is another integration trend that O’Brien and Ringanathan are leading this year. Active, metadata-based smart self-data management “understands” data ratios, improves data quality, and enables self-service data integration.
“[People are] Go to data pipelines and deconstruct things to understand the difference between ingestion and processing patterns,” O’Brien said.
ETL and ELT processing are additional data processing methods that companies can use to transform data. O’Brien noted that both have their benefits and should be taken into account in different situations.
The architecture of the event flow increases the opportunity to take advantage of the advantages of machine learning and artificial intelligence. This type of platform can reuse segments of data pipelines to increase speed and quality.
O’Brien and Rinjathan noted that DataOps is a journey for express delivery teams to increase customer analytics production. Constant shortage of talent leads to tool/platform adoption for faster development and training, and MLOps are needed as ML models can be trained and deployed faster throughout the organization.
An on-demand archived replay of this webinar is available here.