XOps is emerging as a key automation strategy to leverage business value from data and AI/Machine Learning workflows into the broader enterprise technology package. XOps has been named one of the hottest data and analytics trends for 2021 by Gartner, examples include DataOps, MLOps, DevOps, BIOps, PlatformOps, and ModelOps. As many tools and platforms have evolved to solve specialized problem areas, they have been somewhat separated, creating further confusion for data leaders to identify the tools and platforms needed to support their overall business needs.
In simple terms, XOps can be divided into ‘X’ related to data, infrastructure, business intelligence (BI), and machine learning (ML) models, and ‘Ops’ being automation via code. The single component has been around for years, but the difference now is that they are interconnected to drive flexibility and innovation by removing silos.
Data, BI, and ML Ops are the primary focus of this article because they are linked together to create business value. Considered as functional pillars, DevOps is the interconnected glue that ensures consistency of tools, security, and governance requirements to drive continuous integration (CI), continuous development (CD), and continuous training (CT).
What do Data, BI, and MLOps have in common?
Organizations build their strategy around data and technology, and information technology is the main driver of those initiatives. They collaborate with different business functions to gather requirements to implement BI and ML solutions. The adoption and success of these solutions is often measured by the accuracy and timeliness of the data products. The usage and consumption pattern of data products such as users, queries, frequently used tables, and attributes is often overlooked. In addition, lineage display is only partially complete without metadata from BI and ML applications. Metadata integration from these tools is required with DataOps to provide complete visibility and provide governance requirements for data sources, business rules, consumption, users, access levels, PII data, business owner, typical endpoints, and potential data leaks.
As organizations continue to adopt the best solutions as part of their transformation journey, implementation and technology migration are essential components to consider. Having an integrated metadata and a unified observation solution allows them to quickly define a migration roadmap and assess business impact without investing significant resources for a months-long discovery process.
What is the difference between data, BI, and MLOps?
DataOps focuses on reading structured data from SaaS applications and databases and unstructured data from documents and images to provide business and technology metadata. In addition, it provides formatting and transformation capabilities to create a reliable data store. It is a base layer for downstream BI and ML applications. Many organizations have created ETL/ELT tools, and in these cases, pedigrees can be recorded in DataOps via API integration. It should be a strategic asset to provide reliable and verified data in the data market, enabling data citizens to demand access to the tools of their choice. The primary users of the tools are business analysts, data administrators, and data engineers. The metadata captured during the data transition from the initial format to the trusted format are entities, tables, attribute layout, matching rules, business and technical contact, business glossary, data quality rules, quality score, and runtime users.
BIOps focuses on coordinating data into a logical structure compatible with a tool, and building reports and dashboards by BI developers and business users. Most DataOps tools provide native integration with BI tools and pull metadata such as tables, queries, runtime username, metric definition, attributes, derived rules, and number of records.
Machine learning engineers and data scientists use MlOps tools. Data is orchestrated into a structure via feature engineering for model development, training, and running experiments. Metadata built into DataOps tools includes model names, registered models, model versions, experiments, metrics, outcomes, and endpoints.
Are DataOps and DevOps the same?
They differ in terms of tools, skill set, and focus. However, DevOps’ flexibility must be applied to quickly build, test, deploy, and add value to users in the data ecosystem. Ideally, both teams should be part of the same organization or need stakeholders from both teams to be aligned with product and/or project objectives. Most organizations have mature CI/CD processes to handle infrastructure provisioning, user access, and security controls configurations to include business application changes such as workflows, artifacts, and metadata scripts. Application level change management scripts can be called from Git, script using Terraform, Ansible, or other scripts. Each implementation provides a code base and testing plan to ensure changes are applied to a higher environment, validated correctly, and rollback without data loss if something goes wrong during deployment.
What are the potential barriers to adoption?
In most parts, people and processes are the main drivers of non-adoption. Alignment of resources across organizations and stakeholder endorsement is not part of the initial strategy, resulting in a lack of commitment if these stakeholders are involved in the engagement or program at a late stage. Ideally, every data leader wants to own the budget, build a team, and take responsibility for overall program and product delivery, but in reality, the key stakeholders may be IT Security, Governance, and the DevOps team. Matching key stakeholders, active participation from those teams, and clarity of roles is critical to success.
Organizational maturity in embracing change at scale, driving cultural change, and hiring people who have done it for a living are all critical to a successful product launch.
Establishing data literacy is an essential part of successful adoption. Data literacy includes user preparation, training (in person or on demand), documentation, short how-to videos, and documentation. It’s a team that needs to stay focused on creating and delivering content to users in every product release. In most cases, the documentation is outdated, and there are no references due to lack of bandwidth from the development team. User documentation should be part of the sprint plan for every developer and reviewer.
Create a central tool that can be considered as a home tool for business users to get answers to most of their questions and move to other apps for detailed analysis with a single login, helping to ensure a seamless experience.
Define a clear role for each tool in the broader data system. Each tool must be determined by aligning business value with new revenue, cost savings, process efficiencies, and compliance requirements. Tool interoperability is part of the basic design and choice of tools to drive an efficient and mature data organization.
In essence, XOps reinforces the idea that different development teams are cross-functional and work with each other. As data solutions, business intelligence, and machine learning increasingly become core business functions, silos of information that previously served as barriers to the data-driven enterprise may begin to collapse.