Harris’ survey of 200 senior engineering professionals indicates that it may take some time before most DevOps teams are able to achieve real observation in their IT environments.
The survey was conducted on behalf of LogDNA, a provider of the monitoring platform, and found that nearly three-quarters (74%) of respondents struggle to meet their observational goals despite investing hundreds of thousands of dollars. In fact, more than a third (38%) said they already invest $300,000 annually on monitoring tools.
Despite these investments, less than half of respondents are fully satisfied with their ability to use log data. Key areas of frustration include the challenges of collaborating with colleagues on multiple teams (67%), the fact that tools are not easy to use (66%) and routing security events (58%).
However, most survey respondents said they were optimistic about the possibility of the observation. 85% of participants said they believed real observation was possible.
Tucker Calaway, CEO of LogDNA, said that a fundamental issue that many organizations fail to realize is that observability is really a challenge to data literacy and management. Organizations collect logs, metrics, and tracks from a wide variety of tools and platforms without thinking about creating the optimal data pipeline. The result, he said, is the aggregation of data that has not been normalized to simplify the emergence of practical intelligence. In fact, many organizations are quickly discovering that all they’ve really accomplished is increasing data storage costs, Callaway said.
In the absence of data normalization, Callaway noted that it becomes very difficult for cross-functional teams to query all the data collected within the repository.
In essence, monitoring capability promised to make it easier to spot IT issues before they cause a disruption. Observability in one form or another has always been one of the core principles of DevOps best practice. Initially, DevOps teams focused on continuous monitoring as the most effective way to proactively manage application environments. However, it can take days, sometimes weeks, to discover the root cause of the problem. Monitoring platforms allow events to be linked in a way that makes it easier for analysis tools to identify anomalies that may be indicative of the root cause of an IT problem.
In contrast, traditional IT monitoring focuses on predetermined metrics to determine when a particular platform or application is performing within expectations. Monitoring systems combine metrics, logs, and traces—a specialized form of logging—to make it easier for IT professionals to interrogate data generated by a wide range of DevOps tools and platforms.
Of course, the assumption is that DevOps teams already know which queries to launch to determine the root cause of an IT problem. In the long term, machine learning algorithms should facilitate the identification of anomalies within IT environments. In the meantime, however, achieving observability still requires a fair amount of DevOps expertise that isn’t always readily available.
Eventually, the possibility of observation will become much more automated than it is today in general. When this transition occurs, IT teams should find that the ROI for observability is steadily improving.