{"id":45396,"date":"2021-09-29T12:10:05","date_gmt":"2021-09-29T16:10:05","guid":{"rendered":"https:\/\/sdtimes.com\/?p=45396"},"modified":"2021-10-07T11:37:18","modified_gmt":"2021-10-07T15:37:18","slug":"why-opentelemetry-is-driving-a-new-wave-of-innovation-on-top-of-observability-data","status":"publish","type":"post","link":"https:\/\/sdtimes.com\/monitor\/why-opentelemetry-is-driving-a-new-wave-of-innovation-on-top-of-observability-data\/","title":{"rendered":"Why OpenTelemetry is driving a new wave of innovation on top of observability data"},"content":{"rendered":"

The last decade has brought a progressive transition from monolithic applications that run on static infrastructure to microservices that run on highly dynamic cloud-native infrastructure. This shift has led to the rapid emergence of lots of new technologies, frameworks, and architectures and a new set of monitoring and observability tools that give engineers full visibility into the health and performance of these new systems.\u00a0<\/span><\/p>\n

Visibility is essential to ensure that a system and its dependencies behave as expected and to identify and speed resolution of any issues that may arise. To that end, teams need to gather complete health and performance telemetry data (metrics, logs, and traces) from all those components. This is accomplished through <\/span>instrumentation<\/b><\/a>.<\/span><\/p>\n

Why do we need OpenTelemetry?<\/b><\/p>\n

For many years there have been a wide variety of open-source and proprietary instrumentation tools like StatsD, Nagios plugins, Prometheus exporters, Datadog integrations, or New Relic agents. Unfortunately, while there are lots of open-source tools, there hasn’t been alignment about <\/span>specific instrumentation standards<\/span><\/i>, such as StatsD, in the developer community and between vendors. This makes interoperability a challenge.<\/span><\/p>\n

The lack of instrumentation standards and interoperability has required every monitoring and observability tool to build their own collection of integrations to instrument the technologies developers use and need visibility into. For example, many monitoring tools have built integrations to instrument widely used databases like MySQL, including <\/span>Prometheus MySQL Exporter<\/span><\/a>, <\/span>Datadog MySQL integration<\/span><\/a>, and <\/span>New Relic MySQL integration<\/span><\/a>.\u00a0<\/span><\/p>\n

This is also true for application code instrumentation, where New Relic, Dynatrace, Datadog and other vendors have built complex agents that automatically instrument popular application frameworks and libraries. Developers spend years building instrumentation, and it requires a sizable investment to build a large enough catalog of integrations and maintain it as new versions of the technologies monitored are released. Not only is this a very inefficient use of global developer resources, it also creates vendor lock-in since you need to re-instrument your systems if you want to change your observability tool.\u00a0<\/span><\/p>\n

Finally, the value of (and where customers most benefit from!) innovation is not innovation on the instrumentation itself. It’s improvements and advancements on what you can<\/span> do<\/span><\/i> with the data that gets collected. The requirement to make a large investment on instrumentation – i.e., the area that delivers little benefit to end users – for new tools to enter the market has created a big barrier to entry and has severely limited innovation in the space.<\/span><\/p>\n

This is all about to dramatically change, thanks to <\/span>OpenTelemetry<\/span><\/a>: an emerging open-source standard that is democratizing instrumentation.\u00a0<\/span><\/p>\n

OpenTelemetry has already gained a lot of momentum, with support from <\/span>all major observability vendors, cloud providers, and many end users<\/span><\/a> contributing to the project. It has become the <\/span>second most active CNCF project<\/span><\/a> in terms of number of contributions only behind Kubernetes. (It’s also <\/span>recently been accepted as a CNCF incubating project<\/span><\/a>, which reiterates its importance to engineering communities.).<\/span><\/p>\n

Why is OpenTelemetry so popular?<\/b><\/p>\n

OpenTelemetry approaches the instrumentation “problem” in a different way. Like other (usually proprietary) attempts, it provides a lot of out-of-the-box instrumentation for application frameworks and infrastructure components, as well as SDKs for developers to add their own instrumentation.<\/span><\/p>\n

Unlike other instrumentation frameworks, OpenTelemetry covers metrics, traces, and logs, defines an API, semantic conventions, and a standard communication protocol (OpenTelemetry protocol or OTLP). Moreover, it\u00a0is <\/span>completely vendor agnostic, <\/span><\/i>with a plugin architecture to export data to any backend.\u00a0<\/span><\/p>\n

Even more, OpenTelemetry’s goal is for developers who build technologies for others to use (e.g., application frameworks, databases, web servers, and service meshes) to bake instrumentation directly into the code they produce. This will make instrumentation readily available to anyone who uses the code in the future and avoid the need for another developer to learn the technology and figure out how to write instrumentation for it (which in some cases requires the use of complex techniques like bytecode injection.)<\/span><\/p>\n

OpenTelemetry unlocks a lot of new value to all developers:<\/span><\/p>\n

    \n
  1. Interoperability.<\/b>\u00a0Analyze the entire flow of requests to your application as they go through your microservices, cloud services, and third party SaaS in your observability tool of choice. Effortlessly send your observability data to a data warehouse to be analyzed alongside your business data. OpenTelemetry’s common API, data semantics, and protocol make all of the above – and more – possible, out-of-the-box.<\/span><\/li>\n
  2. Ubiquitous instrumentation<\/b>. Thanks to a much larger community working together vs. siloed duplicative efforts, everyone benefits from the broadest, deepest, and highest quality instrumentation available.<\/span><\/li>\n
  3. Future-proof<\/b>. You can instrument your code once and use it anywhere since the vendor-agnostic approach enables you to send data to and run analysis in your backend of choice. Before OpenTelemetry, changing observability backends typically required a time-consuming reinstrumentation of your system.<\/span><\/li>\n
  4. Lower resource footprint<\/b>. More and more instrumentation is directly baked into frameworks and technologies instead of injected, resulting in reduced CPU and memory utilization.\u00a0<\/span><\/li>\n
  5. Improved uptime<\/b>. With OpenTelemetry’s shared metadata, observability tools deliver better correlation between metrics, traces, and logs, so you troubleshoot and resolve production problems faster.\u00a0<\/span><\/li>\n<\/ol>\n

    More importantly, companies no longer have to devote time, people, and money to developing their own product-specific instrumentation and can focus on improving developer experience. With access to a broad, deep, and high-quality observability data set of metrics, traces, and logs with no multi-million dollar investment in instrumentation, a new wave of new solutions that leverage observability data is about to come.\u00a0<\/span><\/p>\n

    Let’s look at some examples to demonstrate what OpenTelemetry will – and is already – enabling developers to do:<\/span><\/p>\n