What is OpenTelemetry? The future of instrumentation

What is OpenTelemetry? 

OpenTelemetry is an open source project and unified standard for service instrumentation, or a way of measuring performance. It provides a single application programming interface (API), libraries, integrations, and a software development kit (SDK) for programming languages including Java, Go, and Python. DevOps teams that use the standardized API don’t need to worry about code changes when switching between SDKs. This saves time, reduces effort, and streamlines performance enhancement and troubleshooting activities.

Sponsored by the Cloud Native Computing Foundation (CNCF), OpenTelemetry replaced OpenTracing and OpenCensus. The goal is to standardize how you collect and send telemetry data to a backend platform.  It defines a centralized collector service that you can use for collecting telemetry data from your applications and services. It also includes exporters to send that data to the observability platform that you choose.

OpenTelemetry covers all three aspects of monitoring software: implementing the instrumentation, capturing data, and exporting the information for analysis. It’s also vendor neutral. Development teams don’t need to manually create monitoring solutions if they want a vendor neutral solution. With OpenTelemetry, you aren’t forced to use the solution provider's instrumentation libraries to meet client specifications.

What is telemetry data?

Telemetry data is the data collected from remote or inaccessible devices and transmitted to a central location for monitoring, analysis and reporting. This data is typically collected from various sources, such as sensors, software applications, and other monitoring devices, and is used to monitor the performance, behavior, and usage patterns of systems and applications.

Telemetry data is crucial for several reasons:

1. Performance monitoring: telemetry data provides real-time insights into resource usage, response times, and other performance metrics, allowing for proactive maintenance and optimization.
2. Issue detection and troubleshooting: telemetry data analysis helps teams detect issues, errors, and anomalies in systems and applications. 
3. Capacity planning: by understanding current usage patterns and trends, IT administrators can make informed decisions about resource allocation and infrastructure upgrades.
4. Security monitoring: telemetry data is used to detect and respond to security threats, breaches, and unauthorized access attempts.
5. User experience improvement: telemetry data provides insights into user behavior and preferences. This information is valuable for developers and product teams to enhance user experience and optimize their products based on real user data.

Compliance and Reporting: organizations must collect and store specific data to comply with industry regulations and standards. Telemetry systems can automate the process of data collection and reporting, ensuring compliance with relevant laws and regulations.

The most valuable OpenTelemetry use cases

Teams find OpenTelemetry the most valuable for three priority use cases:

• You have data across multiple platforms
• You need vendor neutrality
• You want to optimize valuable team hours

In this blog post, you get an overview of what OpenTelemetry is and why this open standard is the future of instrumentation.  Let's start with a video:

Benefits of OpenTelemetry

1. Unified observability: OpenTelemetry provides a unified approach to observability by offering consistent APIs and instrumentation libraries for multiple programming languages. 
2. Distributed tracing: OpenTelemetry enables distributed tracing, allowing developers to track requests as they traverse through different services and components of a distributed application. 
3. Metrics collection: OpenTelemetry supports the collection of various metrics, such as request rates, error rates, and resource usage metrics.
4. Context propagation: OpenTelemetry supports context propagation, ensuring that contextual information, such as trace and span IDs, is passed between different services and components. 
5. Plugin ecosystem: OpenTelemetry has a rich ecosystem of plugins and integrations that allow users to export telemetry data to various backends and observability platforms. 
6. Vendor-neutral: Being an open-source project, OpenTelemetry is vendor-neutral, meaning it is not tied to any specific vendor or technology stack. 
7. Community-driven innovation: This collaborative environment encourages innovation, leading to the continuous improvement of the framework and its capabilities.
8. Standardization: By adopting OpenTelemetry, organizations can adhere to a widely accepted set of practices and standards for collecting telemetry data, ensuring consistency and compatibility across their applications and services.

Why OpenTelemetry is the future of instrumentation

As described in the video, there are four reasons why OpenTelemetry is changing instrumentation:

1. You have access to a broad and deep set of instrumentation, because it is built and supported by a large open source community. More than 300 companies contributed to the OpenTelemetry project in the past year, with more than 100,000 contributions. And the OpenTelemetry community had more than 2,000 contributors over the past year. As an open standard, OpenTelemetry encourages broad collaboration and promotes better coverage, flexibility, and ubiquity as engineers from all over the world contribute to the instrumentation.  
2. The standard instrumentation means that different tools and services operate together, so you get visibility for your entire system. OpenTelemetry is interoperable, which means that various systems can use this standard form of instrumentation across all services to exchange information. Without a standard instrumentation across all services of your application, you have  blind spots or data silos that affect troubleshooting and reduce your ability to quickly detect and resolve issues. OpenTelemetry bridges gaps in visibility.
3. You have flexibility to change your backend observability tools. Most proprietary monitoring solutions require you to install specific instrumentation agents to gather telemetry data. If you switch to a different tool, you have to spend more time and resources installing different proprietary agents or re-instrumenting your code. With OpenTelemetry, your path is independent from the various proprietary tools. You can change your backend observability tool without having to change your instrumentation. You have full visibility into the OpenTelemetry source code and insight into how the community develops features and fixes bugs.
4. You can monitor emerging technologies easily.  When new technologies are built, OpenTelemetry instrumentation becomes a part of libraries and frameworks that your applications are already using. This means you don't need to wait for individual monitoring tools to support the latest technologies.

The necessity to make orchestrated systems like Kubernetes more understandable and application performance monitoring (APM) more transparent are two of the several development threads that gave rise to the discipline of observability. The goal of observability is to offer rich context and incredibly granular insights into how systems behave. 

Why you need OpenTelemetry for your observability

Two main software development threads that gave rise to the discipline of observability are the need to make orchestrated systems like Kubernetes more understandable and the need to make application performance monitoring (APM) more transparent. The goal of observability is to offer rich context and incredibly granular insights into how systems behave. 

For observability, you need: 

• to create systems and applications that collect relevant data like metrics, traces, and logs 
• to be able to easily or automatically export this information to a platform outside of the program you are analyzing 
• the analysis to provide visualizations and insights into the system as a whole, including how the end user experience is affected by changes in the development of the application

Observability provides holistic monitoring of systems and platforms for developers and stakeholders to determine how they are functioning. But observability is hard to measure and analyze without help. Tools like OpenTelemetry streamline performance monitoring across vendors and languages and help devs watch observability metrics and report on them. To determine if OpenTelemetry is right for your product, developers should ask: 

• What programming language, platform, and frameworks are we using?  
• What types of data are we looking to analyze? 
• What analytical tools will we be pushing the data to for analysis? 

OpenTelemetry needs to combine with observability backends to function at its best. When parts of OpenTelemetry’s Collector (the receivers, processors, and exporters) take in, compile, and distribute meaningful data and information in a format that can be analyzed properly for performance and troubleshooting purposes, developers are able to access full observability and monitoring in one tool.

New Relic and OpenTelemetry

New Relic has been one of the major contributors to OpenTelemetry since its inception. In 2020, we announced our full commitment to open source instrumentation including open sourcing our agents and standardizing our observability offerings on OpenTelemetry. With our native OpenTelemetry protocol (OTLP) support and curated user experiences,  you can use this new standard of instrumentation to understand, troubleshoot, and optimize your systems.