Observability - a way to monitor system behavior based on the signals it emits.

For the past few years, one standard has dominated the market and keeps gaining wider adoption in APM services: OpenTelemetry.

OpenTelemetry provides not just a specification, protocols, or naming conventions. It's also a collection of libraries and SDKs enabling "easy" adoption in virtually any system.

So why did I decide to write my own library instead of using existing solutions?

It all started when I wanted to integrate Flow PHP's DataFrame (along with a few libraries like postgresql and filesystem) with OpenTelemetry.

The first problem I ran into was dependencies. The OpenTelemetry SDK requires:

• php: ^8.1
• ext-json: *
• nyholm/psr7-server: ^1.1
• open-telemetry/api: ^1.7
• open-telemetry/context: ^1.4
• open-telemetry/sem-conv: ^1.0
• php-http/discovery: ^1.14
• psr/http-client: ^1.0
• psr/http-client-implementation: ^1.0
• psr/http-factory-implementation: ^1.0
• psr/http-message: ^1.0.1|^2.0
• psr/log: ^1.1|^2.0|^3.0
• ramsey/uuid: ^3.0 || ^4.0
• symfony/polyfill-mbstring: ^1.23
• symfony/polyfill-php82: ^1.26
• tbachert/spi: ^1.0.5

Considering that Flow PHP (in its base version) only requires:

• php: ~8.3.0 || ~8.4.0 || ~8.5.0
• composer-runtime-api: ^2.0
• ext-json: *
• brick/math: ^0.14.2
• flow-php/array-dot: 0.32.0
• flow-php/filesystem: 0.32.0
• flow-php/types: 0.32.0
• psr/clock: ^1.0
• psr/simple-cache: ^1.0 || ^2.0 || ^3.0
• symfony/string: ^6.4 || ^7.3 || ^8.0
• webmozart/glob: ^3.0 || ^4.0

The number of dependencies would practically double. For a framework that users need to add to their systems' dependency list, this becomes somewhat problematic due to potential version conflicts.

Theoretically, I could make Flow depend only on open-telemetry/api and add the SDK as an optional dependency. But when I started looking deeper into how the API is built, I realized it wasn't compatible with the architecture I adopted in Flow.

The main issue for me was all the global states, singletons, and other mechanisms designed to make auto-instrumentation easier. Example: OpenTelemetry Globals.

I could probably try to work around Globals, but looking at instrumentation implementations based on OpenTelemetry, I'd quickly run into problems - the SDK is very tightly coupled with the API, and the API is very tightly coupled with global state.

Here I had to make a choice. Either separate Flow's code using an additional abstraction, or build my own API that would also serve as an abstraction for telemetry.

Given that the API itself isn't that complex, I decided to build an independent API and transport layer that comply with the OTLP protocol. And so a tiny library called flow-php/telemetry was born.

The flow-php/telemetry API is very non-invasive, doesn't rely on global states, depends mainly on contracts, and provides very lightweight InMemory and Void implementations useful in tests.

But the API alone isn't enough to send signals anywhere. We need a serializer and transport for that (this is what the OpenTelemetry SDK does).

Flow provides something very similar: flow-php/telemetry-otlp-bridge.
This library's job is to:

• Serialize signals according to the OTLP protocol
• Send signals to OpenTelemetry Collector

flow-php/telemetry-otlp-bridge can serialize signals to:

• JSON
• Protobuf - requires the protobuf extension

Signals can be transmitted via:

• curl - built-in fully asynchronous transport
• gRPC - requires the gRPC extension
• HTTP - requires psr/http-client and psr/http-factory implementations

What are signals?

Metrics - numerical data about system state, e.g., requests per second, average response time, etc.

Traces / Spans - represent individual operations in the system, along with their attributes, duration, etc.
A single Trace is actually a collection of Spans that are linked together, forming a tree of operations.
On top of that, a Trace can be linked to metrics, logs, or other Traces, providing a fuller picture of what's happening in the system.

Logs - textual data about events in the system, often containing additional attributes like log level or event context.

One of the main benefits of using OpenTelemetry is auto instrumentation.

According to the design, library authors (like myself) should integrate their tools with the OpenTelemetry API/SDK. This way, if we configure serialization and the transport layer in a system using such libraries, we automatically get basic telemetry without any extra effort.

If the PHP community shared the vision of the API/SDK creators - in almost every project you'd just need to configure the SDK to get insight into how the system works in production.

As of now, however, there isn't a single library that's natively integrated with OpenTelemetry.

I'm not sure why that is. Environments like Java or .NET have much better OpenTelemetry adoption than PHP.

Maybe if the OpenTelemetry SDK were more popular, I wouldn't have decided to write my own library.

Nevertheless, slowly but surely I've started integrating all the libraries I maintain with flow-php/telemetry.
I've also created several extensions for existing frameworks/libraries - enabling equally effective auto instrumentation.

Libraries with native integration:

• flow-php/etl
• flow-php/filesystem
• flow-php/postgresql

Extensions for existing libraries:

• flow-php/psr7-telemetry-bridge
• flow-php/psr18-telemetry-bridge
• flow-php/monolog-telemetry-bridge
• flow-php/symfony-http-foundation-telemetry-bridge
• flow-php/symfony-telemetry-bundle
• flow-php/phpunit-telemetry-bridge

OTEL Collector is a fantastic tool serving as a central hub for receiving / processing / forwarding signals.
Placing the collector between your system and APM (Application Performance Monitoring System) helps you avoid vendor lock-in.

We also avoid unnecessary, blocking I/O operations. Our signals are buffered in memory, and when the buffer fills up or the script ends, they're sent to the collector.

Below are example collector configurations from the Flow monorepo:

• docker compose
• collector config

There are many popular solutions on the market for monitoring systems compatible with the OTLP protocol.
These include both paid and completely free tools, available as services or open source. Before choosing the right APM, though, it's worth getting familiar with the OTLP protocol itself.

Of course, most APMs provide their own libraries for monitoring systems, but using them ties you very tightly to a specific vendor.

My preferred solution (at least for now) for local development is Aspire

Aspire is also configured as the APM in the Flow monorepo. You can find the configuration example here.

You can find flow-php/telemetry code examples at https://flow-php.com.
Each of them can be run in the browser using Flow's Playground.

But there's no better way to learn a tool than getting your hands on it, so I encourage everyone interested to set up Aspire, OTEL Collector locally, and add flow-php/telemetry along with flow-php/telemetry-otlp-bridge.

Then all that's left is to start emitting signals!

Below is a screenshot from local Aspire showing telemetry collected during the entire Flow monorepo test execution.