Native support for gRPC in Grafana remains a gap
The demand for real-time visualization of gRPC streams is paramount, especially considering its increasing adoption in IoT applications. This plugin effectively bridges the technological gap, ensuring that the insights extracted from expansive IoT networks are promptly visualized, thereby enhancing system observability and enabling timely interventions.
In the complex realm of real-time data streaming, where precision and speed are essential, our Grafana plugin stands as a technical solution to these challenges, providing an innovative approach compared to traditional streaming integrations.
Grafana: Translating Data into Insight
Grafana, an open-source platform for monitoring and observability, acts as a sophisticated translator. It takes the intricate web of data from diverse sources and weaves it into interactive and insightful dashboards.
In the vast world of the Internet of Things (IoT), data flows like a rushing river. Grafana steps in as a knowledgeable companion, skillfully translating this flood of data into clear visual insights. This process equips users to quickly understand and act on the information, ensuring that issues are spotted early and dealt with effectively.
This is our “IOT” architecture simplified:
gRPC & Protocol Buffers: Redefining Data Communication
gRPC stands as a robust, open-source RPC framework renowned for its high performance. It harnesses the power of Protocol Buffers, a dynamic method for structuring data.
Crafting the gRPC IoT Service
The decision to utilize gRPC is founded on specific technological advantages that the protocol brings to the table:
- Performance Efficiency: gRPC, rooted in the HTTP/2 protocol, offers significant benefits in latency reduction and throughput optimization.
- Bidirectional Streaming Capabilities: gRPC's inherent support for both client and server-side streaming makes it apt for real-time data transmission scenarios.
- Language Agnosticism: Given the diverse development languages in IoT, gRPC's wide support spectrum ensures seamless integration across platforms.
Creating gRPC services that handle “Iot” streams begins with Protocol Buffers (protobuf), delineating both the service architecture and the IoT event message types.
We have generated the following protobuf schema using Sylk CLI:
// Generated by sylk.build
syntax = "proto3";
package sylklabs.iot.v1;
option go_package = "github.com/sylk-build/sylk-examples/sylk-iot/services/protos/sylklabs/iot/v1;iotv1";
import "google/protobuf/timestamp.proto";
import "google/protobuf/empty.proto";
service IotService {
rpc Subscribe (sylklabs.iot.v1.Topic) returns (stream sylklabs.iot.v1.SensorData);
rpc Publish (sylklabs.iot.v1.PublishRequest) returns (google.protobuf.Empty);
}
message PublishRequest {
string topic = 1;
sylklabs.iot.v1.SensorData data = 2;
}
// The Topic message represents the topic to which
// the IoT server and Grafana plugin will subscribe and publish.
// It will have a single field called name,
// which is a string representing the name of the topic.
message Topic {
string name = 1;
}
message SensorData {
string sensor_id = 1;
double value = 2;
google.protobuf.Timestamp timestamp = 3;
}
Protocol Buffer schema for "PubSub" service
Subsequently, server configurations are set up, leveraging the Go language for its concurrency strength.
func (IotServiceServicer *IotService) Subscribe(Topic *iotv1.Topic, stream iotv1.IotService_SubscribeServer) (err error) {
topicName := Topic.Name
// Create a channel for this client's stream
subscriberChannel := make(chan *iotv1.SensorData)
defer func() {
// Close the channel when the stream is closed
close(subscriberChannel)
}()
// Register the channel for this topic
IotServiceServicer.mu.Lock()
IotServiceServicer.Subscribers[topicName] = append(IotServiceServicer.Subscribers[topicName], subscriberChannel)
IotServiceServicer.mu.Unlock()
// Send data to the client's stream
for payload := range subscriberChannel {
err := stream.Send(payload)
if err != nil {
// Client disconnected, remove the subscriber channel
IotServiceServicer.mu.Lock()
channels := IotServiceServicer.Subscribers[topicName]
for i, ch := range channels {
if ch == subscriberChannel {
// Remove the subscriber channel from the slice
IotServiceServicer.Subscribers[topicName] = append(channels[:i], channels[i+1:]...)
break
}
}
IotServiceServicer.mu.Unlock()
break
}
}
return nil
}
Go implementation for Publisher Subscriber mini architecture
With this light setup we have tested the server with ghz a load test toolset for gRPC based servers.
Plugin Architecture: The Synapse between Grafana and gRPC
The Grafana-gRPC plugin acts as an intermediary, ensuring that real-time data streaming via gRPC finds its rightful place on Grafana dashboards.
A Grafana Datasource Plugin allows Grafana to connect to and query data from various external sources. Each plugin is tailored to a specific data source, ensuring that Grafana can communicate with it, fetch the necessary data, and then render that data into graphs and dashboards.
Streams from our gRPC IoT service displayed on Grafana dashboard using the gRPC data source plugin
Assessing the Grafana & gRPC Integration
The fusion of Grafana with gRPC marks a revolutionary leap in the visualization of real-time IoT data. This integration not only caters to existing needs but also charts a promising course for future advancements in this domain.
While this Grafana datasource for our gRPC server offers a streamlined pathway for streaming data into Grafana, additional refinements can address broader use cases:
- Time-Range Analytics: Extending capabilities to store incoming data allows for time-based analytical insights.
- Network Resilience: Enhancements to handle network errors and connectivity disruptions ensure robust performance.
- Query Customization: Implementing query filters on incoming data adds flexibility to data retrieval, currently serving all events from a specific topic.
For those keen on delving deeper, there are comprehensive resources available, including guides, codebases, and more: