In today's world of microservices, efficient communication between services is more crucial than ever. One technology that has been gaining traction for enabling this is gRPC (gRPC Remote Procedure Calls). Built on top of HTTP/2 and leveraging Protocol Buffers for data serialization, gRPC offers a performant and robust way of building scalable APIs.

Why Protocol Buffers?

But what makes gRPC so efficient? A lot of the magic lies in its use of Protocol Buffers (protobuf), a high-performance, language-neutral, and extensible mechanism for serializing structured data. Unlike JSON or XML, which are text-based and can be inefficient for large datasets, Protocol Buffers are binary and thus far more efficient in both size and speed. They also offer strong typing and versioning capabilities, which makes it easier to maintain your APIs as they evolve.

Why TypeScript?

TypeScript adds a strong type system to JavaScript, offering better compile-time error checking and enabling more robust and maintainable codebases. In the realm of gRPC, TypeScript provides several advantages, such as type safety, autocompletion, and better refactoring capabilities, which make the development process more streamlined and efficient.

In this article, we will cover everything you need to know to get started with gRPC in TypeScript, from setting up your development environment to building a simple gRPC service. We'll also take a deep dive into the specifics of using TypeScript in the gRPC domain, such as type safety and modular code structure. Additionally, given the repetitive nature of compiling and managing .proto files, we'll explore best practices and automation techniques to make your development process more efficient.

Whether you're new to gRPC and TypeScript or an experienced developer looking to refine your skills, this guide aims to provide you with both the foundation and the advanced knowledge needed to build robust and scalable gRPC services with TypeScript.

Setting Up the Development Environment

Before diving into code, it's essential to set up your development environment correctly. This will ensure that you have all the necessary tools and packages to start building gRPC services using TypeScript. In this section, we will cover the installation of Node.js, TypeScript, and essential gRPC libraries, as well as how to structure your project directory for optimal development.

Installing Node.js and TypeScript

1. Install Node.js: If you haven't installed Node.js, you can download it from nodejs.org. Choose the version that is appropriate for your operating system.

# Verify Node.js installation
node -v

2. Install TypeScript: Once Node.js is installed, you can install TypeScript globally using npm (Node Package Manager), which comes pre-installed with Node.js.

# Install TypeScript globally
npm install -g typescript

# Verify TypeScript installation
tsc -v

Installing gRPC Libraries and Tools

1. Install gRPC package: The core gRPC library for Node.js can be installed using npm. Open your terminal and run the following command:

npm install grpc

2. Protocol Buffers Compiler (protoc): You'll also need the Protocol Buffers compiler, commonly known as protoc, to compile .proto files. You can download the compiler from the official Protocol Buffers GitHub repository.

Project Directory Structure

For a standard gRPC TypeScript project, a clean directory structure can aid in better code management and readability. Here's a sample structure you might want to consider:

my-grpc-project/
├── src/
│   ├── protos/
│   │   └── my_service.proto
│   ├── client/
│   │   └── client.ts
│   └── server/
│       └── server.ts
├── build/
├── package.json
└── tsconfig.json

• src/: Contains the source TypeScript files.
• protos/: Houses the .proto files that define the gRPC service.
• client/: Contains the TypeScript code for the gRPC client.
• server/: Contains the TypeScript code for the gRPC server.
• build/: A directory where compiled JavaScript files will be output.
• package.json: Lists package dependencies and scripts.
• tsconfig.json: Configuration file for TypeScript compiler options.

In subsequent sections, we'll delve into each of these directories and files to help you understand how to scaffold your gRPC service in TypeScript.

Introduction to gRPC and Protocol Buffers

Understanding the underlying principles of gRPC and Protocol Buffers is fundamental to effectively leveraging them in your TypeScript projects. In this section, we'll cover what RPCs (Remote Procedure Calls) are, dive into the world of Protocol Buffers, and discuss why gRPC uses Protocol Buffers for its operations.

Brief Explanation of RPC (Remote Procedure Call)

At its core, a Remote Procedure Call (RPC) is a powerful method for executing a procedure (or a function) on a remote server, rather than on a local computer. The process involves making a request to the remote server, which then processes the request and sends back the computed result. RPCs enable you to develop distributed, client-server based applications, where tasks and computing resources are distributed across multiple endpoints.

Introduction to Protocol Buffers

Protocol Buffers, often abbreviated as protobuf, are Google's language-neutral, platform-neutral, extensible mechanism for serializing structured data. Protobuf offers benefits like smaller payload size, higher serialization and deserialization speed, and strong typing. A .proto file serves as the interface definition language for describing your data structure and service methods. This file can be compiled into client and server code for various programming languages, making it incredibly flexible and language-agnostic.

Here's a quick example of what a .proto file might look like:

syntax = "proto3";

service MyService {
  rpc SayHello (HelloRequest) returns (HelloResponse);
}

message HelloRequest {
  string name = 1;
}

message HelloResponse {
  string message = 1;
}

Compiling .proto Files Manually

To transform your .proto files into TypeScript (or any other language), you'll need to use the Protocol Buffers Compiler, protoc. Here's a sample command that compiles a .proto file to generate TypeScript code:

protoc --proto_path=src/protos --js_out=import_style=commonjs,binary:src/generated --grpc_out=src/generated --plugin=protoc-gen-grpc=`./node_modules/.bin/grpc_tools_node_protoc_plugin` src/protos/my_service.proto

For typescript support this isn’t enough you should download and install a plugin that was specifically developed for typescript and compile the .proto files with him, there are 2 main plugins for this:

• stephenh/ts-proto
• improbable-eng/ts-protoc-gen

Both plugins should compile your protobuf and generate the TS code files and modules, but the are some differences in the signature they are generating.

This is how you should compile the .proto files for Typescript:

bin/proto.js

const path = require("path");
const { execSync } = require("child_process");
const rimraf = require("rimraf");
const PROTO_DIR = path.join(__dirname, "../protos");
const MODEL_DIR = path.join(__dirname, "../services/protos");
const PROTOC_PATH = path.join(__dirname, "../node_modules/grpc-tools/bin/protoc");
const PLUGIN_PATH = path.join(__dirname, "../node_modules/.bin/protoc-gen-ts_proto");

rimraf.sync(`${MODEL_DIR}/*`, {
  glob: { ignore: `${MODEL_DIR}/tsconfig.json` },
});

const protoConfig = [
  `--plugin=${PLUGIN_PATH}`,

  // https://github.com/stephenh/ts-proto/blob/main/README.markdown
  "--ts_proto_opt=outputServices=grpc-js,env=node,useOptionals=messages,exportCommonSymbols=false,esModuleInterop=true",

  `--ts_proto_out=${MODEL_DIR}`,
  `--proto_path ${PROTO_DIR} ${PROTO_DIR}/*.proto`,
];

// https://github.com/stephenh/ts-proto#usage
execSync(`${PROTOC_PATH} ${protoConfig.join(" ")}`);

console.log(`> Proto models created: ${MODEL_DIR}`);

Directory Structure for Managing Multiple .proto Files

When your project begins to scale, you might end up with multiple .proto files. A thoughtful directory structure can help manage these efficiently. For instance:

src/
├── protos/
│   ├── v1/
│   │   └── my_service_v1.proto
│   ├── v2/
│   │   └── my_service_v2.proto
│   └── common/
│       └── common_types.proto
└── generated/
    ├── v1/
    └── v2/

How to Deal with Repetitive Compilation Tasks

Repeatedly running protoc commands can be cumbersome. Here are some strategies to automate this:

NPM Scripts: Use the scripts section of your package.json to define a compilation step.

"scripts": {
    "compile-protos": "your-protoc-command-here"
}

Now you can run npm run compile-protos whenever you need to compile.

Watchers: Use file watcher utilities like chokidar to automatically run your protoc commands when .proto files change.

Build Tools: Integrate protoc compilation into build tools like Webpack or Gulp.

Pre-commit Hooks: Use Git hooks to ensure .proto files are compiled before commits.

Continuous Integration: Make .proto compilation a step in your CI/CD pipeline to ensure that the generated code is always up to date.

You can read more about mastering protobuf in our blog post

Why gRPC uses Protocol Buffers

The question that might arise is, why does gRPC specifically use Protocol Buffers? Here are some reasons:

1. Efficiency: Protocol Buffers are highly efficient, both in terms of speed and size, as they are binary rather than text-based formats like JSON or XML.
2. Strong Typing: Protobuf allows for strong typing, making it easier to catch errors at compile-time rather than at runtime.
3. Language Neutrality: Protobuf can be compiled into various programming languages, including but not limited to C++, Python, Java, and, of course, TypeScript. This makes it easier to develop cross-language services.
4. Versioning: Protocol Buffers are designed to be forward and backward compatible, meaning that you can modify your data structure without breaking existing deployed services.

By using Protocol Buffers, gRPC ensures that applications are not only performant but also easy to scale and maintain.

Building a Simple gRPC Service in TypeScript

Now that we have a solid understanding of gRPC, Protocol Buffers, and our development environment is set up, let's dive into building a simple gRPC service using TypeScript. In this section, we'll define a gRPC service with Protocol Buffers, implement the service using TypeScript, create a client to consume the service, and walk through the code to ensure a comprehensive understanding.

First define a gRPC Service using Protocol Buffers, for simplicity we going to use the service definition from section Introduction to Protocol Buffers.

After successfully compiling the service we can see few files are generated thanks to the ts-proto plugin discussed above

/* eslint-disable */
import {
  CallOptions,
  ChannelCredentials,
  Client,
  ClientOptions,
  ClientUnaryCall,
  handleUnaryCall,
  makeGenericClientConstructor,
  Metadata,
  ServiceError,
  UntypedServiceImplementation,
} from "@grpc/grpc-js";

export type HelloService = typeof HelloService;

export const HelloService = {
  sayHello: {
    path: "/HelloService/SayHello",
    requestStream: false,
    responseStream: false,
    requestSerialize: (value: HelloRequest) => Buffer.from(HelloRequest.encode(value).finish()),
    requestDeserialize: (value: Buffer) => HelloRequest.decode(value),
    responseSerialize: (value: HelloResponse) => Buffer.from(HelloResponse.encode(value).finish()),
    responseDeserialize: (value: Buffer) => HelloResponse.decode(value),
  }
} as const;

export interface HelloServiceServer extends UntypedServiceImplementation {
  sayHello: handleUnaryCall<HelloRequest, HelloResponse>;
}

export interface HelloServiceClient extends Client {
  sayHello(
    request: CreateFieldRequest,
    callback: (error: ServiceError | null, response: CreateFieldResponse) => void,
  ): ClientUnaryCall;
  sayHello(
    request: CreateFieldRequest,
    metadata: Metadata,
    callback: (error: ServiceError | null, response: CreateFieldResponse) => void,
  ): ClientUnaryCall;
  sayHello(
    request: CreateFieldRequest,
    metadata: Metadata,
    options: Partial<CallOptions>,
    callback: (error: ServiceError | null, response: CreateFieldResponse) => void,
  ): ClientUnaryCall;
}

export const HelloServiceClient = makeGenericClientConstructor(HelloService, "HelloService") as unknown as {
  new (address: string, credentials: ChannelCredentials, options?: Partial<ClientOptions>): HelloServiceClient;
  service: typeof HelloService;
};

We can see some nice things on the generated code:

• We have a complete “classes” that abstract the service and it’s endpoints including the proper mapped Serialize / Deserialize functions
• We have both our client interface and server interface to start implementing them
• Routes are handled automaticlly no more need for boilerplate code

Now for the fun part, lets implement the service class to hold our business logic:

import { 
    handleUnaryCall,
    sendUnaryData,
    ServerUnaryCall,
    status,
    UntypedHandleCall,
    Metadata
} from '@grpc/grpc-js';

import { HelloServiceServer, HelloService } from "./hello.ts"

type HelloServiceApiType<T> = Api<T> & {

    

}

class Hello implements HelloServiceServer, HelloServiceApiType<UntypedHandleCall> {
    [method: string]: any;

    public getHello(): handleUnaryCall<SayHelloRequest, SayHelloResponse> = async (
        call: ServerUnaryCall<SayHelloRequest, SayHelloResponse>,
        callback: sendUnaryData<SayHelloResponse>
    ) { 
        const { name } = call.request;
        console.log(`Hello from client: ${name}`);
        callback(null, { message: `Hello from server: ${name}`}, call.metadata);
    }

}

Import Statements

1. Import gRPC modules: Various functionalities and types are imported from the @grpc/grpc-js package. These are essential for setting up a gRPC service.

import { 
    handleUnaryCall,
    sendUnaryData,
    ServerUnaryCall,
    status,
    UntypedHandleCall,
    Metadata
} from '@grpc/grpc-js';

2. Import service types: This imports HelloServiceServer and HelloService interfaces from another file (./hello.ts), which presumably contains the service and message definitions.

import { HelloServiceServer, HelloService } from "./hello.ts";

Class Definition

The Hello class is declared to implement HelloServiceServer and HelloServiceApiType<UntypedHandleCall>.

class Hello implements HelloServiceServer, HelloServiceApiType<UntypedHandleCall> {
    [method: string]: any;
}

The [method: string]: any; line means that the class can have any string-based property key which can hold any type of value. This offers flexibility but also makes the type less strict.

getHello Method

The getHello() method is a public method that implements the handleUnaryCall type, allowing it to manage unary gRPC calls.

public getHello(): handleUnaryCall<SayHelloRequest, SayHelloResponse> = async (
    call: ServerUnaryCall<SayHelloRequest, SayHelloResponse>,
    callback: sendUnaryData<SayHelloResponse>
) => {
    const { name } = call.request;
    console.log(`Hello from client: ${name}`);
    callback(null, { message: `Hello from server: ${name}`}, call.metadata);
}

Here's what the method does:

It accepts a ServerUnaryCall object call, which contains information about the incoming request.

It also accepts a callback function of type sendUnaryData<SayHelloResponse>, which can be used to send a response back to the client.

The method destructures the name field from the call.request object.

A log message is printed on the server-side console to indicate the name sent from the client.

Finally, the callback function is invoked to send a message back to the client, along with metadata if any.

The types like SayHelloRequest and SayHelloResponse would typically be defined in the gRPC .proto file generated code for TypeScript definitions.

Deep Dive: Specifics of TypeScript in gRPC

TypeScript adds a static typing layer on top of JavaScript, providing various advantages such as better code completion, readability, and compile-time error checking. When using TypeScript with gRPC, these benefits become even more pronounced. In this section, we'll delve into how TypeScript-specific features like type safety, serialization/deserialization, and modularity can significantly improve your gRPC applications.

Type Safety and How It Benefits gRPC Applications

TypeScript's strong typing system can greatly benefit gRPC applications by making it easier to catch errors during development rather than at runtime.

Handling TypeScript Compiler with tsconfig.json Files

The TypeScript compiler is governed by the settings defined in the tsconfig.json file. For a gRPC project, you might want to add some specific configurations:

{
    "compilerOptions": {
        "baseUrl": ".",
        "paths": {},
        "target": "ES2020",
        "outDir": "server",
        "module": "commonjs",
        "moduleResolution": "node",
        "incremental": true,
        "declaration": true,
        "newLine": "lf",
        "strict": true,
        "allowUnreachableCode": false,
        "allowUnusedLabels": false,
        "noFallthroughCasesInSwitch": true,
        "noImplicitOverride": true,
        "noImplicitReturns": true,
        "noPropertyAccessFromIndexSignature": true,
        "noUnusedLocals": false,
        "noUnusedParameters": false,
        "removeComments": false,
        "sourceMap": true,
        "forceConsistentCasingInFileNames": true,
        "esModuleInterop": true,
        "skipLibCheck": true
    },
    "include": [
        "src/**/*",
    ],
    "exclude": [
        "node_modules",
        "src/protos"
    ],
    "references": [
        {
            "path": "src/services/protos"
        }
    ]
}

tsconfig.json file at the src/services/protos/ level:

{
    "extends": "../../tsconfig.json",
    "compilerOptions": {
        "composite": true,
        "outDir": "../../dist/protos",
        "noImplicitReturns": false
    },
    "include": [
        "google/**/*"
    ],
    "exclude": [
        "node_modules"
    ]
}

In this configuration, we specify the target as ES2020 and module as commonjs, both of which are commonly used settings for Node.js projects. We also using another tsconfig.json file for compiler specific options to our generated classes from the .proto files,

Advanced Use Cases

As you grow more comfortable with gRPC and TypeScript, you'll likely want to explore more complex scenarios and advanced functionalities. In this section, we will cover three advanced use-cases: Streaming APIs, gRPC interceptors in TypeScript, and authentication and authorization within a gRPC service.

Streaming APIs

gRPC supports various types of streaming: Unary, Server Streaming, Client Streaming, and Bidirectional Streaming. Each has its use-cases and advantages.

Unary Streaming: This is the simplest form where a client sends a single request and gets a single response.

Server Streaming: The client sends a single request and receives multiple responses from the server.

Client Streaming: The client sends multiple messages and receives a single response.

Bidirectional Streaming: Both the client and server can read and write messages independently of each other.

Here's an example of server streaming in TypeScript:

// Server implementation
class MyStreamingService implements MyStreamService {
  streamData(call: grpc.ServerWritableStream<MyRequest>): void {
    // You can write multiple times to the client
    call.write(response1);
    call.write(response2);
    //...
    call.end();
  }

}

// Client implementation
const call = client.streamData(request);

call.on('data', (data) => {
  console.log('Received:', data);
});

Authentication and Authorization

Security is a primary concern in any production service. With gRPC, you can use different methods like SSL/TLS, token-based, or custom authentication.

Here's an example using token-based authentication in TypeScript:

// Client
const metadata = new grpc.Metadata();

metadata.add('authorization', 'Bearer ' + token);
client.myRpcMethod(myRequest, metadata, (error, response) => {

// Handle the response
});

Conclusion and Further Resources

We've covered quite a bit in this blog post—from setting up your development environment to deep-diving into the specifics of TypeScript in the gRPC domain. Whether you're a seasoned developer or someone new to gRPC or TypeScript, we hope that this article has been a comprehensive guide to get you up and running, and even delve into more advanced topics like Streaming APIs, gRPC interceptors, and authentication and authorization.

Summary of What Has Been Covered

• Setting up the development environment with Node.js, TypeScript, and gRPC.
• Introduction to RPC and Protocol Buffers, and why gRPC relies on them.
• Managing Protocol Buffers effectively within a project.
• Building a simple gRPC service in TypeScript.
• Deep diving into TypeScript specifics and how they benefit gRPC applications.
• Advanced use-cases like streaming, interceptors, and security.

Real-World Applications of Using gRPC with TypeScript

• Microservices Architecture: gRPC is an excellent fit for connecting microservices in a type-safe, efficient manner.
• Real-time Applications: With gRPC's streaming capabilities, building real-time applications becomes more straightforward.
• High-Performance APIs: If you're building an API that needs low latency and high throughput, gRPC is a strong candidate.

Further Resources for Deeper Learning

• Official gRPC Documentation
• TypeScript Handbook
• Sylk CLI for gRPC and TypeScript
• GitHub Repo with TypeScript + gRPC Example
• Awesome gRPC

As a final note, if you'd like to skip the boilerplate and scaffold your gRPC server setup in TypeScript, consider using the Sylk CLI tool. This tool can autogenerate code, manage your .proto files, and get you up and running in no time. It's an invaluable tool for both beginners and experienced developers who want to focus more on the business logic rather than setup.

# Install Sylk CLI
pip install sylk

# Scaffold a new gRPC project
sylk new my-grpc-project

By leveraging tools like Sylk, you can accelerate your development process and ensure that you're following best practices from the get-go.

Thank you for reading, and happy coding!