Build your own integration

When a prebuilt integration such as the Kafka ingress integration does not fit your source, you can build your own. The Integration API lets you push events from any system into Restate as invocations, with durable delivery and configurable semantics, so you get the same guarantees as the prebuilt integrations without hosting the whole ingestion path yourself.

The Integration API is currently available for Java only.

Concepts

An integration is a producer that sends invocations to Restate. Each invocation targets a service and handler, carries a payload, and is assigned an offset: a number that identifies its position in the stream the producer is sending. Restate uses the offset to order deliveries and to deduplicate them, which is what gives the integration its delivery guarantees. You choose the delivery semantics by choosing the producer type:

At-least-once, with Producer. The client assigns the offset for you. Deduplication is off, so a record can be delivered more than once across restarts.
Exactly-once, with ExactlyOnceProducer. You supply a deterministic offset per invocation, and Restate drops replays after a restart.

Both producers are AutoCloseable. A producer is not thread-safe and fails fast if used from multiple threads. Create one producer per sending thread.

At-least-once producer

Use Producer when your source does not have a stable, replayable position for each event. The producer assigns a monotonically increasing offset to each invocation, and returns that offset from send. Deduplication is disabled, so an invocation can be delivered more than once after a restart. Add an idempotency key on the invocation if you need handler-level deduplication.

// Placeholder: replace with a loaded snippet.
public interface Producer extends AutoCloseable {
  CompletableFuture<Long> send(Invocation invocation);
  long trySend(Invocation invocation) throws ProducerNotReadyException;
  long lastSentOffset();
  CompletableFuture<Void> waitReady();
  CompletableFuture<Long> waitAcknowledged();
  CompletableFuture<Long> waitAcknowledged(long offset);
}

A basic send loop:

// Placeholder: replace with a loaded snippet.
try (Producer producer = /* build the producer */) {
    for (Event event : source) {
        // Await each send before the next one to keep ordering and apply backpressure.
        producer.send(
            Invocation.target("MyService", "handle")
                .payload(event.toJson())
        ).join();
    }
    // Wait until Restate has acknowledged everything sent so far.
    producer.waitAcknowledged().join();
}

Exactly-once producer

Use ExactlyOnceProducer when your source can give each event a deterministic, strictly increasing offset, for example a log sequence number or a Kafka partition offset. You pass that offset to send, and Restate deduplicates on the producer id and offset, dropping any replay that arrives after a restart.

// Placeholder: replace with a loaded snippet.
public interface ExactlyOnceProducer extends AutoCloseable {
  CompletableFuture<Void> send(long offset, Invocation invocation);
  void trySend(long offset, Invocation invocation) throws ProducerNotReadyException;
  long lastSentOffset();
  CompletableFuture<Void> waitReady();
  CompletableFuture<Long> waitAcknowledged();
  CompletableFuture<Long> waitAcknowledged(long offset);
}

A send loop that reuses the source offset:

// Placeholder: replace with a loaded snippet.
try (ExactlyOnceProducer producer = /* build the producer */) {
    for (Event event : source) {
        producer.send(
            event.offset(),
            Invocation.target("MyService", "handle")
                .payload(event.toJson())
        ).join();
    }
    producer.waitAcknowledged().join();
}

Sending in order and applying backpressure

Await each send future before starting the next one. This keeps the invocations in order and applies backpressure, since the future completes only once the producer has room to accept the next invocation. For non-blocking sends, use trySend. It attempts a send and throws ProducerNotReadyException when the send window is full. Await waitReady() to be notified when capacity is available again:

// Placeholder: replace with a loaded snippet.
try {
    producer.trySend(invocation);
} catch (ProducerNotReadyException e) {
    producer.waitReady().join();
    producer.trySend(invocation);
}

Stream defaults

To set fields shared by every invocation once, pass an InvocationMetadata when building the producer. Fields set per invocation override the stream defaults.

// Placeholder: replace with a loaded snippet.

Acknowledgements

send completes once the invocation has been sent. To confirm that Restate has durably accepted it, await an acknowledgement:

waitAcknowledged() waits for everything up to lastSentOffset().
waitAcknowledged(offset) waits for a specific offset.

Wait for acknowledgements before you commit progress in your source, so a crash after committing never loses an unacknowledged invocation.

SDKs

Services

Restate Cloud

Restate BYOC

Self-hosted Restate

References

Build your own integration

Concepts

At-least-once producer

Exactly-once producer

Sending in order and applying backpressure

Stream defaults

Acknowledgements

​Concepts

​At-least-once producer

​Exactly-once producer

​Sending in order and applying backpressure

​Stream defaults

​Acknowledgements

Concepts

At-least-once producer

Exactly-once producer

Sending in order and applying backpressure

Stream defaults

Acknowledgements