Skip to main content
Version: 0.0.0 (Beta)

EchoPost

Overview​

EchoPost is a short-lived agent that exists only when the main server is unavailable.

Its only responsibility is to avoid data loss by temporarily buffering data locally and replaying it once the main server becomes available again.

EchoPost is intentionally minimal.
It is not configurable, not long-running, and not meant to do anything beyond buffering and replay.

Once its job is completed, EchoPost shuts down.

Lifecycle​

Startup​

  • EchoPost starts when the main server is detected as unavailable.
  • A local gRPC server is started over a UNIX socket.
  • Incoming data is accepted immediately.

Buffering​

  • All incoming data is written to local storage in timestamp order.
  • No data is deleted during this phase.

Health checks​

  • EchoPost periodically checks if the main server is reachable.
  • No replay happens until the server is confirmed healthy.

Replay​

  • Buffered data is replayed in the same order it was received.
  • Records are deleted only after successful replay.

Shutdown​

  • When no buffered data remains, EchoPost shuts down gracefully.
  • EchoPost does not remain idle after replay.

Why is this required?​

  • When the main server goes down, there is a possibility of data loss.
  • Avoiding this is the primary goal.
  • Using an external fallback server increases latency and adds failure points.
  • EchoPost runs on the same machine as the SDK/application, making communication faster and local.

How is this different from agents out there today?​

  • Most agents are designed to send data to multiple destinations and require user configuration.
    EchoPost does not do that. It only talks to the Data Nadhi server.
  • Other agents are long-running and live until stopped manually or due to an exception.
    EchoPost is short-lived and behaves more like a process.
  • Long-running agents consume more memory because they do more work.
    EchoPost is designed to consume minimal memory and only for a short duration.

Documentation structure​

  • Design decisions
    Why EchoPost is short-lived, local, minimal, and opinionated.
    See: Design Decisions

  • Data usage, storage, and operations
    Data ordering, delivery semantics, PebbleDB usage, file layout, logging, and runtime behavior.
    See: Data Usage & Operations

  • Error handling & failure modes
    Explicit definitions of failures and how EchoPost reacts.
    See: Error Handling

What EchoPost is not​

  • Not a general message broker
  • Not a long-running daemon
  • Not user-configurable
  • Not a fallback analytics pipeline
  • Not a replacement for Kafka, Redis, or similar systems

EchoPost exists for one reason only: to avoid data loss when the main server is unavailable.

Reference​

Github Repository - Echopost