Ingest, workers and metrics

A flow diagram. Parts with no edge between them, side by side: an ingest path into an object store, a job queue and its workers, a metrics agent and its dashboard, and a runbook.

Parts with no edge between them, side by side: an ingest path into an object store, a job queue and its workers, a metrics agent and its dashboard, and a runbook

     ╭────────╮                ┌────────────┐             ┌────────────────┐
     │ Ingest │                │ Job Queue  │             │ Metrics Agent  │
     ╰────────╯                └────────────┘             └────────────────┘
          │                        │    │                          ╎
          │                        │    │                          ╎ scrape
          │                  ┌─────┘    └─────┐                    ╎
          ▼                  │                │                    ▼
     ┌────────┐     ┌────────┼─── Workers ────┼───────┐     ╭────────────╮
     │ Parse  │     │        ▼                ▼       │     │ Dashboard  │
     └────────┘     │  ┌──────────┐     ┌──────────┐  │     ╰────────────╯
          │         │  │ Worker A │     │ Worker B │  │
          │         │  └──────────┘     └──────────┘  │
          │ write   └─────────────────────────────────┘
          │
          │
          ▼
  ╒══════════════╕         ┌──────────┐
  │ Object Store │         │ Runbook  │
  └──────────────┘         └──────────┘

Spec

JSON

{
  "type": "flow",
  "direction": "DOWN",
  "nodes": [
    {"id": "ingest", "label": "Ingest", "shape": "rounded"},
    {"id": "parse", "label": "Parse", "shape": "rect"},
    {"id": "store", "label": "Object Store", "shape": "cylinder", "color": "blue"},
    {"id": "queue", "label": "Job Queue", "shape": "hexagon", "color": "orange"},
    {"id": "worker_a", "label": "Worker A", "shape": "rect"},
    {"id": "worker_b", "label": "Worker B", "shape": "rect"},
    {"id": "metrics", "label": "Metrics Agent", "shape": "rect"},
    {"id": "dashboard", "label": "Dashboard", "shape": "rounded", "color": "green"},
    {"id": "notes", "label": "Runbook", "shape": "parallelogram", "color": "gray"}
  ],
  "edges": [
    {"id": "e-ingest-parse", "from": "ingest", "to": "parse"},
    {"id": "e-parse-store", "from": "parse", "to": "store", "label": "write"},
    {"id": "e-queue-worker_a", "from": "queue", "to": "worker_a"},
    {"id": "e-queue-worker_b", "from": "queue", "to": "worker_b"},
    {
      "id": "e-metrics-dashboard",
      "from": "metrics",
      "to": "dashboard",
      "label": "scrape",
      "style": "dashed"
    }
  ],
  "groups": [
    {"id": "workers", "label": "Workers", "contains": ["worker_a", "worker_b"], "color": "purple"}
  ],
  "ignore": ["isolated-node"]
}

Draw it yourself

diago render ingest-and-workers.json > ingest-and-workers.svg

Open it in the playground to edit it in your browser.