220 lines
7.4 KiB
Markdown
220 lines
7.4 KiB
Markdown
# System Architecture Overview
|
||
|
||
This document is the single source of truth for the platform’s architecture, protocols, data flows, and operational details. It supersedes previous scattered docs.
|
||
|
||
## Contents
|
||
- Components & Topology
|
||
- Decentralized Layer (Membership, Replication, Metrics)
|
||
- Upload/Conversion Pipeline
|
||
- Content View & Purchase Flow
|
||
- API Surface (selected endpoints)
|
||
- Data Keys & Schemas
|
||
- Configuration & Defaults
|
||
- Observability & Metrics
|
||
- Sequence Diagrams (Mermaid)
|
||
|
||
---
|
||
|
||
## Components & Topology
|
||
|
||
- Backend API: Sanic-based service (Telegram bots embedded) with PostgreSQL (SQLAlchemy + Alembic).
|
||
- Storage: Local FS for uploaded/derived data; IPFS used for discovery/pinning; tusd for resumable uploads.
|
||
- Converter workers: Dockerized ffmpeg pipeline (convert_v3, convert_process) driven by background tasks.
|
||
- Frontend: Vite + TypeScript client served via nginx container.
|
||
- Decentralized overlay (in-process DHT): Membership, replication lease management, windowed content metrics.
|
||
|
||
```mermaid
|
||
flowchart LR
|
||
Client -- TWA/HTTP --> Frontend
|
||
Frontend -- REST --> API[Backend API]
|
||
API -- tus hooks --> tusd
|
||
API -- SQL --> Postgres
|
||
API -- IPC --> Workers[Converter Workers]
|
||
API -- IPFS --> IPFS
|
||
API -- DHT --> DHT[(In-Process DHT)]
|
||
DHT -- CRDT Merge --> DHT
|
||
```
|
||
|
||
---
|
||
|
||
## Decentralized Layer
|
||
|
||
### Identity & Versions
|
||
- NodeID = blake3(Ed25519 public key), ContentID = blake3(encrypted_blob)
|
||
- schema_version = v1 embedded into DHT keys/records.
|
||
|
||
### Membership
|
||
- Signed `/api/v1/network.handshake` with Ed25519; includes:
|
||
- Node info, capabilities, metrics, IPFS metadata.
|
||
- reachability_receipts: (issuer, target, ASN, timestamp, signature).
|
||
- State: LWW-Set for members + receipts, HyperLogLog for population estimate.
|
||
- Island filtering: nodes with `reachability_ratio < q` are excluded (`k=5`, `q=0.6`, TTL=600s).
|
||
- N_estimate: `max(valid N_local reports)` across sufficiently reachable peers.
|
||
|
||
### Replication & Leases
|
||
- Compute prefix `p = max(0, round(log2(N_estimate / R_target)))` with `R_target ≥ 3`.
|
||
- Responsible nodes: first `p` bits of NodeID equal first `p` bits of ContentID.
|
||
- Leader = min NodeID among responsible.
|
||
- Leader maintains `replica_leases` with TTL=600s and diversity: ≥3 IP first octets and ≥3 ASN if available.
|
||
- Rendezvous ranking: blake3(ContentID || NodeID) for candidate selection.
|
||
- Heartbeat interval 60s, miss threshold 3 → failover within ≤180s.
|
||
|
||
### Metrics (Windowed CRDT)
|
||
- On view: PN-Counter for views; HyperLogLog for uniques (ViewID = blake3(ContentID || device_salt)); G-Counter for watch_time, bytes_out, completions.
|
||
- Keys are windowed by hour; commutative merges ensure deterministic convergence.
|
||
|
||
```mermaid
|
||
stateDiagram-v2
|
||
[*] --> Discover
|
||
Discover: Handshake + receipts
|
||
Discover --> Active: k ASN receipts & TTL ok
|
||
Active --> Leader: Content prefix p elects min NodeID
|
||
Leader --> Leased: Assign replica_leases (diversity)
|
||
Leased --> Monitoring: Heartbeats every 60s
|
||
Monitoring --> Reassign: Missed 3 intervals
|
||
Reassign --> Leased
|
||
```
|
||
|
||
---
|
||
|
||
## Upload & Conversion Pipeline
|
||
|
||
1) Client uploads via `tusd` (resumable). Backend receives hooks (`/api/v1/upload.tus-hook`).
|
||
2) Encrypted content is registered; converter workers derive preview/low/high (for media) or original (for binaries).
|
||
3) Derivative metadata stored in DB and surfaced via `/api/v1/content.view`.
|
||
|
||
```mermaid
|
||
sequenceDiagram
|
||
participant C as Client
|
||
participant T as tusd
|
||
participant B as Backend
|
||
participant W as Workers
|
||
participant DB as PostgreSQL
|
||
|
||
C->>T: upload chunks
|
||
T->>B: hooks (pre/post-finish)
|
||
B->>DB: create content record
|
||
B->>W: enqueue conversion
|
||
W->>DB: store derivatives
|
||
C->>B: GET /content.view
|
||
B->>DB: resolve latest derivatives
|
||
B-->>C: display_options + status
|
||
```
|
||
|
||
---
|
||
|
||
## Content View & Purchase Flow
|
||
|
||
- `/api/v1/content.view/<content_address>` resolves content and derivatives:
|
||
- For binary content without previews: present original only when licensed.
|
||
- For audio/video: use preview/low for unauth; decrypted_low/high for licensed users.
|
||
- Frontend shows processing state when derivatives are pending.
|
||
- Purchase options (TON/Stars) remain in a single row (UI constraint).
|
||
- Cover art layout: fixed square slot; image fits without stretching; background follows page color, not black.
|
||
|
||
```mermaid
|
||
flowchart LR
|
||
View[content.view] --> Resolve[Resolve encrypted/decrypted rows]
|
||
Resolve --> Derivations{Derivatives ready?}
|
||
Derivations -- No --> Status[processing/pending]
|
||
Derivations -- Yes --> Options
|
||
Options -- Binary + No License --> Original hidden
|
||
Options -- Media + No License --> Preview/Low
|
||
Options -- Licensed --> Decrypted Low/High or Original
|
||
```
|
||
|
||
---
|
||
|
||
## Selected APIs
|
||
|
||
- `GET /api/system.version` – liveness/protocol version.
|
||
- `POST /api/v1/network.handshake` – signed membership exchange.
|
||
- `GET /api/v1/content.view/<content_address>` – resolves display options, status, and downloadability.
|
||
- `GET /api/v1.5/storage/<file_hash>` – static file access.
|
||
- `POST /api/v1/storage` – legacy upload endpoint.
|
||
|
||
---
|
||
|
||
## Data Keys & Schemas
|
||
|
||
- MetaKey(content_id): tracks `replica_leases`, `leader`, `conflict_log`, `revision`.
|
||
- MembershipKey(node_id): LWW-Set of members & receipts, HyperLogLog population, N_reports.
|
||
- MetricKey(content_id, window_id): PN-/G-/HLL serialized state.
|
||
|
||
All DHT records are signed and merged via deterministic CRDT strategies + LWW dominance (logical_counter, timestamp, node_id).
|
||
|
||
---
|
||
|
||
## Configuration & Defaults
|
||
|
||
- Network: `NODE_PRIVACY`, `PUBLIC_HOST`, `HANDSHAKE_INTERVAL_SEC`, TLS verify, IPFS peering.
|
||
- DHT: `DHT_MIN_RECEIPTS=5`, `DHT_MIN_REACHABILITY=0.6`, `DHT_MEMBERSHIP_TTL=600`, `DHT_REPLICATION_TARGET=3`, `DHT_LEASE_TTL=600`, `DHT_HEARTBEAT_INTERVAL=60`, `DHT_HEARTBEAT_MISS_THRESHOLD=3`, `DHT_MIN_ASN=3`, `DHT_MIN_IP_OCTETS=3`, `DHT_METRIC_WINDOW_SEC=3600`.
|
||
- Conversion resources: `CONVERT_*` limits (CPU/mem), `MAX_CONTENT_SIZE_MB`.
|
||
|
||
---
|
||
|
||
## Observability & Metrics
|
||
|
||
Prometheus (exported in-process):
|
||
- dht_replication_under / dht_replication_over / dht_leader_changes_total
|
||
- dht_merge_conflicts_total
|
||
- dht_view_count_total / dht_unique_view_estimate / dht_watch_time_seconds
|
||
|
||
Logs track replication conflict_log entries and HTTP structured errors (with session_id/error_id).
|
||
|
||
---
|
||
|
||
## Sequence Diagrams (Consolidated)
|
||
|
||
### Membership & N_estimate
|
||
```mermaid
|
||
sequenceDiagram
|
||
participant A as Node A
|
||
participant B as Node B
|
||
A->>B: POST /network.handshake {nonce, ts, signature}
|
||
B->>B: verify ts, nonce, signature
|
||
B->>B: upsert member; store receipts
|
||
B-->>A: {node, known_public_nodes, n_estimate, signature}
|
||
A->>A: merge; recompute N_estimate = max(N_local, peers)
|
||
```
|
||
|
||
### Replication Leader Election
|
||
```mermaid
|
||
sequenceDiagram
|
||
participant L as Leader
|
||
participant Peers as Responsible Nodes
|
||
L->>L: compute p from N_estimate
|
||
L->>Peers: rendezvous scores for ContentID
|
||
L->>L: assign leases (diversity)
|
||
Peers-->>L: heartbeat every 60s
|
||
L->>L: reassign on 3 misses (≤180s)
|
||
```
|
||
|
||
### Metrics Publication
|
||
```mermaid
|
||
sequenceDiagram
|
||
participant C as Client
|
||
participant API as Backend
|
||
participant M as MetricsAggregator
|
||
participant D as DHT
|
||
|
||
C->>API: GET content.view?watch_time&bytes_out
|
||
API->>M: record_view(delta)
|
||
M->>D: merge MetricKey(ContentID, Window)
|
||
M->>API: update gauges
|
||
```
|
||
|
||
---
|
||
|
||
## Run & Test
|
||
|
||
```bash
|
||
# Spin services
|
||
docker compose -f /home/configs/docker-compose.yml --env-file /home/configs/.env up -d --build
|
||
|
||
# Backend unit tests (DHT integration)
|
||
cd uploader-bot
|
||
python3 -m unittest discover -s tests/dht
|
||
```
|
||
|