Fault-Tolerant High-Scale Ad Scraping & Semantic Search Architecture
Bottleneck & Risk Diagnosis
Ingesting 250k+ daily creatives directly against an unbuffered relational database creates severe thread contention, redundant row writes from duplicate Apify actor runs, and unindexed full-table searches across millions of marketing vectors. Furthermore, exposing raw partner endpoints invites billing abuse and data scraping.
Engineered Solution Overview
A decoupled, queue-first architecture using Redis 7 and BullMQ isolates Apify webhooks. An atomic SHA-256 deduplication cache intercepts duplicates in under 5ms, while worker pools process OpenAI/CLIP embeddings asynchronously into PostgreSQL 16 with HNSW pgvector indexing. Partner access is enforced at the edge via HMAC-SHA256 signatures with Stripe usage tracking.
Recommended Production Tech Stack
Selected for deterministic reliability, strict type safety, and low cloud overheadIdempotent Webhook Deduplication
Atomic Redis `SETNX` fingerprint locks prevent duplicate ad creative writes from parallel scraper runs, reducing DB write strain by over 18%.
pgvector Sub-50ms Ad Search
PostgreSQL 16 with HNSW (Hierarchical Navigable Small World) indexes over 1536-dim vector embeddings, enabling lightning-fast agency ad discovery.
HMAC-SHA256 Partner & Usage Gateway
Cryptographic signature validation on all incoming agency partner endpoints with atomic Stripe usage metering counters to prevent over-quota API abuse.
System Dataflow & Topology (Mermaid.js)
End-to-end request lifecycle, persistence boundaries, and asynchronous workersRendering system topology...
Sardana Webhook Relay & DLQ Engine
Production webhook ingestion engine demonstrating cryptographic HMAC verification, Redis idempotency deduplication, and zero-loss dead-letter queuing.