Plus234Feed

// the problem

Nigerian market, macro and civic data is scattered across publishers, regulators and PDFs, and most of it is stale or paywalled by the time anyone can use it. Plus234Feed aggregates it, keeps it current, and serves it fast from the edge.

// context

The core of everything else here: SignalDesk, the MCP server and the Intel API all read the Supabase database it fills. Traffic is modest, so the engineering effort goes into correct, sourced data and cheap operation rather than scale.

// what I built

1. A FastAPI backend with 200+ endpoints across 30+ route modules: articles, auth and user engagement, NGX market data, CBN rates, FX, NBS statistics, NPFL football, legislators and newsletters.
2. 36 scheduled GitHub Actions pipelines running Python scrapers and ETL into Supabase Postgres, with separate dev and production environments.
3. 13 Cloudflare Workers in front of the API: edge caching with per-endpoint TTLs, homepage/category/page aggregators, image resizing, Open Graph image generation, a URL shortener on KV and rate-limited third-party proxies.
4. Node/TypeScript services on Render: an AI daily brief (OpenAI), article processing, unified search, newsletters through AWS SES, image processing and link shortening.
5. A Next.js frontend (App Router, SSR + ISR), migrated in gradually beside the original Vite app, plus a Flutter mobile app in development. The platform is operated from SignalDesk, the internal portal below.
6. Plus234Feed Intel: a developer data API over the same tables with API-key issuing, hashing and revocation, daily usage limits, and a response envelope carrying source and freshness.

// key decisions

1. Judge pipelines by what they inserted, not their exit code

   A health job reads scraper run rows: zero inserted with duplicates counts as alive, zero inserted and zero duplicates counts as broken. Only a critical result fails the job; degraded states post to Slack. Added after two scrapers returned HTTP 200 and inserted nothing for weeks.

2. Per-path cache TTLs at the edge, with a stale fallback

   The cache Worker in front of the API sets a TTL per path, from 60 seconds for search and live results to a day for the constituency register, and never caches per-user paths. A stale copy is kept at six times the TTL so pages still render when the origin is down. Purges are by exact key.

3. Provenance as a merge rule

   Every ingested row records its source, source URL and fetch time, and matched fields carry the method and evidence. Hand decisions live in a table read at run time, and AI-extracted facts need a quote the code verifies. Adopted after a matching error put the wrong senator on three districts while every link count looked valid.

4. Embed once, centrally, and cap the run

   One scheduled job embeds new articles with text-embedding-3-small and marks near-duplicates from the last 7 days (cosine 0.85 or higher) as superseded. Runs are capped at 500 articles to bound cost, and search embeds queries with the same model so scores stay comparable.

// results

1. 263 route handlers across 38 route modules in the FastAPI backend.
2. 38 scheduled GitHub Actions workflows and 13 Cloudflare Workers in production.
3. All 469 federal legislative seats mapped to INEC codes, and 4,881 candidates for 2027 loaded.