AI Cost Optimisation Gateway MVP
Budget / Salary$5,000–10,000
TypeFreelance project
LocationRemote
Posted1 hour ago
I need an experienced backend engineer to build a pilot-ready MVP of an AI cost-optimisation gateway for UK companies, built on LiteLLM. It sits between clients’ apps and OpenAI, Anthropic and Google to cut AI costs, protect personal data and prove the savings.
STACK (decided)
LiteLLM as the single gateway, with Python hooks; TypeScript/NestJS services; Next.js dashboards; PostgreSQL; Redis; Docker; AWS London across two availability zones. No second gateway or proxy in front of LiteLLM.
MILESTONES (fixed price each, paid via Milestone Payments when acceptance tests pass)
M1 Foundation: company GitHub repo, Docker, LiteLLM with multi-tenant keys, OpenAI/Anthropic/Google connections, per-client budgets and spend tracking, basic CloudWatch monitoring, design for the gateway-down setting.
M2 Privacy and reliability: PII masking and restoration with Presidio plus UK detectors (NI numbers, NHS numbers with Modulus 11 checksum, UK postcodes); tenant isolation tests; retries and backup models; metadata-only audit log (never prompt text).
M3 Caching and ledger: exact and semantic caching (stored masked, isolated per client); versioned provider price table; shadow-mode baseline; savings ledger that excludes providers’ own caching discounts and reconciles with test calculations.
M4 Differentiators and reporting: basic Batch Autopilot (“can wait” requests sent to provider batch APIs), Cache Doctor (detect cache-breaking prompt patterns), Runaway Agent Guard (spend caps and spike alerts), engineer dashboard and monthly savings statement for finance.
M5 Pilot readiness: AWS London two-zone deployment, per-client fail-open/fail-closed setting (fail-closed by default), production alerting, client onboarding checklist, documentation and handover.
KEY REQUIREMENTS
Multiple client organisations on one install, with strict isolation (keys, cache, budgets, logs, ledger).
PII acceptance: 50+ test prompts, 95%+ of personal data removed before the provider, answers correct after restore.
Under 50 ms added on cache hits; target under 200 ms on cache misses.
All code in my company GitHub from day one, with tests and setup notes. Permissive open source only; ask before any GPL/AGPL code.
Weekly written updates and a demo at the end of each milestone.
TIMELINE
Start within a week; pilot-ready by end of December.
YOUR PROPOSAL MUST INCLUDE
A gateway, proxy or middleware project you built yourself (your role and code, not just a company website).
Your experience with LiteLLM, Presidio and AWS ECS/Fargate.
Your total fixed price and delivery date for the whole project.
Weekly hours and start date.
How you would handle the gateway-down fallback without bypassing PII protection.
Shortlisted candidates sign a short NDA before seeing the full brief. The best one or two may do a small paid trial (the UK PII detectors). Generic proposals will not be considered.
STACK (decided)
LiteLLM as the single gateway, with Python hooks; TypeScript/NestJS services; Next.js dashboards; PostgreSQL; Redis; Docker; AWS London across two availability zones. No second gateway or proxy in front of LiteLLM.
MILESTONES (fixed price each, paid via Milestone Payments when acceptance tests pass)
M1 Foundation: company GitHub repo, Docker, LiteLLM with multi-tenant keys, OpenAI/Anthropic/Google connections, per-client budgets and spend tracking, basic CloudWatch monitoring, design for the gateway-down setting.
M2 Privacy and reliability: PII masking and restoration with Presidio plus UK detectors (NI numbers, NHS numbers with Modulus 11 checksum, UK postcodes); tenant isolation tests; retries and backup models; metadata-only audit log (never prompt text).
M3 Caching and ledger: exact and semantic caching (stored masked, isolated per client); versioned provider price table; shadow-mode baseline; savings ledger that excludes providers’ own caching discounts and reconciles with test calculations.
M4 Differentiators and reporting: basic Batch Autopilot (“can wait” requests sent to provider batch APIs), Cache Doctor (detect cache-breaking prompt patterns), Runaway Agent Guard (spend caps and spike alerts), engineer dashboard and monthly savings statement for finance.
M5 Pilot readiness: AWS London two-zone deployment, per-client fail-open/fail-closed setting (fail-closed by default), production alerting, client onboarding checklist, documentation and handover.
KEY REQUIREMENTS
Multiple client organisations on one install, with strict isolation (keys, cache, budgets, logs, ledger).
PII acceptance: 50+ test prompts, 95%+ of personal data removed before the provider, answers correct after restore.
Under 50 ms added on cache hits; target under 200 ms on cache misses.
All code in my company GitHub from day one, with tests and setup notes. Permissive open source only; ask before any GPL/AGPL code.
Weekly written updates and a demo at the end of each milestone.
TIMELINE
Start within a week; pilot-ready by end of December.
YOUR PROPOSAL MUST INCLUDE
A gateway, proxy or middleware project you built yourself (your role and code, not just a company website).
Your experience with LiteLLM, Presidio and AWS ECS/Fargate.
Your total fixed price and delivery date for the whole project.
Weekly hours and start date.
How you would handle the gateway-down fallback without bypassing PII protection.
Shortlisted candidates sign a short NDA before seeing the full brief. The best one or two may do a small paid trial (the UK PII detectors). Generic proposals will not be considered.
Apply on Freelancer →
Project sourced from Freelancer.com. Applications happen directly on the original platform — we never collect your data.