2 posts tagged with "engineering"

LiteLLM isn't an infrastructure detail. It's the money bottleneck for almost everything we do with models: the Resizes AI chat, customer agents, internal pipelines. When the bill went up, the instinct was "the chat is expensive." The truth was broader: a shared proxy, no map, and an agent that turns one message into a loop of calls.

This post covers how we saw it, how we almost broke it by blindly cutting turns, and what ended up in code: per-agent and per-conversation attribution, less context, fewer tools, a real cap in the runtime, and a Grafana dashboard covering everything that goes through LiteLLM.

Most organisations do not have a “secrets strategy.” They have an AWS line item that grows every time someone creates another credential — and a platform team that still opens tickets to wire the next service into IAM.

AWS Secrets Manager charges $0.40 per secret, every month, plus API usage (official pricing). There is no volume discount. Forgotten secrets still bill. Cross-region replicas bill again. Copy the same logical secret into dev, staging, and prod and you pay three times.

That is not only a FinOps problem. It is an efficiency problem: two sources of truth (password manager for people, Secrets Manager for apps), cloud-locked access paths, and more glue each time you add a cluster.

We help organisations quantify that cost, redesign how secrets are delivered, migrate safely, and leave teams with an operating model that is cheaper and easier to run.