Deployment and Project Integration · Lesson 6
Orchestrating Modular GenAI Systems
Production GenAI apps are made of modules: ingestion, embedding, retrieval, LLM gateway, agents, caching and the user-facing API. Keeping them separate lets you scale and swap parts independently.
An LLM gateway centralises calls to providers, adding routing, fallbacks, retries, caching and cost tracking. Queues (Celery, Redis, Pub/Sub) handle long jobs like document ingestion in the background.
Define clear interfaces between modules so you can change a model or vector store without rewriting the app.
Key points
- Split ingestion, retrieval, generation and API
- Gateway for routing, fallback and cost
- Queues for long background jobs
- Clean interfaces allow swapping parts