We started with a monolithic ML application.

One codebase, one deployment, everything tightly coupled.

It worked great until we needed to update the embedding model without redeploying the entire system. Or scale the inference service independently of the data processing pipeline. Or use different GPU types for different components.

That's when we moved to microservices. And it was both the right decision and a painful one.

What we split: document processing service, embedding service, retrieval service, LLM inference service, and the orchestration layer. Each deployable and scalable independently.

What went well: individual services can be updated without affecting others. Scaling is granular — we can add more inference pods during peak hours without over-provisioning everything else. Different services can use different hardware.

What was harder than expected: inter-service communication adds latency. Debugging across services is more complex. Data consistency between services requires careful design. The deployment pipeline became 5x more complex.

My recommendation: start monolithic. Split only when you feel specific pain — scaling constraints, deployment coupling, or team coordination issues. And split surgically — separate the component causing the pain, not everything at once.

Premature microservices are as costly as premature optimization. Let the pain guide the architecture.