Topic hub
Putting AI inside software that already has customers: which workflow to start from, how to integrate it, and what it takes to make the feature worth paying for.
Start with the pillar guide
Start from a workflow your users already repeat, not a chat box. Pick the simplest integration pattern that fits, enforce tenant isolation in code rather than in the prompt, and build the eval harness before the feature. What decides whether an AI feature survives is rarely the model.
Read the guideDecide first whether AI is a core differentiator or a supporting feature, because most products are the second and plan as though they are the first. For a first feature the structure that usually wins is a partner building it with your engineers embedded, handing over the gateway, evals and telemetry, with hiring deferred until you know what the role requires.
Most AI features fail in the interface, not the model. Put the output where the work already happens rather than in a side chat panel, show sources the user can check in seconds, make editing the expected path, and confirm before the feature acts. Then measure repeat use and acceptance, not activation.
Model APIs fail, throttle, and slow down, so treat one as a remote call that can be slow or unavailable rather than a function call. Set a latency budget per feature, retry with backoff and jitter under a capped budget, fail over to a second provider that passed the same evals, and make sure the product still works with every AI feature off.
Build on the cloud where your data, identity, and spending commitments already live. Go direct to OpenAI only if you need its newest features and limits the day they ship. Either way, put a thin gateway between your product and the provider, so the platform choice stays reversible.
There is no single best LLM, only the best model for each task your product performs. Choose the smallest one that clears a written quality bar on your own data, at a cost and latency you can live with, and name a fallback from a second provider. Put model IDs behind a gateway: releases, price cuts and retirements will change the answer within months.
First estimates count one model call with a short prompt. Production runs several per user action, with growing context, retries and regenerations, which is where the bill actually comes from. Map the call graph, measure real token counts, then cache the static prefix, trim context, and route easy work to a smaller model.
