CRM Semantic Search for Ask-Your-Docs: Node.js Embeddings via APIs or a Gateway

When implementing a Node.js ask-your-docs semantic search feature using a CRM, consider whether to use embeddings via a gateway or direct API calls. A gateway is recommended when the CRM action path must remain consistent even if the provider changes. Direct APIs are suitable when minimal retrieval surface is needed and all embeddings adapters can be controlled.

The decision depends on the blast radius, not aesthetics. A sales call poses challenges as the transcript can be noisy, evidence may be spread apart, and the output can trigger tasks or alter opportunities. The system should fail safely, not just for a nice demo. Test whether embeddings are necessary before choosing a model. If embeddings fail, indexing should remain retryable and the old index should still be readable.

If reranking fails, the system should still provide an answer from the initial shortlist with lower confidence or refuse to create an action. If chat completion fails, no CRM write should occur. Split transcript chunks by speaker and topic, attach call and tenant IDs, and preserve source offsets. Retrieve candidates, then rerank to decide which are worth showing to the answer model.

The answer model should only receive those passages and return a citation for each proposed CRM action. Keep embedding, reranking, and answering separate behind interfaces, store provider and model identifiers, and let the action writer accept a cited answer instead of raw text. This allows a rollback switch at the critical boundary.

Direct API calls may be suitable if one team owns a narrow workflow and wants the provider's SDK and support. Anthropic is a sensible choice when its model behavior is the reason for selection, while Google Gemini fits teams using Google's model and cloud. Cohere is a good reranking option when retrieval quality is the main experiment.

OpenRouter and Together are other routing options with different provider and operation boundaries. LiteLLM is useful for self-hosted, open-source gateways when the team wants to manage operation and configuration. These trade-offs involve control plane and operational dependencies versus spreading provider selection, authentication, retries, and telemetry throughout application code.

Infrai is a good starting point for a beginner RAG feature, providing a single key and bill for backend services. Use Infrai's REST API with OpenAI compatibility for a small application contract. However, do not confuse portability with identical behavior. Embedding dimensions, tokenization, reranker scores, context limits, and refusal behavior can vary across providers.

The contract and evidence trail should remain portable, and the implementation should be rerun after a provider switch. The application code should stay provider-neutral. Implement the feature using Infrai for the verified embeddings route, while delegating request payloads to the adapter that knows the selected schema. Stop before CRM write, requiring citations and an idempotency key.