PrivEmbed: private embedding models

PrivEmbed is an internal project we built to evaluate, deploy, and benchmark embedding models as an alternative to managed embedding services. It exists because embeddings are one of the few places where going fully private is both practical and clearly worth it. Embedding models are far smaller and easier to serve than large language models, so a team that wants a fully private AI stack, and does not want to keep paying per call for managed inference, can host its own with very little overhead.

The short version of what we measured: served through Hugging Face Text Embeddings Inference on a single GPU, the fastest open models we tested reached roughly 136,000 tokens per second under sustained load, and answered a single idle request in about 180 milliseconds.

Owning the embedding layer instead of renting it

The embedding layer is the one part of a retrieval stack a team can own outright without taking on the cost of hosting a large language model, and three things follow from that.

Text never leaves the client’s infrastructure. Every document sent for embedding goes, in full, to whoever runs the model, and for regulated data or for a company whose knowledge base is the product, that is the whole argument on its own.

The economics invert once volume is steady. Managed embedding is billed per call forever, while a self-hosted model is a fixed cost already paid, and because embedding models are compact, one GPU absorbs a great deal of throughput before a second is needed. There is no per-token meter running against a corpus the client already owns.

The lock-in is worse than it looks, and this is the part teams underestimate. When embeddings come from a managed provider, changing the embedding model means re-embedding and reindexing the entire corpus through that provider at their per-call price, so the larger the vector database grows the more expensive it becomes to leave, and the more committed the team is to whatever models that provider chooses to offer. With a self-hosted model the same reindex is compute already owned, which preserves the freedom to swap models as better ones ship or to fine-tune one to a specific domain. The embedding layer becomes something owned rather than rented.

Benchmarking every model under identical load

We serve each candidate model with Hugging Face Text Embeddings Inference in Docker, then drive load against its /v1/embeddings endpoint with Locust, running the identical protocol against every model so the comparison is apples to apples.

Text comes from the public HLE questions dataset, chunked with each model’s own tokenizer and validated so no chunk exceeds the 512-token limit once special tokens are added. Chunking with the model’s own tokenizer rather than a generic splitter matters, because a chunk that is valid for one model can overflow another.

We run two separate tests, because they answer different questions.

Throughput in tokens per second is derived rather than measured directly, as requests per second multiplied by batch size multiplied by average tokens per chunk. Locust reports request rates, so this is the honest conversion rather than an instrumented token counter.

What five open models deliver on one GPU

Five current multilingual models, all measured on the same single-GPU deployment under the throughput test above. MTEB rank comes from the public leaderboard and stands in as a proxy for retrieval quality; every other column is ours.

Idle latency, measured with a single user, tells a different and more flattering story.

These are relative comparisons on identical hardware, which is what the table is for; absolute figures would need the exact GPU stated alongside them to mean anything.

Three things fall out of the results. Model size predicts throughput almost perfectly, with the two 0.3B models at 768 dimensions running at roughly 2.2 times the throughput of the 0.6B, 1024-dimension models and doing it at a third of the idle latency, and nothing else in the table explains the spread as cleanly as parameter count.

Models of the same class are interchangeable on performance. gte-multilingual-base and arctic-embed-m-v2.0 are separated by 0.1 percent on throughput and are identical on latency, and the same holds for bge-m3 against arctic-embed-l-v2.0, so once a size class is settled the choice should come down to retrieval quality and licence rather than speed.

Latency under load is not the latency measured in isolation. Every model answers in well under a second when idle and takes between 4.8 and 11 seconds at 30 concurrent users pushing batches of 96, which is queueing rather than slowness, and it is the number that bites in production. The deployment has to be sized for the concurrency actually expected, or capped.

Trading retrieval quality against throughput

Across this set the relationship is close to linear, and it is a decision worth making deliberately rather than by default. Moving from arctic-embed-m-v2.0 at MTEB rank 47 up to Qwen3-Embedding-0.6B at rank 8 costs 54 percent of throughput and triples idle latency.

That trade is worth making for a small, high-value corpus where retrieval precision decides whether the product works at all. It is a bad trade for a large corpus with heavy ingest, where the difference gets spent on hardware to embed the same documents more slowly. Our default recommendation is to start at the 0.3B tier, measure retrieval quality on the client’s own data, and only move up if the evaluation says it is needed.

Worth being explicit about the split in what is measured here. Throughput and latency are ours, taken on identical hardware. Retrieval quality is the public MTEB ranking rather than something we evaluated independently, and while MTEB is a reasonable proxy it is not anyone’s actual corpus, which is exactly why the recommendation above ends with measuring on real data.

How the cost shape differs from managed

The honest answer on absolute cost is that it depends on hardware and volume, and any figure quoted without both is marketing. What we can say structurally is that the shape of the cost is different, not merely smaller.

Managed embedding is a variable cost that never stops and scales with success. Self-hosted embedding is a fixed cost provisioned once, with a marginal cost near zero until the GPU saturates. The crossover point is the only number that matters, and it arrives sooner than most teams expect, because embedding models are small enough that a single GPU absorbs a lot of corpus.

The reindex case is where the gap is starkest. Re-embedding a large corpus through a managed API is a line item that has to be justified. Re-embedding it on hardware already owned is an afternoon.

Frequently asked

Does this need a GPU? For production throughput, yes. TEI does ship a CPU image, and PrivEmbed includes a CPU compose override that runs on machines without an NVIDIA card, including Apple Silicon. It is slower by a wide margin and it is fine for development, evaluation, and small corpora.

Can this run inside our own AWS account? Yes, and that is the point. TEI is a container, so it deploys on GPU instances inside the client VPC with no public egress and no third-party API in the path. Documents never leave the account.

Which model would you pick for us? A 0.3B model at 768 dimensions unless there is a specific reason not to, because it costs nothing in speed and very little in rank. Then evaluate retrieval on real data before committing, since MTEB rank is a proxy and no client corpus is MTEB.

If we switch models later, do we have to re-embed everything? Yes. Embeddings from different models are not comparable, so changing the model means reindexing the corpus. That is true whether the model is self-hosted or not, which is precisely why owning the compute matters: the same reindex is a fixed cost on your own hardware and a variable bill on someone else’s.