# Models

> A technical guide to building with open-weight models from Hugging Face, private frontier models and in-house models: licences, fine-tuning, serving, evaluation, routing, cost and sovereignty. Boltcliq is model-agnostic and deploys where you decide.

Canonical URL: https://www.boltcliq.com/solutions/models
Part of: https://www.boltcliq.com/llms.txt

Build your own, adapt open-weight models, or use private frontier models. We work with all three, deploy where you decide, route between them through one gateway and test them on your own cases.
- The model is a component. You should be able to build it, borrow it or rent it, move it, and replace it, without rebuilding the system around it.
- Three paths: Build in-house, adapt open-weight, or use private frontier models
- Model-agnostic: Any of them, mixed, behind one gateway
- Your location: On premises, private cloud, Boltcliq cloud or a local provider
- Your tests: Evaluation on your own cases before and after every change
### Three ways to get a model, and why you may use all three
- Almost every organization we meet wants one of three things. Some want to build their own models in-house, because the data is theirs and the use case is narrow. Some want to use open-weight models, the kind published on Hugging Face, because they can run them anywhere and adapt them. Some want the best private models from the frontier labs, because quality matters most and they do not want to run GPUs.
- All three are reasonable. None of them is right for every task. A bank may use a frontier model to draft internal research, an open-weight model on its own servers to read customer files, and a small in-house model to score transactions. The mistake is picking one path for the whole company and then bending every use case to fit it.
- Boltcliq is model-agnostic. We build with all three, deploy where you decide, route between them through one gateway, and evaluate them on your own cases. This page is a technical explainer: what each path involves, what to check, which tools matter, and where the traps are. It is written for engineers and for the people who sign the contracts.
- A note on names. We mention model families, labs and tools so that you can see what we work with. A name on this page means we design for compatibility. It does not mean the vendor endorses us, partners with us or has reviewed this page. Licences, prices and capabilities change often, so confirm current terms with the source before you commit.
### The ecosystem we work across
- Open-weight and hubs
- huggingface
- meta
- mistral-ai
- qwen
- deepseek
- google
- microsoft
- Private frontier labs
- openai
- anthropic
- google
- xai
- Cloud and hardware
- aws
- microsoft
- google
- nvidia
- Names and logos show compatibility only. They do not imply partnership, endorsement or certification by the companies shown.
### Build, adapt or rent: the honest comparison
- Build or adapt (in-house and open-weight)
- Use a private frontier model
- Cost: capital and engineering up front, then GPU-hours or hardware. Cheap per call at steady, high utilisation.
- Cost: pay per token, nothing up front. Cheap at low or spiky volume, grows with usage.
- Control: you choose the weights, version, location and update date. Nothing changes unless you change it.
- Control: the lab sets the model, versions and retirement dates. You control the prompt and the contract.
- Privacy: data can stay inside your boundary, including air-gapped sites.
- Privacy: data goes to a provider under contract terms. Retention, region and training use must be checked.
- Quality ceiling: strong open models are close for many tasks, but the very hardest reasoning tasks often favour frontier models. Confirm on your own cases.
- Quality ceiling: usually the highest on broad, hard tasks, and it changes every few months.
- Speed to value: weeks for adaptation, months for a model built from scratch.
- Speed to value: days. An API key and a prompt.
- Lock-in: low on weights you hold. Some tooling and hardware lock-in remains.
- Lock-in: moderate. Prompts, tool formats and fine-tunes are tied to one vendor unless you abstract them.
- Skills needed: ML engineering, data engineering, GPU operations, evaluation.
- Skills needed: prompt design, integration, evaluation, contract and risk review.
### When to choose which
- Choose in-house
- The task is narrow and valuable: scoring, forecasting, speech or vision in a local language, document classification at volume. You have labelled data nobody else has.
- We design, train and validate a compact model, often a few million to a few billion parameters, with a repeatable training recipe, and deploy it on hardware you choose.
- You decide whether the data may be used for training, what accuracy is acceptable, and who owns the weights.
- Choose open-weight
- Data cannot leave your boundary, or you need predictable cost at scale, or you need to adapt a model to your domain and language and keep the result.
- We select candidates, check licences, evaluate them on your cases, then adapt with retrieval or fine-tuning and serve them on your infrastructure.
- Legal confirms the licence fits your use. Risk decides which tasks the model may handle without a person in the loop.
- Choose private frontier
- You need the strongest general reasoning, coding or multimodal ability, volume is moderate, and the data involved is not restricted from leaving your boundary under a suitable contract.
- We integrate through APIs or your cloud provider's private endpoints, add redaction and cost caps at the gateway, and build prompts and evaluations that survive a model change.
- Security and procurement approve the vendor terms, region and retention settings. Owners set the monthly budget.
- Choose a mix
- You have several use cases with different sensitivity, volume and difficulty. This is the common case.
- We route by policy: small local model first, escalate to a frontier model when confidence is low and the data allows, with every call logged. The mix can change as models and prices change.
- You approve the routing rules and review the monthly report of what went where and at what cost.
### Open-weight model families
- Open-weight means you can download the weights and run them yourself. It does not always mean open source. Training data and code are often not released, and licences differ. This is a snapshot of families we work with. New releases arrive monthly, so treat it as a map, not a ranking.
**Llama (Meta)**
- A widely used family of general language models, with strong tooling support. Released under a community licence that carries usage conditions, including thresholds for very large services and an acceptable use policy. Confirm current terms.
- General
- Community licence
**Mistral and Mixtral**
- Compact and efficient models from Mistral AI, including mixture-of-experts designs. Some releases use Apache-2.0, others use research or commercial licences. Check each release separately.
- Efficient
- Licence varies
**Qwen (Alibaba)**
- A broad family covering general language, code, vision-language and embeddings, with good multilingual coverage. Several sizes use Apache-2.0, others use a custom licence. Confirm the licence for the exact checkpoint.
- Multilingual
- Sizes vary
**DeepSeek**
- Reasoning-focused and general models, some under MIT-style terms. Large mixture-of-experts checkpoints need serious hardware, but distilled smaller versions exist. Review provenance and your own policy on model origin.
- Reasoning
- Large
**Gemma (Google)**
- Smaller models designed to run on modest hardware. Released under Google's own terms of use that include a prohibited use policy. Confirm current terms before commercial use.
- Small
- Custom terms
**Phi (Microsoft)**
- Small language models trained with a strong focus on data quality, useful for on-device and low-cost serving. Several releases use the MIT licence. Confirm per release.
- Small
- Often MIT
**Whisper and speech models**
- Whisper is an open speech recognition model with permissive licensing at the time of writing. Quality in low-resource African languages varies a lot, so measure it, and plan to fine-tune with your own labelled audio.
- Speech
- Permissive
**Embedding and rerank models**
- Open embedding models such as BGE, E5 and GTE style families, plus open rerankers, power retrieval. They are small, fast and easy to host. Multilingual coverage matters more than leaderboard rank for local-language corpora.
- Retrieval
- Small
**Vision and others**
- Open vision-language, segmentation and detection models (for example SAM and YOLO style families, ViT backbones), plus OLMo-style fully open research models. Licences range from permissive to copyleft. Check each.
- Vision
- Mixed
### Licence terms to check before you build on a model
- What it usually allows
- What to check
- Apache-2.0 / MIT
- What it usually allows: Commercial use, modification and redistribution, with notice requirements. Apache-2.0 adds a patent grant. | What to check: That the licence applies to the weights and not only the code. That attribution notices ship with your product.
- Community licences with usage limits
- What it usually allows: Commercial use for most organizations, often with an acceptable use policy and a user-count threshold. | What to check: Thresholds, naming and attribution rules, limits on using outputs to train other models, and what happens if you grow past a limit.
- OpenRAIL-style licences
- What it usually allows: Broad use with behavioural restrictions that pass down to derivatives. | What to check: The use restrictions list, how it flows to your customers, and whether your use case touches a restricted area.
- Research or non-commercial
- What it usually allows: Evaluation and academic work. | What to check: Do not ship in a product. Look for a separate commercial grant.
- Training data and outputs
- What it usually allows: Varies. Some licences speak to outputs, many do not speak to training data provenance. | What to check: Data provenance statements, opt-out handling, and your own sector rules on data origin.
- This is general information, not legal advice. Licences change between releases. Confirm current terms for the exact checkpoint, and have counsel review before production use.
### Working with the Hugging Face hub, safely
- The hub is an excellent resource and also an open registry that anyone can upload to. Treat a downloaded model like a downloaded dependency: pin it, scan it, and store a copy you control.
**Read the model card**
- Check the licence field, intended use, training data notes, known limits, languages and evaluation claims. Treat claims as hints, not proof. A missing card is a warning.
**Prefer safetensors**
- Safetensors files hold only tensors and cannot execute code on load. Python pickle files (.bin, .pt, .pkl) can run arbitrary code when loaded. Prefer safetensors, and refuse pickles unless they are scanned and from a trusted source.
- Never enable trust_remote_code on an untrusted repository without reading the code first.
**Pin an exact revision**
- Download by commit hash, not by branch name, so the weights cannot change under you. Record the hash, the licence and the date in your model register.
**Scan and verify**
- Run a pickle scanner and a malware scan on every artifact, verify file hashes, and check the tokenizer and config files for unexpected custom code. Run the first load in a sandbox with no network.
**Mirror into your registry**
- Copy the approved revision into your own artifact store, such as an S3-compatible bucket or an OCI registry. Production pulls from the mirror only. Air-gapped sites receive the mirror by approved media transfer.
**Version datasets the same way**
- Datasets from the hub get the same treatment: licence check, pinned revision, and a record of what was filtered. Your own training data is versioned and hashed so a model can be traced to the exact data behind it.
### How you adapt a model to your work
- Start with the cheapest technique that solves the problem. Most projects need retrieval and good prompts first. Fine-tuning comes after you have an evaluation set that proves it is needed.
**Retrieval (RAG) with pgvector**
- Index your documents in PostgreSQL with the pgvector extension, retrieve relevant passages at question time and give them to the model with citations. Knowledge stays in your database, can be updated in minutes and respects access rules.
- Chunking and metadata that match how people search
- Hybrid search: keyword plus vectors
- Permission filters applied before retrieval
- Fresh, traceable answers without retraining.
**LoRA, QLoRA and PEFT**
- Parameter-efficient fine-tuning trains small adapter matrices and leaves the base weights frozen. QLoRA does it on a quantised base, so a single workstation GPU can tune models that would otherwise need a cluster. Adapters are small files you can version and swap.
- One base model, many task adapters
- Adapters can be merged or kept separate
- Cheap to retrain when data changes
- Domain behaviour at a fraction of full tuning cost.
**Supervised fine-tuning (SFT)**
- Train on prompt and response pairs that show the format, tone and judgement you want. Quality beats quantity: a few thousand clean, reviewed examples often beat a large noisy set.
- Expert-written or expert-reviewed examples
- Held-out evaluation split
- Train on formats, not facts
- Reliable structure and style.
**Preference tuning**
- Methods such as DPO and related approaches teach a model to prefer better answers over worse ones using ranked pairs. Useful for tone, refusal behaviour and reducing unwanted patterns. Needs careful preference data.
- Pairs ranked by domain experts
- Guardrails against reward gaming
- Re-evaluate for regressions
- Shapes behaviour that examples alone do not.
**Distillation**
- Use a large teacher model, where its licence permits, to produce training data for a small student model that handles one task cheaply. Check the teacher's terms: some forbid using outputs to train competing models.
- Task-specific student models
- Lower cost and latency
- Licence check on the teacher
- Frontier-level behaviour on a narrow task, at small-model cost.
**Quantisation and merging**
- Quantisation stores weights in fewer bits (GGUF for llama.cpp, AWQ and GPTQ for GPU serving) so models fit smaller hardware, with some quality loss that you must measure. Merging combines weights or adapters from related models. It can help, and it can quietly break behaviour.
- 4-bit and 8-bit variants compared on your tests
- Merged models re-evaluated in full
- Record the recipe
- Fit the hardware you actually have.
### Building a model in-house
- Building a model means you train it on your data and own the result. It does not mean training a general chatbot from scratch. That costs enormous compute and data, and almost no organization needs it. Be honest about this early.
- Where building makes sense: a domain-specific small model such as a classifier, a ranker or an extractor. Speech recognition or text-to-speech for languages the big models serve poorly. Vision for local conditions, such as crop disease, damaged roads or medical images. Tabular and forecasting models for demand, credit, churn, energy load or fraud. These tasks are well served by models with millions to a few billion parameters.
- Continued pretraining sits between adapting and building. You take an open base model and keep training it on a large body of domain text, such as legal, medical or a language it saw little of. It needs far more data and compute than fine-tuning, and it can cause the model to forget things. Do it only when retrieval and fine-tuning clearly fall short.
- Training from scratch is rarely justified for language models. The cost is in data curation, compute, failed runs and the team to run them. For tabular, vision and speech tasks it is more common and much cheaper.
- Data: labelled, representative, legally usable, versioned
- Compute: usually a few GPUs for days, not a cluster for months
- Team: data engineer, ML engineer, domain expert, evaluator
- Output: weights, training recipe, data lineage, evaluation report
### Using private frontier models well
- Frontier models from OpenAI, Anthropic, Google and other labs are often the strongest option for hard, general tasks. You can reach them through the lab's own API, or through a cloud provider's private endpoint such as Amazon Bedrock, Azure OpenAI or Google Vertex AI. Availability of each model on each route changes, so confirm it.
**Direct API**
- The fastest route and the newest models. Data goes to the lab's infrastructure under its commercial terms. Check retention, training use, region and subprocessors.
- Zero-data-retention options may exist on request
- Regional endpoints may be limited
- Enterprise terms differ from consumer terms
**Cloud private endpoints**
- Models served through your existing cloud agreement, with network controls such as private links, your own keys and regional hosting. Often easier for procurement and for data residency discussions.
- Private networking and customer-managed keys
- Regional deployment choices
- Model catalogue may lag the lab's own API
**Why not route everything here**
- Cost grows with usage. Models are versioned and retired on the lab's schedule. Outages and rate limits are outside your control. Some data should never leave the building regardless of contract.
- Budget caps at the gateway
- Fallback models for outages
- Sensitive classes stay local
### Data-handling questions and a contract checklist
- Your data protection officer and your counsel should lead this, with engineering support. Boltcliq helps you structure the questions and test the answers, and we do not give legal advice. The aim is simple: know exactly what happens to every prompt and every output.
- Ask each provider in writing and keep the answers with the contract. Terms differ between consumer products, API products and cloud-hosted versions of the same model, so ask about the specific route you will use.
- Is customer content used to train or improve models? Is the opt-out the default?
- How long are prompts and outputs retained, and who can read them? Can zero retention be agreed in writing?
- In which regions is data processed and stored? Is regional-only processing offered for your region?
- Who are the subprocessors, and how are changes notified?
- What security attestations does the provider hold? Ask for current reports; do not rely on marketing pages.
- What are the breach notification times and the audit rights?
- Which model versions are guaranteed, for how long, and with what notice before retirement?
- What are the rate limits, uptime commitments and support terms?
- Who owns the outputs, and what indemnities cover them?
- What are the exit terms: data deletion confirmation and export?
### The hybrid pattern: one gateway, many models
- Monolith is the tool and model gateway in the Boltcliq platform. It is where provider routing, fallbacks, redaction and cost caps live, so that applications call one interface and policy decides which model answers. This is how a request flows.
**Request arrives**
- An application or agent sends a request to the gateway using a familiar OpenAI-compatible interface, with a task label and the identity of the caller.
**Policy check**
- Odin authenticates the caller and checks what this identity may do. Policy decides which models are allowed for this data class, region and budget.
**Small model first**
- For tasks marked routine, the gateway tries a small local or open-weight model first. Cheap, fast, and inside your boundary.
**Escalate on low confidence**
- If the answer fails a confidence or validation check, and policy allows, the request escalates to a larger or frontier model. Sensitive classes never escalate outside the boundary.
**Redact**
- Before any request leaves the boundary, the gateway removes or masks defined fields such as names, account numbers and national IDs. Mappings stay local so answers can be restored.
**Log and learn**
- Cyclopes records who called what, which model answered, the cost and the outcome. Escalated cases feed the next evaluation set and the next fine-tune.
### What a call through the gateway could look like
- OpenAI-compatible chat API
- Lets existing SDKs and tools point at the gateway.
- Model registry and routing policy
- Declares allowed models per task and data class.
- Eval and cost reports
- Per route, per team, per month.
- Connectors to hub mirrors and private endpoints
- The idea is stable: applications ask for a task, not a vendor, and policy picks the model.
- The route name, headers and metadata fields are placeholders.
### Serving and hardware, in plain terms
- Serving is where many pilots stall. A model that works in a notebook still has to meet latency, uptime and cost targets for real users.
**Memory arithmetic**
- A rough rule: weights need about the parameter count times bytes per parameter. A 7 billion parameter model in 16-bit needs about 14 GB just for weights. At 4-bit it needs about 3.5 to 4 GB. Add memory for the KV cache, which grows with context length and the number of concurrent users, plus runtime overhead. A 70 billion parameter model at 4-bit needs roughly 35 to 40 GB for weights before the cache, so it spans one large GPU or two smaller ones. Treat these as estimates and test on the real stack.
**GPU classes**
- Data-centre GPUs with high memory and fast interconnect suit large models and heavy traffic. Mid-range data-centre or workstation GPUs suit small and medium models. Consumer GPUs can serve small models for tests and low traffic, but check the vendor licence terms for data-centre use.
**CPU-only options**
- Quantised small models (roughly 1 to 8 billion parameters) run on modern server CPUs with llama.cpp or ONNX Runtime. Throughput is lower and latency higher, but it works with no GPU. A fit for back-office batch jobs, branch sites and fallbacks.
**Batching**
- Continuous batching serves many requests on the same GPU at once. It is the biggest lever on cost per token. Better utilisation means lower unit cost, but it needs enough traffic to fill the batch.
**Caching**
- Prefix caching reuses work on shared system prompts and documents. Response caching avoids repeat calls for identical questions. Embedding caches avoid recomputing vectors. Cache keys must respect access rules.
**Autoscaling**
- Scale replicas on queue depth and latency, not CPU. GPU nodes start slowly, so keep warm capacity and scale to zero only for non-urgent workloads. Plan a fallback route for when capacity runs out.
### Evaluation: how you know a model is good enough
- Public leaderboards tell you little about your documents, your customers and your languages. The only evidence that counts is a test built from your own work.
**Golden sets**
- A curated set of real inputs with expert-approved answers or grading rules. Start with 100 to 300 cases and grow it. Cover easy, hard and adversarial cases, and each language you serve. Keep a hidden slice that is never used for tuning.
- Built with the people who do the work
- Versioned like code
- Refreshed from production failures
**LLM-as-judge, with caveats**
- A strong model can grade answers at scale, and this is useful for screening. Judges favour longer answers, favour their own style and can be wrong in specialist domains and low-resource languages. Calibrate against human grades and keep a human-reviewed sample.
- Calibrate on a human-labelled slice
- Use clear rubrics
- Never the only gate for high-stakes tasks
**Regression gates**
- Every change to a prompt, adapter, quantisation, base model or retrieval index runs the golden set before release. If a score drops below the threshold, the release stops. This is what makes a model swap safe.
- Automatic in the release pipeline
- Per-task thresholds
- Compare against the current production model
**Red-teaming**
- Test for prompt injection, data leakage, jailbreaks, bias, harmful output and unsafe tool use. Include local-language and code-mixed attacks, which generic safety tests miss. Repeat after every significant change.
- Injection through retrieved documents
- Leakage of private training data
- Tool actions that need approval
### The cost model, framed honestly
- We do not publish a break-even number, because there is no honest universal one. It depends on utilisation, model size, context length, hardware prices in your region and what you count as cost.
**Per-token pricing (APIs)**
- You pay for input and output tokens. No idle cost, no ops team, and prices have tended to fall over time. Cost grows linearly with usage, and long documents or agent loops can multiply tokens quickly.
- Best for low, spiky or uncertain volume
- Set budgets and per-task caps
- Model the cost of retries and long contexts
**GPU-hours or owned hardware**
- You pay for capacity whether it is used or not. At steady, high utilisation the cost per token can be much lower. At low utilisation it can be much higher, because an idle GPU still costs money.
- Best for steady high volume or restricted data
- Include staff, power, cooling and spares
- Quantisation and batching lower unit cost
**The break-even question**
- Compare the monthly API bill for a realistic workload with the monthly cost of the smallest hardware that meets your latency target, including people. The crossover depends on utilisation. We build this model with your numbers and show the assumptions.
**Costs people forget**
- Evaluation runs, data preparation, re-tuning after model updates, security reviews, monitoring and failover capacity. In-house paths shift cost from tokens to people and time.
### What we do and do not do
- Help you choose between building, adapting and renting, task by task
- Select, evaluate and adapt open-weight models, and mirror them into your registry
- Build compact in-house models on data you own
- Integrate private frontier models through APIs and private endpoints
- Route, redact, cap cost and log across all of them through one gateway
- Deploy on premises, in private cloud, on Boltcliq cloud or with a local provider
- Build evaluation sets and regression gates on your own cases
- Claim a model is best without testing it on your cases
- Give legal advice on licences, residency or contracts
- Promise that a licence or capability will not change
- Train a general frontier model from scratch for you
- Imply endorsement or partnership by any model lab or vendor
- Send restricted data to a provider your policy does not allow
### Frequently asked questions
- What does open-weight mean, and is it the same as open source?
- Open-weight means the trained parameters are downloadable. Open source in the strict sense also includes the right to use, study, modify and share under an approved licence, and often training code and data. Many open-weight models carry custom licences with conditions. Read the licence for the exact model.
- Can we use Llama or Gemma commercially?
- Often yes, under their own terms, which include usage conditions and acceptable use policies. Terms change between versions. Confirm current terms and have counsel review before production use.
- Are open models as good as frontier models?
- For many focused tasks, a well-chosen and adapted open model is good enough. For the hardest general reasoning, frontier models often lead, and the gap moves over time. The only reliable answer comes from testing on your own cases.
- Do we need GPUs?
- Not always. Small quantised models run on CPUs for light or batch workloads. Interactive, high-volume or large-model use usually needs GPUs, which you can own, rent from a cloud or a local provider, or avoid by using APIs.
- When is fine-tuning worth it?
- When retrieval and prompt work have been tried, an evaluation set shows a clear gap, and you have enough clean examples. Fine-tuning teaches style, format and judgement, and it is not a good way to add fast-changing facts.
- Is it safe to send data to a frontier provider?
- It depends on the data, the provider's terms, the region and your regulator. Ask the contract questions in writing, redact what you can, and keep restricted data classes on models you host. Counsel and your security team decide.