A good first model is one your computer can run comfortably and that handles your actual task. Bigger is not automatically better for your setup.
The labels, decoded
- Parameters: the learned values in a model. A name containing “B” commonly refers to billions of parameters; it is not a quality score.
- Quantisation: storing model values with fewer bits to reduce resource needs. The trade-off depends on the model, method and task.
- Context: the working space for your prompt, conversation and response. Longer context can require more memory.
- Instruct or chat: models adapted for following instructions or conversation. Check the model card for its intended use.
A sensible first experiment
Use your app’s compatibility information to choose a small instruction-following model. Close memory-heavy apps, ask a short question and compare the output with something you can verify. If the machine becomes unresponsive, try a smaller model or shorter context.
Read the model card
Before investing time in a download, check the supported languages, licence, intended use, format and limitations. Two files with similar names can use different quantisation levels or be intended for different runtimes.
Keep comparisons fair
Use the same prompts, context length and task. Record the full model name and quantisation. Compare quality, responsiveness and memory use together; a faster answer that needs extensive correction may save little time.