top of page
Step 1 - Select 'On-device LLM' from the AI Selection list
Step 2 - Select a model from the Model Selection list
If you have already downloaded a model, it will appear under Model Selection
If you only see 'No Model Selected' then you will need to download a model
-
The app provides an option to download a suggested model, or if you have already downloaded one, load it from the file system.
-
You also have the ability to download a language model (e.g. from HuggingFace) in the .gguf format by providing a URL to a model
-
Selecting a model causes it to load into your device's memory. Large models take a long time. A green tick badge will display next to the model once it has loaded.
Step 3 - Configure your local AI Model
This can be tricky when using devices with limited memory. 16GB is the minimum recommended. Changes to settings can cause undefined behaviour
-
Adjust the slider on the Context Size. If suitable, an 'Apply Recommended Settings' button should appear.
-
Look at the provided model information for the memory pressure. Do not exceed the GPU budget as this will cause undefined behaviour.
Remember that large, more capable models require lots of memory and processing power. We recommend that you use Cloud-based services to get the best behaviour out of your assistants.
The capabilities of small language models are improving all the time. Sign up to keep up to date on the best models to try. Sign up here.
bottom of page