On-Device Multimodal Inference with llama.cpp and SmolVLM

Background

Our app uses image understanding in several places: searching user-created images, adding detail to image prompts, and drafting captions for posts. Most of this work currently runs on server-side models. To cut API costs, avoid uploading images, and reduce network latency, we began looking at models that could run directly on the device.

After looking at inference frameworks such as MLX and llama.cpp, we chose SmolVLM running in llama.cpp as a concrete case to study. This post starts with the model architecture, then traces a complete inference pass through the llama.cpp code. There were two practical reasons for choosing this pair:

  • SmolVLM comes in several quantized variants that are small enough for on-device use, and llama.cpp supported the model early on.
  • Because llama.cpp is written in C++, we can share the core inference code between iOS and Android instead of maintaining another platform-specific implementation.

VLM Architecture

This section draws on the SmolVLM paper.

A typical VLM has three core components: a vision encoder, a projector, and a large language model. The vision encoder extracts image features, the projector converts them into embeddings the language model can consume, and the LLM combines those embeddings with text tokens to understand the input and generate a response.

On-Device Multimodal Inference with llama.cpp and SmolVLM: VLM Architecture (figure 1)

SmolVLM keeps this basic layout, with a few changes to reduce memory use and make inference cheaper. Images or video frames are encoded into visual features and combined with text embeddings before the mixed sequence enters the language model. In SmolVLM, the three parts are:

  1. Vision Model: a SigLIP encoder processes input images or video frames and extracts visual features.
  2. Projector: Pixel Shuffle first compresses the visual features, which are then projected into the LLM’s embedding space.
  3. LLM: based on SmolLM2, it consumes the mixed visual-text sequence and generates the response.

Image Splitting in SmolVLM

SmolVLM splits an image before visual encoding. Here, splitting does not mean object-level semantic segmentation. The model divides the image into a regular grid based on its dimensions and input limits, while also keeping a low-resolution thumbnail of the whole image. The crops preserve local detail; the thumbnail preserves the broader context.

  • Encoding the full image directly: resizing a high-resolution image such as 1920x1080 straight to the encoder’s fixed 512x512 input can erase small text and chart details.

  • Regular tiling: dividing the resized image into a fixed grid and encoding each crop separately preserves more local detail without changing the encoder’s input size.

Vision Encoder

SmolVLM’s vision encoder is based on SigLIP, short for Sigmoid Loss for Language-Image Pretraining. SigLIP uses a Vision Transformer architecture and learns compact visual representations through image-text pretraining. Unlike CLIP’s global softmax, SigLIP computes a sigmoid loss independently for each image-text pair. It does not need to normalize all similarities across a batch and is therefore less dependent on large batch sizes.

On-Device Multimodal Inference with llama.cpp and SmolVLM: Vision Encoder (figure 2)

On-Device Multimodal Inference with llama.cpp and SmolVLM: Vision Encoder (figure 3)

Dimension CLIP SigLIP
Introduced OpenAI, 2021 Google, 2023
Training objective Applies global softmax normalization to image-text similarities within a batch Computes sigmoid loss independently for each image-text pair
Batch-size dependency Usually needs a large batch to provide enough negative samples Does not depend on global normalization and remains effective with smaller batches

The SigLIP variant depends on the SmolVLM model size and its intended use:

  • SigLIP base patch-16/512: approximately 93M parameters with 16x16-pixel patches. It is used by SmolVLM-256M and SmolVLM-500M for resource-constrained devices.

  • SigLIP-SO400M: approximately 428M parameters with 14x14-pixel patches. It is used by SmolVLM-2.2B for higher-performance scenarios.

SmolVLM’s experiments show that a larger vision encoder does not automatically make the whole model better. Paired with a 135M language model, the 428M vision encoder actually performed worse than the 93M version. Paired with a 360M language model, it improved performance by 11.6%, but increased the total parameter count by 66%.

For a compact VLM, scaling the vision encoder alone may not be a good trade-off. If the language model is too small, it may not be able to make full use of the additional visual features.

On-Device Multimodal Inference with llama.cpp and SmolVLM: Vision Encoder (figure 4)

This trade-off also explains why compact VLMs often devote more parameters to the language model than to the vision encoder.

Large Language Model

On-Device Multimodal Inference with llama.cpp and SmolVLM: Large Language Model (figure 5)

SmolLM2.jpg

Hugging Face’s SmolLM2 family contains 0.1B, 0.3B, and 1.7B models designed for on-device applications. Like SmolLM1, it focuses on compact models for situations where latency, privacy, and limited hardware matter.

SmolLM2-1.7B was trained on roughly 11 trillion tokens from datasets including FineWeb-Edu, DCLM, and The Stack. The corpus is primarily English, with additional work on text rewriting, summarization, and function calling. Its published training recipe is notable in several ways:

  • Long training runs: despite the model’s small size, it was trained on far more tokens than was common for earlier models of a similar scale.

  • Multi-stage training: training began with large-scale web text, then gradually introduced high-quality mathematics, code, and instruction data while dynamically adjusting the data mixture.

  • Dataset evaluation and selection: web, code, mathematics, and instruction-following data were evaluated separately, and the results guided the composition of the training set.

  • Dataset augmentation: FineMath, Stack-Edu, and SmolTalk were added to strengthen mathematical reasoning, coding, and instruction-following capabilities, respectively.

  • Online rebalancing: dataset proportions were continuously adjusted according to the model’s performance on key benchmarks.

Projector

SmolVLM’s projector takes the SigLIP features after Pixel Shuffle and maps them into the language model’s embedding space. Pixel Shuffle expands each visual token’s feature dimension by a factor of ; a linear layer then projects it to the LLM’s hidden size so it can be processed alongside text tokens.

This is a small, inexpensive layer. Its job is mainly to align dimensions and feature spaces, so some detail may be lost during projection, and it does not explicitly model richer cross-modal relationships.

Pixel Rearrangement

Pixel Shuffle was originally introduced for image super-resolution. It reshapes a feature map by rearranging its spatial and channel dimensions without discarding elements. SmolVLM runs the rearrangement in the opposite direction: some spatial information is folded into the channel dimension, leaving wider feature vectors and a shorter sequence of visual tokens for the language model.

On-Device Multimodal Inference with llama.cpp and SmolVLM: Pixel Rearrangement (figure 7)

For standard Pixel Shuffle, the shape transformation can be written as .

  • B: batch size.

  • : the total number of channels, containing sub-units, where is the scaling factor.

  • H: image height.

  • W: image width.

The expression above describes the super-resolution direction. SmolVLM uses the inverse rearrangement, which can be written as .

For example:

. This operation reduces spatial resolution while proportionally increasing the channel count.

In SmolVLM-256M, a 512x512 image is divided into 64 independent image blocks. Each block is encoded as one visual token, so the entire image can be represented with only 64 tokens. This design substantially reduces the number of visual tokens entering the language model, lowering memory usage and speeding up inference.

SmolVLM-256M uses SigLIP base patch-16. A 512x512 input with 16x16 patches initially produces 1,024 visual tokens. Pixel Shuffle then compresses the sequence to 64.

On-Device Multimodal Inference with llama.cpp and SmolVLM: Pixel Rearrangement (figure 8)

The 1,024 patches form a 32x32 spatial grid. Reducing the token count from 1,024 to 64 gives a compression ratio of:

. The spatial scaling factor is therefore 4, corresponding to , where is the token feature dimension. Folding spatial information into the channel dimension this way shortens the visual-token sequence and reduces the cost of both attention and the KV cache:

  • In a Transformer, attention computation usually grows quadratically with sequence length. If the token sequence length is Len and the feature dimension is Dim, then and have shape , while the attention matrix has shape .

  • During inference, the model generates and caches Key and Value vectors in the KV cache. The size of this cache is directly proportional to the input sequence length and batch size.

By the time they reach the language model, the visual tokens have been flattened into a one-dimensional sequence. Their two-dimensional relationships are carried mainly by the vision encoder and positional information; the LLM itself receives an ordered sequence of visual features. The SmolVLM paper describes this process in more detail.

Inference Framework

On-Device Multimodal Inference with llama.cpp and SmolVLM: Inference Framework (figure 9)

llama.cpp is a C/C++ inference framework for running large models locally across a wide range of hardware, with few dependencies.

Its main features include:

  • A pure C/C++ implementation with no external dependencies.

  • Native support for Apple silicon, with extensive optimization through ARM NEON, Accelerate, and Metal.

  • Architecture-specific instruction-set support for x86 and RISC-V.

  • Support for 1.5-bit, 2-bit, 3-bit, 4-bit, 5-bit, 6-bit, and 8-bit integer quantization, improving inference speed while reducing memory use.

  • Custom CUDA kernels for running large language models on NVIDIA GPUs, HIP support for AMD GPUs, and MUSA support for Moore Threads GPUs.

  • Vulkan and SYCL backends.

  • Hybrid CPU+GPU inference, allowing models larger than the available VRAM to be partially accelerated.

Our integration uses llama.cpp’s multimodal path, which exposes the language model, vision projector, and mixed image-text input through the same set of interfaces. The framework runs on both iOS and Android, so the model code can be shared while each client keeps only a thin platform-specific layer.

GGUF Model Files

GGUF is the model interchange and storage format used by the GGML ecosystem. Designed for fast loading and inference, it succeeds the GGML, GGMF, and GGJT formats. A GGUF file can hold model structure, configuration, and weights, while leaving room to extend its metadata without breaking compatibility. It is now used by many local inference tools beyond llama.cpp.

On-Device Multimodal Inference with llama.cpp and SmolVLM: GGUF Model Files (figure 10)

A GGUF file has three main sections: the header, metadata, and tensor data.

  1. Header: contains the GGUF magic number and format version, allowing the loader to identify the file and handle version differences.

  2. Metadata: stores the model architecture, name, embedding dimension, vocabulary, tensor information, and other configuration as key-value pairs. The loader therefore does not need a separate configuration file.

  3. Tensor data: stores weights, biases, and normalization parameters contiguously. Quantized models also store low-precision data and the information required for dequantization, organized into types such as Q4_0, Q5_K, Q8_0, F16, and F32. Tensors are usually aligned to specific byte boundaries for efficient memory mapping and loading.

Compared with Safetensors, GGUF is more self-contained for inference. Alongside the weights, it can store the model architecture, quantization method, vocabulary, and other metadata. Safetensors also supports file-level metadata, but the model configuration and tokenizer usually live in separate files. A multimodal model may still keep its language model and vision projector in separate GGUF files.

  • Self-describing: weights, architecture, and quantization information are kept within the same model component, so the loader can read the required parameters directly from the file.

  • Memory mapping: tensors are aligned according to defined rules and can be mapped with mmap, avoiding a full copy of the model at startup.

  • Extensible: metadata uses a key-value structure, so new fields can be added without redesigning the entire file format.

For a concrete example, the SmolVLM-256M-Instruct-GGUF repository on Hugging Face shows the metadata and tensor descriptions for both the language and vision models.

On-Device Multimodal Inference with llama.cpp and SmolVLM: GGUF Model Files (figure 11)

On-Device Multimodal Inference with llama.cpp and SmolVLM: GGUF Model Files (figure 12)

Inference Flow

With the model architecture out of the way, we can start at the mtmd-cli.cpp entry point and trace where image splitting, visual encoding, Pixel Shuffle, and language-model inference happen in code.

Loading the Models

llama.cpp initializes the multimodal CLI in the main() function of mtmd-cli.cpp.

1
2
3
4
5
6
int main(int argc, char ** argv) {
...
if (!common_params_parse(argc, argv, params, LLAMA_EXAMPLE_MTMD, show_additional_info)) {
}
...
}

common_params_parse() collects environment variables and command-line arguments into common_params, downloads the model if needed, and validates the resulting options. Model loading, context creation, and inference all use this shared structure.

Building the Context

1
2
3
4
5
int main(int argc, char ** argv) {
...
mtmd_cli_context ctx(params);
...
}

The mtmd_cli_context constructor loads the language and vision models, then initializes the model parameters, chat template, vocabulary, and caches. It also creates the llama_context used by the rest of the inference pipeline.

Prompt Processing

Chat templates vary between models. SmolVLM is based on the IDEFICS3 architecture, so its prompt must follow the official chat template. The template is also embedded in the model file and can be read when the model is loaded.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
<|im_start|>
{% for message in messages %}
{{message['role'] | capitalize}}
{% if message['content'][0]['type'] == 'image' %}{{':'}}{% else %}{{': '}}{% endif %}
{% for line in message['content'] %}
{% if line['type'] == 'text' %}
{{line['text']}}
{% elif line['type'] == 'image' %}
{{ '<image>' }}
{% endif %}
{% endfor %}
<end_of_utterance>
\n
{% endfor %}
{% if add_generation_prompt %}
{{ 'Assistant:' }}
{% endif %}

On-Device Multimodal Inference with llama.cpp and SmolVLM: Prompt Processing (figure 13)

The order of the image marker, user prompt, and system prompt matters. Swapping the image and text positions can produce incorrect output.

1
2
3
<|im_start|>System: You are a helpful assistant.<end_of_utterance>
User:<image>Describe this image.<end_of_utterance>
Assistant:

For SmolVLM, llama.cpp’s image marker can be set to \<image\> to match the official template. The marker is not converted into an ordinary token ID. It becomes a separate image chunk and is replaced with image embeddings before the sequence enters the Transformer.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
// mtmd.cpp
const char * mtmd_default_marker() {
return "<image>";
// return "<__media__>";
}

struct mtmd_tokenizer {
int32_t tokenize(mtmd_input_chunks * output) {
// Convert text to token IDs and <image> to an image chunk
...
*output = std::move(cur);
return 0;
}
}

In the llama.cpp revision used for this integration, the system prompt and user prompt each call eval_message() instead of being processed together.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
// mtmd-cli.cpp
int main(int argc, char ** argv) {
...
auto eval_system_prompt_if_present = [&] {
...
msg.role = "system";
// Run one prefill pass when a system prompt is present
return eval_message(ctx, msg);
};
...
if (is_single_turn) {
...
// Run one prefill pass for the user prompt
if (eval_message(ctx, msg)) {
return 1;
}
...
}
}

On-Device Multimodal Inference with llama.cpp and SmolVLM: Prompt Processing (figure 14)

That revision did not append \<end_of_utterance\> to the system prompt, so we added it:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
// llama-chat.cpp
int32_t llm_chat_apply_template(
llm_chat_template tmpl,
const std::vector<const llama_chat_message *> & chat,
std::string & dest, bool add_ass) {
...
else if (tmpl == LLM_CHAT_TEMPLATE_SMOLVLM) {
// SmolVLM
ss << "<|im_start|>"; // uses <|im_start|> as BOS, but the actual content is NOT chatml
for (auto message : chat) {
std::string role(message->role);
if (role == "system") {
// ss << "System: " << message->content << "\n\n";
// Use the correct message terminator
ss << "System: " << message->content << "<end_of_utterance>\n";
} else if (role == "user") {
ss << "User: " << message->content << "<end_of_utterance>\n";
} else {
ss << "Assistant: " << message->content << "<end_of_utterance>\n";
}
}
if (add_ass) {
ss << "Assistant:";
}
}
....
}

Prefill

For autoregressive models such as LLaMA and GPT, the initial pass over the complete input sequence is called prefill. It computes the hidden states, logits, and KV cache needed for generation.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
static int eval_message(mtmd_cli_context & ctx, common_chat_msg & msg) {
...
int32_t res = mtmd_tokenize(ctx.ctx_vision.get(),
chunks.ptr.get(), // output
&text, // text
bitmaps_c_ptr.data(),
bitmaps_c_ptr.size());
...
llama_pos new_n_past;
if (mtmd_helper_eval_chunks(ctx.ctx_vision.get(),
ctx.lctx, // lctx
chunks.ptr.get(), // chunks
ctx.n_past, // n_past
0, // seq_id
ctx.n_batch, // n_batch
true, // logits_last
&new_n_past)) {
LOG_ERR("Unable to eval prompt\n");
return 1;
}
ctx.n_past = new_n_past;
...
}

Each call to eval_message() runs one prefill pass. mtmd_tokenize() tokenizes the prompt and divides it into chunks; mtmd_helper_eval_chunks() then runs the forward pass.

Prefill Flow

The system prompt and user prompt each call eval_message(), so prefill runs in two segments. This does not change the final sequence as long as token positions remain contiguous and the KV cache is extended in order. The first segment’s logits_0 are not sampled; generation uses the logits from the final chunk. The intermediate logits are still computed, even though they are discarded.

mtmd_helper_eval_chunks() sends image and text chunks to the model in sequence, handling visual encoding and text decoding separately.

In this example, the system prompt becomes text chunk0; one forward pass produces logits_0. The user prompt is split into text chunk1, image chunk2, and text chunk3, then fed to the model in that order. Only the final text chunk’s logits_last are used for autoregressive sampling.

Image Preprocessing in llama.cpp

SmolVLM divides an image into local crops before sending them through the SigLIP encoder. Each crop is encoded into image embeddings. In llama.cpp, this path starts in clip_image_preprocess().

1
2
3
4
5
6
7
8
9
10
11
12
13
14
clip.h
bool clip_image_preprocess(struct clip_ctx * ctx, const clip_image_u8 * img, struct clip_image_f32_batch * res_imgs) {
case PROJECTOR_TYPE_IDEFICS3:{
...
// Keep one global overview image
instructions.overview_size = clip_image_size{params.image_size, params.image_size};
// Scale the source proportionally so its longest edge stays within the configured limit
instructions.refined_size = refined_size;
// Determine how many tiles the refined image should be split into
instructions.grid_size = clip_image_size{
static_cast<int>(std::ceil(static_cast<float>(refined_size.width) / params.image_size)),
static_cast<int>(std::ceil(static_cast<float>(refined_size.height) / params.image_size)),
};
...

The code first sets overview_size for the global preview. It then computes refined_size, the source image scaled to the model’s limits, and derives grid_size, the number of tiles needed at the target crop size.

In SmolVLM’s preprocessor_config.json, size.longest_edge is 2048, which caps the longest edge after scaling. max_image_size.longest_edge is 512, the target size for each image passed to the vision encoder. A 1024x512 image, for example, is resized to 2048x1024 and divided into eight 512x512 crops. With the global preview, nine images are encoded in total. The llama.cpp revision and vision GGUF used here did not store a separate longest_edge value, so runtime preprocessing produced only the global preview. To enable the full tiling path, the corresponding longest_edge must also be loaded with the vision model’s hyperparameters.

Llama Decode

llama_context::decode() is where the main inference work happens. It accepts a batch of tokens, runs the Transformer forward pass, updates the KV cache, and produces logits.

The call has four main steps:

  1. Build the computation graph
  2. Run the Transformer
  3. Write to the KV cache
  4. Produce logits
1
2
3
4
5
6
7
8
9
10
11
12
13
14
int llama_context::decode(const llama_batch & batch_inp) {
...
const auto * res = process_ubatch(ubatch, LLM_GRAPH_TYPE_DECODER, mctx.get(), status);
...
auto *t_logits = res->get_logits();
...
if (logits.data && t_logits && n_outputs > 0 && needs_raw_logits(ubatch, sampling.samplers)) {
...
float * logits_out = logits.data + n_outputs_prev*n_vocab;
if (n_outputs) {
ggml_backend_tensor_get_async(backend_res, t_logits, logits_out, 0, n_outputs * n_vocab * sizeof(float));
}
}
}

process_ubatch() builds the computation graph, binds the inputs, dispatches the graph to the Metal backend, and saves the KV cache generated in this pass. Once computation finishes, the logits are copied asynchronously into the CPU-side llama_context::logits.data buffer for sampling. I will leave backend tensors and graph scheduling for a separate discussion.

Autoregressive Decoding

Once prefill is complete, generation switches to autoregressive decoding. The model emits one token at a time, with each token conditioned on the prompt, the KV cache, and everything generated so far.

After prefill produces the logits for the first sampling position, the model samples a token, appends it to the batch, and runs the next decode pass. The loop continues until a stopping condition is reached.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
static int generate_response(mtmd_cli_context & ctx, int n_predict) {

// Sample the latest token ID
llama_token token_id = common_sampler_sample(ctx.smpl, ctx.lctx, -1);
generated_tokens.push_back(token_id);
...
// Convert it to a string
std::string piece = common_token_to_piece(ctx.lctx, token_id);
...
// Continue with the next inference step
common_batch_add(ctx.batch, token_id, ctx.n_past++, {0}, true);
if (llama_decode(ctx.lctx, ctx.batch)) {
on_token(piece, GEN_DECODE_FAIL);
LOG_ERR("failed to decode token\n");
return 1;
}
...
}

Summary

Images and text take different paths before they meet in the language model. An image is split and encoded, Pixel Shuffle shortens its visual-token sequence, and the projector maps the features into the LLM’s embedding space. Text is broken into tokens and chunks according to the chat template. Prefill merges both streams in prompt order and writes them to the KV cache; autoregressive decoding then generates the response one token at a time.

When the output looks wrong, I would check the chat template, image marker, and preprocessing parameters first. It is also worth verifying that every prefill segment advances token positions and extends the KV cache without a gap. Any one of these details can change the generated text, image-understanding quality, or memory footprint.

-------------本文结束 感谢您的阅读-------------
If this article helped you, your support encourages me to keep writing.

Title:On-Device Multimodal Inference with llama.cpp and SmolVLM

Author:lingyun

Published:2026-10-07 19:05

Updated:2026-10-11 00:00

Original link:https://tsuijunxi.github.io/en/2026/10/07/on-device-multimodal-inference-with-llama-cpp-and-smolvlm/

License: Attribution-NonCommercial-NoDerivatives 4.0 International Please keep the original link and author when sharing.