ailia-llm

ailia LLM Python API

!! CAUTION !! “ailia” IS NOT OPEN SOURCE SOFTWARE (OSS). As long as user complies with the conditions stated in License Document, user may use the Software for free of charge, but the Software is basically paid software.

About ailia LLM

ailia LLM is a library for running LLMs on edge devices. It provides bindings for C++ and Unity.

Install from pip

You can install the ailia LLM free evaluation package with the following command.

pip3 install ailia_llm

Install from package

You can install the ailia LLM from Package with the following command.

python3 bootstrap.py
pip3 install ./

Usage

import ailia_llm

import os
import urllib.request

model_file_path = "gemma-2-2b-it-Q4_K_M.gguf"
if not os.path.exists(model_file_path):
	urllib.request.urlretrieve(
		"https://storage.googleapis.com/ailia-models/gemma/gemma-2-2b-it-Q4_K_M.gguf",
		model_file_path
	)

model = ailia_llm.AiliaLLM()
cpu_backend = next(backend for backend in model.get_backend_list()
                   if backend["backend_name"] == "CPU")
model.set_backend(cpu_backend["idx"])  # Optional; call before open().
model.open(model_file_path, n_ctx=8192)

messages = []
messages.append({"role": "system", "content": "語尾に「わん」をつけてください。"})
messages.append({"role": "user", "content": "あなたの名前は何ですか？"})

stream = model.generate(messages)

text = ""
for delta_text in stream:
	text = text + delta_text
print(text)

if model.context_full():
	raise Exception("Context full")

messages.append({"role": "assistant", "content": text})

Tool Use (JSON API)

Use a model that supports tool calling, such as Gemma 4. Define tools with set_tools(), generate with generate_json(), and read the complete assistant message with get_response_json(). Append the response and tool results to the history, then generate again for the final answer.

import json
import ailia_llm

model = ailia_llm.AiliaLLM()
model.open("gemma-4-E2B-it-Q4_K_M.gguf", n_ctx=8192)
model.set_tools([{"type":"function","function":{
    "name":"get_weather", "description":"Get the current weather.",
    "parameters":{"type":"object","properties":{"city":{"type":"string"}},"required":["city"]}
}}])
messages = [{"role":"user","content":"What is the weather in Tokyo?"}]
try:
    for _ in range(8):
        for _ in model.generate_json(messages):
            pass  # Consume generation; collecting text deltas is optional.
        if model.context_full():
            raise RuntimeError("Context full; shorten the history.")
        response = model.get_response_json()
        messages.append(response)
        if not response.get("tool_calls"):
            print(response["content"])
            break
        for call in response["tool_calls"]:
            args = json.loads(call["function"]["arguments"])
            # Validate the requested tool and arguments, then execute it here.
            result = {"city": args["city"], "weather": "sunny"}  # Mock result
            messages.append({"role":"tool", "tool_call_id":call["id"],
                             "content":json.dumps(result)})
finally:
    model.set_tools(None)

While tools are enabled, use generate_json() from the first turn instead of generate(). Read the response before clearing tools. Invalid input or incomplete output raises AiliaLLMError; only append successfully parsed responses and execute their complete tool calls.

Images and audio

Load a compatible projector with open_multimodal_projector() and pass media as content parts to generate_json():

model.open_multimodal_projector("mmproj.gguf")
messages = [{"role": "user", "content": [
    {"type": "text", "text": "Describe this image."},
    {"type": "image", "file_path": "/path/to/image.jpg"},
]}]
for delta in model.generate_json(messages):
    print(delta, end="", flush=True)

For audio, use "type": "audio" with a supported file such as WAV, MP3 or FLAC. Use separate prompts for images and audio. Each media part accepts either file_path or data (standard Base64-encoded file bytes). Media works with or without tools, and matching media and conversation prefixes can reuse cached encoder outputs and KV cache.

API specification

https://github.com/ailia-ai/ailia-sdk