How to Integrate Local LLMs With Ollama and Python

Get Ollama Running

To get started, navigate to Ollama’s download page and grab the installer for your current operating system. You’ll find installers for Windows 10 or newer and macOS 14 Sonoma or newer. Run the appropriate installer and follow the on-screen instructions. For Linux users, the installation process differs slightly, as you’ll learn soon.

On Windows, Ollama will run in the background after installation, and the CLI will be available for you. If this doesn’t happen automatically for you, then go to the Start menu, search for Ollama, and run the app.

On macOS, the app manages the CLI and setup details, so you just need to launch Ollama.app.

If you’re on Linux, install Ollama with the following command:

$ curl -fsSL https://ollama.com/install.sh | sh

Once the process is complete, you can verify the installation by running:

$ ollama -v

If this command works, then the installation was successful. Next, start Ollama’s service by running the command below:

$ ollama serve

That’s it! You’re now ready to start using Ollama on your local machine. In some Linux distributions, such as Ubuntu, this final command may not be necessary, as Ollama may start automatically when the installation is complete. In that case, running the command above will result in an error.

Pull the Required Models

Once Ollama is installed and running, you can pull the required models. You must have sufficient free disk space to complete the process. For example, the llama3.2:latest model requires 2.0 GB of space, and the codellama:latest model needs 3.8 GB.

To pull the models, run the following commands:

$ ollama pull llama3.2:latest
$ ollama pull codellama:latest

These commands will take some time to complete because they download the models to your local drive.

Now that you’ve downloaded the models, you can try them out from the command line:

$ ollama run llama3.2:latest
>>> Explain what Python is in one sentence.
Python is a high-level, interpreted programming language known for its
simplicity, readability, and versatility, often used for web development,
data analysis, machine learning, automation, and more.

If you ran this prompt successfully, then you’re ready to connect from Python. You can press Ctrl+D to exit the chat.

Install Ollama’s Python SDK

Ollama provides a Python library that enables you to use local models within your Python applications and projects. You can install the library from PyPI into a virtual environment with the following command:

(venv) $ python -m pip install ollama

This library is also known as the Ollama Python SDK (Software Development Kit) and provides the recommended way to integrate Python projects with Ollama.

Interact With Ollama’s Chat Interface

Here’s a minimal chat example in Python that resembles the command-line example from the previous section:

>>> from ollama import chat
>>> messages = [
...     {
...         "role": "user",
...         "content": "Explain what Python is in one sentence.",
...     },
... ]
>>> response = chat(model="llama3.2:latest", messages=messages)
>>> print(response.message.content)
Python is a high-level, interpreted programming language that is widely used
for its simplicity, readability, and versatility, making it an ideal choice
for web development, data analysis, machine learning, automation, and more.

In this example, you should consider a few things about the messages:

• messages is a list of dictionaries because the chat interface is built for multi-turn conversations.
• Each message is represented as a Python dictionary, which has a role and a content key with the corresponding values.

The call to chat() with the model and your messages as arguments generates a ChatResponse that contains several attributes. To get the generated answer, you can access the response.message.content attribute.

To keep the context of the current conversation, you can append the model’s reply and ask a follow-up question. Consider the following example where you keep a conversation about Python list comprehensions:

>>> messages = [
...     {"role": "system", "content": "You are an expert Python tutor."},
...     {
...         "role": "user",
...         "content": "Define list comprehensions in a sentence."
...     },
... ]
>>> response = chat(model="llama3.2:latest", messages=messages)
>>> print(response.message.content)
List comprehensions are a concise and expressive way to create new lists
by performing operations on existing lists or iterables, using a compact
syntax that combines conditional statements and iteration.
>>> messages.append(response.message)  # Keep context
>>> messages.append(
...     {
...         "role": "user",
...         "content": "Provide a short, practical example."
...     }
... )
>>> response = chat(model="llama3.2:latest", messages=messages)
>>> print(response.message.content)
Here's an example of a list comprehension:
```python
numbers = [1, 2, 3, 4, 5]
double_numbers = [num * 2 for num in numbers if num % 2 == 0]
print(double_numbers)  # Output: [2, 4, 6]
```

In this example, you take each number in the numbers list, double it, and include only the results where the original number is even.

Here, you start a conversation about list comprehensions in Python. Once the model replies, you append the answer to the list of messages so that the model uses it as context. Next, you run another chat interaction and get the response grounded in the provided context.

If you’re building CLI and chat apps, then the chat streaming capabilities make the model responses feel much more interactive and fluid. Consider the following script:

from ollama import chat
stream = chat(
    model="llama3.2:latest",
    messages=[
        {
            "role": "user",
            "content": "Explain Python dataclasses with a quick example."
        }
    ],
    stream=True,
)
for chunk in stream:
    print(chunk.message.content, end="", flush=True)

The key in this code is the stream argument. When you set it to True, chat() returns an iterator that yields partial response chunks as they arrive. That way, you can print or render the assistant’s message incrementally, rather than waiting for the full response.

Go ahead and run the script from your command line to check how it behaves!

Use Ollama’s Text Generation Interface

If you don’t need to keep a conversation with context, then generate() is a great fit. This function allows you to run one-shot prompts on the target model. Here’s how it works:

>>> from ollama import generate
>>> response = generate(
...     model="llama3.2:latest",
...     prompt="Explain what Python is in one sentence."
... )
>>> print(response.response)
Python is a high-level, interpreted programming language known for its
simplicity, readability, and versatility. It is widely used in various
fields such as web development, data analysis, artificial intelligence,
and more.

The generate() function is quite convenient for one-shot tasks, such as summarizing, rewriting, or rephrasing text.

You can also use this function to generate Python code. For example, say that you want the model to help you write a custom function that implements the FizzBuzz challenge, where you iterate over a range of numbers and replace certain values with words as follows:

• “Fizz” for multiples of 3
• “Buzz” for multiples of 5
• “FizzBuzz” for multiples of both 3 and 5

Here’s a prompt that asks the model to generate a basic implementation of this challenge:

>>> from ollama import generate
>>> prompt = """
... Write a Python function fizzbuzz(n: int) -> List[str] that:
...
... - Returns a list of strings for the numbers 1..n
... - Uses "Fizz" for multiples of 3
... - Uses "Buzz" for multiples of 5
... - Uses "FizzBuzz" for multiples of both 3 and 5
... - Uses the number itself (as a string) otherwise
... - Raises ValueError if n < 1
...
... Include type hints compatible with Python 3.8.
... """
>>> response = generate(model="codellama:latest", prompt=prompt)
>>> print(response.response)
```
from typing import List
def fizzbuzz(n: int) -> List[str]:
    if n < 1:
        raise ValueError("n must be greater than or equal to 1")
    result = []
    for i in range(1, n+1):
        if i % 3 == 0 and i % 5 == 0:
            result.append("FizzBuzz")
        elif i % 3 == 0:
            result.append("Fizz")
        elif i % 5 == 0:
            result.append("Buzz")
        else:
            result.append(str(i))
    return result
```

For this example, you’ve used the codellama:latest model, which is intended to generate code from natural language prompts.

Now that the model has generated the code, review it and run a quick test. Copy the generated code into the REPL session and call it with an integer as an argument:

>>> from typing import List
>>> def fizzbuzz(n: int) -> List[str]:
...     if n < 1:
...         raise ValueError("n must be greater than or equal to 1")
...     result = []
...     for i in range(1, n+1):
...         if i % 3 == 0 and i % 5 == 0:
...             result.append("FizzBuzz")
...         elif i % 3 == 0:
...             result.append("Fizz")
...         elif i % 5 == 0:
...             result.append("Buzz")
...         else:
...             result.append(str(i))
...     return result
...
>>> fizzbuzz(16)
['1', '2', 'Fizz', '4', 'Buzz', 'Fizz', ..., 'FizzBuzz', '16']

Great! The code works as expected. You can experiment with other prompts and get the model to build helper functions, classes, and more.

Understand the Tool Calling Workflow

Before attempting to use tool calling in your code, be aware that this feature depends on the model’s capabilities. Therefore, make sure you use a model known to support tool calling. In this tutorial, you’re using the llama3.2:latest model, which supports this feature.

The high-level flow for tool calling looks something like this:

1. Define relevant tools as Python functions.
2. Pass the tools along with the prompt.
3. Execute the selected tools in your code.
4. Append the tool result as a role="tool" message.
5. Generate the final answer that uses the tool’s result.

Now that you understand the high-level workflow, you can implement tool calling in Python and walk through a complete example.

Implement Tool Calling in Python

To see tool calling in action, you define a simple Python function and pass it to the model as a tool. Here’s an example using the square_root() function:

import math
from ollama import chat
# Define a tool as a Python function
def square_root(number: float) -> float:
    """Calculate the square root of a number.
    Args:
        number: The number to calculate the square root for.
    Returns:
        The square root of the number.
    """
    return math.sqrt(number)
messages = [
    {
        "role": "user",
        "content": "What is the square root of 36?",
    }
]
response = chat(
    model="llama3.2:latest",
    messages=messages,
    tools=[square_root]  # Pass the tools along with the prompt
)
# Append the response for context
messages.append(response.message)
if response.message.tool_calls:
    tool = response.message.tool_calls[0]
    # Call the tool
    result = square_root(float(tool.function.arguments["number"]))
    # Append the tool result
    messages.append(
        {
            "role": "tool",
            "tool_name": tool.function.name,
            "content": str(result),
        }
    )
    # Obtain the final answer
    final_response = chat(model="llama3.2:latest", messages=messages)
    print(final_response.message.content)

When defining tools, clear docstrings and type hints can help the model decide which tool to call and how. That’s why the square_root() function has both.

Next, you call chat() with your prompt and a list of tools. You should append the response message to your list of messages to keep the context of your conversation. Then, you call the tools. In this specific example, you have only one tool, square_root(). The result should be added to the list of messages with the role set to "tool".

Finally, you call chat() again with all the context provided by messages. If you run the script, then you’ll get a response like the following:

(venv) $ python tool_calling.py
The square root of 36 is 6.

As you can see, the model is properly using the tool’s result to provide a grounded answer to your original question.

If you’re not seeing any output, the model may have decided to provide a direct answer without calling the tool. Alternatively, it may be returning the tool call as text rather than populating the .tool_calls attribute. In these cases, try using a larger model, such as llama3.1:8b, or tweaking your prompt to encourage tool usage.