Ollama Notes

Table of Contents

1. Prerequisites
2. Install
- 2.1. Open Ports
- 2.2. Podman Config
3. Config
- 3.1. Get Models
- 3.2. Delete Models

1. Prerequisites

Ollama works best with a graphics card.
Follow this Nvidia notes doc to install the Nvidia drivers.

2. Install

2.1. Open Ports

Firewall ports

Ollama

sudo firewall-cmd --add-port=11434/tcp --permanent

OpenWeb UI

sudo firewall-cmd --add-port=3010/tcp --permanent

Comfy UI

sudo firewall-cmd --add-port=3011/tcp --permanent

Reload FW
```
sudo firewall-cmd --reload
```

2.2. Podman Config

Create Podman Compose file

Expand for source

# Ollama API: https://hub.docker.com/r/ollama/ollama
# GUI: https://docs.openwebui.com/
# Models: https://ollama.com/library
# NVidia Support: https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html#installation

services:
  ollama-api:
    container_name: Ollama-API
    image: docker.io/ollama/ollama:latest
    privileged: true
    ports:
      - "11434:11434"
    environment:
      - TZ=America/New_York
      #- OLLAMA_HOST=0.0.0.0
      #- HTTPS_PROXY=https://chat.xackleystudio.com
    volumes:
      - ./Ollama-data:/root/.ollama/models
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities:
                - gpu
    restart: always


  # https://docs.openwebui.com/getting-started/env-configuration

  open-webui:
    container_name: OpenWeb-UI
    image: ghcr.io/open-webui/open-webui:main
    privileged: true
    #image: ghcr.io/open-webui/open-webui:cuda
    #runtime: nvidia
    ports:
      - "3010:8080"
    environment:
      - TZ=America/New_York
      - OLLAMA_BASE_URL=http://ollama-api:11434
    volumes:
      - ./OpenWebUI-data:/app/backend/data
    depends_on:
      - ollama-api
    #extra_hosts:
    #  - host.docker.internal:host-gateway
    restart: always

    # https://comfyui-wiki.com/en/install
  comfyui:
    image: ghcr.io/jemeyer/comfyui:latest  # Or another image of your choice
    container_name: Comfy-UI
    restart: unless-stopped
    volumes:
      - ./ComfyUI-data/data:/app/ComfyUI/data
      - ./ComfyUI-data/models:/app/models
      - ./ComfyUI-data/input:/app/input
      - ./ComfyUI-data/output:/app/output
      - ./ComfyUI-data/user:/app/user
      - ./ComfyUI-data/temp:/app/temp
    ports:
      - "3011:8188" # use http://comyui:8188 when configuring OpenWeb-UI to use ComfyUI
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities:
                - gpu
    #environment:
    #  - NVIDIA_VISIBLE_DEVICES=all # Use all available GPUs
    #  - NVIDIA_DRIVER_CAPABILITIES=compute,utility,video,graphics # Enable all necessary capabilities

3. Config

3.1. Get Models

Models can be downloaded via the OpenWebUI interface as well.

Model names can be found here

Via the Ollama-API container

podman exec -it Ollama-API ollama pull deepseek-coder-v2:16b

3.2. Delete Models

Via the Ollama-API container

podman exec -it Ollama-API ollama rm deepseek-coder-v2:16b