demo/docs/INSTRUCTLAB.adoc

== Testing InstructLab Models Locally ==

=== What is InstructLab? ===

There's a large variety of https://huggingface.co/models[models] available from https://huggingface.co[HuggingFace], and https://huggingface.co/instructlab[InstructLab] is an open-source collection of LLMs with tools that allow users to both use, and improve, LLMs based on Granite models.

There are also OCI model images available on https://catalog.redhat.com/search?gs&q=granite%208b[Red Hat Ecosystem Catalog] (the link is just for the Granite 8b family).

A https://developers.redhat.com/articles/2024/08/01/open-source-ai-coding-assistance-granite-models[Red Hat blog] by Cedric Clyburn shows how you can use Ollama and InstructLab to run LLMs locally in a lot more detail, so I'll keep it short and with a focus on Conda here.

=== Setting Up the Environment ===

You can use one of the provided environment files, `env-ilab-25.yml`, to create a Conda environment with the `instructlab` package version `0.25.x`.

This gives you the basic environment that enables you to start serving and chatting to various HuggingFace (and other) Transformer-based models.

Just like with any other Conda environment, start by creating the desired configuration.

[subs="+quotes"]
----
$ *source conda-init.sh*

(base) $ *mamba env create -y -f envs/env-ilab-25.yml*
Channels:
 - conda-forge
Platform: osx-arm64
Collecting package metadata (repodata.json): done
Solving environment: done

Downloading and Extracting Packages:
...

Preparing transaction: done
Verifying transaction: done
Executing transaction: done
...
----

====
NOTE: The installation uses `pip` to install `instructlab` as there are no Conda Forge packages for it. Be patient, it takes quite some time.
====

Activate the environment and create a `bash` completion file.

[subs="+quotes"]
----
(base) $ *mamba activate ilab-25*

(ilab-25) $ *_ILAB_COMPLETE=bash_source ilab > ilab.completion*

(ilab-25) $ *source ilab.completion*
----

Check the system information.

[subs="+quotes"]
----
(ilab-25) $ *ilab system info*
Platform:
  sys.version: 3.11.12 | packaged by conda-forge | (main, Apr 10 2025, 22:18:52) [Clang 18.1.8 ]
  sys.platform: darwin
  os.name: posix
  platform.release: 24.4.0
  platform.machine: arm64
  platform.node: foobar
  platform.python_version: 3.11.12
  platform.cpu_brand: Apple M1 Max
  memory.total: 64.00 GB
  memory.available: 25.36 GB
  memory.used: 14.97 GB

InstructLab:
  instructlab.version: 0.25.0
  ...

Torch:
  torch.version: 2.5.1
  ...
  __torch.backends.mps.is_built: True
  torch.backends.mps.is_available: True__

llama_cpp_python:
  llama_cpp_python.version: 0.3.6
  _llama_cpp_python.supports_gpu_offload: True_
----

The PyTorch `mps` and Llama `supports_gpu_offload` settings show that InstructLab is capable of using the M1 Max GPU for serving.

=== Downloading Models ===

Visit the InstructLab page and choose a model to download (for this demo, I selected `granite-3.0-8b-lab-community`).

Use the `ilab model download` command to pull it.

By default, models will be stored in `~/.cache/instructlab/models/`, unless you say otherwise with the `--model-dir` option to `ilab model` command.

[subs="+quotes"]
----
(ilab-25) $ *ilab model download -rp instructlab/granite-3.0-8b-lab-community*
INFO 2025-04-14 13:29:59,724 instructlab.model.download:77: Downloading model from Hugging Face:
    Model: instructlab/granite-3.0-8b-lab-community@main
    Destination: /foo/bar/.cache/instructlab/models
...
INFO 2025-04-14 13:36:13,171 instructlab.model.download:288:
ᕦ(òᴗóˇ)ᕤ instructlab/granite-3.0-8b-lab-community model download completed successfully! ᕦ(òᴗóˇ)ᕤ

INFO 2025-04-14 13:36:13,171 instructlab.model.download:302: Available models (\`ilab model list`):
+------------------------------------------+...+---------+--------------------------+
| Model Name                               |...| Size    | Absolute path            |
+------------------------------------------+...+---------+--------------------------+
| instructlab/granite-3.0-8b-lab-community |...| 15.2 GB | .../models/instructlab   |
+------------------------------------------+...+---------+--------------------------+
----

====
NOTE: LLMs are usually quite large (as the name suggests) so be patient and set aside sufficient amount of disk space. The above model is a total download of 17 GiB, so even on a fast link it takes a couple of minutes to download.
====

Note that the absolute path to model is a directory - if you look inside it, there will be a subdirectory containing the actual download.

The format of the model is HuggingFace _safetensors_, which requires the https://github.com/vllm-project/vllm.git[vLLM] serving backend, and is not supported on macOS by default.

From here on, there are two options: either install vLLM manually, or use `llama.cpp` to convert the model to GGUF.

Personally, I prefer the second option as it very often also results in a smaller model, and does not require too much manual hacking about. You can even have a separate Conda environment just for `llama.cpp`.

=== Installing vLLM on macOS ===

If you used the InstructLab env file provided in this repo, you should already have `cmake`, `torch`, and `torchvision` modules in the environment. If not, ensure they are available.

During the compilation, `pip` in particular may complain about some incompatibilities. Just ignore it.

First, clone Triton and install it.

[subs="+quotes"]
----
(ilab-25) $ *git clone https://github.com/triton-lang/triton.git*
Cloning into 'triton'...
...

(ilab-25) $ *cd triton/python*

(ilab-25) $ *pip install -e .*
Obtaining file:///foo/bar/baz/triton/python
...
Successfully built triton
Installing collected packages: triton
Successfully installed triton-3.3.0+git32b42821

(ilab-25) $ *cd ../..*
(ilab-25) $ *rm -rf ./triton/*
----

====
NOTE: Triton compilation takes quite a long time and it appears to be doing nothing. Don't worry.
====

Clone vLLM and build it.

[subs="+quotes"]
----
(ilab-25) $ *git clone https://github.com/vllm-project/vllm.git*
Cloning into 'vllm'...
...

(ilab-25) $ *cd vllm*

(ilab-25) $ *sed -i 's/^triton==3.2/triton==3.3/' requirements/requirements-cpu.txt*
(ilab-25) $ *pip install -e .*
Obtaining file:///foo/bar/baz/vllm
...
Successfully built vllm
Installing collected packages: vllm
Successfully installed vllm-0.8.5.dev3+g7cbfc1094.d20250414

(ilab-25) $ *cd ..*
(ilab-25) $ *rm -rf ./vllm/*
----

====
NOTE: vLLM 0.8.5 somehow imposes a restriction of maximum version of Triton being 3.2.0, which is not necessary.
====

References:

* https://github.com/triton-lang/triton[Triton Development Repository]
* https://docs.vllm.ai/en/stable/getting_started/installation/cpu.html?device=apple[Building vLLM for Apple Silicon]

=== Converting Models to GGUF ===

You can use https://github.com/ggerganov/llama.cpp.git[`llama.cpp`] to convert models from HF, GGML, and LORA model formats to GGUF, which InstructLab can serve even on a Mac.

Clone and build `llama.cpp`.

[subs="+quotes"]
----
(ilab-25) $ *git clone https://github.com/ggerganov/llama.cpp.git*
Cloning into 'llama.cpp'...
...

(ilab-25) $ *cd llama.cpp*

(ilab-25) $ *pip install --upgrade -r requirements.txt*
Looking in indexes: https://pypi.org/simple, ...
...
Successfully installed aiohttp-3.9.5 ...
----

You can now use the various `convert_*.py` scripts. In our case, it would be HF (HuggingFace) to GGUF conversion.

[subs="+quotes"]
----
(ilab-25) $ *./convert_hf_to_gguf.py \*
                *~/.cache/instructlab/models/instructlab/granite-3.0-8b-lab-community/ \*
                *--outfile ~/.cache/instructlab/models/granite-3.0-8b-lab-community.gguf \*
                *--outtype q8_0*
INFO:hf-to-gguf:Loading model: granite-3.0-8b-lab-community
INFO:gguf.gguf_writer:gguf: This GGUF file is for Little Endian only
INFO:hf-to-gguf:Exporting model...
INFO:hf-to-gguf:gguf: loading model weight map from 'model.safetensors.index.json'
INFO:hf-to-gguf:gguf: loading model part 'model-00001-of-00004.safetensors'
...
INFO:hf-to-gguf:Model successfully exported to /foo/bar/.cache/instructlab/models/granite-3.0-8b-lab-community.gguf

(ilab-25) $ ilab model list
+------------------------------------------+...+---------+---------------------------------------+
| Model Name                               |...| Size    | Absolute path                         |
+------------------------------------------+...+---------+---------------------------------------+
| instructlab/granite-3.0-8b-lab-community |...| 15.2 GB | .../instructlab                       |
| granite-3.0-8b-lab-community.gguf        |...| 8.1 GB  | .../granite-3.0-8b-lab-community.gguf |
+------------------------------------------+...+---------+---------------------------------------+
----

Reference: https://github.com/ggml-org/llama.cpp/discussions/2948[Tutorial: How to convert HuggingFace model to GGUF format] on GitHub.

=== Serving Models ===

Start the model server.

[subs="+quotes"]
----
(ilab-25) $ *ilab model serve \*
            *--model-path /foo/bar/.cache/instructlab/models/granite-3.0-8b-lab-community.gguf*
INFO 2025-04-14 14:49:05,624 instructlab.model.serve_backend:79: Setting backend_type in the serve config to llama-cpp
INFO 2025-04-14 14:49:05,633 instructlab.model.serve_backend:85: Using model '/foo/bar/.cache/instructlab/models/granite-3.0-8b-lab-community.gguf' with -1 gpu-layers and 4096 max context size.
...
INFO 2025-04-14 14:49:12,050 instructlab.model.backends.llama_cpp:233: Starting server process, press CTRL+C to shutdown server...
INFO 2025-04-14 14:49:12,050 instructlab.model.backends.llama_cpp:234: After application startup complete see http://127.0.0.1:8000/docs for API.
----

In another terminal, start a chat.

[subs="+quotes"]
----
(ilab-25) $ *ilab model chat*
╭─────────────────────────────────────── system ────────────────────────────────────────╮
│ Welcome to InstructLab Chat w/ GRANITE-3.0-8B-LAB-COMMUNITY.GGUF (type /h for help)   │
╰───────────────────────────────────────────────────────────────────────────────────────╯
>>> *what are your specialties?*
My specialties include providing assistance with general tasks such as setting up a new device, troubleshooting software issues, and answering basic questions about using technology.

I can also help with more specific tasks related to Linux, such as configuring network settings, managing users and groups, and installing software packages. I have experience working with various Linux distributions, including Red Hat Enterprise Linux, Fedora, Ubuntu, and Debian.

Additionally, I am familiar with a wide range of programming languages, tools, and frameworks, including Python, Java, C++, Ruby on Rails, AngularJS, React, and Node.js.

I hope this information is helpful! Let me know if you have any other questions.
----

Congratulations!