== Testing InstructLab Models Locally ==
=== What is InstructLab? ===
There's a large variety of https://huggingface.co/models[models] available from https://huggingface.co[HuggingFace], and https://huggingface.co/instructlab[InstructLab] is an open-source collection of LLMs with tools that allow users to both use, and improve, LLMs based on Granite models.
There are also OCI model images available on https://catalog.redhat.com/search?gs&q=granite%208b[Red Hat Ecosystem Catalog] (the link is just for the Granite 8b family).
A https://developers.redhat.com/articles/2024/08/01/open-source-ai-coding-assistance-granite-models[Red Hat blog] by Cedric Clyburn shows how you can use Ollama and InstructLab to run LLMs locally in a lot more detail, so I'll keep it short and with a focus on Conda here.
=== Setting Up the Environment ===
You can use one of the provided environment files, `env-ilab-25.yml`, to create a Conda environment with the `instructlab` package version `0.25.x`.
This gives you the basic environment that enables you to start serving and chatting to various HuggingFace (and other) Transformer-based models.
Just like with any other Conda environment, start by creating the desired configuration.
[subs="+quotes"]
----
$ *source conda-init.sh*
(base) $ *mamba env create -y -f envs/env-ilab-25.yml*
Channels:
- conda-forge
Platform: osx-arm64
Collecting package metadata (repodata.json): done
Solving environment: done
Downloading and Extracting Packages:
...
Preparing transaction: done
Verifying transaction: done
Executing transaction: done
...
----
====
NOTE: The installation uses `pip` to install `instructlab` as there are no Conda Forge packages for it. Be patient, it takes quite some time.
====
Activate the environment and create a `bash` completion file.
[subs="+quotes"]
----
(base) $ *mamba activate ilab-25*
(ilab-25) $ *_ILAB_COMPLETE=bash_source ilab > ilab.completion*
(ilab-25) $ *source ilab.completion*
----
Check the system information.
[subs="+quotes"]
----
(ilab-25) $ *ilab system info*
Platform:
sys.version: 3.11.12 | packaged by conda-forge | (main, Apr 10 2025, 22:18:52) [Clang 18.1.8 ]
sys.platform: darwin
os.name: posix
platform.release: 24.4.0
platform.machine: arm64
platform.node: foobar
platform.python_version: 3.11.12
platform.cpu_brand: Apple M1 Max
memory.total: 64.00 GB
memory.available: 25.36 GB
memory.used: 14.97 GB
InstructLab:
instructlab.version: 0.25.0
...
Torch:
torch.version: 2.5.1
...
__torch.backends.mps.is_built: True
torch.backends.mps.is_available: True__
llama_cpp_python:
llama_cpp_python.version: 0.3.6
_llama_cpp_python.supports_gpu_offload: True_
----
The PyTorch `mps` and Llama `supports_gpu_offload` settings show that InstructLab is capable of using the M1 Max GPU for serving.
=== Downloading Models ===
Visit the InstructLab page and choose a model to download (for this demo, I selected `granite-3.0-8b-lab-community`).
Use the `ilab model download` command to pull it.
By default, models will be stored in `~/.cache/instructlab/models/`, unless you say otherwise with the `--model-dir` option to `ilab model` command.
[subs="+quotes"]
----
(ilab-25) $ *ilab model download -rp instructlab/granite-3.0-8b-lab-community*
INFO 2025-04-14 13:29:59,724 instructlab.model.download:77: Downloading model from Hugging Face:
Model: instructlab/granite-3.0-8b-lab-community@main
Destination: /foo/bar/.cache/instructlab/models
...
INFO 2025-04-14 13:36:13,171 instructlab.model.download:288:
ᕦ(òᴗóˇ)ᕤ instructlab/granite-3.0-8b-lab-community model download completed successfully! ᕦ(òᴗóˇ)ᕤ
INFO 2025-04-14 13:36:13,171 instructlab.model.download:302: Available models (\`ilab model list`):
+------------------------------------------+...+---------+--------------------------+
| Model Name |...| Size | Absolute path |
+------------------------------------------+...+---------+--------------------------+
| instructlab/granite-3.0-8b-lab-community |...| 15.2 GB | .../models/instructlab |
+------------------------------------------+...+---------+--------------------------+
----
====
NOTE: LLMs are usually quite large (as the name suggests) so be patient and set aside sufficient amount of disk space. The above model is a total download of 17 GiB, so even on a fast link it takes a couple of minutes to download.
====
Note that the absolute path to model is a directory - if you look inside it, there will be a subdirectory containing the actual download.
The format of the model is HuggingFace _safetensors_, which requires the https://github.com/vllm-project/vllm.git[vLLM] serving backend, and is not supported on macOS by default.
From here on, there are two options: either install vLLM manually, or use `llama.cpp` to convert the model to GGUF.
Personally, I prefer the second option as it very often also results in a smaller model, and does not require too much manual hacking about. You can even have a separate Conda environment just for `llama.cpp`.
=== Installing vLLM on macOS ===
If you used the InstructLab env file provided in this repo, you should already have `cmake`, `torch`, and `torchvision` modules in the environment. If not, ensure they are available.
During the compilation, `pip` in particular may complain about some incompatibilities. Just ignore it.
First, clone Triton and install it.
[subs="+quotes"]
----
(ilab-25) $ *git clone https://github.com/triton-lang/triton.git*
Cloning into 'triton'...
...
(ilab-25) $ *cd triton/python*
(ilab-25) $ *pip install -e .*
Obtaining file:///foo/bar/baz/triton/python
...
Successfully built triton
Installing collected packages: triton
Successfully installed triton-3.3.0+git32b42821
(ilab-25) $ *cd ../..*
(ilab-25) $ *rm -rf ./triton/*
----
====
NOTE: Triton compilation takes quite a long time and it appears to be doing nothing. Don't worry.
====
Clone vLLM and build it.
[subs="+quotes"]
----
(ilab-25) $ *git clone https://github.com/vllm-project/vllm.git*
Cloning into 'vllm'...
...
(ilab-25) $ *cd vllm*
(ilab-25) $ *sed -i 's/^triton==3.2/triton==3.3/' requirements/requirements-cpu.txt*
(ilab-25) $ *pip install -e .*
Obtaining file:///foo/bar/baz/vllm
...
Successfully built vllm
Installing collected packages: vllm
Successfully installed vllm-0.8.5.dev3+g7cbfc1094.d20250414
(ilab-25) $ *cd ..*
(ilab-25) $ *rm -rf ./vllm/*
----
====
NOTE: vLLM 0.8.5 somehow imposes a restriction of maximum version of Triton being 3.2.0, which is not necessary.
====
References:
* https://github.com/triton-lang/triton[Triton Development Repository]
* https://docs.vllm.ai/en/stable/getting_started/installation/cpu.html?device=apple[Building vLLM for Apple Silicon]
=== Converting Models to GGUF ===
You can use https://github.com/ggerganov/llama.cpp.git[`llama.cpp`] to convert models from HF, GGML, and LORA model formats to GGUF, which InstructLab can serve even on a Mac.
Clone and build `llama.cpp`.
[subs="+quotes"]
----
(ilab-25) $ *git clone https://github.com/ggerganov/llama.cpp.git*
Cloning into 'llama.cpp'...
...
(ilab-25) $ *cd llama.cpp*
(ilab-25) $ *pip install --upgrade -r requirements.txt*
Looking in indexes: https://pypi.org/simple, ...
...
Successfully installed aiohttp-3.9.5 ...
----
You can now use the various `convert_*.py` scripts. In our case, it would be HF (HuggingFace) to GGUF conversion.
[subs="+quotes"]
----
(ilab-25) $ *./convert_hf_to_gguf.py \*
*~/.cache/instructlab/models/instructlab/granite-3.0-8b-lab-community/ \*
*--outfile ~/.cache/instructlab/models/granite-3.0-8b-lab-community.gguf \*
*--outtype q8_0*
INFO:hf-to-gguf:Loading model: granite-3.0-8b-lab-community
INFO:gguf.gguf_writer:gguf: This GGUF file is for Little Endian only
INFO:hf-to-gguf:Exporting model...
INFO:hf-to-gguf:gguf: loading model weight map from 'model.safetensors.index.json'
INFO:hf-to-gguf:gguf: loading model part 'model-00001-of-00004.safetensors'
...
INFO:hf-to-gguf:Model successfully exported to /foo/bar/.cache/instructlab/models/granite-3.0-8b-lab-community.gguf
(ilab-25) $ ilab model list
+------------------------------------------+...+---------+---------------------------------------+
| Model Name |...| Size | Absolute path |
+------------------------------------------+...+---------+---------------------------------------+
| instructlab/granite-3.0-8b-lab-community |...| 15.2 GB | .../instructlab |
| granite-3.0-8b-lab-community.gguf |...| 8.1 GB | .../granite-3.0-8b-lab-community.gguf |
+------------------------------------------+...+---------+---------------------------------------+
----
Reference: https://github.com/ggml-org/llama.cpp/discussions/2948[Tutorial: How to convert HuggingFace model to GGUF format] on GitHub.
=== Serving Models ===
Start the model server.
[subs="+quotes"]
----
(ilab-25) $ *ilab model serve \*
*--model-path /foo/bar/.cache/instructlab/models/granite-3.0-8b-lab-community.gguf*
INFO 2025-04-14 14:49:05,624 instructlab.model.serve_backend:79: Setting backend_type in the serve config to llama-cpp
INFO 2025-04-14 14:49:05,633 instructlab.model.serve_backend:85: Using model '/foo/bar/.cache/instructlab/models/granite-3.0-8b-lab-community.gguf' with -1 gpu-layers and 4096 max context size.
...
INFO 2025-04-14 14:49:12,050 instructlab.model.backends.llama_cpp:233: Starting server process, press CTRL+C to shutdown server...
INFO 2025-04-14 14:49:12,050 instructlab.model.backends.llama_cpp:234: After application startup complete see http://127.0.0.1:8000/docs for API.
----
In another terminal, start a chat.
[subs="+quotes"]
----
(ilab-25) $ *ilab model chat*
╭─────────────────────────────────────── system ────────────────────────────────────────╮
│ Welcome to InstructLab Chat w/ GRANITE-3.0-8B-LAB-COMMUNITY.GGUF (type /h for help) │
╰───────────────────────────────────────────────────────────────────────────────────────╯
>>> *what are your specialties?*
My specialties include providing assistance with general tasks such as setting up a new device, troubleshooting software issues, and answering basic questions about using technology.
I can also help with more specific tasks related to Linux, such as configuring network settings, managing users and groups, and installing software packages. I have experience working with various Linux distributions, including Red Hat Enterprise Linux, Fedora, Ubuntu, and Debian.
Additionally, I am familiar with a wide range of programming languages, tools, and frameworks, including Python, Java, C++, Ruby on Rails, AngularJS, React, and Node.js.
I hope this information is helpful! Let me know if you have any other questions.
----
Congratulations!