Unsloth更快训练大模型并导出GGUF - Windows

环境搭建

系统环境

需要Nvidia显卡，至少8G显存，且专用显存与共享显存之和大于20G

建议将非安装版的环境文件都放到非系统盘，方便重装或移植

以Windows11为例，非安装环境文件都放在 E 盘下

设置自定义Path文件夹

创建 E:\mypath 文件夹，将其添加进用户环境变量Path中，之后会用

CMake

下载 CMake 的 Windows x64 ZIP 对应文件：Download CMake

解压到E:\environment\cmake

将E:\environment\cmake\bin添加到用户环境变量Path中

C++编译

下载 Community 版 Visual Studio：Visual Studio 2022 IDE

运行后选择桌面c++板块内容安装，若中途取消安装了，可以在开始菜单下栏的新增项目（或推荐项目）中找到该安装程序

这里只需要c++的编译环境，把除了C++板块以外的安装项目全部取消勾选（该板块自动勾选系统SDK等其他相关组件，不要取消勾选）

python环境

python安装

下载新版Anaconda安装程序：Download Anaconda

安装过程中将添加到环境变量等选项全部打勾

用uv管理pip包

将uv所有文件装至 E:\uv 下

符合国内环境，设置环境变量和更新源，安装到脚本常用硬盘，方便项目共用包可以硬链接节省硬盘空间

设置环境变量

PowerShell指令

[Environment]::SetEnvironmentVariable("UV_INDEX_URL","https://mirrors.huaweicloud.com/repository/pypi/simple","User")
[Environment]::SetEnvironmentVariable("UV_CACHE_DIR","E:\uv\cache","User")
[Environment]::SetEnvironmentVariable("UV_TOOL_DIR","E:\uv\tools","User")
[Environment]::SetEnvironmentVariable("UV_TOOL_BIN_DIR","E:\uv\toolsbin","User")
[Environment]::SetEnvironmentVariable("UV_PYTHON_INSTALL_DIR","E:\uv\python","User")
[Environment]::SetEnvironmentVariable("UV_PYTHON_INSTALL_MIRROR","https://ghproxy.cn/https://github.com/indygreg/python-build-standalone/releases/download","User")
[Environment]::SetEnvironmentVariable("Path","%UV_TOOL_BIN_DIR%;"+[Environment]::GetEnvironmentVariable("Path"),"User")

改变安装路径并安装

PowerShell指令

$env:UV_INSTALL_DIR = "E:\uv\bin"
powershell -ExecutionPolicy ByPass -c "irm https://astral.sh/uv/install.ps1 | iex"

[!tip] 如果提示在此系统上禁止运行脚本

以管理员身份打开 PowerShell，输入

Set-ExecutionPolicy Remotesigned

回到前面的 PowerShell 终端继续操作

[!tip] 下载速度慢或连不上

可能需要代理连接

自主更新

uv self update

添加 python 3.11 独立文件

uv python install 3.11
uv python list # 查看是否安装成功

Git环境

安装Git：Git - 安装 Git (git-scm.com)

Nvidia CUDA 工具包

安装12.1版：CUDA Toolkit 12.1 Update 1 Downloads | NVIDIA Developer

注意运行安装程序后在安装选单界面只安装CUDA驱动和工具包，不选其他显卡驱动和Nsignt

安装完后重启电脑

输入指令查看信息

nvidia-smi # 看当前驱动最大可支持的CUDA版本
nvcc -V # 看当前安装的CUDA运行时版本

部署训练框架

创建unsloth_env文件夹，并在里面操作

uv venv -p 3.11
uv pip install .\torch-2.4.0+cu121-cp311-cp311-win_amd64.whl
uv pip install .\torchvision-0.19.0+cu121-cp311-cp311-win_amd64.whl .\torchaudio-2.4.0+cu121-cp311-cp311-win_amd64.whl
uv pip install .\flash_attn-2.6.3+cu123torch2.4.0cxx11abiFALSE-cp311-cp311-win_amd64.whl
uv pip install --no-cache-dir "unsloth[huggingface] @ git+https://github.com/unslothai/unsloth.git"
uv pip install "bitsandbytes>=0.43.3" packaging ninja
uv pip install --no-deps "xformers==0.0.27.post2"
uv pip install .\triton-3.1.0-cp311-cp311-win_amd64.whl # 官方没有win版，win版下载：https://github.com/woct0rdho/triton-windows
	"triton\runtime\jit.py" 修改第717行，加入try，except self.src = None
uv pip install setuptools
uv pip uninstall transformers
uv pip install --upgrade --no-cache-dir "git+https://github.com/huggingface/transformers.git"
uv pip uninstall numpy # 不支持numpy2
uv pip install numpy==1.24.0
git clone --recursive https://github.com/ggerganov/llama.cpp

下载 llama.cpp 的可执行程序包 https://github.com/ggerganov/llama.cpp/releases 解压覆盖至 llama.cpp 目录下

unsloth更新：

uv pip install --upgrade --no-cache-dir --no-deps unsloth unsloth_zoo

".venv\Lib\site-packages\unsloth\save.py" 1028行左右修改：

from platform import system as platform_system
if platform_system() == "Windows":
	quantize_location = get_executable(["llama-quantize.exe", "quantize.exe"])
else:
	quantize_location = get_executable(["llama-quantize", "quantize"])

".venv\Lib\site-packages\unsloth\save.py" 1072行左右添加：

elif os.path.exists("llama.cpp/quantize.exe"):
	quantize_location = "llama.cpp/quantize.exe"
elif os.path.exists("llama.cpp/llama-quantize.exe"):
	quantize_location = "llama.cpp/llama-quantize.exe"

微调训练代码

from unsloth import FastLanguageModel
# from unsloth import is_bfloat16_supported
import torch
from trl import SFTTrainer
from transformers import TrainingArguments
from datasets import load_dataset
from unsloth import to_sharegpt
from unsloth import standardize_sharegpt
model_name = r"E:\AI\models\Qwen2.5-7B-Instruct"
max_seq_length = 1024 # Supports RoPE Scaling interally, so choose any!
load_in_4bit = True # 在 4 位量化中进行微调
dtype = torch.float16 # 保留此值为 None，但可以选择 torch.float16 或 torch.bfloat16，适用于较新的 GPU
r = 8 # 调整微调过程的等级。数字越大，使用的内存越多，速度越慢，但可以在更困难的任务上提高准确性。我们通常建议使用 8（适用于快速微调）到 128 之间的数字。数字过大可能导致过拟合，损害模型质量。同步 LoRA 系数。
max_steps = 60 # num_train_epochs = 1
num_train_epochs = 3 # 训练轮数
learning_rate = 5e-5 # 学习率
datasetpath = r".\alpaca_dataset.json" # Get Alpaca dataset
output_dir = "outputs"
gguf_dir = "ggufs"

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name = model_name,
    max_seq_length = max_seq_length,
    dtype = dtype,
    load_in_4bit = load_in_4bit,
)

# Load Alpaca dataset
dataset = load_dataset("json", data_files = {"train" : datasetpath}, split = "train")
dataset = to_sharegpt(
    dataset,
    merged_prompt = "[[{system}\n]]{instruction}[[\n{input}]]",
    output_column_name = "output",
)
dataset = standardize_sharegpt(dataset)

# Do model patching and add fast LoRA weights
model = FastLanguageModel.get_peft_model(
    model,
    r = r,
    target_modules = ["q_proj", "k_proj", "v_proj", "o_proj",
                      "gate_proj", "up_proj", "down_proj",],
    lora_alpha = r,
    lora_dropout = 0, # Supports any, but = 0 is optimized
    bias = "none",    # Supports any, but = "none" is optimized
    # [NEW] "unsloth" uses 30% less VRAM, fits 2x larger batch sizes!
    use_gradient_checkpointing = "unsloth", # True or "unsloth" for very long context
    random_state = 3407,
    max_seq_length = max_seq_length,
    use_rslora = False,  # We support rank stabilized LoRA
    loftq_config = None, # And LoftQ
)

trainer = SFTTrainer(
    model = model,
    train_dataset = dataset,
    dataset_text_field = None,
    max_seq_length = max_seq_length,
    tokenizer = tokenizer,
    args = TrainingArguments(
        per_device_train_batch_size = 2,
        gradient_accumulation_steps = 4,
        warmup_steps = 10,
        max_steps = max_steps,
        num_train_epochs = num_train_epochs,
        learning_rate = learning_rate,
        fp16 = True,
        bf16 = False,
        logging_steps = 1,
        output_dir = output_dir,
        optim = "adamw_8bit",
        seed = 3407,
        save_strategy = "steps",
        save_steps = 50,
    ),
)
trainer.train()
# trainer.train(resume_from_checkpoint = True) # start from the latest checkpoint and continue training.

# Save the model
model.save_pretrained_gguf(gguf_dir, tokenizer, quantization_method = "q4_k_m")

手动量化

若前步报错无法完成量化，则执行以下命令进行量化

在 E:\AI\llama.cpp 目录下

.\llama.cpp\llama-quantize.exe "ggufs\unsloth.F16.gguf" "ggufs\unsloth.Q4_K_M.gguf" q4_k_m