Type Your Question

How to use Llama AI with Python?

 Thursday, 20 March 2025

---

Llama 2, developed by Meta AI, is a family of open-source large language models (LLMs) that provides state-of-the-art performance on many benchmarks and is available for research and commercial use. Integrating Llama 2 with Python allows developers and researchers to harness its capabilities for various applications. This guide will provide a detailed explanation of how to use Llama 2 with Python, covering installation, inference, fine-tuning, and practical examples. We will primarily focus on leveraging popular libraries such as transformers and torch for a seamless experience.

1. Prerequisites

Before getting started, ensure you have the following:

• Python 3.8+: Make sure you have Python 3.8 or a newer version installed.
• Pip: Python package installer.
• A Machine with sufficient resources: Llama 2 requires substantial computational resources (RAM and potentially a GPU) depending on the model size and task. Consider using cloud-based GPU instances (AWS, GCP, Azure) if you lack sufficient local resources.
• Meta Llama 2 Access: You need to request access to download the Llama 2 models from the Meta AI website. After approval, you'll receive instructions for accessing the weights.

2. Setting Up Your Environment

It is highly recommended to create a virtual environment to manage project dependencies. This prevents conflicts with other Python projects.

 python -m venv llama2_env
 source llama2_env/bin/activate # On Linux/macOS
 # llama2_env\Scripts\activate # On Windows
 

Now, install the necessary libraries:

 pip install torch transformers accelerate sentencepiece
 

* torch: PyTorch, a deep learning framework.
* transformers: Hugging Face's Transformers library, providing pre-trained models and tools.
* accelerate: Hugging Face's Accelerate library, allows to easily distribute computations on multiple GPUs. This package helps you execute the model more efficiently and quickly.
* sentencepiece: SentencePiece is a text tokenization library required by Llama 2.

If you have a GPU, install the appropriate CUDA version of PyTorch:

bash
# Example (check the PyTorch website for the latest command for your CUDA version)
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118

3. Accessing Llama 2 Models

After requesting access and receiving approval from Meta, you'll likely need to log in via the Hugging Face CLI to gain access to download Llama 2 weights. Follow the instructions provided by Meta and Hugging Face. Typically this involves:

1. *Obtaining an Access Token:* From your Hugging Face profile, generate a read access token.

2. *Logging in via the CLI:*
bash
huggingface-cli login
# Enter your access token when prompted

4. Performing Inference with Llama 2

Once you have access and your environment is set up, you can start using Llama 2 for inference. We'll use the transformers library for this.

python
from transformers import AutoTokenizer, AutoModelForCausalLM

model_name = "meta-llama/Llama-2-7b-chat-hf" # Or a different Llama 2 variant

# Load tokenizer and model
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name, device_map='auto') # Automatically maps layers to available GPUs

# Prepare the prompt
prompt = "Write a short story about a talking cat who goes on an adventure."

# Tokenize the input
input_ids = tokenizer(prompt, return_tensors="pt").to(model.device) #Move to GPU if available

# Generate text
generation_output = model.generate(
*input_ids,
max_new_tokens=256,
do_sample=True,
top_p=0.9,
temperature=0.6,
repetition_penalty=1.15,
pad_token_id=tokenizer.eos_token_id
)

# Decode the output
output = tokenizer.decode(generation_output[0], skip_special_tokens=True)

print(output)


*Explanation:*

* AutoTokenizer.from_pretrained(model_name): Loads the appropriate tokenizer for the Llama 2 model.
* AutoModelForCausalLM.from_pretrained(model_name, device_map='auto'): Loads the pre-trained Llama 2 model. device_map='auto' attempts to utilize available GPUs, if present. It intelligently maps layers to the optimal GPU(s). If no GPU is found, the model runs on CPU (though it will be significantly slower).
* tokenizer(prompt, return_tensors="pt"): Tokenizes the input prompt using the loaded tokenizer. return_tensors="pt" specifies that the output should be PyTorch tensors. The tensor is moved to the model's device (model.device).
* model.generate(...): Generates text based on the input tokens. The generate method accepts various parameters:
* max_new_tokens: Limits the number of newly generated tokens (controls the length of the output).
* do_sample: Enables sampling for diverse and creative output. If set to False it will perform greedy decoding.
* top_p: Nucleus sampling; controls the diversity by considering only tokens whose cumulative probability exceeds this threshold. 0.9 is a common value.
* temperature: Controls randomness; lower values yield more deterministic and predictable outputs. Higher values increase randomness and creativity. 0.6 is usually good.
* repetition_penalty: Deters the model from repeating phrases. 1.0 means no penalty; values greater than 1.0 penalize repetition. A good starting point is 1.15
* pad_token_id: Some model implementations requires to explicitly pass pad token id to properly decode and generate consistent result. This param is available starting from 4.35 version of transformers

* tokenizer.decode(generation_output[0], skip_special_tokens=True): Decodes the generated tokens back into human-readable text. skip_special_tokens=True removes special tokens like padding tokens.

*Choosing a Model Variant:*

The model_name variable specifies which Llama 2 variant you want to use. Available options include:

* meta-llama/Llama-2-7b-chat-hf : 7 billion parameter model optimized for dialogue.
* meta-llama/Llama-2-13b-chat-hf: 13 billion parameter model optimized for dialogue.
* meta-llama/Llama-2-70b-chat-hf: 70 billion parameter model optimized for dialogue (requires significant resources).
* meta-llama/Llama-2-7b-hf (Base model).
* meta-llama/Llama-2-13b-hf (Base model).
* meta-llama/Llama-2-70b-hf (Base model).

Generally, larger models (e.g., 70B) produce better results but require more resources. The "chat" versions are specifically trained for conversational applications. Base models can perform generic tasks, but often excel after being finetuned for a specific need.

5. Optimizing Inference for Speed

Running inference with large language models can be computationally intensive. Here are some techniques to optimize for speed:

* Quantization: Reduces the memory footprint and computation by using lower-precision data types (e.g., converting weights from float32 to int8).
* GPU Acceleration: Offloads computations to a GPU. The device_map="auto" helps manage distributing across GPUs if available.
* Mixed Precision Training: Utilizing lower-precision datatypes where acceptable, using a flag during the from_pretrained() load method for the model: torch_dtype=torch.float16. Requires recent PyTorch and CUDA installations. This approach drastically reduce resource usage and can greatly reduce computation time when combined with GPU use.

Example:

python
from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

model_name = "meta-llama/Llama-2-7b-chat-hf"

tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name, device_map='auto', torch_dtype=torch.float16)

# (Rest of the inference code remains the same)

6. Fine-Tuning Llama 2 with Python

Fine-tuning involves training a pre-trained model on a specific dataset to improve its performance on a particular task. This process adapts the model to a specialized domain or style. Tools like transformers, accelerate and PEFT (Parameter-Efficient Fine-Tuning) make this process more accessible and efficient.
We'll use the PEFT library to perform QLoRA (Quantization-Aware Low-Rank Adaptation), a parameter efficient tuning method.

First install all necessary libraries:
bash
pip install peft trl accelerate datasets

Sample implementation:

python
from datasets import load_dataset
from transformers import AutoModelForCausalLM, AutoTokenizer, TrainingArguments
from trl import SFTTrainer
from peft import LoraConfig

# 1. Load Dataset
dataset_name = "Abirate/english_quotes" # Sample dataset
dataset = load_dataset(dataset_name, split="train")

# 2. Model and Tokenizer Setup
model_name = "meta-llama/Llama-2-7b-chat-hf"
tokenizer = AutoTokenizer.from_pretrained(model_name, use_fast=True) #Using fast tokenizer will make the performance better.
tokenizer.pad_token = tokenizer.eos_token #Crucial, add to avoid errors
model = AutoModelForCausalLM.from_pretrained(
model_name,
load_in_4bit=True, # Load in 4-bit quantization for memory efficiency
device_map="auto",
)
model.config.use_cache = False # Necessary for QLoRA

# 3. LoRA Configuration (PEFT)
lora_config = LoraConfig(
r=16, # Rank of the update matrices
lora_alpha=32, # Scaling factor for the LoRA weights
lora_dropout=0.05, # Dropout probability for LoRA layers
bias="none", # Disable bias training
task_type="CAUSAL_LM", # Task type (causal language modeling)
target_modules=["q_proj", "v_proj"] #Linear layers to finetune using qLoRA

)

# 4. Training Arguments
training_arguments = TrainingArguments(
output_dir="./llama-2-finetuned", #The place where finetuned results are saved
per_device_train_batch_size=4, #Adjust according to your system
gradient_accumulation_steps=4,
optim="paged_adamw_32bit", #Optimiser is recommended with QLoRA.
save_steps=100, #Checkpoints saving every 100 steps.
logging_steps=50, #Output some logging every 50 steps
learning_rate=2e-4, #Keep this number between 1e-5, 2e-4 depending of the dataset size.
fp16=True, #Flag to enable fp16, usually gives 2x performance increase for some hardware, check.
max_steps=500, #Tuning on a dataset should takes a long time to provide very accurate responses, but will improve accuracy a bit.

)

# 5. Trainer Setup (TRL)
trainer = SFTTrainer(
model=model,
train_dataset=dataset,
dataset_text_field="quote", #Column with your dataset text field.
max_seq_length=512, #Dataset records long, needs enough max length otherwise you may skip long records if shorter.
tokenizer=tokenizer,
args=training_arguments,
peft_config=lora_config,

)

# 6. Training
trainer.train()

# 7. Save the Model
trainer.save_model()

print("Fine-tuning complete! Model saved to ./llama-2-finetuned")


*Explanation:*
* Datasets load. Load training datasets with training or custom data to improve model precision, you can change Abirate/english_quotes to others dataset name, or use with the same dataset for code execution simplicity
* PEFT config (LoraConfig)
Here's a breakdown of the arguments passed:

* r: The rank of the update matrices. A higher rank allows for more complex updates but requires more parameters. A rank of 16 to 32 often is reasonable.
* lora_alpha: The scaling factor for the LoRA weights. It adjusts the magnitude of the LoRA updates, allowing for fine-grained control over the fine-tuning process. It's recommended to equal to the double size from "r".
* lora_dropout: The dropout probability for LoRA layers. It prevents overfitting and encourages the model to learn more robust features. Recommended to be around 5% - 10% max (0.05-0.1).
* Training Arguments. Parameters to make sure the PEFT is configured
* The target_modules refers to layers inside of model which are "Linear". Target_modules is essential to choose, and different LLMs architecture varies regarding target_modules linear layers (Q_proj, V_proj and O_proj mostly but that's dependent). Inspecting and choosing linear layer target module for the specified Llama 2 variant. (e.g., q_proj, v_proj from transformers) can be crucial to training accurately.

*Key Considerations:*

*Dataset Quality:* The quality and relevance of the fine-tuning dataset are critical.
*Hardware:* Fine-tuning LLMs is computationally demanding and benefits significantly from a GPU. Consider cloud-based solutions like AWS SageMaker, Google Cloud AI Platform, or Azure Machine Learning. Using CUDA (if you have Nvidia based video card) during this processing will save up more processing and power cost on calculations
*Evaluation:* Monitor the model's performance on a validation set to prevent overfitting.
*LoRA Rank* This value determines how big fine tuned version weights gets generated
*Save the generated/ finetuned version for future loading after the processes, in order to avoid repetitive actions over same code in the long future*
## 7. Advanced Topics

* Reinforcement Learning (RLHF)
* For an AI which has been properly trained in terms of general knowledge in the field and can interact over certain question/ subject using text. In some conditions or requests the language used may be inaccurate, unethical (racist etc.) and unsafe (e.g: criminal and drugs usage instructions). Thus RLHF comes into place, being reinforcement learning an approach which make an additional finetuning level by providing negative feedback (usually given manually) that reduce unethical use of it, this method relies with "Reinforcement learning". Note that to complete the RLHF on the ai model, a very expert team, knowledgable of the model should be prepared or the results might go south really soon causing the opposite behaviour on certain circunstances
* Custom Training Loop and datasets Instead of loading pretrained methods, you can program dataset manually including training and loop in order to get fine tuned output as wished
* Using other AI and LLM related open libraries or APIs Like for vector databases, embedding text and LLM orchestration there exists langchain with its expression language which supports Llama V2 implementations using Python

8. Conclusion

This guide provided a comprehensive overview of how to use Llama 2 AI with Python. By leveraging libraries like transformers, and PEFT, developers and researchers can readily harness the power of these models for various AI applications. By following proper instructions, instructions and testing methods on those implementations we may contribute on AI use cases which respects all ethical regulations required and bring only well created projects without unexpected uses to make a great AI environment between humans and machines

Llama Python Api Sdk 

---

Related

---