Performance Benchmarks¶

Comprehensive benchmarks for llcuda v2.1.0 on Tesla T4 GPUs across different models and configurations.

Executive Summary¶

Gemma 3-1B (Verified)¶

Test Configuration¶

model = "unsloth/gemma-3-1b-it-GGUF:gemma-3-1b-it-Q4_K_M.gguf"
gpu_layers = 35
ctx_size = 2048
batch_size = 512
ubatch_size = 128

Results¶

Detailed Performance by Prompt Length¶

Observation: Performance remains consistent across varying input/output lengths.

Quantization Comparison¶

GPU Layer Offload Impact¶

Recommendation: Always use full GPU offload (gpu_layers=99 or 35 for Gemma 3-1B).

Context Size Impact¶

Batch Size Effect¶

Recommendation: Use batch_size=512, ubatch_size=128 for optimal performance.

Llama 3.2-3B (Estimated)¶

Test Configuration¶

model = "unsloth/Llama-3.2-3B-Instruct-Q4_K_M"
gpu_layers = 99
ctx_size = 2048
batch_size = 256
ubatch_size = 64

Results¶

Quantization Comparison¶

Qwen 2.5-7B (Estimated)¶

Test Configuration¶

model = "Qwen/Qwen2.5-7B-Instruct-GGUF:Q4_K_M"
gpu_layers = 99
ctx_size = 2048
batch_size = 128
ubatch_size = 32

Results¶

Llama 3.1-8B (Estimated)¶

Test Configuration¶

model = "unsloth/Llama-3.1-8B-Instruct-Q4_K_M"
gpu_layers = 99
ctx_size = 2048
batch_size = 128
ubatch_size = 32

Results¶

Model Size vs Performance¶

Recommendation: Gemma 3-1B Q4_K_M offers best performance for T4.

Flash Attention Impact¶

Benchmarks with and without FlashAttention:

Key Finding: FlashAttention provides 1.3-2x speedup for contexts > 4096 tokens.

Concurrent Requests¶

Performance with parallel request handling:

Use Case: n_parallel=4 optimal for serving applications.

Temperature vs Speed¶

Impact of sampling parameters on performance:

Impact: Higher temperatures slightly reduce speed due to sampling overhead.

Comparison with Other Solutions¶

vs PyTorch Native¶

Advantage: llcuda is 3x faster than PyTorch, 1.5x faster than vLLM.

vs Other GGUF Runners¶

Hardware Requirements¶

Minimum Requirements¶

• GPU: Tesla T4 (SM 7.5)
• VRAM: 2 GB (for 1B models)
• CUDA: 12.0+
• RAM: 8 GB

Recommended Requirements¶

• GPU: Tesla T4
• VRAM: 15 GB (full T4)
• CUDA: 12.2+
• RAM: 16 GB

Reproducibility¶

All benchmarks can be reproduced using:

import llcuda
import time

# Setup
engine = llcuda.InferenceEngine()
engine.load_model("gemma-3-1b-Q4_K_M", auto_start=True)

# Warmup
for i in range(5):
    engine.infer("Warmup", max_tokens=10)

# Benchmark
engine.reset_metrics()
prompts = ["Test prompt"] * 100

start = time.time()
results = engine.batch_infer(prompts, max_tokens=100)
elapsed = time.time() - start

# Results
metrics = engine.get_metrics()
print(f"Throughput: {metrics['throughput']['tokens_per_sec']:.1f} tok/s")
print(f"Latency: {metrics['latency']['p50_ms']:.0f} ms")

Next Steps¶

• T4 Results - Detailed T4 analysis
• Optimization Guide - Performance tuning
• Performance Tutorial - Hands-on optimization

Benchmark Data¶

Full benchmark data available at: - GitHub Repository - Colab Notebook