Inferno AI Documentation

Welcome to the comprehensive documentation for Inferno AI - a high-performance, privacy-first AI inference server that runs 100% locally on your machine.

What is Inferno?

Inferno is a powerful AI inference server designed for local AI model execution with enterprise-grade features. It provides:

🔒 Complete Privacy: All inference happens on your device - your data never leaves your machine
⚡ Exceptional Performance: GPU acceleration with Metal (13x faster on Apple Silicon), CUDA, and ROCm support
🔌 OpenAI Compatible: Drop-in replacement for OpenAI API - works with existing tools
📦 Multi-Format Support: Production-ready GGUF support, ONNX in active development
🌐 Cross-Platform: Available for macOS, Linux, Windows, and Docker

Documentation Sections

🚀 Getting Started

New to Inferno? Start here for installation guides, quick start tutorials, and initial configuration.

Installation for all platforms
Quick start guide
Basic configuration

📚 Guides

In-depth guides for common workflows, production deployment, and best practices.

Production deployment
Model management
Troubleshooting

🎓 Tutorials

Step-by-step tutorials for specific use cases and advanced features.

Performance optimization
Package manager integration
Advanced configurations

📖 Reference

Complete technical reference for APIs, CLI commands, and architecture.

API Reference (OpenAI-compatible)
CLI Command Reference
System Architecture
Changelog

💡 Examples

Practical code examples and integration guides for popular frameworks.

Python integration
REST API usage
Streaming responses
Docker deployment

🔧 Troubleshooting

Common issues, solutions, and debugging techniques.

Installation issues
Performance problems
Platform-specific fixes

Quick Links

GitHub Repository - Source code and contributions
Report Issues - Bug reports and feature requests
Contributing Guide - How to contribute
Download Inferno - Get the latest release

Version Information

Current version: v0.7.0

Inferno is in active development. We encourage you to report any issues you encounter and contribute to the project!

Key Features at a Glance

Privacy-First Architecture

Run AI models completely offline. Perfect for sensitive data, confidential work, or when you value your privacy.

GPU Acceleration

Metal (Apple Silicon: M1/M2/M3/M4) - Production ready, 13x faster
CUDA (NVIDIA GPUs) - Supported
ROCm (AMD GPUs) - Supported
Intel Arc - Experimental support

OpenAI-Compatible API

Works seamlessly with existing tools and libraries. Simply change the base URL:

from openai import OpenAI
 
client = OpenAI(
    base_url="http://localhost:8080/v1",
    api_key="not-needed"
)
 
response = client.chat.completions.create(
    model="llama-2-7b-chat",
    messages=[{"role": "user", "content": "Hello!"}]
)

Model Format Support

GGUF: Production-ready (llama.cpp compatible)
ONNX: Active development

Getting Help

Check the Getting Started guide for installation help
Browse the Guides section for in-depth tutorials
Search the documentation using the search bar above
Visit GitHub Issues for support

Ready to get started? Head to the Getting Started section →