The complete map of edge AI: where inference moves from the cloud to the device
For most of the last decade, artificial intelligence meant sending data to a distant data center, waiting for a GPU cluster to respond, and trusting that the round trip would be fast enough and private enough. Edge AI inverts that model. Instead of carrying data to the model, it carries the model to the data — running inference directly on the phone in your pocket, the camera over a doorway, the microcontroller inside a sensor, or a small accelerator card bolted to an industrial gateway. EdgeAIDirectory is a curated, continuously updated catalog of the hardware, runtimes, and platforms that make this possible, organized so engineers and product teams can compare real options side by side instead of wading through marketing pages.
Why inference is moving to the edge
Three forces are driving the shift. The first is latency: a self-driving forklift, a factory defect detector, or a voice assistant cannot afford a hundred-millisecond network round trip, and on-device inference answers in single-digit milliseconds. The second is privacy and data sovereignty: when audio, video, and biometric data never leave the device, entire categories of regulatory and security risk disappear. The third is cost and connectivity: streaming continuous camera feeds to the cloud is expensive and impossible in places with intermittent networks, while a chip drawing a couple of watts can run all day on a battery. Together these forces have turned edge AI from a niche into one of the fastest-growing segments of the semiconductor and software industries.
The five layers of the edge AI stack
This directory is organized around the layers a real deployment touches. Edge hardware covers the silicon — NVIDIA Jetson modules, Google Coral Edge TPUs, and dedicated neural processing units from Hailo, Kneron, Axelera, SiMa.ai, DEEPX, MemryX, and neuromorphic and ultra-low-power designs from BrainChip and Syntiant. Inference frameworks are the runtimes that execute models efficiently across that silicon: TensorFlow Lite and its successor LiteRT, Microsoft's ONNX Runtime, Intel OpenVINO, PyTorch's ExecuTorch, and vendor toolchains such as NXP eIQ. Local LLM tools — Ollama, llama.cpp, LM Studio, Jan, LocalAI, and Cortex — bring large language models down to laptops and workstations with no cloud dependency. Mobile AI spans Apple Core ML, Qualcomm AI Hub, and MediaTek NeuroPilot, the frameworks that exploit the neural engines inside billions of phones. And IoT AI ties it together with embedded and orchestration platforms like Edge Impulse, AWS IoT Greengrass, Azure IoT Edge, and the STM32 and Texas Instruments TinyML toolchains.
How to use this directory
Every listing carries a description of at least fifty words explaining what the tool actually does, the hardware or models it targets, and where it fits in a real deployment — written from the documentation and public technical material of each vendor, not recycled press-release copy. Use the category filters to narrow by layer, or the search bar to jump straight to a name, a chipset, or a capability such as quantization, TinyML, or on-device LLM. The right answer is rarely a single product: a typical edge project pairs a runtime with an accelerator and an orchestration layer, and the goal here is to help you assemble that shortlist quickly.
EdgeAIDirectory is a free, independent resource. We do not sell hardware or licenses, and listings are chosen on technical relevance to the edge-inference market. New entries are added as the field evolves — and it is evolving fast, as generative models that once required a data center increasingly run on a chip you can hold between two fingers.
Inference Frameworks
End-to-end TinyML platform for IoT and embedded devices — model training, optimization, and deployment to MCUs.
Edge Impulse is an end-to-end development platform for TinyML — machine learning that runs on microcontrollers and other resource-constrained embedded devices. Engineers collect and label sensor data, design signal-processing and neural-network pipelines in a visual Studio, then export optimized C++ libraries that fit in kilobytes of RAM. It supports hundreds of boards from vendors such as Nordic, STMicroelectronics, Espressif, and Arduino, and its EON Compiler reduces model memory use further. Acquired by Qualcomm in 2025, Edge Impulse is widely used for predictive maintenance, audio keyword spotting, gesture recognition, and embedded vision.
Inference Frameworks
iot mcu edge tinyml
Intel's open-source toolkit for optimizing and deploying deep learning inference across Intel hardware.
OpenVINO (Open Visual Inference and Neural network Optimization) is Intel's open-source toolkit for optimizing and deploying deep-learning inference across Intel CPUs, integrated and discrete GPUs, and the NPUs built into Core Ultra processors. Its Model Optimizer converts and quantizes models from PyTorch, TensorFlow, ONNX, and PaddlePaddle into an intermediate representation, while the runtime automatically distributes workloads across available devices. OpenVINO is widely used for computer-vision, generative-AI, and video-analytics workloads on edge servers and PCs, and integrates with Hugging Face Optimum to accelerate transformer and diffusion models on Intel silicon.
Inference Frameworks
intel open-source inference optimization
Microsoft's cross-platform inference engine — runs ONNX models on CPU, GPU, and specialized accelerators.
ONNX Runtime is Microsoft's high-performance, cross-platform engine for running models in the open Open Neural Network Exchange format. Through a pluggable Execution Provider architecture it accelerates inference on CPUs, NVIDIA and AMD GPUs, Intel hardware via OpenVINO, Qualcomm NPUs via QNN, Arm processors, and mobile devices with NNAPI and Core ML. It powers AI features across Windows, Office, and Azure, and supports both server-side deployment and a lightweight build (ORT Mobile) for phones and embedded systems. Because ONNX is framework-agnostic, models trained in PyTorch or TensorFlow can be exported once and deployed everywhere.
Inference Frameworks
onnx microsoft cross-platform open-source
Google's lightweight inference framework for mobile and embedded — cross-platform model deployment for on-device AI.
TensorFlow Lite is Google's open-source runtime for running machine-learning models on phones, microcontrollers, and embedded Linux devices. It converts trained TensorFlow models into a compact FlatBuffer format and executes them through hardware delegates for GPU, the Android Neural Networks API, Core ML, and Hexagon DSPs. Post-training quantization to 8-bit integers shrinks models and speeds inference. In 2024 Google began rebranding the runtime as LiteRT while keeping the same APIs, and it remains one of the most widely deployed edge-inference engines in the world.
Inference Frameworks
tflite mobile embedded google
Local Llm
Local AI inference server — run multiple LLMs locally with an OpenAI-compatible API for app development.
Cortex is an open-source local AI engine from Menlo Research, the team behind Jan, designed to be embedded in applications that need private on-device inference. Written in C++ for low overhead, it runs GGUF models through a llama.cpp backend and can also load ONNX and TensorRT-LLM engines, exposing an OpenAI-compatible HTTP server plus a command-line interface. It supports model pulling from Hugging Face, automatic hardware detection, and multi-model serving, making it a lightweight alternative to heavier inference stacks for developers building local-first AI features into desktop and server software.
Local Llm
llm local api open-source
Open-source alternative to ChatGPT that runs offline — cross-platform desktop app for private local AI inference.
llama.cpp, created by Georgi Gerganov, is a dependency-free C/C++ implementation of LLM inference that made it practical to run large language models on ordinary laptops and even Raspberry Pis. It introduced the GGUF model format and a family of quantization schemes (from 2-bit to 8-bit) that dramatically reduce memory footprint with minimal quality loss. The project supports CPU SIMD, CUDA, Metal, Vulkan, and ROCm backends, and underpins many downstream tools including Ollama, LM Studio, and Jan. Its ggml tensor library is the engine behind much of the local-AI ecosystem.
Local Llm
llm local open-source gui
Pure C/C++ LLM inference with GGUF quantization — run large language models on CPU and consumer GPUs.
LM Studio is a free desktop application for discovering, downloading, and running open-source large language models entirely on a personal computer. Available for macOS, Windows, and Linux, it offers a friendly graphical interface for browsing Hugging Face models, selecting quantization levels, and chatting with them offline. Under the hood it uses llama.cpp and Apple's MLX engine to take advantage of GPU and Apple-silicon acceleration. LM Studio can also run a local OpenAI-compatible server so other applications connect to private models, and its lms command-line tool and SDK support developer workflows and document chat with retrieval.
Local Llm
llm cpp quantization cpu-inference
Desktop app for running open-source LLMs locally — GUI for model discovery, download, and inference.
LocalAI is a self-hosted, drop-in replacement for the OpenAI API that runs entirely on consumer hardware without a GPU requirement. It exposes the same REST endpoints used by OpenAI clients, so applications can switch from cloud to local inference by changing only the base URL. Beyond text generation it supports audio transcription with Whisper, text-to-speech, image generation with Stable Diffusion, and embeddings, drawing on multiple backends including llama.cpp and Transformers. It ships as a single binary or Docker image and is designed for privacy-conscious teams that want full control over their data.
Local Llm
llm local gui mac-windows
Free, open-source alternative to OpenAI APIs running locally — supports LLaMA, Whisper, Stable Diffusion, and more.
Ollama packages local large-language-model inference behind a single command-line tool and a Docker-style model registry. Built on top of llama.cpp, it lets users pull and run models such as Llama, Mistral, Gemma, Phi, and DeepSeek with one command, automatically handling GGUF downloads, quantization variants, and GPU offload on macOS, Windows, and Linux. It exposes an OpenAI-compatible REST API on localhost so existing applications can swap cloud endpoints for private on-device inference, and a Modelfile system lets users customize system prompts and parameters.
Local Llm
llm local open-source openai-compatible
The easiest way to run open-source LLMs locally — Docker-style model management for local AI inference.
Core ML is Apple's framework for running machine-learning models entirely on-device across iPhone, iPad, Mac, Apple Watch, and Vision Pro. At runtime it automatically schedules computation across the CPU, GPU, and the dedicated Apple Neural Engine to maximize performance while minimizing power and memory. Developers convert PyTorch and TensorFlow models with the coremltools package, and the format supports quantization, palettization, and on-device personalization. Core ML underpins many system features and third-party apps, and recent releases added stateful models and efficient KV-caching to support running large language models locally on Apple silicon.
Local Llm
llm local easy-setup docker-like
Mobile Ai
Apple's framework for running ML models on-device — hardware-accelerated via Neural Engine on iPhone, iPad, and Mac.
Axelera AI is a Netherlands-based semiconductor company building the Metis AI Platform for high-performance, energy-efficient computer-vision inference at the edge. Its Metis AI Processing Unit uses a digital in-memory computing architecture combined with RISC-V control cores to deliver large amounts of INT8 throughput at low power, shipped as PCIe and M.2 accelerator cards. The accompanying Voyager SDK lets developers deploy standard models from PyTorch and ONNX with quantization handled automatically. Backed by significant European funding, Axelera targets smart retail, security, robotics, and industrial vision applications that need data-center-class accuracy without cloud round-trips.
Mobile Ai
apple ios macos on-device
Edge Hardware
Arm's ML platform and Ethos NPU IP — the AI compute backbone of most mobile and IoT chips.
Coral is Google's hardware-and-software platform for building products with fast, private, on-device AI. At its core is the Edge TPU, a small ASIC that performs 4 trillion operations per second while drawing roughly 2 watts, delivering about 2 TOPS per watt. Coral ships the Edge TPU in several form factors including a USB Accelerator, M.2 and Mini PCIe modules, the Dev Board single-board computer, and system-on-modules. The Edge TPU runs INT8-quantized TensorFlow Lite models compiled with the Edge TPU Compiler, and is popular for embedded vision, smart cameras, and offline keyword spotting.
Edge Hardware
arm npu embedded ethos
European edge AI chip startup — Metis AI Platform using phase-change memory for ultra-efficient inference.
Hailo is an Israeli fabless semiconductor company that designs purpose-built AI processors for edge inference. Its flagship Hailo-8 delivers up to 26 TOPS at typical power consumption of a few watts, and the newer Hailo-10 family targets generative AI on devices; both ship as M.2 and PCIe modules as well as system-on-modules. Hailo's dataflow architecture maps each layer of a neural network onto dedicated compute resources for high efficiency, and the Hailo Dataflow Compiler ingests models from TensorFlow and ONNX. The chips are widely deployed in smart cameras, automotive ADAS, industrial automation, and the Raspberry Pi AI Kit.
Edge Hardware
ai-chip edge pcm european
Google's Edge TPU hardware — USB and M.2 accelerators for running TensorFlow Lite models at the edge with low power.
Kneron is a San Diego-based edge-AI company that designs reconfigurable neural processing units for fully on-device inference. Its KL520, KL630, and KL730 chips combine NPU, CPU, and image-signal-processing blocks to run computer-vision and increasingly transformer workloads at low power without a network connection. Kneron emphasizes a heterogeneous, reconfigurable architecture that adapts to convolutional and transformer models alike, and provides the KNEO software platform and toolchain for porting standard models. Its silicon appears in smart locks, security cameras, access-control systems, automotive applications, and AIoT devices that require private, latency-free processing.
Edge Hardware
tpu embedded google tflite
Hailo AI processors for edge devices — high-performance inference chips for smart cameras and embedded vision.
NVIDIA Jetson is a family of system-on-modules and developer kits that bring GPU-accelerated AI computing to robots, drones, smart cameras, and autonomous machines. The current Orin lineup ranges from the Jetson Orin Nano up to the AGX Orin, which delivers as much as 275 TOPS, while the entry-level Orin Nano Super made capable edge AI available for around 250 dollars. Every Jetson runs the same CUDA, cuDNN, and TensorRT software stack used on data-center GPUs, plus the JetPack SDK and Isaac robotics frameworks, so models scale from the cloud to the edge with a consistent toolchain.
Edge Hardware
ai-chip edge vision video-analytics
Edge AI chip and solution provider — KL series NPUs for smart cameras, IoT devices, and embedded systems.
Qualcomm AI Hub is a developer platform for optimizing, validating, and deploying machine-learning models on Snapdragon and Qualcomm-powered devices. Developers upload a PyTorch or ONNX model and the service automatically converts and compiles it for the target chipset, then runs profiling and inference jobs on real cloud-hosted devices to report latency and accuracy on the Hexagon NPU, Adreno GPU, and Kryo CPU. It ships with a library of more than a hundred pre-optimized models spanning vision, audio, and generative AI, helping mobile and IoT developers take full advantage of the Qualcomm AI Engine.
Edge Hardware
ai-chip edge vision iot
NVIDIA's embedded computing platform for AI at the edge — Jetson Orin and Nano for robotics, vision, and autonomous systems.
Edge Hardware
gpu embedded nvidia robotics
Qualcomm's AI model deployment hub — optimize and deploy AI models on Snapdragon-powered devices.
Edge Hardware
mobile-ai qualcomm snapdragon npu