← all repositories
zhongkaifu/TensorSharp

Someone wrote a llama.cpp competitor in C#

Local inference is C++ and Python territory; TensorSharp bets a full GGUF engine — text, image, video, embeddings, agents — can live in .NET.

TensorSharp
Not currently ranked.
star history

What it does TensorSharp is an LLM inference engine and agent runtime built on .NET, loading GGUF models across a broad roster: DeepSeek V4/V4.1 Flash, GLM 5.x, Gemma 4, Qwen 3.5 through 3.8, GPT OSS, Nemotron-H, Mistral 3, and more. It goes past autoregressive text into DiffusionGemma-style text diffusion, Qwen-Image-2.1 image generation and editing, MiniMax-H3 video with native 32 kHz stereo audio, and BERT/XLM-R embeddings. Everything is reachable from a CLI, a browser chat UI, Ollama/OpenAI-compatible HTTP APIs, and an iPhone app, with CUDA, Metal, and Vulkan backends spanning Windows, macOS, Linux, and iOS.

The interesting bit The unusual part is where the code lives: the CPU path is managed .NET, with native GGML/CUDA/Metal/Vulkan backends underneath for the GPU heavy lifting, and the authors publish head-to-head benchmarks against llama.cpp on identical GGUF files and hardware. It’s also a teaching artifact — two books walk through building this engine from tensors and tokenizers to serving. The agent layer stands on its own: bounded tool loops, subagents with private workspaces and read-only defaults, so sandboxing is a design constraint rather than an afterthought.

Key highlights

  • Continuous batching and a paged, Radix prefix-shared KV cache ship on by default; speculative decoding and tensor parallelism are opt-in.
  • Tensor parallelism spans multiple GPUs and multiple machines — two nodes × two GPUs yields a global TP degree of 4.
  • TensorSharp.AgentHost adds Agent Skills, sandboxed file and shell tools, and automatic subagent delegation with independent contexts.
  • Ollama/OpenAI-compatible APIs cover chat plus batch embeddings (Snowflake Arctic Embed L, MiniLM).
  • TensorAgent extends the same chat and agent experience to iPhone and iPad via the iOS ggml_metal backend.

Caveats

  • Support is uneven by design: backend, modality, and validation coverage vary model by model, so check the model cards before committing.
  • The GB10/DGX Spark build’s one real-hardware check predates the current code, and hosted CI only re-verifies CPU and archive paths — GPU paths aren’t continuously certified.
  • The llama.cpp comparison is honest but scoped: results are specific to the measured model, backend, and workload.

Verdict .NET teams wanting local inference — or anyone who wants to learn how an inference engine actually works, with two books as the lab manual — should pay attention. If llama.cpp already covers your needs, this is a well-documented curiosity with a genuinely interesting agent runtime attached.

Frequently asked

What is zhongkaifu/TensorSharp?
Local inference is C++ and Python territory; TensorSharp bets a full GGUF engine — text, image, video, embeddings, agents — can live in .NET.
Is TensorSharp open source?
Yes — zhongkaifu/TensorSharp is open source, released under the BSD-3-Clause license.
What language is TensorSharp written in?
zhongkaifu/TensorSharp is primarily written in C#.
How popular is TensorSharp?
zhongkaifu/TensorSharp has 681 stars on GitHub.
Where can I find TensorSharp?
zhongkaifu/TensorSharp is on GitHub at https://github.com/zhongkaifu/TensorSharp.

heatdrop uses Google Analytics to see which pages get read — nothing else. Your call. How we handle data.