Over the past twelve months, inference compute demand has surged roughly 3.7x. Yet average GPU utilization across major data centers sits below 35%. That gap is not a supply problem. It is a protocol problem. Wang Dong, co-founder of Chinese GPU maker Moore Threads, recently dropped a thesis that cuts straight to the bone: there is no universal chip for inference. The market will run on combinations. For anyone who builds infrastructure protocols, this should set off alarms and incentives both.
Context: The Fragmented State of Inference
Wang Dong’s core claim—that no single architecture can optimally serve all inference scenarios—matches what we observe at the instruction level. NVIDIA’s H100 dominates training, but inference is a zoo: Groq’s LPU for latency-critical tasks, Cerebras’ wafer-scale chips for monolithic models, AMD’s MI300X for memory-bandwidth-heavy workloads, and a dozen ASICs for edge deployment. Moore Threads itself positions its MTT series as a general-purpose GPU, but realistically it plays best in Chinese government cloud projects and cost-sensitive environments. The fragmentation is not temporary; it is a structural consequence of diverging model architectures and deployment constraints.
Core: Breaking the Block to See What Spins
Wang Dong’s real contribution is framing this as a “combination of solutions”—a concept that should be deeply familiar to anyone who has ever designed a composable DeFi protocol. The idea: an Inference Service Provider (ISP) aggregates different hardware backends, routes each inference request to the optimal chip based on latency, batch size, and precision requirements, and abstracts the heterogeneity away from the model developer. This is essentially an execution layer for AI compute. But here’s where the protocol architect’s skepticism kicks in.
At the silicon level, the differences between inference engines are not just performance deltas; they are semantic. A Groq LPU executes operations in a deterministic dataflow graph, while a NVIDIA GPU relies on warp scheduling and shared memory. An AMD MI300X uses a different memory coherence model. Wrapping all these under a single virtual instruction set architecture (ISA) is non-trivial. The last time this was attempted at scale—Intel’s iAPX 432—it failed because hardware abstraction layers always leak. The question is not whether the abstraction can be built, but whether the overhead and security guarantees are acceptable.
This is where Wang Dong’s thesis intersects with blockchain infrastructure. The ISP model is essentially a centralized version of what decentralized compute networks (Akash, io.net, Render) attempt to do: dynamically allocate hardware to inference jobs. But the centralized ISP can cheat. It can lie about which chip executed the model. It can downgrade precision to save costs. It can reuse model weights across tenants without permission. The decentralized version suffers from the opposite problem: verifiability of execution is computationally expensive—especially when different chips produce floating-point results that differ at the last bit.
Contrarian: The Blind Spots in Wang Dong’s Vision
Wang Dong is correct that the market will move toward combinations. But his prescription—independent ISPs aggregating heterogeneous hardware—ignores a critical economic and security blind spot. ISPs, like any intermediary, will optimize for their own profit. In practice, this means they will use the cheapest hardware that barely meets the SLA—likely older-generation chips with higher power consumption and lower reliability. The combination becomes a race to the bottom on hardware quality. Worse, the customer has no way to audit which physical chip executed their model. This is a trust problem, not a code problem.
From my 2020 audits of DeFi composability, I learned that every abstraction layer introduces new attack surfaces. The “hook” in Uniswap V4 is a controlled explosion of complexity. The ISP’s hardware abstraction layer is no different. It will contain zero-day exploits—maybe a side-channel via timing variance between chip models, or a shared memory leak when switching between Groq and NVIDIA backends. Wang Dong’s team at Moore Threads is not known for pushing security patches fast; their software stack (MUSA, PyTorch adapter) still lags months behind upstream. Building a reliable ISP on top of that foundation is futuristic at best.
Another hidden assumption: that Chinese model companies will continue to have a cost advantage. That advantage relies on aggressive quantization and distillation, which introduce model instability. When you couple an unstable model with hardware that has inconsistent numerical precision, you get outputs that vary not just in quality but in safety. For applications like legal AI or medical triage, this is a liability. Regulators will demand audit trails that map each inference to a specific hardware identity—something the current ISP proposals do not provide.
Takeaway: The Protocol That Proves Execution
The real opportunity is not in building another ISP. It is in building a verification layer—a decentralized oracle that attests to which hardware executed a given inference, with what precision, and under what timing constraints. Think of it as a proof-of-execution protocol that uses a mix of trusted execution environments (TEEs) and economic bonds to hold ISPs accountable. The model developer submits a hash of the model weights and a request; the ISP responds with a proof that includes the chip ID, the execution trace, and a zero-knowledge proof of correct operation (for a subset of ops). If the ISP cheats, the bond slashes.
This is hard. It requires aligning incentives across hardware vendors, ISPs, and model owners. But Wang Dong’s thesis makes it inevitable. When hardware becomes a commodity, trust becomes the scarce resource. And on a technical level, trust is just a protocol waiting to be written.
Signatures: - Breaking the block to see what spins. - Silicon ghosts in the machine, verified. - Building on chaos, then locking the door. - Composability is just controlled anarchy.
