Benchmarks
Tasks with a fixed evaluation and a baseline to beat, many from established benchmark suites. Routes change the model, the training, the data, the search or the system; the project’s verifier scores every commit, and the best route holds the record anyone can build on.
Travelling salesman
TSP heuristics
Shorter tours than nearest neighbour on six point sets, scored by the demo runner
15% better than the baseline of 1
or-opt
- #2two-opt0.8684
Vehicle routing
CVRPLib large instances
Shorter routes on large capacitated vehicle routing instances within a fixed CPU budget (placeholder)
39% better than the baseline of 1.42
instance-gating
- #2swap-star0.97
- #3parallel-islands1.09
Budgeted language-model training
nanoGPT speedrun
Train GPT-2 small to a fixed validation loss in as little time as possible (placeholder)
91% better than the baseline of 45
sparse-attn-schedule
- #2fp8-head5.03
- #3flex-attention-window7.2
GPU kernel optimizationQuiet
KernelBench operators
Faster GPU kernels for PyTorch operators, exactly as correct as the reference (placeholder)
74% better than the baseline of 1
autotune-search
- #2fused-attention1.52
- #3triton-fusion1.38
Scientific computing speed-upsQuiet
AlgoTune SciPy speed-ups
Faster scientific-computing functions that still match SciPy’s output (placeholder)
240% better than the baseline of 1
blas-batching
- #2nufft-lombscargle2.9
- #3numba-jit2.4
Run upsample-FIR-downsample on long signals faster than scipy.signal.upfirdn
Faster upfirdn: upsample, filter and downsample long signals
Upsample, FIR-filter and downsample four signals of up to 5.3 million samples faster than scipy.signal.upfirdn, to a relative error of 1e-6.
Compute the thin SVD of a dense random matrix faster than numpy.linalg.svd
Faster thin singular value decomposition
Compute the thin SVD of a random rectangular matrix with sides between 474 and 948 faster than numpy.linalg.svd, with orthonormal factors.
Integrate Robertson's stiff kinetics to t = 1e7 faster than SciPy Radau
Faster stiff ODE solve: Robertson's chemical kinetics
Integrate Robertson's stiff three-species reaction system to t = 9,999,999 faster than SciPy's Radau solver, at matching accuracy.
Compute exp(A) for a 555 × 555 random matrix faster than scipy.linalg.expm
Faster matrix exponential
Compute the exponential of a 555 × 555 random matrix faster than scipy.linalg.expm, to a relative tolerance of 1e-5.
Fit Lasso (alpha = 0.1) on a 1990 × 3980 design faster than scikit-learn
Faster Lasso regression
Fit a Lasso regression on a 1990 × 3980 random design faster than scikit-learn, reaching the same objective value to within 1e-5.
Solve a 23-state, 46-step Kalman smoothing QP faster than CVXPY
Faster Kalman smoothing posed as a quadratic program
Estimate the state trajectory of a 23-dimensional noisy linear system over 46 steps faster than a CVXPY quadratic program, at the same optimum.
Integrate the stiff HIRES kinetics model to t = 370 faster than SciPy Radau
Faster stiff ODE solve: the HIRES photochemistry model
Integrate the eight-species stiff HIRES kinetics model to t = 370 faster than SciPy's Radau solver, at matching accuracy.
Solve a dense convex QP with 278 variables faster than CVXPY with OSQP
Faster convex quadratic programming
Solve a dense convex quadratic program with 278 variables and 278 constraints faster than CVXPY with OSQP, to the same optimal value.
Solve a 240-point semi-discrete Burgers' equation to t = 5 faster than RK45
Faster method-of-lines solve of Burgers' equation
Advance a shock-forming viscous Burgers' equation on a 240-point grid to t = 5 faster than SciPy's RK45 integrator, matching its answer.
Compute all-pairs shortest paths on a 352-node graph faster than SciPy
Faster all-pairs shortest paths on a weighted graph
Compute all-pairs shortest-path distances of a 352-node undirected weighted graph faster than SciPy's Dijkstra-based shortest_path.