Scheduling middleware that batches and routes long-context LLM jobs intelligently, cutting inference costs 60% without new hardware.

1. Fractile's $220M raise from Accel and Founders Fund in May 2026 establishes top-tier investor consensus that inference throughput is the new AI infrastructure battleground, creating a buyer narrative that software-layer scheduling tools can ride immediately.
2. Fractile's reported jump from 40 to 1,200 tokens per second quantifies a 30x throughput gap; software scheduling can capture roughly half that gap on existing GPUs today while Fractile hardware remains 12–18 months out.
3. Some long-context inference jobs already take weeks on conventional chips per Fractile's own pitch disclosure—a failure mode enterprise AI teams are hitting right now, creating immediate budget urgency for software fixes.
4. Investor CapEx is explicitly rotating from training to inference infrastructure, as evidenced by Fractile's post-training hardware positioning—software scheduling tools are the first layer ML teams will budget for while specialized chips are still pre-commercial.

Catalyst. Fractile's $220M raise in May 2026 crystallizes investor and operator consensus that inference throughput is the new frontier; ML teams are actively seeking software-layer fixes while next-generation hardware is still 12–18 months from general availability.

The product is a cloud-hosted inference scheduling control plane deployed as a thin proxy in front of any OpenAI-compatible endpoint. When a request arrives, the scheduler profiles the job by context length, priority tier, and deadline, then dynamically batches it with compatible in-flight requests and selects the cheapest endpoint that meets the latency SLA. Teams instrument their code with a single SDK call and immediately see per-job cost breakdowns, queue depth, and throughput metrics in a dashboard. The scheduler's batching engine achieves GPU utilization rates above 80% compared to the industry-typical 30–40% seen in over-provisioned setups. For jobs with relaxed deadlines—overnight batch analysis, large-corpus ingestion—the scheduler automatically downgrades to preemptible compute and reduces effective per-token cost by 60–70%.

What's different. Existing inference serving frameworks (vLLM, TGI, LiteLLM) optimize for median online latency and treat every request as equally urgent; this product is the first to apply throughput-aware batch scheduling with SLA-tiered prioritization designed specifically for long-context jobs. Unlike cloud providers' native batch APIs, the scheduler is model-agnostic and works across any OpenAI-compatible endpoint including self-hosted models on private infrastructure. The percentage-of-savings pricing model aligns incentives with engineering teams who must justify new tooling to a CFO without upfront commitment risk.

Jobs to be done

flowchart LR
  Client[ML Engineer / Agent App] --> Proxy[Scheduling Proxy SDK]
  Proxy --> Profiler[Job Profiler\nContext Length and Priority]
  Profiler --> Batcher[Dynamic Batcher\nSLA-Tier Queue]
  Batcher --> Router[Endpoint Router]
  Router --> GPU1[Current GPUs\nH100 / A100]
  Router --> GPU2[Preemptible Compute\nSpot / Batch]
  GPU1 --> Dashboard[FinOps Dashboard\nCost Attribution]
  GPU2 --> Dashboard

Executive takeaways

• Long-context inference is now a real operations problem, but incumbents mostly offer point optimizations—batch APIs, prompt caching, or provider-specific serving stacks—rather than a neutral scheduler that classifies jobs by context length, SLA, and cheapest viable endpoint [18][2][4][40].
• The best initial buyer is still the ML infra lead already paying for production inference, because enterprise adoption is moving from pilots to scaled deployments while organizations still struggle to fully scale experiments and governance [11][13].
• Competition is intense but fragmented: hyperscalers sell throughput reservations, managed platforms sell faster inference on their own clouds, and open-source stacks expose primitives; the whitespace is cross-provider cost attribution plus SLA-aware routing for long-context jobs [1][30][9][16][28].
• The technology thesis is credible because the literature already shows meaningful gains from continuous batching, prefix caching, and prefill/decode disaggregation; a startup can commercialize orchestration before custom chips like Fractile or Cerebras become broadly standard [37][38][39][12].
• Regulatory friction is manageable but non-trivial: buyers routing sensitive documents across multiple endpoints will need auditable governance, regional controls, and data-protection posture rather than pure latency wins [31][14][25][29].

Market definition

Control-plane software for long-context LLM inference that sits above model APIs, self-hosted runtimes, and managed inference clouds. It profiles job shape, queues compatible requests, applies batch/caching-aware routing, and attributes spend by workload. The market excludes general model hosting and broad observability unless they explicitly optimize long-context batch economics and cross-provider scheduling [2][30][35][41].

Customer and buyer

Primary users are ML infrastructure engineers and platform teams operating production document-analysis, code-analysis, or agentic workflows with large prompts and variable deadlines. The economic buyer is usually a Head of ML Infrastructure, VP Engineering, or central platform owner responsible for throughput, latency, and cloud GPU spend [11][13][9].

Buying triggers

• Monthly inference spend becomes material enough that batch, cache, and reservation discounts are worth operationalizing systematically rather than manually. [1][29][4][16][36]
• Long-context jobs or overnight document runs cause visible queueing, slow decode speed, or unstable tail latency on current vLLM/TGI-based stacks. [19][9][5]
• Leadership sees inference as the next infrastructure bottleneck and budgets for optimization before next-generation hardware is broadly deployed. [18][22][7][26]

Willingness to pay

The market already accepts explicit price discrimination for optimization features. Anthropic charges cache reads at a fraction of base input cost, AWS and Azure both advertise 50% batch discounts, Fireworks discounts cached and batch tokens, and Together markets batch inference at 50% lower cost. That creates a credible budget envelope for a scheduler that proves savings beyond what one provider can offer alone [4][1][29][16][36]. [4][1][29][16][36]

Category dynamics

Growth signal 19.2% CAGR

Validation signals

• Fractile’s $220M financing explicitly argues inference latency is the next bottleneck after training.
• Baseten and Groq both raised large rounds around inference infrastructure, indicating continued capital formation in the category.
• Cloud vendors and managed platforms now advertise batch and cache discounts directly, showing buyers already optimize around workload class rather than raw model quality alone.
• Community and operator materials still surface long-context performance pain and continuous-batching gains, suggesting the operational problem is not abstract.

Regulatory & technical constraints

• Batch APIs often exclude interactive features like tool calling or structured output, which limits where a scheduler can defer work without changing application behavior.
• Prompt caching gains depend on stable prefixes and sometimes session affinity or replica locality, so software scheduling must understand workload shape, not just price tables.
• Throughput reservations and regional deployment modes are provider-specific, complicating a neutral control plane that must normalize quotas, latencies, and residency options.
• Sensitive enterprise document flows need auditable governance and data-protection controls when requests may cross regions or model providers.

The field breaks into four groups. Hyperscalers (AWS, Azure, Vertex) sell batch, caching, and reserved throughput inside one cloud [1][30][21]. Managed inference platforms like Baseten, Fireworks, and Together optimize latency and cost, but within their own infrastructure and commercial incentives [9][16][36]. Open-source orchestration layers like Anyscale, vLLM, LiteLLM, BentoML, and KServe expose primitives yet still push integration and policy work onto the buyer [6][41][28][10][27]. Specialized hardware vendors like Groq, Cerebras, SambaNova, and Fractile raise the performance ceiling but also increase endpoint heterogeneity, which strengthens the case for a neutral routing layer [22][12][26][18].

Why incumbents do not win by default

• Cloud platforms. AWS, Azure, and Vertex increasingly expose batch and reserved-throughput controls, but those controls are cloud-native and do not solve cross-provider scheduling or hardware-neutral FinOps by default.
• Managed inference platforms. Baseten, Fireworks, and Together already monetize speed and cost-efficiency, but they win by keeping workloads on their own stack rather than brokering the cheapest endpoint across many stacks.
• Open source and in-house. vLLM, LiteLLM, Ray Serve, BentoML, and KServe give sophisticated teams the raw pieces, yet a buyer still has to design routing policy, savings attribution, and reliability operations internally.
• Specialized inference hardware. Groq, Cerebras, SambaNova, and Fractile can improve tokens-per-second, but they do not eliminate the need to classify workloads, arbitrate among endpoints, or explain spend by job type.

Long Context Inference Scheduler is a control plane for AI teams whose document-analysis and code-analysis workloads now hit long-context inference cost and queueing limits before they can justify new hardware. The first customer is a 10-80 engineer Series A-C AI-native startup already spending roughly $50k-$120k per month on inference and seeing either a budget shock or an SLA breach on 64k-200k token jobs. The product wedges in as a drop-in SDK and proxy that classifies requests by context length, priority, and deadline, then batches and routes deferrable jobs to the cheapest endpoint that still meets policy. Research supports a focused starting market rather than a broad inference platform story, with an estimated $600.0M TAM, $72.0M SAM, and $5.4M year-3 SOM for the initial long-context control-plane category. The buyer case is credible because clouds, model vendors, and managed inference platforms already train customers to pay for batching, caching, and throughput optimization, but most tools remain provider-specific or require substantial internal platform work. The deliberate strategy is to win one narrow workflow first, prove incremental savings after native provider optimizations are enabled, and only then expand into broader inference FinOps and multi-hardware brokerage. The main reasons to stay cautious are intense substitution risk from hyperscalers and open source, governance friction for sensitive document routing, and limited direct customer evidence beyond the research corpus and one headline catalyst around Fractile's financing. This should therefore be treated as a strong operating thesis with clear falsification tests, not yet as a de-risked infrastructure investment.

Problem

• AI-native teams running 64k-200k token document or code-analysis jobs on vLLM, TGI, or cloud model APIs end up overprovisioning GPUs, absorbing queue spikes, and losing cost visibility because current serving stacks treat long-context workloads like low-latency requests.
• Provider-native batch, caching, and provisioned-throughput controls help inside one stack, but buyers still lack a neutral system that decides which jobs are deferrable, which endpoint is cheapest under policy, and how spend maps back to workload-level business value.

Solution

• Deploy a thin proxy plus Python SDK in front of OpenAI-compatible endpoints so each request is profiled by context length, SLA tier, and deadline before execution.
• Use shadow mode first, then controlled routing to batch compatible long-context jobs, steer overnight or low-priority work toward cheaper capacity, and expose job-level cost attribution that proves savings beyond native vendor features.

Why we win

• The beachhead is a narrow but urgent workflow where the buyer already owns a visible monthly bill and can measure savings, queue reduction, and deployment speed faster than in a broad platform sale.
• A neutral scheduler can sit above AWS, Azure, self-hosted vLLM stacks, and emerging hardware instead of forcing workload migration onto one vendor's commercial stack.
• Proprietary telemetry on context shape, deadline slack, cache reuse, and realized savings can compound into a routing and policy moat that open-source primitives do not accumulate by default.

Milestones

flowchart LR
  Wedge[Long-context document and code workloads] --> MVP[Shadow-mode scheduler and savings dashboard]
  MVP --> Proof[Paid pilots with verified savings and SLA protection]
  Proof --> Expansion[Inference FinOps and multi-hardware brokerage]

Founding team

Experiment roadmap

Risk assessment

What must be true

• At least one quarter of qualified target accounts must have enough deferrable long-context work to save money through batching and routing without breaking latency commitments.
• Buyers must still see at least 15% incremental savings after they enable native batch, cached-input, or throughput-reservation features on their current provider.
• A shadow-mode deployment must reach first benchmark output within one day and first controlled routing within two weeks in the majority of pilots.
• The economic buyer must fund the purchase from infrastructure or platform budget rather than an experimental AI budget in at least the first several paid pilots.
• Early customers must show willingness to expand into broader FinOps or multi-provider brokerage once the first workload is proven, or the wedge will cap out as a narrow utility.

Open diligence questions

• What share of the target customer's long-context volume is truly deadline-flexible rather than user-facing and latency-critical?
• How much savings remains after the customer fully enables native batch, prompt caching, and reserved-throughput options on its existing provider?
• Why would a buyer install a neutral proxy instead of moving more traffic to Baseten, Fireworks, Together, or cloud-native controls?
• Which governance objections appear first in real deals: hot-path reliability, provider approval, or data residency?
• Does one workload land at $30k-$80k annual spend without requiring services-heavy custom deployment?

Model sanity

• Revenue engine. The base case reaches 67 active paid deployments by Q4Y3 at roughly $60K blended annual revenue per deployment, with most of the lift coming from repeat pilot conversion after Year 1.
• Must go right. The product must keep proving savings beyond native batch and caching features so net adds can sustain 4-6 per quarter in Y2 and 8-12 per quarter in Y3.
• Model breaks if. If pricing slips toward $52K and gross margin stalls near 68%, downside cash falls below zero before the next round case is proven.
• Next-round proof. The next financing is justified if the company exits Y2 with 25 active deployments, ~68% gross margin, and multiple production references that shorten the sales cycle.

Scenarios

Sensitivity

flowchart LR
  Leads --> PaidPilots
  PaidPilots --> ActiveDeployments
  ActiveDeployments --> PlatformRevenue
  PlatformRevenue --> GrossProfit
  GrossProfit --> Cash

Flags: The model still burns more than $1.0M in Y1 and $1.17M in Y2, so execution risk is concentrated in proving faster pilot conversion before cash falls under $1M. · Revenue per exit FTE only reaches the low end of SaaS benchmarks because solutions work and deployment support remain meaningful through Y3. · Q4Y3 is the first positive EBITDA quarter, so a one-to-two quarter slip in conversion timing would likely pull fundraising forward. · Customer counts are modeled as net of churn and small expansions; if real gross churn exceeds the 2.0% heuristic without offsetting upsell, Y3 revenue will miss materially.

• Cloud provider counter-move. AWS, Azure, and GCP could bundle native long-context batch scheduling into their managed inference APIs, commoditizing the core wedge within 12–18 months. Mitigation: Build multi-cloud and self-hosted model compatibility before any single provider ships competing batch APIs and shift moat to proprietary job-shape datasets and FinOps integrations cloud providers cannot easily replicate.
• Open-source substitution. The vLLM or LiteLLM communities could add throughput-aware batch scheduling as an open-source feature, undermining the commercial value proposition. Mitigation: Contribute core scheduling primitives to open source as a distribution channel while keeping the managed control plane, cost attribution engine, and multi-endpoint routing proprietary.
• Single-source evidence gap. Fractile's hardware performance claims (40 to 1,200 tokens/sec) are unverified by independent benchmarks; if the hardware thesis proves overblown, urgency signals for inference scheduling may weaken. Mitigation: Validate customer pain directly through ML infra interviews before Series A; the scheduling product's value case is independent of Fractile's hardware shipping on schedule.