Plausible Code Is Not an Integration: Why We Built Vise

We watched this happen over and over in our own runs: a pile of session transcripts from agents integrating our SDK across five platforms. An ungrounded agent doesn't know your SDK. It knows what SDKs like yours tend to look like. So it guesses, hits an error, guesses a fix, hits a different error, guesses again. Sometimes the loop lands on working code. Just as often it lands on something worse: code that compiles, runs, and is quietly wrong.

We call that plausible code, and we treat it as the enemy. It's more dangerous than a loud failure because it survives review. The SDK version is one major release stale. The handler listens for an event that got renamed two versions ago. The ID was copied from a docs example and returns nothing against your environment. A clean empty state hides a silently swallowed error. Plausible code doesn't look like a bug. It looks like a finished feature that happens to have no data yet.

Your compiler is not a witness

The obvious rebuttal: "that's what type checking is for." But a compiler only validates that your code agrees with itself and with the type declarations it can see. It has no opinion about whether an ID identifies anything real, whether a runtime response matches what the agent assumed, or whether a subscription will ever emit. Build success is a claim about syntax and types. An integration is a claim about behavior against a live service. No amount of static analysis converts one into the other.

Everything we catalogued lives in that gap: tenant-specific IDs, data shapes that quietly drift, secrets hiding in fallback values, and, most subtly, features that were scaffolded but never actually exercised. A green build says the code could work. It's not evidence that it does. Linters check form. We needed something that demands proof.

Why we called it Vise

The name isn't clever, and that's the point. A vise is the least glamorous tool in a workshop, and the one every other tool leans on. It doesn't cut, doesn't shape, doesn't do the work itself. It holds the workpiece so force lands precisely, so the saw doesn't wander and the drill doesn't walk. Nobody admires a vise. Everybody trusts one.

That's the job description we arrived at after reviewing the transcripts. The models are good; craftsmanship isn't the problem. The problem is precision work on a workpiece that keeps sliding around the bench: no fixed truth about the SDK, no fixed definition of done, no fixed record of what was verified. Vise clamps the work.

In practice, social.plus Vise governs the agent's entire workflow, from before the first line of code to the run through the last. It turns a vague ask into a concrete build plan and grounds the agent in current SDK facts and versions before generation starts, which kills the guess-and-retry loop at the root. This builds on our MCP server, which made our documentation agent-readable in the first place. Think of MCP as the reference layer and Vise as the workflow layer that sits on top of it.

From there, social.plus Vise runs 400+ platform-specific checks across all five platforms. Did the feature actually get exercised? Are there stale assumptions? Are there hardcoded values or leaked secrets? When the run ends, Vise writes everything down. Each run produces a versioned, reviewer-ready verification record that can gate a CI pipeline.

That last piece is the one we'd defend most stubbornly. The industry's current answer to "how do I review agent output?" is to read the diff harder, or, worse, to read the prompt transcript. A transcript is a record of what was said. A verification record is a record of what was proven: machine-checkable, versioned, diffable. If AI-assisted development is going to scale past demos, the unit of trust has to change, from "the agent seemed confident" to an artifact a human can audit in five minutes. A vise holds the work steady. It also holds it still enough to inspect.

What the benchmark told us (with the error bars on)

Because we'd spent months arguing that plausibility isn't proof, we had to hold Vise to the same rule. So we ran real end-to-end integrations: agents building actual social.plus features across all five platforms, with and without Vise governing the run.

The headline: with Claude Sonnet 5, the composite score went from 53.1 without Vise to 82.2 with it, a +29.1 lift. The ungrounded runs looked exactly like our original transcripts: confident scaffolding, stale assumptions, features that demoed well and verified badly. The Vise runs converged faster, and more importantly, they converged on integrations where the capability had demonstrably been exercised, not just scaffolded.

Now the caveats, stated plainly. A post about the difference between plausible and proven is a strange place to oversell a benchmark. This is n=1 per arm, one run per configuration, so treat the delta as directional, not a distribution. Capability activation is roughly a quarter of the composite score, so part of the lift reflects "the feature provably ran." That's the dimension we care most about, but you should know how it's weighted. And the Codex comparison ran on an earlier version of the harness, so cross-model numbers aren't apples-to-apples. Directional, not audited. Still, the result matched what every transcript had already told us: grounding plus systematic verification beats raw model capability applied blind.

Demand proof from your tools

Here's the flag we'll plant. As AI-native development becomes the default, the interesting question about any piece of infrastructure is shifting from "does it have an SDK?" to "can an agent integrate it provably?" Documentation was built for human readers. MCP made it readable to agents. But that only fixes the input side. The output side still defaults to vibes in most stacks. Did the integration actually work? Can someone verify that without re-deriving the whole thing? Most stacks can't answer.

Every platform vendor is going to run into the same uncomfortable truth: their real integration surface is no longer a human with a browser open to documentation. It's an agent in a loop, and that agent will produce exactly as much correctness as the workflow around it demands. Plausible code is what you get by default. Proof is what you get by design.

So when you evaluate infrastructure, ours included, ask the nerdy question: where is the artifact? Not the demo, not the transcript, not the green build, but the thing a skeptical reviewer can actually check. If a vendor can't answer that, you're not evaluating an integration. You're evaluating a very fluent guess.

Vise is available now and works with the AI coding tools your team already uses: Claude Code, Cursor, VS Code, GitHub Copilot, and OpenAI Codex.