Ace

An outage shouldn’t stopyour whole team.

Your engineers should be building great things, not fixing outages or running cloud ops.Ace is a team of AI agents that does that work for you.

In production with

7DIGITUNICEFWarestack

How Ace investigates an incident

AcmeInvestigations / ACE-842
Investigatingorders-platform

Orders stalled after a schema change

orders-api / request trace8.4 s
02 s4 s6 s8 s
HTTP request
8.4 s
Authentication
38 ms
Acquire connection
42 ms
SELECT orders
8.1 s
Publish outbox
waiting
Orders databaseorders table · lock queueReportingHolds relation lockMigration #842DDL waiting for lockOrders API180 reads queuedOutboxPublisher stalledFulfillmentDelivery delayedOrders databaselock queueReportingHolds relation lockMigration #842DDL waiting for lockOrders API180 reads queuedOutboxPublisher stalledFulfillmentDelivery delayed
Lock chain Downstream impact
Amazon RDSpg_locksE1

A migration waits on a lock held by reporting

Infrastructure agent
Datadogrequest traceE2

Order delivery waits behind the same session

Telemetry agent
GitHubdeployment 842E3

The migration runner retries without a timeout

Code agent
GitHubdb/migrations/runner.tsPR #843
24await db.transaction(async (tx) => {
25+ await tx.sql`SET LOCAL lock_timeout = '2s'`;
26 await applyMigration(tx, migration);
28− retry: untilSuccess()
28+ retry: boundedBackoff({ attempts: 3 })
3 checks passedApproval required
Orders API · p99184 ms
14:03Recovery applied14:12
Lock queue clearedOutbox caught up
Saved to system memory
Agents investigating

A whole team. On the same case.

Infrastructure agent

Checks the resources behind the failure and the services that depend on them.

Telemetry agent

Follows the failing request and shares the trace that sends the others to the right service.

Code agent

Tests recent changes against the findings, then prepares the patch and a regression test.

Recovery agent

Rechecks the original failure and downstream impact, and reopens the work if recovery fails.

Know what depends on
the failing service.

Ace maps the resources and dependencies in your infrastructure. Its agents use that graph to follow a failure across services, even when the alert only names one.

System map
Opening system map

Review the fix
where you ship code.

Ace opens a pull request with the patch, the evidence behind it and a test for the failure. Your engineers review and merge in GitHub.

GitHubacme / orders-platform

Bound migration locks and retries #843

Openace wants to merge 2 commits into main

db/migrations/runner.ts+8 −2
24await db.transaction(async (tx) => {
25+ await tx.sql`SET LOCAL lock_timeout = '2s'`;
26 await applyMigration(tx, migration);
27});
28− retry: untilSuccess()
28+ retry: boundedBackoff({ attempts: 3 })
3 checks passed
  • Fix pull request

    A minimal code or infrastructure-as-code repair, tested in an isolated sandbox and opened for review.

  • Revert pull request

    An exact reversal of the change behind the incident, ready to merge.

  • Rollback

    Redeploys a known-healthy revision through a workflow your administrator has confirmed.

  • Kubernetes changes

    Restart, scale or patch workloads in the namespaces you grant, under the same approvals.

Memory that stays
with your system.

Ace keeps the failure patterns it has seen, the hypotheses it ruled out and the fixes that held, along with your engineers’ feedback. The next investigation starts from all of it.

  • Patterns

    A schema migration waits behind an open transaction

    Recognized the next time the same shape appears in your system.

  • Ruled out

    Database CPU was 19%. Capacity was not the cause.

    Dead ends and false alarms stay closed, with the evidence that closed them.

  • Fixes that held

    Lock timeout and bounded retries, recovery verified

    Reused for similar changes. Fixes that failed are remembered too.

Delegate more
as Ace earns it.

Start with investigations. Approve each change while you review the results. Turn on automatic actions only where Ace has already proven itself, within limits your team sets.

How earned autonomy works
Autonomy policyorders-platform
  1. ShadowRecords what Ace would have done
  2. ApproveEvery plan approved in Slack or the app
  3. AutomaticWithin a daily cap, with time to cancel

Required before automatic actions

Verified recoveries in this scope
5 of 5
Worsened outcomes
0
Daily cap
3 actions
Cancellation window in Slack
60 s

Connects to the tools your engineers already use

SOC 2 Type II compliant

Give your engineers
their time back.

Watch Ace investigate an incident, prepare the fix
and verify that service recovered.