Troubleshooting¶

This guide is for support engineers investigating and recovering from pipeline failures in Aunsight Data Services. It covers recovering and cancelling workgraph submissions, scoping a recovery, and resolving blocked submissions, using both the Console UI and the Toolbelt CLI.

Toolbelt

Many procedures here use the Toolbelt CLI. The examples use au3 as an alias for the toolbelt executable. Before running them, make sure your context is set to the same project as the work you're recovering. See the Reference section for installing and configuring the Toolbelt.

Understanding submissions and runs¶

When a workgraph is submitted for execution, it creates a submission — the logical unit of work. Because a submission may not succeed on the first attempt, it can have multiple runs: an initial run, plus any recovery runs. Each run executes the nodes (tasks) in the workgraph.

When a run fails, the submission enters the AWAITING_RECOVERY state. From there you can recover it (re-run the work) or cancel it. The submission ID (a siwgsub-... value) is the primary handle for all recovery operations.

Recovering a failed submission¶

From the Console¶

Find any task belonging to the failed submission and open its action menu, then choose Recover Submission. This navigates you to the recovery run in the UI, where you can watch it progress. The Submission Trace tab on a task shows the full submission → run → node hierarchy and also exposes recover and cancel actions.

From the Toolbelt¶

First, get the submission ID from a failed task:

au3 task view --id [task_id] --fields name,id,state,error,trace

If trace.submission is present, use that siwgsub-... ID for recovery.

There are three recovery modes:

Remaining recovery (default) — re-runs only the nodes that have not yet succeeded in prior attempts. This is the most common path and should be your first choice:

au3 sln inst wg sub recover --id [submission_id]

Partial recovery — re-runs only specific remaining (non-succeeded) nodes matching a filter. Use this to skip certain blocked nodes and recover only a subset of the remaining work:

au3 sln inst wg sub recover --id [submission_id] --node-filter '{"field": "id", "op": "in", "val": ["node-id-1", "node-id-2"]}'

Get node IDs from the node summary (see below).

Full recovery — re-runs all nodes, including ones that already succeeded. Use this when you need a clean slate:

au3 sln inst wg sub recover --id [submission_id] --mode full

Cancelling a submission¶

If you do not intend to recover a submission, cancel it rather than leaving it in AWAITING_RECOVERY.

From the Console, use the Cancel Submission action in a task's action menu (available while the submission is in a non-final state such as PENDING, RUNNING, AWAITING_RECOVERY, or RECOVERING).

From the Toolbelt:

au3 sln inst wg sub cancel --id [submission_id]

Inspecting a submission¶

These commands help you understand a submission's state and scope a recovery.

View the submission:

au3 sln inst wg sub view --id [submission_id]

View with node summary — adds completed_node_ids and remaining_runnable_node_ids so you can see exactly which nodes have finished and which are still expected to run. This is especially useful for scoping a targeted recovery:

au3 sln inst wg sub view --id [submission_id] --node-summary

View the submission trace — shows the full submission → run → node → task hierarchy:

au3 sln inst wg sub trace view --id [submission_id]

Resolving a blocked submission¶

A new submission can be blocked by an earlier one that is still open. The error message identifies the blocking submission by its siwgsub-... ID.

If wg sub list returns nothing: by default the list only shows submissions from the last 24 hours. If the blocking submission was created earlier, look back further with --since:

au3 sln inst wg sub list --since 2026-05-25
au3 sln inst wg sub list --since 2026-05-25T00:00:00Z

Inspect the blocker directly using the ID from the error message, without needing to find it in a list:

au3 sln inst wg sub view --id [submission_id] --node-summary

This shows the submission state, which nodes completed, and which remain.

Unblock by cancelling the stuck submission if it cannot be recovered:

au3 sln inst wg sub cancel --id [submission_id]

Investigating failed tasks¶

To find failed tasks and their traces, list tasks and filter for the failed state:

au3 task list --fields name,state,id,trace

To view a single task in detail:

au3 task view --id [task_id]

If a task failed, look for operations_with_unhandled_errors in the output to identify which component failed:

operations_with_unhandled_errors:
  - wfop-73

You can also inspect a task's Logs tab in the Console, which is often the fastest way to see why a task failed. See Tasks for the full task inspection UI.

Tasks — finding and inspecting tasks and submission traces in the Console
Stored runs — scheduling and re-running pipeline work