Buckets¶

A bucket is where Aunsight Data Services stores unstructured data — arbitrary files and folders, like an object store or filesystem. Where a warehouse holds structured, queryable tables, a bucket holds raw files: documents, exports, and similar content that later pipeline steps can process.

All bucket operations are currently performed through the AU3 Toolbelt CLI (the au3 command); there is no UI for buckets yet. The concepts below apply to any reader; the command reference further down is aimed at developers and support engineers.

Concepts¶

What a Local Bucket is¶

A Local Bucket (LBKT) is a managed file-storage container that lives inside an ADS compute environment, backed by S3-compatible storage. It's the piece that lets ADS take in raw files — documents, exports, and similar — and keep them, unaltered, inside an installation so later pipeline steps can process them.

Bucket types¶

Every transfer has a source and a destination, and each is one of three types:

Local Bucket (local_bucket_sync) — a bucket inside the compute environment / ADS, referenced by bucket ID.
Remote Bucket (remote_bucket_sync) — a bucket in an external system reached via a Sync Agent (for example, a client's file store).
User Bucket (user_bucket_sync) — files on a user's local machine (or a pipeline module's working folder), referenced by a local directory path. This is how files are pushed into ADS or pulled back out.

The transfer directions supported today are user → local (upload from a user's machine), local → user (download to a user's machine), remote → local, and local → local. The local → remote and user → remote directions are not currently supported.

Bucket Locations¶

A Bucket Location is a saved, named reference to a specific path inside a bucket — optionally with transfer directives baked in. Instead of repeating the same bucket ID, path, and settings in every command, you define the location once and reference it by ID. Think of it as a bookmark with defaults.

Manifests¶¶

S3 natively tracks only modification time, so ADS keeps its own metadata in hidden .ads-lbkt-manifest files — one per folder that contains objects. These track content checksums, logical timestamps, and which task created or last modified each file. These files are not managed by hand; ADS creates and reconciles them automatically. The toolbelt deliberately never uploads a file literally named .ads-lbkt-manifest from a user's machine, and never writes manifests into a user bucket — manifests are LBKT-internal, and ADS cannot guarantee their integrity on a user system.

How transfers behave¶

Transfers into a bucket are atomic: the new content is assembled fully before anything is committed to storage, so a failed transfer won't leave a half-written result. What happens to each individual file is governed by merge settings (detailed in the command reference below), which decide what to do when a file exists only in the source, only in the destination, or in both with matching or differing content.

AU3 Toolbelt command reference¶

The AU3 Toolbelt is invoked with the au3 command (for example, au3 bucket transfer-to-bucket --spec path/to/args.yaml).

Spec commands: Commands marked (spec) take --spec, which accepts either an inline JSON/YAML string or a path to a .json/.yaml file. Examples below are shown as JSON for readability; YAML files work equally well. Any spec command also accepts --spec-schema to print its exact schema — run this to confirm current field names and options before relying on an example.

Bucket lifecycle¶

Command	Args	Purpose
`bucket create`	flags (from schema)	Create a bucket
`bucket list` (`l`)	`--limit --skip --all --stream`	List buckets
`bucket view` (`v`)	`--id` (req), `--fields`	View one bucket
`bucket update`	`--spec`	Update a bucket
`bucket delete`	flags (from schema)	Delete a bucket
`bucket list-history`	`--id` (req), `--fields`	Bucket change history

Simple commands (create, delete, view, list, list-history) take their input as individual command-line flags — for example --id or --name. update instead takes a single --spec with the fields to change. Because create and delete build their flags from the schema, the fastest way to see what they accept is --help.

# Create a bucket (run with --help to see required fields)
au3 bucket create --name "<bucket-name>" --organization "<org>" --project "<project>"

# List buckets
au3 bucket list

# View a bucket, limited fields
au3 bucket view --id "<bucket-id>" --fields "id,name,created_at"

# Update a bucket (see fields with: au3 bucket update --spec-schema)
au3 bucket update --spec ./update.yaml

# History
au3 bucket list-history --id "<bucket-id>"

Bucket Locations¶

Command	Args	Purpose
`bucket-location create`	flags (from schema)	Create a location
`bucket-location list` (`l`)	`--limit --skip --all --stream`	List locations
`bucket-location view` (`v`)	`--id` (req), `--fields`	View one location
`bucket-location update`	`--spec`	Update a location
`bucket-location delete`	flags (from schema)	Delete a location
`bucket-location list-history`	`--id` (req), `--fields`	Location change history

au3 bucket-location create --help    # see required fields
au3 bucket-location list
au3 bucket-location view --id "<location-id>"

Listing objects — `bucket management list-objects` (spec, runs as a task)¶

au3 bucket management list-objects --spec-schema        # print the schema first
au3 bucket management list-objects --spec ./list.yaml   # JSON or YAML file, or inline

Representative list.yaml:

source:
  type: local_bucket_mgmt
  bucket: '<bucket-id>'
  paths:
    - '/<prefix>/<file-or-folder>'
  depth: 1
  walk: DFS
  include_patterns:
    - '.*\.json$'
  exclude_patterns:
    - '.*\.psv$'

Runs asynchronously as a task, waits for completion, and returns the object list as JSON. Alias: lo.

Removing objects — `bucket management remove-objects` (spec, runs as a task)¶

au3 bucket management remove-objects --spec-schema
au3 bucket management remove-objects --spec ./remove.yaml

Representative remove.yaml:

source:
  type: local_bucket_mgmt
  bucket: '<bucket-id>'
  paths:
    - '/<prefix>/<file-or-folder>'
  depth: 1
  walk: DFS
  include_patterns:
    - '.*\.json$'
  exclude_patterns:
    - '.*\.psv$'

Returns the list of removed objects (same shape as list-objects). Alias: ro. Destructive — verify the target paths first with list-objects.

Transfers — `bucket transfer-to-bucket` (spec)¶

The core operation. Alias: transfer-to-bkt. Structure:

{
  "source":      { "type": "<sync-type>", "...": "..." },
  "destination": { "type": "<sync-type>", "write_settings": { "...": "..." } }
}

Upload from a user's machine into an LBKT (user → local)¶

{
  "source": {
    "type": "user_bucket_sync",
    "path": "/<local/path/to/files>",
    "include_patterns": ["\\.pdf$", "\\.docx$"],
    "exclude_patterns": ["~\\$"],
    "depth": 5,
    "concurrency": 10
  },
  "destination": {
    "type": "local_bucket_sync",
    "bucket": "<bucket-id>",
    "path": "/<destination/prefix>",
    "write_settings": {
      "handle_merge_src_only": "copy",
      "handle_merge_dst_only": "keep",
      "handle_merge_conflict": "src",
      "handle_merge_no_conflict": "skip"
    }
  }
}

au3 bucket transfer-to-bucket --spec ./upload.yaml

Notes:

source.path may be a single file or a directory. A directory is globbed recursively (honoring depth).
include_patterns / exclude_patterns are regexes matched against the file name (not the full path).
The transfer runs as a task; the CLI streams progress and waits for completion.

Download from an LBKT to a user's machine (local → user)¶

{
  "source":      { "type": "local_bucket_sync", "bucket": "<bucket-id>", "path": "/<prefix>" },
  "destination": {
    "type": "user_bucket_sync",
    "path": "/<local/dest/dir>",
    "write_settings": {
      "handle_merge_src_only": "copy",
      "handle_merge_dst_only": "keep",
      "handle_merge_conflict": "src",
      "handle_merge_no_conflict": "skip"
    }
  }
}

au3 bucket transfer-to-bucket --spec ./download.yaml

Practical notes for this direction:

The destination directory must already exist as a real directory, or the command errors immediately.
The LBKT→UBKT task only stages the files as artifacts. The actual copy into the local folder happens client-side in the toolbelt, after the task has already reported success. So if a write-setting assertion fails during that client-side step, the au3 command fails but the task still shows "succeeded." Judge success by the command's output, not by task status alone.
Hidden files count as objects. State assertions (assert_dst_state_pre: empty / not_empty) include dotfiles when checking whether a directory is empty. A stray .DS_Store (common on macOS) or other hidden file makes a directory "not empty." If an empty assertion behaves unexpectedly, check for hidden files first (ls -a).

Remote → local, and local → local¶

Same shape, swapping the type and identifiers (remote_bucket_sync uses a remote bucket reached via a Sync Agent). local → remote is not currently supported.

Download from an RBKT to a user's machine (remote → user)¶¶

{
  "source": {
    "type": "remote_bucket_sync",
    "bucket": "<bucket-id>",
    "path": "/<prefix>",
    "include_filters": ["<exact-file-name-with-escaped-dots>\\.txt"]
  },
  "destination": {
    "type": "user_bucket_sync",
    "path": "/<local/dest/dir>",
    "write_settings": {
      "handle_merge_src_only": "copy",
      "handle_merge_dst_only": "keep",
      "handle_merge_conflict": "src",
      "handle_merge_no_conflict": "skip",
      "handle_merge_type_conflict": "fail"
    }
  }
}

au3 bucket transfer-to-bucket --spec ./download-remote.yaml

Same shape as local → user, swapping source.type to remote_bucket_sync. include_filters is optional — omit it to transfer everything under path; include it (as above) to pull just one file.

There is one important difference when transferring a single file: point source.path at the containing folder, not the file itself, and filter down with a regex. Setting source.path directly to a single file's full path on a remote_bucket_sync source reliably fails at transfer time with NotFoundError: No files found in bucket — even though bucket management list-objects shows that exact path as present.

The fix:

source:
  type: remote_bucket_sync
  bucket: '<bucket-id>'
  path: '<folder-containing-the-file>'
  include_filters:
    - '<exact-file-name-with-escaped-dots>\.txt'
destination:
  type: user_bucket_sync
  path: '<local/dest/dir>'
  write_settings:
    handle_merge_src_only: copy
    handle_merge_dst_only: keep
    handle_merge_conflict: src
    handle_merge_no_conflict: skip
    handle_merge_type_conflict: fail

Note

include_filters is a distinct field from include_patterns used elsewhere on this page — confirm the exact field name against --spec-schema for your version. Also, handle_merge_type_conflict has no working default in this direction despite the table below listing one — omitting it causes UnsupportedError: Unsupported handle_merge_type_conflict value: undefined, so set it explicitly.

`write_settings` reference¶

All merge policy lives under destination.write_settings.

State assertions (fail fast if preconditions are not met):

Setting	Choices	Default	Meaning
`assert_src_state_pre`	`empty` / `not_empty` / `undefined`	`undefined`	Assert source has 0 / 1+ objects before transfer
`assert_dst_state_pre`	`empty` / `not_empty` / `undefined`	`undefined`	Assert destination state before transfer
`assert_dst_state_post`	`empty` / `not_empty` / `undefined`	`undefined`	Assert destination state before commit

Merge handling (per file, based on where it exists and whether content matches by size + checksum):

Setting	Choices	Default	Applies when a file…
`handle_merge_src_only`	`copy` / `skip` / `fail`	`copy`	exists only in source
`handle_merge_dst_only`	`keep` / `remove` / `fail`	`keep`	exists only in destination
`handle_merge_conflict`	`src` / `dst` / `newer` / `older` / `rename` / `fail`	`src`	exists in both, different content
`handle_merge_no_conflict`	`newer` / `touch` / `skip` / `fail`	`skip`	exists in both, same content
`handle_merge_type_conflict`	`src` / `dst` / `obj` / `prefix` / `newer` / `fail`	`fail`	path is a file on one side, folder on the other

Common recipes:

Mirror source exactly (delete extras in destination): dst_only: remove, conflict: src.
Additive only, never overwrite: conflict: dst (or skip via no-conflict), dst_only: keep.
Safe first load into an empty prefix: assert_dst_state_pre: empty.

Not yet supported¶

local → remote and user → remote transfers, bucket↔warehouse changes, two-way sync, versioned buckets, and advanced merge modes are not available yet.

Buckets¶

Concepts¶

What a Local Bucket is¶

Bucket types¶

Bucket Locations¶

Manifests¶¶

How transfers behave¶

AU3 Toolbelt command reference¶

Bucket lifecycle¶

Bucket Locations¶

Listing objects — bucket management list-objects (spec, runs as a task)¶

Removing objects — bucket management remove-objects (spec, runs as a task)¶

Transfers — bucket transfer-to-bucket (spec)¶

Upload from a user's machine into an LBKT (user → local)¶

Download from an LBKT to a user's machine (local → user)¶

Remote → local, and local → local¶

Download from an RBKT to a user's machine (remote → user)¶¶

write_settings reference¶

Not yet supported¶

Listing objects — `bucket management list-objects` (spec, runs as a task)¶

Removing objects — `bucket management remove-objects` (spec, runs as a task)¶

Transfers — `bucket transfer-to-bucket` (spec)¶

`write_settings` reference¶