Populate

Getting your own portfolio in. CSV or JSON, validated as a batch, idempotent, and with no assumption about where the data came from.

Discover the contract, do not guess it
What it accepts
The six rules
The smallest useful batch
Blank files to fill in
The datasets
Loading incrementally
Loading over HTTP
Things that are easy to get wrong
Observations and decisions are not the same thing
Evidence
Adapters
Collecting from GitHub
Other CI/CD platforms
When an import fails
1. Errors, findings and conflicts are three different things

Discover the contract, do not guess it

crane import spec --json

This prints every dataset, every column, every accepted value and every rule. It is generated from the same code that validates the import, so it cannot drift out of date and it is the authority when this page and it disagree.

There is also a human-readable form:

crane import spec

What it accepts

A directory of CSV files, one per dataset, named <dataset>.csv
A single .csv file, named after its dataset
A single .json file whose top level keys are dataset names and whose values are arrays of row objects

The JSON bundle is usually easiest to generate and the easiest to diff.

The six rules

Unknown columns are an error, not ignored, so a typo is never silent data loss.
Only the datasets you provide are touched. Loading distribution.csv alone against an existing database is normal and is not a partial overwrite.
References resolve against rows in the same batch and rows already in the database.
Everything loads as an observation needing review, never as an approved decision, unless source_kind says otherwise and names an authority.
Loading is idempotent. The same input twice changes nothing the second time.
Validation runs over the whole batch before anything is written. Either every row loads or none do.

Rule 6 is why a failed import leaves nothing behind to clean up.

The smallest useful batch

Three datasets are enough to have a portfolio that scores:

cat > data/batch.json <<'JSON'
{
  "products": [
    {"slug": "gateway", "canonical_name": "Aurora Gateway", "lifecycle": "released"}
  ],
  "components": [
    {"slug": "crypto-lib", "canonical_name": "Crypto Library"}
  ],
  "product_components": [
    {"product_slug": "gateway", "component_slug": "crypto-lib"}
  ]
}
JSON

crane import data /data/batch.json --dry-run --json   # writes nothing
crane import data /data/batch.json --json

/data/batch.json, not data/batch.json. The path is resolved inside the container. Write the file into a directory you have mounted, and refer to it by its container path.

Always dry run first. It reads, resolves and reports, then rolls back.

Blank files to fill in

crane import template /data/incoming

Writes one empty CSV per dataset with the correct header row. Fill in the ones you have, delete the ones you do not, and point import data at the directory. There is no requirement to provide all twenty.

The datasets

Dataset	What it is	Required columns
`source_systems`	A system that holds metadata about the portfolio, and how to link back to it	`key`, `display_name`
`org_units`	The accountability hierarchy: division, organization, team	`name`, `level`
`parties`	People, teams and service principals that can own things	`display_name`, `party_type`
`products`	A reviewable shipping or distributed software unit	`slug`, `canonical_name`
`components`	A building block that may ship in several products	`slug`, `canonical_name`
`build_workflows`	A build, release, packaging, signing or publication definition	`slug`, `canonical_name`
`source_assets`	A repository, package, service record, catalog entry, image or binary	`canonical_locator`, `asset_type`
`identities`	The same thing as known by another system. One row per subject per system	`subject_type`, `subject_slug`, `source_system`, `external_id`
`product_components`	Which components ship in which products	`product_slug`, `component_slug`
`component_workflows`	Which build workflows produce which components	`component_slug`, `workflow_slug`
`product_relationships`	Product to product: containment, bundling, succession	`from_product_slug`, `to_product_slug`, `relationship_type`
`workflow_assets`	What a build workflow reads and what it produces	`workflow_slug`, `asset_locator`, `relationship_type`
`workflow_dependencies`	A build that depends on another build. The fan in and fan out case	`downstream_slug`, `upstream_ref`
`responsibilities`	Who is accountable for what	`subject_type`, `subject_slug`, `party`, `role`
`aliases`	Other names a subject is known by, so search and matching find it	`subject_type`, `subject_slug`, `alias`
`scope`	Whether a subject is in scope, under one named definition	`subject_type`, `subject_slug`, `lens`, `status`
`build_topology`	Where code lives and where the build runs. Two different questions	`subject_type`, `subject_slug`
`distribution`	Where something enters and leaves the organization	`subject_type`, `subject_slug`, `direction`, `channel_type`
`claims`	Anything else known: a support end date, a disclosure contact, a licence	`subject_type`, `subject_slug`, `claim_kind`, `value`
`evidence`	Pointers to evidence. The file itself is never copied into the database	`subject_type`, `subject_slug`, `kind`, `uri`

The four subject types referenced throughout are product, component, build_workflow and source_asset.

Loading incrementally

References resolve against what is already stored, so adding one dataset later needs no other file:

echo '{"distribution": [{"subject_type": "product", "subject_slug": "gateway",
  "direction": "outbound", "channel_type": "nuget",
  "channel_locator": "pkg:nuget/Aurora.Gateway@2.4.0", "visibility": "public"}]}' \
  > data/dist.json

crane import data /data/dist.json --json

This is the normal shape of an ongoing feed: a nightly job writes whatever it knows, and CRAne merges it without disturbing anything it was not told about.

Loading over HTTP

Everything above works over the API as well as the command line, with no filesystem access and nothing fetched from a URL: the request body carries the rows.

# One request, same importer, same all-or-nothing validation.
curl -s -X POST $CRANE/api/v1/import/data \
  -H "Authorization: Bearer $CRANE_API_TOKEN" \
  -d '{"data": {"products": [...]}, "dry_run": true}'

That is the shorter path and usually the right one. Use the two-step flow when you need to be able to prove that what was loaded is what passed validation — an audited nightly feed, or an integration that has to record the digest of what it shipped:

# 1. The contract, and the fingerprint that identifies it.
curl -s $CRANE/api/v1/import/spec | jq '{fingerprint, limits}'

# 2. Validate and stage. Writes nothing. An Idempotency-Key makes a retry one import.
curl -s -X POST $CRANE/api/v1/imports \
  -H "Authorization: Bearer $CRANE_API_TOKEN" \
  -H "Idempotency-Key: nightly-$(date +%F)" \
  -d '{"data": {...}, "requested_by": "nightly-sync", "source_system": "catalogue",
       "contract_fingerprint": "<from step 1>"}'
# -> 201 with import_id, bundle_sha256, ok, errors[], warnings[], reconciliation

# 3. Apply it. No bundle here: the id names the one that was validated.
curl -s -X POST $CRANE/api/v1/imports/$IMPORT_ID/apply \
  -H "Authorization: Bearer $CRANE_API_TOKEN" \
  -d '{"requested_by": "nightly-sync"}'
# -> 200 applied, or 202 applying when the worker takes it

# 4. The receipt, whenever it is wanted. It persists.
curl -s $CRANE/api/v1/imports/$IMPORT_ID

The apply step accepts no bundle, so there is nothing that could differ from what passed validation. What to know before writing the integration:

ok: false still returns 201. The bundle was received and refused, and errors[] names every row to fix. The attempt is on record either way, which is the point of a receipt.
Only a validated import can be applied. Fix the rows and stage again.
Applying twice returns the first receipt. It does not load twice.
The fingerprint is checked at both steps. If the contract moved in between, the apply is refused rather than reinterpreted. Re-read the spec and re-stage.
A 202 means the worker has it. Poll the receipt for status: applied.

There are no scoped import permissions and no per-tenant isolation: the token is all or nothing, so anyone who can import can import anything.

Things that are easy to get wrong

Three questions, three answers. Where code lives, where the build runs, and where output goes are separate. A component built by a vendor from their own source and shipped back as binaries has code_hosting: vendor, build_location: vendor_external, and an outbound release_asset distribution row. Do not collapse them into one.

Direction matters. outbound means you make it available, which is what puts a product on the market. inbound means you consume it. A third party library has an inbound row and often no build at all, which is normal and not a gap.

Record “not distributed” explicitly. For something used only internally, write an outbound row with channel_type: not_distributed. An absent row means nobody looked, and the report has to tell those two apart.

Leave is_signed empty when unknown. Empty means not established. false means known to be unsigned. The first is a coverage gap, the second is a finding.

Attach ownership at the most specific level that is true. A product owned by a team reports under that team’s organization and division automatically. Do not repeat the hierarchy on every row.

One scope row per lens. The same product can be in scope under cra.regulatory and undecided under commercial. Both are kept. Do not reconcile them yourself.

Do not invent values. A field you do not know is left empty. Empty means nobody has looked, which is a state CRAne reports honestly. A guess becomes a fact nobody can later distinguish from a real one.

Dates are ISO, and anything else is rejected. 2026-08-01, or a full ISO timestamp whose date part is read. 08/01/2026 is refused because CRAne will not guess whether that is January or August, and because a date is compared as text once stored — an end date that is not a date decides whether a relationship is still in force by comparing something that cannot be compared. This used to be a warning that still loaded the row, which meant a feed was told it had succeeded and then read back wrong.

Observations and decisions are not the same thing

Everything you import is an observation. It is what some system says, recorded with where it came from. It is not an approval, and CRAne will not treat it as one.

A decision is a separate, recorded act with a named authority and a rationale. Scope is the common case:

crane scope set product gateway cra.regulatory in_scope \
    --by "Regulatory counsel" \
    --rationale "Made available on the Union market in the course of a commercial activity"

--by and --rationale are both required. An approved decision with no name against it is not one, and the rationale must be at least eight characters because it is the sentence somebody reads in a year. The status is one of in_scope, out_of_scope, undecided, not_applicable or disputed.

That writes the decision first, then the claim citing it. The two are linked, and the report can always answer “who said so, and when”.

Never write source_kind: approved in an imported scope row unless a human actually approved it. Inventing an authority corrupts the audit trail this exists to keep. This matters especially when an agent is doing the loading.

When two sources disagree, both statements are kept. A lower authority claim never silently overwrites a higher authority one. Conflicts surface at /api/v1/conflicts and on the /findings page, where a person settles them.

Evidence

Evidence is registered by URI. The file is never copied into the database:

{"evidence": [{
  "subject_type": "product", "subject_slug": "gateway",
  "kind": "sbom", "uri": "/data/sboms/gateway-2.4.0.spdx.json"
}]}

Register the path the worker will see. The worker resolves that URI on its own filesystem. Register /home/me/sboms/x.spdx.json and then run the worker in a container, and every validation fails with “the SBOM file does not exist”, because inside the container that path is not there. Mount the documents and register the container path. crane job show <id> prints the path it tried.

Uploading an SBOM through the interface (/import), POST /api/v1/import/sbom, or crane import sbom PATH stores the file under /data/uploads and registers it for you, which sidesteps the whole problem. All three call the same function: the document is identified by content, stored by hash, attached to a product, its packages become inventory unless you ask for evidence only, and validation plus a rescore are queued behind it.

Adapters

An adapter reads a source system directly instead of you exporting from it.

Published images carry no adapters at all. Each one describes somebody else’s system, its table names and by implication its data model, and none of that belongs in a public image. A deployment that needs one installs its own.

crane import adapter                    # what this deployment actually has
crane import adapter <name> /sources/legacy.db --dry-run
crane import adapter <name> /sources/legacy.db --report /exports/migration-report.md

Read the reconciliation report before doing anything else. Its unresolved list is the work queue for the first review pass, and its counts have to add up to the source row counts. If they do not, that is a bug worth reporting rather than working around.

Re-running an adapter import is safe: identifiers are derived from the source rows, so a second run updates in place rather than duplicating the portfolio.

Collecting from GitHub

Where the code lives on GitHub, the inventory can build itself:

crane collect github acme --dry-run     # scan one batch, report, write nothing
crane collect github acme               # queue the scan; a worker does the reading

Repositories become source assets with external identities, workflow files become build workflows with their last run, releases become outbound distribution, SBOM release assets become evidence for the ordinary validator path, and the catch-all CODEOWNERS rule becomes unreviewed maintainer assignments. Everything goes through the same importer as the files above, so the six rules apply unchanged — in particular, nothing a scanner writes is approved, and no products are created: attaching repositories to products is a decision you record afterwards, not one a scanner makes. Private repositories need CRANE_GITHUB_TOKEN in the worker’s environment.

Other CI/CD platforms

GitHub is the only platform with a built-in collector. GitLab, Azure DevOps, Jenkins and the rest are populated through the bulk contract on this page, which is not a lesser path: the GitHub collector itself writes nothing directly. It scans, builds a bundle in exactly the format documented above, and hands it to the same importer, subject to the same six rules. Anything it can express, your own script can express.

So the shape for another platform is the shape of any ongoing feed:

Read the platform’s API with whatever already has credentials for it.
Write the datasets you can fill — source_assets for the repositories, build_workflows, workflow_assets, component_workflows, distribution, evidence, identities — leaving out what you do not know. Absent is silence, not a denial.
Send it to /api/v1/imports, then apply it, as under Loading over HTTP.
Run it on a schedule. Re-sending the same bundle is a no-op, and a bundle that carries no review_status never disturbs an approval somebody recorded.

The things worth getting right are the same ones a collector has to get right: derive stable slugs so a second run updates rather than duplicates, mark everything imported rather than approved, and do not invent products — attaching repositories to products is a decision a person records, not one a scanner makes.

Native endpoints for the common platforms are tracked in issue #58. Nothing waits on them; they would remove the script, not unlock the data.

When an import fails

Exit code 4 means the input was read and did not validate. The errors are records, not prose, each naming a dataset, a row number, a column and what is wrong:

crane import data /data/batch.json --json

{
  "ok": false,
  "written": {},
  "errors": [
    {
      "dataset": "products",
      "row": 1,
      "column": "nonsense_column",
      "value": null,
      "message": "unknown column. Unknown columns are rejected so a typo is never silent data loss"
    }
  ],
  "warnings": []
}

"written": {} is the important part. Nothing was written, so there is no half-loaded state to undo. Fix the rows and retry; retrying unchanged input fails identically.

Errors, findings and conflicts are three different things

A successful import can still tell you about problems, and the distinction decides who has to do something about it:

In the result	Exit	Means
`errors[]`	`4`	The batch was rejected. Nothing was written
`findings{}`	`0`	Loaded, and something about the data is worth a person’s attention
`conflicts`	`0`	Loaded, and it disagrees with what a different source already said

A value that is not on the accepted list for its column is usually a finding, not an error: the row loads, and the finding says the value could not be interpreted. Structural problems, like an unknown column or an unresolvable reference, are errors and stop the batch. So ok: true is not the same as “nothing to look at” — check findings too.