Myelin · Partner Ingestion API
Developer documentation
Push R&D data deliveries into Myelin from your pipeline — validated against the client’s quality rules before a byte moves, resumable at any scale, reviewed and audited exactly like a portal delivery.
Quick start
Your bridge owner creates an API key in Bridge → API. From key to first submitted delivery is a few minutes:
export MYELIN_API_KEY=myl_live_… npx @myelinbridge/cli ping # verify auth, list your projects npx @myelinbridge/cli check ./run_042 --dataset onco1-wes # preflight — nothing uploaded npx @myelinbridge/cli push ./run_042 --dataset onco1-wes --submit
Prefer raw HTTP? Everything the CLI does is plain REST + presigned S3 uploads — see the reference and the OpenAPI 3.1 spec.
Authentication
Every request carries Authorization: Bearer myl_live_…. A key is scoped to one bridge and a chosen set of projects; behind it stands a named service identity that appears in member lists and in the tamper-evident audit trail. Keys are shown once, rotate in one click, and the client can disable a bridge’s API access instantly. Errors are RFC 9457 application/problem+json with a stable code; the rate limit is 120 requests/min per key (uploads don’t count — they go direct to storage).
Delivery flow
POST /v1/datasets/{id}/preflight # 0. validate your file list — no upload, exit early on blockers
POST /v1/datasets/{id}/batches # 1. create or resume THE draft (one per dataset)
POST /v1/batches/{id}/files # 2. declare files (≤500/call) → storage paths + part size
POST /v1/files/{id}/upload-parts # 3. get presigned S3 URLs for the parts you still need
PUT {presigned url} # push each part direct to EU staging — resumable, no credential stored
POST /v1/files/{id}/confirm # 4. finalize (upload_id + part ETags); server verifies the declared size
POST /v1/batches/{id}/submit # 5. runs the client's auto-checks, enters review
GET /v1/batches/{id}/findings # 6. on changes_requested: failed files + hints, machine-readable
# fix → re-declare (replacement) → re-upload → re-submitDeclares are idempotent (same path + size = same file, untouched) — a crashed pipeline re-runs the same command and only the delta uploads. A batch that is submitted or in review is locked (409 delivery_locked); recall it or wait.
Webhooks
Register an HTTPS endpoint (portal, or POST /v1/webhook-endpoints) and Myelin pushes signed events — batch.review_started, batch.changes_requested, batch.validated, batch.transferred, … Verify the signature:
// Node — header: Myelin-Signature: t=<unix>,v1=<hex>
import { createHmac } from 'node:crypto'
function verify(secret, header, rawBody, toleranceSec = 300) {
const { t, v1 } = Object.fromEntries(header.split(',').map((p) => p.split('=')))
if (Math.abs(Date.now() / 1000 - Number(t)) > toleranceSec) return false
const expected = createHmac('sha256', secret).update(`${t}.${rawBody}`).digest('hex')
return expected === v1
}# Python
import hmac, hashlib, time
def verify(secret: str, header: str, raw_body: bytes, tolerance: int = 300) -> bool:
parts = dict(p.split("=") for p in header.split(","))
if abs(time.time() - int(parts["t"])) > tolerance:
return False
signed = f"{parts['t']}.".encode() + raw_body
expected = hmac.new(secret.encode(), signed, hashlib.sha256).hexdigest()
return hmac.compare_digest(expected, parts["v1"])Deliveries retry at +5 min / +30 min / +24 h; use Myelin-Event-Id to deduplicate. No HTTPS receiver? Poll GET /v1/events?cursor=… (pass next_cursor back verbatim).
Endpoint reference
Rendered from the same spec served at /api/v1/openapi.json (1.0.0-phase4).
/v1/pingIs my key alive? Returns key, bridge, and scoped project ids.
/v1/meKey identity, bridge (lifecycle, connection mode), scoped projects.
/v1/projectsProjects in the key's scope.
/v1/projects/{projectId}/datasetsDatasets of a scoped project: id, name, slug, description, lifecycle_status, sample_depth, current quality-check version, open-draft pointer.
/v1/datasets/{datasetId}Dataset detail: the list fields plus project_id and quality_checks_published_at. The object is the delivery contract only — what a delivery must satisfy comes from the quality-check version, never from a descriptive label (`data_type`, `data_domain` and `metadata_mode` are all gone — see the Dataset schema).
/v1/datasets/{datasetId}/sample-depthDeclare at which folder depth a sample sits (PRD §4.3) — the only dataset write a partner has. Body: { sample_depth: 0-5 }. You own your output structure, so you declare the depth your deliveries use; everything else about a dataset is created and described by the client in the portal. Idempotent: re-declaring the current value succeeds with changed:false, so a pipeline can assert its depth on every run. Locks once any batch in the dataset leaves draft — after that the value is frozen so sample-scoped quality verdicts stay comparable across deliveries. Set it BEFORE your first submit.
/v1/datasets/{datasetId}/quality-checksThe delivery contract: the published quality-check snapshot preflight/submit will evaluate, each check carrying a plain-language assertion, its quality dimension, whether it can be checked BEFORE upload, and a pass/fail example.
/v1/batchesList scoped batches. Filter by dataset_id/status; cursor pagination.
/v1/batches/{batchId}Batch detail: status, whose-move, round number, files with upload state, submit-readiness.
/v1/batches/{batchId}Delete a draft batch (files, rows, and staging blobs). Draft only.
/v1/batches/{batchId}/findingsQuality findings; in changes_requested also per-file reviewer verdicts + request-changes comment.
/v1/batches/{batchId}/eventsBatch timeline (submitted, review_started, changes_requested, …).
/v1/datasets/{datasetId}/batchesCreate OR resume the dataset's draft batch (one draft per dataset).
/v1/batches/{batchId}/filesDeclare files (single {path, size_bytes, checksum?} or {files: [...]} ≤500). Mints storage paths, creates 'queued' rows, returns the S3-multipart upload descriptor (protocol, part_size, max_parts_per_sign). Get presigned part URLs per file from /files/{id}/upload-parts. Re-declaring identical files is idempotent; a different size at an existing path needs replace=true.
/v1/files/{fileId}/upload-partsMint presigned S3 multipart UploadPart URLs for a declared file (SEC-TOKEN-01: no bearer token — each URL authorizes exactly one part of one object for a few minutes). Body: { part_numbers: number[] } (1-based, ≤ max_parts_per_sign). Returns the upload_id, the part_size, which parts already landed (resume), and a presigned url per still-missing requested part. PUT each part to its url; the ETag response header is what confirm needs. Gated to draft/changes_requested batches.
/v1/files/{fileId}/confirmFinalize the upload: completes the S3 multipart upload from its parts, then verifies the staging object exists with the declared size, then flips the row to uploaded (audit batch.file_uploaded / file_reuploaded). Body: { upload_id, parts: [{part_number, etag}], checksum?, checksum_algorithm? }.
/v1/files/{fileId}Remove a file (row + staging blob) from a draft / changes_requested batch.
/v1/batches/{batchId}/submitSubmit (or resubmit from changes_requested): round snapshot, frozen fields, synchronous auto-checks, status flip, reviewer notification. Blocks while declared files are still un-uploaded.
/v1/batches/{batchId}/recallRecall a submitted batch (only before review starts). Restores draft or changes_requested and invalidates the submit snapshots.
/v1/datasets/{datasetId}/preflightEvaluate a file list ({files: [{path, size_bytes}]}, ≤10k) against the dataset's published quality checks BEFORE uploading. Read-only, no audit. Returns per-check verdicts with remediation hints; content-dependent checks come back in `deferred` (they run at submit), manual checks in `manual`.
/v1/bridges/{bridgeId}/sandboxPartner-side test transfer, API mode: multipart `file` ≤10 MB (hosting platform may cap request bodies earlier, ~4.5 MB on Vercel — a small sample file is the point). Satisfies the same activation checklist item as the portal sandbox (same tracking row, same audit event). Works while the bridge is still configuring.
/v1/webhook-endpointsList this bridge's webhook endpoints (url, events, active, failure_count, last delivery) plus the available event_types.
/v1/webhook-endpointsCreate an endpoint: {url (HTTPS, public host), events?} — empty events = all partner events. The HMAC signing secret is returned ONCE. Deliveries carry `Myelin-Signature: t=<unix>,v1=<hmac_sha256(secret, t + "." + body)>` (5-min replay window recommended) and `Myelin-Event-Id` for dedup. Retries at +5 min/+30 min/+24 h, terminal after 4 attempts; endpoints auto-disable after 30 consecutive terminal failures.
/v1/webhook-endpoints/{endpointId}Remove an endpoint (pending deliveries to it are dropped).
/v1/webhook-endpoints/{endpointId}/testQueue and immediately attempt a synthetic test.ping delivery (signed like a real event).
/v1/eventsPolling fallback for shops without an HTTPS receiver: ascending cursor over batch timeline events across the key's projects. Pass next_cursor back VERBATIM (it carries microsecond precision).
Pipeline recipes
Nightly cron shipping finished runs:
#!/usr/bin/env bash
set -euo pipefail
export MYELIN_API_KEY="$(cat /etc/myelin/key)"
for run in /data/outbox/*/; do
if npx @myelinbridge/cli check "$run" --dataset onco1-wes --json > /tmp/preflight.json; then
npx @myelinbridge/cli push "$run" --dataset onco1-wes --submit && mv "$run" /data/shipped/
else
echo "blocked: $run — see /tmp/preflight.json" >&2 # exit 2 = failing checks
fi
doneNextflow / Snakemake completion hook — same two commands in an onComplete handler; use --json and the exit code to gate your workflow status.
API v1 · additive changes only