Authoring tools in dosho

Why a Python cab format, not cult-cargo’s YAML?

cult-cargo and scabha did the real work of cataloguing this ecosystem’s tools first, and dosho leans on that prior art throughout – this is a new repository, not a rejection of the old one. dosho exists because cult-cargo’s cab YAML format – built for Stimela 2.0/scabha – carries assumptions shinobi deliberately doesn’t carry forward: dynamic_schema (a Python function imported and executed at cab-load time to compute a real tool’s schema), package-scoped _include composition, and dtype coverage gaps that silently degrade to str. Every tool in dosho is instead authored directly in Python, using shinobi’s existing typed schema machinery – no YAML dialect, and none should be added.

Two shapes: Cab and pystep

Not every tool can be a Cab. shinobi only ever executes flavour="binary" cabs – real standalone executables, argv-built and shelled out to. A tool that’s actually a Python-package function call with no standalone binary at all (CASA tasks are the running example: casatasks.listobs, casaplotms.plotms) is instead a @shinobi.pystep-decorated function, producing a StepRef rather than a Cab.

That’s architecturally distinct from, and doesn’t violate, “never import a cab package”: a pystep’s ctx.import_func("<task>", "<package>") imports inside the running container, at step-execution time, calling a real Python function the pystep author wrote directly into trusted source – not shinobi interpreting untrusted cab data on the host at load time.

Both shapes live side by side under dosho/cabs/, and both are first-class for Recipe.add_step – a caller doesn’t need to know or care which one a given tool is.

Defining a Cab

dosho.define_cab() wraps Cab(...) and shinobi.loaders.build_model() to remove the boilerplate of hand-transcribing a real tool’s parameter list from a flat {raw_param_name: (dtype, required, default)} dict – the shape a tool’s own --help/docs naturally give you:

from dosho import define_cab, images

cab = define_cab(
    "mytool",
    "mytool",
    images.MYTOOL,
    {"data-ms": ("MS", True, None), "out-name": ("str", False, "out")},
)

Hyphenated/dotted raw names are sanitised to valid pydantic field names, with the original kept as a nom_de_guerre so the built cab’s argv still uses the tool’s real flag. Raw Cab(...) construction stays fully available for anything the helper doesn’t cover.

Dynamic per-instance parameter families

Some real tools have parameter names that depend on a caller-chosen value, not a fixed field set – CubiCal’s g1-solvable/g-time-int, QuartiCal’s K.time_interval, one family per solvable-term name chosen per pipeline call. shinobi’s ParamPattern expresses this declaratively: a hand-authored table of the real, known attrs (transcribed once from the tool’s own docs/template), plus a regex-matched wildcard segment for the caller-chosen term name – static data, never generated by importing/executing the tool’s own schema function. See dosho/cabs/cubical.py/quartical.py for real examples.

Dynamic output paths

Some tools’ output paths depend on other resolved input values – WSClean’s {prefix}-MFS-image.fits-shaped outputs, for example. A ParamMeta.implicit string template, resolved via plain str.format against a step’s own validated inputs (never eval, never an expression language), covers exactly this. Only the handful of outputs a real pipeline actually wires as a dependency need a resolved implicit template; anything more exotic (WSClean’s open-ended per-band/per-interval combinatorics) stays validation-only via output_patterns. See dosho/cabs/wsclean.py.

Container images

dosho has no image-building infrastructure – cult-cargo already solves that problem well, it’s just not this repository’s job, since dosho is a cab-schema repo, not an image-building one. dosho/images.py is the single place image references are pinned, reusing existing published images; bumping a tool’s version is editing one constant there.

Layout: single-command vs. multi-command tools

A single-command tool gets its own module exporting one object named after the tool itself:

# dosho/cabs/wsclean.py
wsclean = define_cab("wsclean", "wsclean", images.WSCLEAN, {...})

A multi-command tool (several real sub-commands sharing one binary or Python package – CASA tasks, simms) gets one module exporting one object per sub-command:

# dosho/cabs/casatasks.py
@shinobi.pystep(image=images.CASA6)
def listobs(ctx, vis: Path, listfile: Path) -> ListobsOutputs: ...

@shinobi.pystep(image=images.CASA6)
def mstransform(ctx, vis: Path, ...) -> MstransformOutputs: ...

dosho/cabs/__init__.py re-exports every tool-level object by name, so from dosho.cabs import wsclean works without knowing the submodule – see Quickstart.

Two lookup interfaces, one set of objects

dosho.get(name)/dosho.registry is the parallel, string-keyed lookup used by ninja cabs list/show and shinobi’s shinobi.cabs entry-point discovery – for a caller that only knows the tool’s name at runtime. It resolves to the exact same objects the direct-import path gives you; picking one over the other is purely about whether the caller knows the name at write-time or run-time.