roboto.experimental.ingestion_rules#

Request and response models for an org’s ingestion rules, which decide which uploaded files are ingested and with which action.

A rule has path patterns and the action that ingests matching files. A file whose path a rule matches when it is uploaded is ingestable, and while the rule’s auto_ingest is on, Roboto runs that action on it. There is no client class yet: call the /v1/ingestion-rules endpoints with RobotoClient.

Examples

List the org’s rules:

>>> from roboto.experimental.ingestion_rules import IngestionRuleRecord
>>> from roboto.http import RobotoClient
>>> rules = (
...     RobotoClient.defaulted()
...     .get("v1/ingestion-rules")
...     .to_record_list(IngestionRuleRecord, json_path=["data", "items"])
... )

Submodules#

Package Contents#

class roboto.experimental.ingestion_rules.CreateIngestionRuleRequest(/, **data)#

Bases: pydantic.BaseModel

Request payload to create an ingestion rule.

Parameters:: data (Any)

auto_ingest: bool = True#

description: str | None = None#

exclude_patterns: list[str]#

include_patterns: list[str]#

invocation: roboto.domain.triggers.ActionInvocationSpec | None = None#: How each matching upload is ingested. Its invocation_input selects only inputs beyond the uploaded file, which the rule’s trigger always passes first. Required when auto_ingest is on.

name: str#

class roboto.experimental.ingestion_rules.DefaultIngestionRuleRecord(/, **data)#

Bases: pydantic.BaseModel

Wire format of a rule Roboto offers every org as a starting point for a new rule, and whether the caller’s org already has it.

POST /v1/ingestion-rules/defaults creates those with for_new_orgs set that are not already_present.

Parameters:: data (Any)

already_present: bool#: Whether the org has a rule with this name, or a rule with one of these include patterns.

category: str = 'Other'#: The group the rule is listed under, such as ROS or Tabular.

exclude_patterns: list[str]#

for_new_orgs: bool = True#: Whether sign-up offers this rule and POST /v1/ingestion-rules/defaults creates it. When false, the rule is only offered as a starting point for a new rule.

include_patterns: list[str]#

invocation: roboto.domain.triggers.ActionInvocationSpec#

name: str#

needs_settings: bool = False#: Whether the rule’s action likely needs settings for the org’s files, such as which column holds the timestamps, before it ingests them.

class roboto.experimental.ingestion_rules.ExistingFilesPreview(/, **data)#

Bases: pydantic.BaseModel

Files already uploaded that a rule matches, that are not ingested, and whose current upload no rule’s trigger has run on: ones uploaded, or uploaded again, before the rule existed or while its auto-ingest was off. Editing a file’s metadata or tags does not make it one of these.

Response of GET /v1/ingestion-rules/<rule_id>/existing-files.

Parameters:: data (Any)

complete_session_count: int = 0#: Complete sessions holding any of these files. After a session’s files are ingested, it is announced as fully ingested again, and the triggers on session.ingested start for it.

count: int#

count_is_lower_bound: bool = False#: Whether there may be more than count; counting stops after a fixed number of files.

sample_paths: list[str]#: Up to 10 of the files’ paths. Empty for a caller who is not an org admin.

class roboto.experimental.ingestion_rules.IngestFileOutcome#

Bases: roboto.compat.StrEnum

What POST /v1/ingestion-rules/ingest did with one file.

AlreadyRunning = 'already_running'#: An ingestion run on the file is still queued or running, so no new one was started.

AutoIngestOff = 'auto_ingest_off'#: Rules match the file, but none of them auto-ingests.

NoMatchingRule = 'no_matching_rule'#: No rule in the file’s org matches its path.

NotFound = 'not_found'#: No such file in the caller’s org, the caller may not edit it, or it is not in a dataset (rules apply only to dataset files).

NotStarted = 'not_started'#: A matching rule’s trigger declined the file (it is disabled, say).

Started = 'started'#: A matching rule’s trigger started ingesting the file.

class roboto.experimental.ingestion_rules.IngestFileResult(/, **data)#

Bases: pydantic.BaseModel

!!! abstract “Usage Documentation”: [Models](../concepts/models.md)

A base class for creating Pydantic models.

Parameters:: data (Any)

__class_vars__#: The names of the class variables defined on the model.

__private_attributes__#: Metadata about the private attributes of the model.

__signature__#: The synthesized __init__ [Signature][inspect.Signature] of the model.

__pydantic_complete__#: Whether model building is completed, or if there are still undefined fields.

__pydantic_core_schema__#: The core schema of the model.

__pydantic_custom_init__#: Whether the model has a custom __init__ function.

__pydantic_decorators__#: Metadata containing the decorators defined on the model. This replaces Model.__validators__ and Model.__root_validators__ from Pydantic V1.

__pydantic_generic_metadata__#: A dictionary containing metadata about generic Pydantic models. The origin and args items map to the [__origin__][genericalias.__origin__] and [__args__][genericalias.__args__] attributes of [generic aliases][types-genericalias], and the parameter item maps to the __parameter__ attribute of generic classes.

__pydantic_parent_namespace__#: Parent namespace of the model, used for automatic rebuilding of models.

__pydantic_post_init__#: The name of the post-init method for the model, if defined.

__pydantic_root_model__#: Whether the model is a [RootModel][pydantic.root_model.RootModel].

__pydantic_serializer__#: The pydantic-core SchemaSerializer used to dump instances of the model.

__pydantic_validator__#: The pydantic-core SchemaValidator used to validate instances of the model.

__pydantic_fields__#: A dictionary of field names and their corresponding [FieldInfo][pydantic.fields.FieldInfo] objects.

__pydantic_computed_fields__#: A dictionary of computed field names and their corresponding [ComputedFieldInfo][pydantic.fields.ComputedFieldInfo] objects.

__pydantic_extra__#: A dictionary containing extra values, if [extra][pydantic.config.ConfigDict.extra] is set to ‘allow’.

__pydantic_fields_set__#: The names of fields explicitly set during instantiation.

__pydantic_private__#: Values of private attributes set on the model instance.

file_id: str#

invocation_id: str | None = None#: The invocation started, when outcome is started.

outcome: IngestFileOutcome#

rule_id: str | None = None#: The rule whose trigger ran, or would have.

class roboto.experimental.ingestion_rules.IngestFilesRequest(/, **data)#

Bases: pydantic.BaseModel

Request payload to ingest existing files now with the org’s rules that match them.

Parameters:: data (Any)

file_ids: list[str]#

class roboto.experimental.ingestion_rules.IngestFilesResponse(/, **data)#

Bases: pydantic.BaseModel

!!! abstract “Usage Documentation”: [Models](../concepts/models.md)

A base class for creating Pydantic models.

Parameters:: data (Any)

__class_vars__#: The names of the class variables defined on the model.

__private_attributes__#: Metadata about the private attributes of the model.

__signature__#: The synthesized __init__ [Signature][inspect.Signature] of the model.

__pydantic_complete__#: Whether model building is completed, or if there are still undefined fields.

__pydantic_core_schema__#: The core schema of the model.

__pydantic_custom_init__#: Whether the model has a custom __init__ function.

__pydantic_decorators__#: Metadata containing the decorators defined on the model. This replaces Model.__validators__ and Model.__root_validators__ from Pydantic V1.

__pydantic_generic_metadata__#: A dictionary containing metadata about generic Pydantic models. The origin and args items map to the [__origin__][genericalias.__origin__] and [__args__][genericalias.__args__] attributes of [generic aliases][types-genericalias], and the parameter item maps to the __parameter__ attribute of generic classes.

__pydantic_parent_namespace__#: Parent namespace of the model, used for automatic rebuilding of models.

__pydantic_post_init__#: The name of the post-init method for the model, if defined.

__pydantic_root_model__#: Whether the model is a [RootModel][pydantic.root_model.RootModel].

__pydantic_serializer__#: The pydantic-core SchemaSerializer used to dump instances of the model.

__pydantic_validator__#: The pydantic-core SchemaValidator used to validate instances of the model.

__pydantic_fields__#: A dictionary of field names and their corresponding [FieldInfo][pydantic.fields.FieldInfo] objects.

__pydantic_computed_fields__#: A dictionary of computed field names and their corresponding [ComputedFieldInfo][pydantic.fields.ComputedFieldInfo] objects.

__pydantic_extra__#: A dictionary containing extra values, if [extra][pydantic.config.ConfigDict.extra] is set to ‘allow’.

__pydantic_fields_set__#: The names of fields explicitly set during instantiation.

__pydantic_private__#: Values of private attributes set on the model instance.

results: list[roboto.experimental.ingestion_rules.record.IngestFileResult]#: One result per requested file, in request order.

class roboto.experimental.ingestion_rules.IngestionPathRule#

The path half of an ingestion rule: include and exclude Git wildmatch patterns.

File annotation and the rule’s trigger both decide through matches(), so a file is marked ingestable exactly when the rule’s trigger would accept its upload.

exclude_patterns: tuple[str, ...] = ()#

include_patterns: tuple[str, ...]#

matches(relative_path)#

Whether relative_path matches an include pattern and no exclude pattern, and is not a dotfile.

No rule makes a dotfile ingestable, because triggers never run on dotfiles.

Parameters:: relative_path (str)
Return type:: bool

classmethod of(include_patterns, exclude_patterns=())#

Parameters:

include_patterns (collections.abc.Iterable[str])
exclude_patterns (collections.abc.Iterable[str])

Return type:

IngestionPathRule

class roboto.experimental.ingestion_rules.IngestionRuleRecord(/, **data)#

Bases: pydantic.BaseModel

Wire format of an ingestion rule: which file paths in an org are ingestable, and how they are ingested.

A file whose path matches any of include_patterns and none of exclude_patterns is marked ingestable when it is created. The rule owns one trigger (trigger_id) that runs invocation on each matching upload while auto_ingest is on.

Parameters:: data (Any)

auto_ingest: bool#

whether the rule’s trigger is enabled. Turning this off disables the trigger and leaves matching files ingestable.

Type:: Whether matching uploads are ingested automatically

created: datetime.datetime#

created_by: str#

description: str | None = None#

exclude_patterns: list[str]#: Git wildmatch patterns. A file matching any of them is not ingestable by this rule.

include_patterns: list[str]#: Git wildmatch patterns, e.g. **/*.mcap. A file matching any of them is ingestable.

invocation: roboto.domain.triggers.ActionInvocationSpec | None = None#: How each matching upload is ingested. The trigger passes the uploaded file as the first input, followed by whatever invocation.invocation_input selects. None only on a rule that does not auto-ingest.

modified: datetime.datetime#

modified_by: str#

name: str#

org_id: str#

rule_id: str#

trigger_id: str | None = None#: The trigger that runs invocation on matching uploads, enabled while auto_ingest is on. Only the rule can enable, disable, edit or delete it. None while the rule has no invocation.

class roboto.experimental.ingestion_rules.IngestionRun(/, **data)#

Bases: pydantic.BaseModel

The latest time an ingestion rule’s trigger ran on a file’s current upload.

Parameters:: data (Any)

detail: str | None = None#: Why the trigger could not start the action, when it could not.

file_id: str#

invocation_id: str | None = None#: The action invocation the run started. None when the trigger could not start one.

rule_id: str#: The rule whose trigger ran.

started: datetime.datetime#

status: IngestionRunStatus#

trigger_id: str#

class roboto.experimental.ingestion_rules.IngestionRunStatus#

Bases: roboto.compat.StrEnum

Where one run of a rule’s trigger on a file stands.

Failed = 'failed'#: The action failed or was cancelled, or the trigger could not start it.

Finished = 'finished'#: The action finished. The file’s ingestion_status says how much of it was ingested.

Running = 'running'#: The ingestion action is queued or running.

class roboto.experimental.ingestion_rules.LatestIngestionRunsRequest(/, **data)#

Bases: pydantic.BaseModel

Request payload to look up the latest ingestion run on each of some files.

Parameters:: data (Any)

file_ids: list[str]#

class roboto.experimental.ingestion_rules.LatestIngestionRunsResponse(/, **data)#

Bases: pydantic.BaseModel

!!! abstract “Usage Documentation”: [Models](../concepts/models.md)

A base class for creating Pydantic models.

Parameters:: data (Any)

__class_vars__#: The names of the class variables defined on the model.

__private_attributes__#: Metadata about the private attributes of the model.

__signature__#: The synthesized __init__ [Signature][inspect.Signature] of the model.

__pydantic_complete__#: Whether model building is completed, or if there are still undefined fields.

__pydantic_core_schema__#: The core schema of the model.

__pydantic_custom_init__#: Whether the model has a custom __init__ function.

__pydantic_decorators__#: Metadata containing the decorators defined on the model. This replaces Model.__validators__ and Model.__root_validators__ from Pydantic V1.

__pydantic_generic_metadata__#: A dictionary containing metadata about generic Pydantic models. The origin and args items map to the [__origin__][genericalias.__origin__] and [__args__][genericalias.__args__] attributes of [generic aliases][types-genericalias], and the parameter item maps to the __parameter__ attribute of generic classes.

__pydantic_parent_namespace__#: Parent namespace of the model, used for automatic rebuilding of models.

__pydantic_post_init__#: The name of the post-init method for the model, if defined.

__pydantic_root_model__#: Whether the model is a [RootModel][pydantic.root_model.RootModel].

__pydantic_serializer__#: The pydantic-core SchemaSerializer used to dump instances of the model.

__pydantic_validator__#: The pydantic-core SchemaValidator used to validate instances of the model.

__pydantic_fields__#: A dictionary of field names and their corresponding [FieldInfo][pydantic.fields.FieldInfo] objects.

__pydantic_computed_fields__#: A dictionary of computed field names and their corresponding [ComputedFieldInfo][pydantic.fields.ComputedFieldInfo] objects.

__pydantic_extra__#: A dictionary containing extra values, if [extra][pydantic.config.ConfigDict.extra] is set to ‘allow’.

__pydantic_fields_set__#: The names of fields explicitly set during instantiation.

__pydantic_private__#: Values of private attributes set on the model instance.

runs: dict[str, roboto.experimental.ingestion_runs.IngestionRun]#: The latest run per file id. A file whose current upload no rule’s trigger has run on, or that the caller may not view, is absent.

roboto.experimental.ingestion_rules.MAX_INGEST_FILES = 100#: Most files one POST /v1/ingestion-rules/ingest takes.

roboto.experimental.ingestion_rules.MAX_MATCH_PATHS = 200#: The most paths one MatchIngestionRulesRequest may ask about.

class roboto.experimental.ingestion_rules.MatchIngestionRulesRequest(/, **data)#

Bases: pydantic.BaseModel

Request payload to ask which of an org’s ingestion rules match each of some file paths.

Parameters:: data (Any)

paths: list[str]#: File paths relative to their dataset, e.g. logs/run_1.mcap.

class roboto.experimental.ingestion_rules.MatchIngestionRulesResponse(/, **data)#

Bases: pydantic.BaseModel

The rules matching each requested path, oldest rule first. A path no rule matches maps to an empty list.

Parameters:: data (Any)

matches: dict[str, list[roboto.experimental.ingestion_rules.record.IngestionRuleRecord]]#

class roboto.experimental.ingestion_rules.UpdateIngestionRuleRequest(/, **data)#

Bases: pydantic.BaseModel

Request payload to change an ingestion rule. Only the fields present in the payload change.

Parameters:: data (Any)

auto_ingest: bool | None = None#: Turning this on enables the rule’s trigger, creating it if the rule had none; turning it off disables it.

description: str | None = None#

exclude_patterns: list[str] | None = None#

include_patterns: list[str] | None = None#

invocation: roboto.domain.triggers.ActionInvocationSpec | None = None#: Replaces the rule’s whole invocation when present.

name: str | None = None#