mnesis.compaction.file_ids

file_ids

File ID extraction and propagation utilities.

Mnesis's "lossless" guarantee rests on preserving file_xxx identifiers across every compaction round. Even when prose context is discarded, the pointer to the external file content is never lost.

This module provides: - :func:extract_file_ids — pull all file_<hex> references from text. - :func:append_file_ids_footer — attach a [LCM File IDs: ...] footer to a summary string when file IDs are present. - :func:collect_file_ids_from_nodes — aggregate file IDs from a list of SummaryNode objects (for condensation input propagation). - :func:extract_file_id_paths and friends — recover which path each ID belongs to, so the footer can record path (file_<hex>) pairs. Without them a summary can keep an ID and a path that are no longer associated.

Footer format::

[LCM File IDs: services/billing/rates.yaml (file_3fa9c2d17b8e4a60), file_9b1e0c44a7d2f381]

A bare file_<hex> entry means the path was not known. The older footer (bare IDs only) is still parsed everywhere; ID extraction only looks for file_<hex> tokens, so both forms yield the same IDs.

extract_file_ids

extract_file_ids(text: str) -> list[str]

Extract all file_<hex> identifiers from text.

Deduplicates and preserves first-occurrence order.

Parameters:

Name	Type	Description	Default
text	str	Raw text that may contain LCM file ID references.	required

Returns:

Type	Description
list[str]	Ordered, deduplicated list of file ID strings (e.g.
list[str]	["file_a1b2c3d4", "file_deadbeef12345678"]).

message_id_text

message_id_text(msg: MessageWithParts) -> str

Return the full raw text of msg that may carry file_<hex> references.

Unlike the display rendering used for summarisation prompts, nothing is truncated: text parts, tool inputs, tool outputs and tool errors are all included in full. Pruned tool outputs (compacted_at set) are included too -- pruning only tombstones what the context shows; the original output stays in the append-only store, and an ID that appears only there must still reach the summary footer.

Parameters:

Name	Type	Description	Default
msg	MessageWithParts	Message to scan.	required

Returns:

Type	Description
str	The concatenated raw text, parts separated by newlines.

extract_file_ids_from_messages

extract_file_ids_from_messages(
    messages: list[MessageWithParts],
) -> list[str]

Extract all file IDs referenced across a list of messages.

Scans the full raw content of every message (see :func:message_id_text), including pruned tool outputs, and deduplicates.

Parameters:

Name	Type	Description	Default
messages	list[MessageWithParts]	Messages to scan for file ID references.	required

Returns:

Type	Description
list[str]	Ordered, deduplicated list of file ID strings.

most_recent_file_ids

most_recent_file_ids(
    messages: list[MessageWithParts],
) -> list[str]

File IDs ordered most-recently-referenced first, deduplicated.

"Recent" means the position of an ID's last occurrence in the full raw content, scanning messages newest to oldest and each message back to front.

Parameters:

Name	Type	Description	Default
messages	list[MessageWithParts]	Messages in chronological order.	required

most_recent_file_ids_from_nodes

most_recent_file_ids_from_nodes(
    nodes: list[SummaryNode],
) -> list[str]

Like :func:most_recent_file_ids, over summary nodes in chronological order.

strip_file_ids_footer

strip_file_ids_footer(text: str) -> str

Remove a trailing [LCM File IDs: ...] footer from text, if present.

collect_file_ids_from_nodes

collect_file_ids_from_nodes(
    nodes: list[SummaryNode],
) -> list[str]

Aggregate all file IDs already embedded in a list of summary nodes.

Each node's content may contain a [LCM File IDs: ...] footer; this function extracts IDs from every node and returns a deduplicated union.

Parameters:

Name	Type	Description	Default
nodes	list[SummaryNode]	Summary nodes whose content is scanned for file IDs.	required

Returns:

Type	Description
list[str]	Ordered, deduplicated list of file ID strings.

footer_file_id_paths

footer_file_id_paths(text: str) -> dict[str, str]

Return the {file_id: path} pairs recorded in text's footer(s).

Entries without a path (the older bare-ID footer) are skipped.

extract_file_id_paths

extract_file_id_paths(text: str) -> dict[str, str]

Map each file ID in text to its path, where one can be determined.

A pair recorded in a [LCM File IDs: ...] footer wins; otherwise an ID is paired with a path written directly next to it (at most three separator characters between them, e.g. path (file_x) or file_x: path), and only when that is unambiguous: an ID with no adjacent path, or sharing its path with another ID, stays unpaired. The first pairing found for an ID wins.

Parameters:

Name	Type	Description	Default
text	str	Text that may contain file IDs, paths and a footer.	required

Returns:

Type	Description
dict[str, str]	{file_id: path} for the IDs whose path is known.

extract_file_id_paths_from_messages

extract_file_id_paths_from_messages(
    messages: list[MessageWithParts],
) -> dict[str, str]

Like :func:extract_file_id_paths, over the raw text of messages (first pairing wins).

collect_file_id_paths_from_nodes

collect_file_id_paths_from_nodes(
    nodes: list[SummaryNode],
) -> dict[str, str]

Aggregate {file_id: path} pairs from summary nodes (for condensation).

Footer pairs (derived from the original messages when a leaf was written) take precedence over pairs inferred from a node's prose, which an LLM may have mangled. Within each tier the earliest node wins.

append_file_ids_footer

append_file_ids_footer(
    text: str,
    file_ids: list[str],
    paths: Mapping[str, str] | None = None,
) -> str

Append a [LCM File IDs: ...] footer to text when file_ids is non-empty.

If text already contains the footer this function is idempotent — it will not duplicate the block. The footer is always placed at the end.

Parameters:

Name	Type	Description	Default
text	str	The summary text to annotate.	required
file_ids	list[str]	File IDs to include in the footer.	required
paths	Mapping[str, str] | None	Optional {file_id: path}; an ID with a path is written as path (file_<hex>), others as the bare ID.	None

Returns:

Type	Description
str	Annotated text, or the original text unchanged if file_ids is empty.