mnesis.tokens.estimator

estimator

Multi-model token estimation with caching and graceful fallback.

TokenEstimator

TokenEstimator(*, heuristic_only: bool = False)

Multi-model token counting with caching and graceful fallback.

Priority order: 1. tiktoken for OpenAI model families (gpt-4, gpt-3.5, o1, o3) 2. Character-based heuristic (len // 3) for Claude models 3. Character-based heuristic (len // 4) for all other models

Caching: - Encoder objects are cached by encoding name (one load per process). - Token counts are cached by SHA-256 of content for immutable content (file references, summary nodes). Use estimate_cached() for this.

Parameters:

Name	Type	Description	Default
heuristic_only	bool	When True, always use the character-based heuristic and skip tiktoken entirely. Useful for testing, benchmarking, or environments where tiktoken is not installed.	False

for_model

for_model(model: ModelInfo | None) -> TokenEstimator

Return a view of this estimator that tokenises for model by default.

Calls that omit the model argument use model's tokeniser instead of the model-less len // 4 heuristic, so a component that sizes text without threading a model through every call (the compaction engine and its level functions) counts in the same units as :class:~mnesis.context.builder.ContextBuilder. The encoder cache is shared; the content-hash count cache is not, so counts for different models can never collide.

Parameters:

Name	Type	Description	Default
model	ModelInfo | None	Model whose tokeniser to use by default; None returns self unchanged.	required

estimate

estimate(text: str, model: ModelInfo | None = None) -> int

Estimate the token count for a string.

Parameters:

Name	Type	Description	Default
text	str	The text to estimate.	required
model	ModelInfo | None	Optional model info for accurate tokenisation. Uses heuristic when None or model encoding is unknown.	None

Returns:

Type	Description
int	Estimated token count, always >= 1 for non-empty text.

estimate_cached

estimate_cached(
    text: str,
    cache_key: str,
    model: ModelInfo | None = None,
) -> int

Estimate with caching, keyed by cache_key.

Use for immutable content (file references, summary nodes) where the same text will be estimated multiple times across context builds.

Parameters:

Name	Type	Description	Default
text	str	The text to estimate.	required
cache_key	str	A stable identifier for this content (e.g. SHA-256 hash).	required
model	ModelInfo | None	Optional model info for accurate tokenisation.	None

Returns:

Type	Description
int	Estimated token count from cache or fresh computation.

estimate_message

estimate_message(
    msg: MessageWithParts, model: ModelInfo | None = None
) -> int

Estimate total tokens for a message including all non-pruned parts.

Pruned tool outputs (compacted_at set) contribute only the tombstone string length rather than the full output length.

Parameters:

Name	Type	Description	Default
msg	MessageWithParts	The message with its associated parts.	required
model	ModelInfo | None	Optional model info for accurate tokenisation.	None

Returns:

Type	Description
int	Total estimated token count for the message.

content_hash staticmethod

content_hash(text: str) -> str

Return a stable SHA-256 hex digest for use as a cache key.