fix(openviking): cap embedding max_input_tokens at 1536 to stay under llama.cpp nomic-bert 2048 ctx limit
astro-orbiter's llama.cpp router hard-caps nomic-embed-text-v1.5 effective context at 2048 tokens regardless of ctx-size (known nomic-bert/RoPE limitation in llama.cpp, not fixable server-side). OpenViking chunks observed at 2000-3400 tokens were tripping 400 exceed_context_size_error and endless circuit-breaker re-enqueue for viking://temp/default/08140552_5f1c9e/homelab.tar/*. Set embedding.max_input_tokens: 1536 (well under 2048) since OpenViking's chunk-time token estimator uses a different tokenizer than llama.cpp's context counter, so token counts won't match 1:1 - 1536 leaves ~25% headroom. Approved by Ryan as lowest-risk mitigation (does not touch astro-orbiter/ llama.cpp serving config, which is War Machine's domain and already fixed separately for the ubatch-size issue).
This commit is contained in:
@@ -146,6 +146,17 @@ config:
|
||||
dimension: 768
|
||||
input: "text"
|
||||
max_concurrent: 5
|
||||
# max_input_tokens caps the raw text tokens OpenViking sends per chunk to the
|
||||
# embedding model. astro-orbiter's llama.cpp router hard-caps nomic-embed-text-v1.5's
|
||||
# effective context at 2048 tokens regardless of ctx-size config (known llama.cpp
|
||||
# nomic-bert limitation, not fixable via server flags). OpenViking's chunker was
|
||||
# observed producing 2000-3400 token chunks, well over that ceiling, causing
|
||||
# `400 exceed_context_size_error, n_ctx: 2048` and endless circuit-breaker re-enqueues.
|
||||
# Set well under 2048 (1536) to leave headroom: OpenViking's chunk-time token
|
||||
# estimator is not the same tokenizer llama.cpp uses to count context, so token
|
||||
# counts won't match 1:1 between the two. Approved by Ryan as lowest-risk fix
|
||||
# (option 1 of 3) vs. touching the astro-orbiter serving stack further.
|
||||
max_input_tokens: 1536
|
||||
|
||||
# ============================================================================
|
||||
# VLM / Summarization configuration (L0/L1/L2 generation)
|
||||
|
||||
Reference in New Issue
Block a user