While operating Hindsight in Docker, we use cloud APIs for the LLM and embeddings, and run the reranker on a local NPU. Because embedding API costs are low, we saw no need for local operation, whereas the reranker was moved locally to avoid recurring API expenses.
While running Hindsight, a Docker-based long-term memory management system, on a single-board computer (ROCK 5C, RK3588 SoC, 8GB RAM), we adopted a hybrid architecture that splits cloud APIs and the local NPU according to workload characteristics. LLM generation and embeddings are handled via commercial APIs, while candidate reranking during retrieval is processed on the local NPU.
Across 9 complete UTC calendar days, the gateway recorded 12,536 requests of production traffic; based on these logs and hardware benchmark data, this article outlines the economic and technical rationale and latency trade-offs behind this partitioned design.
Table of Contents
- 1. Hybrid Architecture: Partitioning LLM and Embedding APIs from the Local NPU Reranker
- 2. Bifrost Logs: Requests and Cost Scope Across 9 Days
- 3. Why We Kept Embeddings on Cloud APIs: 130 Ten-Thousand Tokens for $0.26
- 4. Why We Moved the Reranker Locally: Qwen3 RKLLM NPU Performance and Measured Latency
- 5. Real-World Operational Strategy: Retaining the Local Reranker Despite Slow NPU Inference
- 6. Operational Limits and Decision Criteria: Workload Partitioning in Single-Board LLM Systems
1. Hybrid Architecture: Partitioning LLM and Embedding APIs from the Local NPU Reranker
The criteria for distributing each inference task in the Hindsight environment stems from the interaction between hardware resource requirements and API pricing models.
- Generative LLM: We use the DeepSeek V4.1 Flash cloud API via OpenCode Go. Within the monthly subscription usage limits, usage is metered at peak rates (input $0.30 per 100 ten-thousand tokens, output $1.20, cache read $0.006) and off-peak rates (with a 50% discount applied).
- Text Embeddings: We use the Google Gemini Embedding 2 (768-dimensional) cloud API.
- Document Reranker: By leveraging the local NPU of the single-board hardware (Rockchip RK3588, RKLLM 1.2.1), we run
Qwen3-Reranker-0.6B.
Model execution and memory constraints on the same board were also examined in Limitations of Anima RKNN Optimization: W8A8 Quality Failure and 8GB Resident Model Experiment and Generating Anima Images on RK3588: RKNN Conversion and Execution Results on ROCK 5C 8GB.
Communication between Hindsight and upstream models is proxied through Bifrost (v2.2.4), a central LLM gateway. For the reranker, to avoid the compounding unit cost of continuously calling commercial cloud APIs, we used the Qwen3-Reranker-0.6B model released by Alibaba Cloud’s Qwen team, converted into the RK3588-dedicated W8A8 quantized format (.rkllm) distributed by the community (Hugging Face dulimov). We wrapped and connected this model as a Cohere-compatible HTTP service based on a native C++ logit bridge (RKLLM_INFER_GET_LOGITS) in an isolated Docker Compose container.
2. Bifrost Logs: Requests and Cost Scope Across 9 Days
In Bifrost’s Hindsight-dedicated virtual key logs from 2026-10-14 00:00:00Z through 2026-10-23 00:00:00Z across 9 UTC dates, we examined requests. The table below summarizes only the successful requests for Gemini Embedding 2 and the active DeepSeek V4.1 Flash within this window.
| Provider and Model | Successful Requests | Prompt Tokens | Completion Tokens | Total Tokens | Bifrost Recorded Cost (USD) |
|---|---|---|---|---|---|
gemini / gemini-embedding-2 |
9,484 | 1,307,222 | 0 | 1,307,222 | $0.2614 |
opencode-go / deepseek-v4.1-flash |
889 | 6,833,856 | 2,754,646 | 9,588,502 | $2.5412 |
Both amounts represent recorded figures calculated from unit rates configured in the gateway. For Gemini as well, actual billing may vary depending on the account’s free tier quotas and other terms.
First, for DeepSeek V4.1 Flash under OpenCode Go, the metered amount ($2.5412) is an equivalent value calculated by the gateway to manage token usage quotas according to the provider’s time-of-day tariff schedule (peak and 50% off-peak discount). It does not reflect the operator’s actual monthly subscription invoice amount.
Second, for Gemini embeddings, we used logs after backfilling token and cost tracking. Previously missing records were not treated as free usage.
Third, across the same observation window, the local Qwen reranker handled 731 successful requests. This configuration does not make external pay-as-you-go API calls for reranking. We did not interpret the absence of reranker token aggregation as meaning that token usage was 0.
The total of 12,536 requests includes initial environment setup, load testing, retries, background jobs, and legacy proxy paths. We did not aggregate routes lacking cost records into an overall operating expense, nor did we extrapolate this table into daily/monthly averages or per-unit costs for storing or recalling a single memory.
3. Why We Kept Embeddings on Cloud APIs: 130 Ten-Thousand Tokens for $0.26
According to Google’s official pricing policy, the standard paid tier for Gemini Embedding 2 text is $0.20 per 100 ten-thousand input tokens (the asynchronous Batch API is $0.10, but Hindsight’s synchronous pipeline is billed at standard rates).
Over 9 UTC days, Gemini prompt tokens consumed by Hindsight for memory storage and query processing totaled 1,307,222. Calculating this using the official standard unit rate yields the following:
\[1,307,222 \times \frac{\$0.20}{1,000,000} = \$0.2614444\]The value calculated at standard unit rates is $0.2614444. The $0.2614 in the table reflects this rounded. Within this observation window, the recorded expense corresponding to 9,484 successful embedding requests was approximately $0.26.
At this usage volume, embedding API costs remain low enough that we saw no need to operate an additional local model. We decided to keep embeddings on the API while focusing local resources and operational maintenance on the reranker. This decision reflects our current setup and traffic volume.
4. Why We Moved the Reranker Locally: Qwen3 RKLLM NPU Performance and Measured Latency
The initial motivation for migrating the reranker locally was API cost pressure. Under our current operating environment, we remain satisfied with Qwen’s retrieval result quality and continue to use it. This was not calculated as cost savings directly compared against the pricing of any specific commercial reranker.
The local reranker environment runs on the ROCK 5C single-board computer using its RK3588 NPU and RKLLM 1.2.1 runtime. We loaded the community-distributed dulimov/Qwen3-Reranker-0.6B-rk3588-1.2.1 W8A8 quantized model, invoking RKLLM_INFER_GET_LOGITS from a native C++ bridge to apply softmax over the tokenizer’s yes/no logits for score generation. We capped the context length to 2048 tokens, adding a guardrail that returns a 422 error when exceeded.
In those benchmarks, runtime initialization took 2.283 seconds excluding Python module imports, and peak process RSS reached 1,437,372 KiB (approximately 1.371 GiB). This value represents the RSS of that process and differs from whole-system memory utilization.
To validate quality, across 12 handcrafted test cases consisting of Korean and cross-lingual question-document pairs, we compared ranking accuracy against the existing CPU-based MiniLM (batch size 32). Out of 12 queries, for 11 queries the Qwen3 NPU model ranked the ground-truth document at rank 1 (ground-truth rank 1 ratio 11/12, MRR 0.9375), exhibiting higher discriminatory performance compared to MiniLM’s ground-truth rank 1 in 6 queries (ground-truth rank 1 ratio 6/12, MRR 0.6417) (note that these 12 test cases serve as pipeline integration validation probes and do not guarantee public IR benchmark scores).
In contrast, inference latency revealed clear limitations stemming from the serialized compute pipeline of the NPU model.
| Candidate Documents | MiniLM CPU (Batch Size 32) | Qwen3 RK3588 NPU (Single Serial Inference) |
|---|---|---|
| 10 | 0.372 s (median of 3 runs) | 5.047 s (median of 3 runs) |
| 30 | 1.175 s (median of 3 runs) | 14.544 s (median of 3 runs) |
| 100 | 4.411 s (1-run observation) | 51.980 s (1-run observation) |
| 300 | 13.423 s (1-run observation) | 114.537 s (1-run observation) |
Tests across candidate counts were constructed by repeating 48 short documents on a shared host. Total prompt input for Qwen averaged approximately 108 tokens. Each implementation used its own prompt structure and tokenizer, with MiniLM executing at batch size 32 while Qwen processed candidates serially. The results in the table reflect empirical observations from benchmarking these two implementations.
5. Real-World Operational Strategy: Retaining the Local Reranker Despite Slow NPU Inference
For processing 100 candidates, a latency of 52 seconds represents significant wait time from the standpoint of real-time conversational services. Nevertheless, the decision to retain this setup in production rests on the operator’s specific workflow characteristics and concrete optimization configurations.
In actual everyday use, the perceived slowdown is negligible, so we continue using Qwen due to its satisfactory quality. This assessment reflects our specific operating workflow; it cannot be generalized to suggest that the latency figures measured above are universally short.
To preserve system stability, we enforced two core operational policies.
1) Upfront Candidate Capping at 50 Documents
In the initial configuration, up to 300 candidate documents were forwarded to the reranker. In a real-world memory recall benchmark where 244 candidates were passed, reranking alone consumed 181.868 seconds, driving total recall latency to 183.612 seconds.
We subsequently applied HINDSIGHT_API_RERANKER_MAX_CANDIDATES=50 in Hindsight’s configuration to restrict the Reciprocal Rank Fusion (RRF) stage so that only the top 50 candidates are passed to the reranker. Re-testing with the identical query reduced reranking latency to 104.259 seconds (a 42.67% reduction) and total memory recall latency to 105.791 seconds (a 42.38% reduction). Because these figures originate from distinct inputs and measurement conditions, they cannot be directly compared against the preceding short-document benchmark latencies.
| Single-Execution Comparison on Identical Query | Previous Configuration (Max 300 Allowed, 244 Evaluated) | Capped Configuration (50 Upfront Cap, 50 Evaluated) |
|---|---|---|
| Evaluated Candidate Documents | 244 | 50 |
| Reranking Execution Time | 181.868 s | 104.259 s |
| Total Memory Recall Time | 183.612 s | 105.791 s |
| Final Returned Memories | 48 | 44 |
| Original Source Memory Rank | Rank 1 | Rank 1 |
Although the candidate count decreased from 244 to 50, reduced to roughly a 5-to-1 ratio, execution time did not scale down proportionally. This disparity is likely driven by compound factors such as prompt cache reuse rates and background resource contention on the shared single-board host; linear latency scaling with respect to candidate counts should not be assumed as a performance guarantee. Capping the candidate pool at 50 carries the trade-off of potentially excluding relevant memories from reranking in exchange for lower latency.
2) Gateway Serial Queueing and Multi-Tier Timeouts
To ensure that the local worker processes only one request at a time, we configured Bifrost’s reranker concurrency to 1 and channel buffer to 10. Under current global policy, additional requests can still wait when the buffer fills, meaning buffer 10 does not represent an absolute upper bound on total queued requests.
In actual synthetic load testing (sequential arrival of 16-document requests (1 call) and 2-document requests (10 calls)), all 11 requests succeeded within the 33.2~35.1-second range with HTTP 200, and the NPU worker process was stably reused without restarts.
We also established multi-tier timeout margins to prevent unexpected NPU lockups.
- Hindsight reranker timeout: 1800 s (30 min)
- Bifrost upstream socket timeout: 780 s
- Local native C++ inference timeout: 600 s
- Native queue wait timeout: 120 s
These timeout values do not represent nominal system response latency, but rather safety margins configured to recover and restart processes safely when requests stall in queues or encounter abnormal delays.
6. Operational Limits and Decision Criteria: Workload Partitioning in Single-Board LLM Systems
Whether to retain a local reranker must be judged by jointly evaluating API expenditure, retrieval quality, and latency overhead. The rationale and operational limitations of maintaining the current architecture are summarized below.
We also benchmarked KURE’s fixed 256-token FP16 RKNN model. Across the identical 50 inputs, HTTP rerank response latency was 10.647 s for KURE and 27.798 s for Qwen. However, KURE truncated documents across 2 inputs, and subsequent live recall tests yielded different final rankings. These samples do not conclusively prove overall quality superiority between the two models. For now, we continue using Qwen.
Second, there are limitations to the benchmark sample. The ranking performance observed in this operational deployment—ground-truth rank-1 hit count of 11/12 (Top-1 hit rate) and MRR 0.9375 ranking performance—reflects only minimum pipeline verification probe results for service deployment, and does not represent generalized retrieval efficacy on large-scale Korean information retrieval (IR) datasets.
Our decision criteria in this operational setup are as follows:
- Embeddings: At current usage levels, API costs remain low enough that we saw no practical necessity for local operation.
- Reranker: We chose local Qwen due to API cost burdens. We continue to run it because we are satisfied with its quality and perceive no noticeable slowdown in real-world use.
Under our present workload, there is ample justification to keep embeddings on cloud APIs, while cost and quality warrant keeping the reranker local. Should traffic volume or response-latency constraints change, this architectural division can be re-evaluated.