feat: add prompt caching to improve response times #134
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR adds caching for enhanced prompts to improve response times and reduce API calls.
Changes
Added new prompt-cache store with 24-hour TTL for cached prompts
Modified enhancer API to check cache before making LLM calls
Added visual indicators in UI to show when responses come from cache:
Clock icon for cached responses
"From cache" text instead of "Prompt enhanced"
Cache status header in API response
Technical Details
Cache entries expire after 24 hours
Cache is stored in memory using nanostores
Cache status is communicated via x-from-cache response header
UI updates dynamically based on cache status
Testing
Create a new prompt: Shows stars icon, makes LLM call
Repeat same prompt: Shows clock icon, returns cached result
Wait 24 hours: Cache expires, makes fresh LLM call
Modify prompt: Makes new LLM call, caches new result
Performance Impact
Reduces API calls for frequently used prompts
Improves response time for cached prompts
Minimal memory footprint with 24h TTL