You'll notice
Useful3 Signal2
Preparing your conversation for each request reuses the unchanged parts instead of rebuilding it.
What's wrong with this entry?
Anonymous. No account, no email.
What
Preparing the message list for a request now caches the previous input and output and reuses the unchanged prefix and suffix when the conversation grows, rather than rebuilding everything each turn.
Details
- Only streaming assistant messages whose stop reason is still null are re-walked.
- That pending set is pruned once it exceeds a threshold constant.
Evidence
PRUNE_PENDING_AT
Strings lifted out of the shipped bundle, so the claim above can be checked against them.
Related
Other releases about the same thing. Found by shared names or similar wording; neither means one caused the other.
-
v2.1.227
Less eager module loading at startup
Both mention performance
-
v2.1.227
Event loop stall detector rewritten as a class
Both mention performance
-
v2.1.239
Large directory scans stop freezing the interface
Both mention performance