Let us pause Auto write, Guided write, and Rewrite mid-inference, alter their output, and resume inference.

This would be a HUGE improvement to Sudowrite and possibly solve some of the other feature requests in this channel (if it’s technically possible). It’s a feature of some local LLM programs like LM Studio that has completely changed the way I use LLMs for fiction. The ability to pause the model mid-inference and alter what it has generated so far (usually to correct issues — changing a the colour of a character’s shirt, so-and-so character definitely doesn’t have a tail, altering a line of dialogue or an important tonal or stylistic wording choice) would be enormously helpful. It’s also helpful to teach the model about what you want your output to be — rewriting the first sentence or two of what has been generated will set the tone for the rest of the inference process.

From a user perspective, this is helpful so that the generation doesn’t continue with an error or something you know you will have to fix later, wasting your valuable credits while you watch the wrong story being written by the model, particularly when generating lots of words and using lots of credits. It’s frustrating knowing you’ve just wasted money on something that means more work for you to fix. Especially since if there’s an error early on in the generation, the rest of the inference will continue taking that error into account, potentially rendering the rest of the generated prose useless (or needing a lot of editing).

The reason for pausing and editing mid-inference, rather than after the response has been generated, is because the model will remember your changes when it continues—if you pause after the sentence: “The bar was dimly lit and filled with sideways glances” and change ‘bar’ to ‘council chambers’ (for instance), the model will continue to write the scene in the council chambers as you’ve altered it’s context.

Chapter Generator kind of does this already by way of you altering the text in your document and asking it to continue where your doc leaves off. But this would be incredibly helpful in auto write and guided write, too, especially when generating 300+ words.

Hope that makes sense! Would give authors finer control over model output, save a lot of credits for authors, and save you a lot of API requests. Thanks!