Vim mode: moving up or down after $ stays at the end of the line

Vim mode lets you edit your prompt with the keys of the vim text editor. The $ key moves to the end of the line. It now also remembers that you want to be at the end, so moving up or down afterwards puts the cursor at the end of each line, however long it is. Before, $ only moved to the end of the current line.