Vertical Position Far doesn’t always reset to the middle. If it’s set to higher than default (slider moved to the right) it will remember the position correctly when switching between modes.
I too would like to play with Action Camera set to a lower position (slider moved to the left) and also find it resetting very annoying. While I would appreciate it remembering the settings for the 2 modes separately, a fix so that it doesn’t reset at all between modes would be just fine too.