Try using the settings button in the AnomalyRO patcher. I've noticed this opens a different settings menu than the opensetup.exe located in the anomalyRO folder. This worked fine for me.
Should this not work, try the reverse. Open the opensetup.exe in your anomalyRO folder and select your resolution there. Ofcourse the above mentioned resolution limitations will still apply regardless.