in my experience image settings is the option that kill fps, from the lowest to the second lowest my game get so much smoother, I think it is the shadows.
I never paid any attention to this setting and wondered what the hell it could be. Since there are so many post render things going on.
But I think you are right with shadows. In order to render real time shadows, the whole scene is rendered to a texture from the perspective of the light source. This is then compared to the actual render to determine if the pixel is in shadow or not. So the image quality may have different shadow texture sizes which would kill FPS for sure.
Lower settings would have more pixelated shadows. Higher settings for smoother shadows. There are also several techniques that try to smooth the edges of the shadow and all of these may add time. So they may add them as your selection quality goes up.
I will have to test this now
EDIT: I did not play a game on each mode.
But minimum settings = NO shadows.
Each setting from there increases the shadow map resolution (texture size).
As I stated there are other things going on like ambient occlusion that may turn on/off at different settings as well.
Cyanara and others state that particle effect density is adjusted here also (another FPS killer).
EDIT 2: If anyone starts testing these settings look for SSAO (Dark regions where objects meet) and see when it turns on/off. May only show up on certain maps since there is control over it in the map editor lighting settings. See example screen shot for dark region example:
https://en.wikipedia.org/wiki/Screen_space_ambient_occlusion