SketchUp’s Zoom tool (and the scroll wheel) doesn’t “Zoom” in the sense that a photographer uses the term, see this post:
What you probably want is to position eye point, (possibly even with the Position Camera tool), and then zoom with the shift key down. That changes the focal length/angle of view without moving the camera/eye point. The Field of View tool is different: it actually moves the eye point and changes the focal length at the same time.