hi,
There are 2 way to avoid this;
1- Don't use product pictures that has same resolution as thumbnail (thumbnail is the picture that you roll over to zoom). Product picture must be large (for instance, the Blue Dress product picture is large and the ZoomEffect works just fine with it)
2- Decreased the Zoom Window (the window that appears on the right side of thumbnail when you roll over it) so the ZoomWindow won't be larger than original product picture. In your screenshot, i am seeing this is happening. The yellow dress picture is smaller than the zoomWindow, which breaks the algorithm and mess up the zoom ratio calculation.
Ideal solution is to use Large Product Picture. Product pictures mush be larger than 900x900 (pixel)
regards.