Correct me if I'm wrong, but isn't this the root problem: that when the user presses the mouse button down, you don't yet know if they're clicking or dragging? I don't know about other OS's but the Windows way of handling mouse messages is this for click events to be fired on mouse-up and drag-start events are fired when the mouse has moved n pixels from the original mouse-down pixel (where n==2 usually -- the area marked by this radius is called "the slop").
If you do this, a click could move the viewport to the code the user actually clicked on. Yes, the highlight may not be under the mouse anymore, but the user likely won't care. If they're dragging, you should scroll based on delta-y instead of abs(y) like you do currently if the user drags the highlight.
Here are a couple of other UX tweaks that might help:
1) Invert the drag operation so that instead of dragging the highlight, you instead drag the code beneath the highlight; i.e. moving the mouse towards the bottom of the screen moves the viewport towards the top of the code. Currently the user drags the highlight, so dragging outside of the highlight doesn't really make sense and that's part of the problem we're running into. If instead the user drags the code map then it's easier to reason about what the right thing to do is.
2) Animation would help. Instead of snapping to something, giving a 250ms, ease-out animation would make it much easier for the user to figure out what just happened.