Moving by sub-pixel amounts will always alter the texture sampling. That you use nearest sampling affects how the texture is sampled, but not at what coordinate it is sampled at in the first place. That coordinate is a function of both the tex-coords, interpolated across the polygon, and the vertices, which determine how the polygon is rasterised to screen pixels.
Remember that to get perfect texture sampling the interpolated tex-coords, when evaluated at pixel centres, must exactly lie on texel centres.
Moving the polygon by a fraction of a pixel therefor changes where the texture is sampled from, as the pixel centres now lie at slightly different point within the polygon.
Now, with nearest sampling, the resulting coordinates will effectively be rounded back to the nearest texels, so if your coordinates are not far from the centres, it'll probably work. However when your coordinates have moved by around half a texel, so they lie right at the point where two texels are almost equidistant, then it only takes tiny errors on the precision of the calculations to nudge some pixels one way, and some the other.
Here's a diagram. I've chosen to show a triangle, mapped with a check pattern, but it applies equally well to a sprite, or anything else.
In the top part of the diagram, I'm just showing how the triangle is mapped to the texture. If you rendered this triangle on an very high-resolution screen (or at a very large size) you'd expect to see that check pattern exactly replicated.
However if you're mapping at a closer ratio, such as 1:1, the sampling is critical. In the bottom part of the diagram I show this situation.
The key thing to look at is where the screen-pixel centres are, because that determines which pixels are drawn, and also where the texture is sampled at. In this case I've shown the polygon being mapped at exactly the right size, but offset by half a pixel. That aligns the pixel centres exactly between adjacent texels, and the sampling will be pretty twitchy.
