Depending on your workflow or end goal the following may work for you.
Use cvFillPoly to create a binary mask of the polygon, then use cvCopy to make a copy of the image using the mask created to a whited out array, thus the only black pixels are those that were in the polygon.
Or instead of using the pointPolygonTest you could use a raycasting algorithm and only check if the pixel is black if you have had an odd number of crossings.
Doing the raycasting algorithm yourself could save alot of time over calling pointPolygonTest.
You could have a situation where the scanline that a point is on has a polygon that occupies the entirety of the line.
The raycasting algorithm counts polygon line crossings from one side, until it reaches the point being tested. If there are m line segments we perform m line segment ray intersections, order them, and count how many have occured.
If you use the pointPolygonTest you are having to do this for every point, but all the rays you test for the n points on this line, all intersect the line, and their intersections with the line segments all occur at the same place, so you can save all this recalculation by writing the algorithm yourself and augmenting it to check for black pixels when the number of crossings has been odd.