The tests probably timed out due to unrelated reasons, can they be retriggered?

@asmorkalov Anything missing from my side before merging?

[edit] I tested the fix on a much larger number of examples, and the proper treatment makes other algorithm assumptions fail. So merging as is, is not a good option. Either another fix is needed or the algorithm needs to be written in a way that is not relying on exact corner treatment in other parts. I will continue looking into it.

So the original idea with putting the "numerical circle" around endpoints failed. I had examples, where a intersection was detected, even though polygons did not touch. For some reason the algorithm went crazy from that point on.

Now the new approach would be to only check "on the polygon". So if one edge goes through the vertex it will have its (let's say) s in [0,1]. Then the issue observed was that for the edge before the vertex we had t > 1 and after the vertex t < 0, but in fact we passed the vertex, so it should have been either t <= 1 or s>= 0.

For now I put the MR to draft status: The original solution would not work, the new approach should circumvent those issues. I will remove the draft status once I arrive at a solution that reliably works for me. In the meantime I invite to iterate on the idea proposed with my second commit.

If you want to reproduce the issue with the original fix yourself, use:
{-0.033171528879342396, -0.007706989388492961},
{1920.2008671642784, -0.12243585190040425},
{1920.2549093866564, 1080.028222999363},
{0.020563744444937603, 1080.1050984957212}}
and
{{0, 0}, {1920, 0}, {1920, 1080}, {0, 1080}};

Ok, so unit tests pass now as before and also on my side I ran some initial automated tests with random rectangles - so far it looks good, so I would say the MR is again ready for review.

Compared to the first solution where I enlarged the vertex, I am now checking whether the intersecting lines start point was outside and its end point inside the polygon. Unless the edge is super tiny in the magnitude of the numeric errors, this is a robust alternative to checking whether a float point ==0.

I am not super happy yet with the names of variables and organization of the updates per "advance". I am open for suggestions making the code more readable. In fact if there is interest I could also propose some cleanup towards more readable code. It took me quite some time to wrap my head around the several a's b's p's and q's.

@klosteraner, thank you for the contribution and sorry for a long delay with the review. I found that if intersectLineSegments() is modified to use double-precision arithmetics more extensively, as shown below, then your test passes as-is, without complex modifications to intersectConvexConvex. If possible, could you please check it?

static LineSegmentIntersection intersectLineSegments( Point2f a, Point2f b, Point2f c,
                                                      Point2f d, Point2f& p, Point2f& q )
{
    double denom = ((double)a.x - b.x) * ((double)d.y - c.y) - ((double)a.y - b.y) * ((double)d.x - c.x);
    // If denom is zero, then segments are parallel: handle separately.
    if( denom == 0. )
        return parallelInt(a, b, c, d, p, q);
    double num = ((double)d.y - a.y) * ((double)a.x - c.x) + ((double)a.x - d.x) * ((double)a.y - c.y);
    double s = num / denom;
    num = ((double)b.y - a.y) * ((double)a.x - c.x) + ((double)c.y - a.y) * ((double)b.x - a.x);
    double t = num / denom;
    p.x = (float)(a.x + s*((double)b.x - a.x));
    p.y = (float)(a.y + s*((double)b.y - a.y));
    q = p;
    return s < 0. || s > 1. || t < 0. || t > 1. ? LS_NO_INTERSECTION :
           s == 0. || s == 1. || t == 0. || t == 1. ? LS_ENDPOINT_INTERSECTION : LS_SINGLE_INTERSECTION;
}