refract with scalar arguments returns NaN instead of 0 on total internal reflection

[paulofduarte]

The scalar overload of glm::refract returns NaN when total internal reflection occurs (k < 0), instead of the zero value defined by the GLSL specification. The vector overloads correctly return the zero vector in the same case.

Reproduction (GLM 1.0.3 and current master):

#include <glm/glm.hpp>
#include <cstdio>

int main()
{
    float r = glm::refract(-0.5f, 1.0f, 2.0f); // k = 1 - 4 * (1 - 0.25) = -2
    std::printf("%f\n", r);                    // prints nan, expected 0

    glm::vec2 v = glm::refract(glm::normalize(glm::vec2(1.0f, -1.0f)), glm::vec2(0.0f, 1.0f), 2.0f);
    std::printf("%f %f\n", v.x, v.y);          // prints 0 0, as specified
}

Expected: refract returns genType(0) when k < 0, as specified by GLSL ("if k < 0.0 the result is 0.0") and as the vector overloads already do.

Actual: the scalar overload returns NaN. In glm/detail/func_geometric.inl:

return (eta * I - (eta * dotValue + sqrt(k)) * N) * static_cast<genType>(k >= static_cast<genType>(0));

sqrt(k) is computed unconditionally, so for k < 0 it produces NaN, and NaN * 0 is still NaN. The multiply-by-boolean form cannot express the branch it replaced.

This is a regression introduced by 20bdab3 ("Branch free refract and reflect", 2014-11-22), which rewrote both the scalar and vector paths this way. The vector path was later restored to a real conditional in 0722404 ("Refactored low level SIMD API, refract SIMD optimization", 2016-05-29) via detail::compute_refract, but the scalar overload kept the broken form. Every scalar instantiation is affected (float and double).

PR with the fix and a regression test to follow.