Verified answer

sin(x) = t

cos(x) * dx = dt

sin(x) * cos(x) * dx / sqrt(1 + 3sin(x)^2) =>

t * dt / sqrt(1 + 3t^2)

t^2 = (1/3) * tan(m)^2

t = sqrt(1/3) * tan(m)

dt = sqrt(1/3) * sec(m)^2 * dm

sqrt(1/3) * tan(m) * sqrt(1/3) * sec(m)^2 * dm / sqrt(1 + 3 * (1/3) * tan(m)^2) =>

(1/3) * tan(m) * sec(m)^2 * dm / sqrt(1 + tan(m)^2) =>

(1/3) * tan(m) * sec(m)^2 * dm / sqrt(sec(m)^2) =>

(1/3) * tan(m) * sec(m)^2 * dm / sec(m) =>

(1/3) * tan(m) * sec(m) * dm =>

(1/3) * sin(m) * dm / cos(m)^2

cos(m) = u

-sin(m) * dm = du

(-1/3) * du / u^2

Integrate

(1/3) * (1/u) + C =>

(1/3) * (1/cos(m)) + C =>

(1/3) * sec(m) + C =>

(1/3) * sqrt(1 + tan(m)^2) + C =>

(1/3) * sqrt(1 + 3t^2) + C =>

(1/3) * sqrt(1 + 3sin(x)^2) + C

In retrospect, there was a simpler substitution. I just didn't see it right away

u = 1 + 3sin(x)^2

du = 3 * 2 * sin(x) * cos(x) * dx = 6 * sin(x) * cos(x) * dx

sin(x) * cos(x) * dx / sqrt(1 + 3sin(x)^2) =>

(1/6) * du / sqrt(u)

Integrate

(1/6) * 2 * u^(1/2) + C =>

(1/3) * sqrt(u) + C =>

(1/3) * sqrt(1 + 3sin(x)^2) + C

You might want to go that route.

From 0 to pi/2

(1/3) * sqrt(1 + 3 * sin(pi/2)^2) - (1/3) * sqrt(1 + sin(0)^2) =>

(1/3) * sqrt(1 + 3 * 1) - (1/3) * sqrt(1 + 0) =>

(1/3) * sqrt(1 + 3) - (1/3) * sqrt(1) =>

(1/3) * sqrt(4) - (1/3) * 1 =>

(1/3) * 2 - 1/3 =>

1/3

∫ (sinx*cosx)/(√(1+3sin^2x)) = 1/3*sqrt(3sin^2(x)+1)+C

Enter both pi/2 and 0 and you will see that the answer is 1/3.