We begin with the integral
I=∫(sin5x+cos3xsin2x+sin3xcos2x+cos5x)2sin2xcos2xdx.
First we simplify the rather complicated expression that appears in the denominator. We write every term in powers of sinx and cosx and then factor out the highest common power of cosx:
sin5x+cos3xsin2x+sin3xcos2x+cos5x=cos5x(tan5x+tan2x+tan3x+1).
Inside the large parentheses we recognise a product. Indeed,
(1+tan3x)(1+tan2x)=1+tan2x+tan3x+tan5x,
which is exactly the bracket above. Hence we may write
sin5x+cos3xsin2x+sin3xcos2x+cos5x=cos5x(1+tan3x)(1+tan2x).
Because 1+tan2x=sec2x, the entire factor becomes
cos5x(1+tan3x)sec2x=cos3x(1+tan3x),
and therefore the square that appears in the denominator of the integrand is
(sin5x+cos3xsin2x+sin3xcos2x+cos5x)2=cos6x(1+tan3x)2.
Now we replace every occurrence of sinx and cosx in the integrand:
cos6x(1+tan3x)2sin2xcos2x=cos6xsin2xcos2x(1+tan3x)21=cos6xtan2xcos4x(1+tan3x)21=cos2xtan2x(1+tan3x)21.
Because cos2x1=sec2x=1+tan2x, the integrand becomes
(1+tan3x)2tan2x(1+tan2x).
At this stage it is very natural to introduce the substitution
t=tanx.
We know the basic differentiation formula dxd(tanx)=sec2x=1+t2, so
dt=(1+t2)dx⟹dx=1+t2dt.
Substituting for both the integrand and dx we obtain
I=∫(1+t3)2t2(1+t2)1+t2dt=∫(1+t3)2t2dt.
The fraction has now become a very clean rational function in t. We recognise the numerator t2dt as essentially the differential of t3. Indeed,
dtd(t3)=3t2.
So we set
u=1+t3⟹du=3t2dt⟹t2dt=3du.
Substituting u and du into the integral we get
I=∫31u2du=31∫u−2du.
We recall the power-rule formula ∫undu=n+1un+1+C for every exponent n=−1. Here n=−2, so
∫u−2du=−1u−1+C=−u−1+C.
Putting the numeric factor back,
I=31(−u−1)+C=−3u1+C.
Finally we reverse all substitutions. First u=1+t3 and then t=tanx, giving
I=−3(1+tan3x)1+C.
This matches Option C in the list provided.
Hence, the correct answer is Option C.