Let us introduce two acute angles so that the inverse-sine symbols disappear and we can work with ordinary trigonometry.
We put α = sin − 1 12 13 \alpha=\sin^{-1}\frac{12}{13} α = sin − 1 13 12 and β = sin − 1 3 5 . \beta=\sin^{-1}\frac{3}{5}\;. β = sin − 1 5 3 .
Because 12 13 < 1 \frac{12}{13}\lt1 13 12 < 1 and 3 5 < 1 \frac{3}{5}\lt1 5 3 < 1 , both angles lie in the first quadrant, that is, 0 < α < π 2 0\lt \alpha\lt \frac{\pi}{2} 0 < α < 2 π and 0 < β < π 2 . 0\lt \beta\lt \frac{\pi}{2}\;. 0 < β < 2 π . By the definition of the arcsine, we have
sin α = 12 13 , sin β = 3 5 . \sin\alpha=\frac{12}{13},\qquad \sin\beta=\frac{3}{5}\;. sin α = 13 12 , sin β = 5 3 .
Now we need the cosines of the same angles. We state the Pythagorean identity first: for any angle θ \theta θ , sin 2 θ + cos 2 θ = 1 . \sin^{2}\theta+\cos^{2}\theta=1\;. sin 2 θ + cos 2 θ = 1 . Using it for α \alpha α we obtain
cos α = 1 − sin 2 α = 1 − ( 12 13 ) 2 = 1 − 144 169 = 25 169 = 5 13 . \cos\alpha=\sqrt{1-\sin^{2}\alpha}=\sqrt{1-\Bigl(\frac{12}{13}\Bigr)^{2}}
=\sqrt{1-\frac{144}{169}}
=\sqrt{\frac{25}{169}}
=\frac{5}{13}\;. cos α = 1 − sin 2 α = 1 − ( 13 12 ) 2 = 1 − 169 144 = 169 25 = 13 5 .
Applying the same identity to β \beta β gives
cos β = 1 − sin 2 β = 1 − ( 3 5 ) 2 = 1 − 9 25 = 16 25 = 4 5 . \cos\beta=\sqrt{1-\sin^{2}\beta}=\sqrt{1-\Bigl(\frac{3}{5}\Bigr)^{2}}
=\sqrt{1-\frac{9}{25}}
=\sqrt{\frac{16}{25}}
=\frac{4}{5}\;. cos β = 1 − sin 2 β = 1 − ( 5 3 ) 2 = 1 − 25 9 = 25 16 = 5 4 .
Our required expression is α − β . \alpha-\beta\;. α − β . To find it we use the sine of the difference of two angles. We first recall the formula:
sin ( α − β ) = sin α cos β − cos α sin β . \sin(\alpha-\beta)=\sin\alpha\cos\beta-\cos\alpha\sin\beta\;. sin ( α − β ) = sin α cos β − cos α sin β .
Substituting the values already obtained, we have
sin ( α − β ) = ( 12 13 ) ( 4 5 ) − ( 5 13 ) ( 3 5 ) = 48 65 − 15 65 = 33 65 . \sin(\alpha-\beta)=\Bigl(\frac{12}{13}\Bigr)\Bigl(\frac{4}{5}\Bigr)
-\Bigl(\frac{5}{13}\Bigr)\Bigl(\frac{3}{5}\Bigr)
=\frac{48}{65}-\frac{15}{65}
=\frac{33}{65}\;. sin ( α − β ) = ( 13 12 ) ( 5 4 ) − ( 13 5 ) ( 5 3 ) = 65 48 − 65 15 = 65 33 .
Because α − β \alpha-\beta α − β is the difference of two first-quadrant angles with α > β , \alpha \gt \beta, α > β , the result is still an acute angle. Hence the principal value of sin − 1 \sin^{-1} sin − 1 applies and we may write
α − β = sin − 1 33 65 . \alpha-\beta=\sin^{-1}\frac{33}{65}\;. α − β = sin − 1 65 33 .
So far we have sin − 1 12 13 − sin − 1 3 5 = sin − 1 33 65 . \sin^{-1}\frac{12}{13}-\sin^{-1}\frac{3}{5}
=\sin^{-1}\frac{33}{65}\;. sin − 1 13 12 − sin − 1 5 3 = sin − 1 65 33 .
This exact form does not appear among the options, therefore we transform it. We make use of the co-function relation that links sine and cosine:
For any angle θ \theta θ in [ − π 2 , π 2 ] , [-\tfrac{\pi}{2},\tfrac{\pi}{2}], [ − 2 π , 2 π ] ,
sin θ = cos ( π 2 − θ ) , \sin\theta=\cos\Bigl(\frac{\pi}{2}-\theta\Bigr)\;, sin θ = cos ( 2 π − θ ) ,
and, taking inverses,
sin − 1 x = π 2 − cos − 1 x for − 1 ≤ x ≤ 1 . \sin^{-1}x=\frac{\pi}{2}-\cos^{-1}x\quad\text{for }-1\le x\le1\;. sin − 1 x = 2 π − cos − 1 x for − 1 ≤ x ≤ 1 .
Instead of rewriting sin − 1 33 65 \sin^{-1}\frac{33}{65} sin − 1 65 33 in terms of cos − 1 , \cos^{-1}, cos − 1 , we notice there is an option of the form π 2 − sin − 1 ( something ) \tfrac{\pi}{2}-\sin^{-1}(\text{something}) 2 π − sin − 1 ( something ) . Hence we try to express sin − 1 33 65 \sin^{-1}\frac{33}{65} sin − 1 65 33 as π 2 − sin − 1 y \tfrac{\pi}{2}-\sin^{-1}\!y 2 π − sin − 1 y for a suitable y y y .
Set θ = sin − 1 33 65 . \theta=\sin^{-1}\frac{33}{65}\;. θ = sin − 1 65 33 . Then sin θ = 33 65 \sin\theta=\frac{33}{65} sin θ = 65 33 and 0 < θ < π 2 . 0\lt\theta\lt\frac{\pi}{2}\;. 0 < θ < 2 π . Using the Pythagorean identity once again, we compute
cos θ = 1 − sin 2 θ = 1 − ( 33 65 ) 2 = 1 − 1089 4225 = 3136 4225 = 56 65 . \cos\theta=\sqrt{1-\sin^{2}\theta}
=\sqrt{1-\Bigl(\frac{33}{65}\Bigr)^{2}}
=\sqrt{1-\frac{1089}{4225}}
=\sqrt{\frac{3136}{4225}}
=\frac{56}{65}\;. cos θ = 1 − sin 2 θ = 1 − ( 65 33 ) 2 = 1 − 4225 1089 = 4225 3136 = 65 56 .
But cos θ = sin ( π 2 − θ ) , \cos\theta=\sin\!\Bigl(\frac{\pi}{2}-\theta\Bigr)\;, cos θ = sin ( 2 π − θ ) , so
π 2 − θ = sin − 1 56 65 . \frac{\pi}{2}-\theta=\sin^{-1}\frac{56}{65}\;. 2 π − θ = sin − 1 65 56 .
Solving for θ , \theta, θ , we get
θ = π 2 − sin − 1 56 65 . \theta=\frac{\pi}{2}-\sin^{-1}\frac{56}{65}\;. θ = 2 π − sin − 1 65 56 .
Remembering that θ = α − β , \theta=\alpha-\beta, θ = α − β , we have proved
sin − 1 12 13 − sin − 1 3 5 = π 2 − sin − 1 56 65 . \sin^{-1}\frac{12}{13}-\sin^{-1}\frac{3}{5}
=\frac{\pi}{2}-\sin^{-1}\frac{56}{65}\;. sin − 1 13 12 − sin − 1 5 3 = 2 π − sin − 1 65 56 .
This expression is exactly the one listed in Option 4.
Hence, the correct answer is Option 4.