We are given AB=5, ∠B=cos−1(53) (so cosB=53 and sinB=54), and circumradius R=5.
By the law of sines, the side opposite to B (i.e., AC) satisfies:
sinBAC=2R⟹AC=2×5×54=8.
Now applying the law of cosines with BC=a:
AC2=AB2+BC2−2⋅AB⋅BC⋅cosB
64=25+a2−2(5)(a)(53)=25+a2−6a.
This gives a2−6a−39=0, so a=26±36+156=26±192=3±43.
Since a=BC>0, we have BC=3+43.
The area of triangle ABC is:
Area=21⋅AB⋅BC⋅sinB=21⋅5⋅(3+43)⋅54=2(3+43)=6+83.