We need to find the relation between a and b given that the coefficient of x7 in (ax2+2bx1)11 equals the coefficient of x−7 in (ax−3bx21)11.
To determine the coefficient of x7 in the expansion of (ax2+2bx1)11, consider the general term
Tr+1=(r11)(ax2)11−r(2bx1)r=(r11)(2b)ra11−r⋅x22−3r.
Requiring the exponent to be 7 gives 22−3r=7⇒r=5, so the coefficient is
(511)(2b)5a6=32b5462⋅a6.
Similarly, for the coefficient of x−7 in the expansion of (ax−3bx21)11, the general term is
Tr+1=(r11)(ax)11−r(3bx2−1)r=(r11)(3b)ra11−r(−1)r⋅x11−3r.
Setting 11−3r=−7 gives r=6 and the coefficient becomes
(611)(3b)6a5⋅(−1)6=729b6462⋅a5.
Equating these two coefficients yields
32b5462⋅a6=729b6462⋅a5
Cancelling 462⋅a5/b5 from both sides leads to
32a=729b1
729ab=32
Therefore, the required relation is 729ab=32, which corresponds to Option A.