Key idea: find each required term with the general term Tr+1=11Cr(first)11−r(second)r.
As printed, the second expression [ax2−bx1]11 has no x−7 term (22−3r=−7 gives no integer r). The original question uses [ax−bx21]11 for the second expansion, so we use that.
- In (ax2+bx1)11: power =22−3r=7, so r=5. Coefficient 11C5b5a6.
- In (ax−bx21)11: power =11−3r=−7, so r=6. Coefficient 11C6b6a5.
- 11C5=11C6, so b5a6=b6a5, giving ab=1.
Answer: (D) ab=1