We need to find the coefficient of x3 in (1+x)p(1−x)q, given that the coefficients of x and x2 are −3 and −5 respectively.
Since the coefficient of x is −3, [x1]:(1p)(1)+(1q)(−1)=p−q=−3⋯(1)
Next, since the coefficient of x2 is −5, [x2]:(2p)−pq+(2q)=2p(p−1)−pq+2q(q−1)=−5
p2−p+q2−q−2pq=−10
(p−q)2−(p+q)=−10
9−(p+q)=−10
p+q=19⋯(2)
From (1) and (2), p=8, q=11. Check: p,q≤15. ✓
Next, [x3]=∑j=03(3−jp)(jq)(−1)j
=(38)−(28)(111)+(18)(211)−(311)
=56−28×11+8×55−165
=56−308+440−165
=23
The correct answer is 23.