The first plane is x=5. Any point on the common line must therefore have x=5.
Let a general point on the common line be (x,y,z). Substituting x=5 in the second plane
2x−5ay+3z−2=0 gives
2(5)−5ay+3z−2=0
10−5ay+3z−2=0
8−5ay+3z=0
3z=5ay−8
z=35ay−38.
Hence every point on the line can be written as
(x,y,z)=(5,t,35at−38),
where t is a real parameter.
This same point must satisfy the third plane
3bx+y−3z=0.
Substituting x=5,y=t,z=35at−38 gives
3b(5)+t−3(35at−38)=0.
Simplify:
15b+t−5at+8=0.
For this to hold for every value of the parameter t, both the coefficient of t and the constant term must be zero:
1. Coefficient of t: 1−5a=0⟹a=51.
2. Constant term: 15b+8=0⟹b=−158.
Therefore(a,b)=(51,−158).
Option B which is: (51,−158)