We have the three vectors
a ⃗ = i ^ + 2 j ^ + 4 k ^ , b ⃗ = i ^ + λ j ^ + 4 k ^ , c ⃗ = 2 i ^ + 4 j ^ + ( λ 2 − 1 ) k ^ . \vec a = \hat i + 2\hat j + 4\hat k,\qquad
\vec b = \hat i + \lambda\hat j + 4\hat k,\qquad
\vec c = 2\hat i + 4\hat j + (\lambda^2-1)\hat k. a = i ^ + 2 j ^ + 4 k ^ , b = i ^ + λ j ^ + 4 k ^ , c = 2 i ^ + 4 j ^ + ( λ 2 − 1 ) k ^ .
Since the vectors are given to be coplanar, their scalar triple product must be zero. The scalar triple product formula is
a ⃗ ⋅ ( b ⃗ × c ⃗ ) = 0. \vec a\cdot(\vec b\times\vec c)=0. a ⋅ ( b × c ) = 0.
First we compute the cross product b ⃗ × c ⃗ . \vec b\times\vec c. b × c . Using the determinant form of the cross product,
b ⃗ × c ⃗ = ∣ i ^ j ^ k ^ 1 λ 4 2 4 λ 2 − 1 ∣ . \vec b\times\vec c=\begin{vmatrix}
\hat i & \hat j & \hat k\\
1 & \lambda & 4\\
2 & 4 & \lambda^2-1
\end{vmatrix}. b × c = i ^ 1 2 j ^ λ 4 k ^ 4 λ 2 − 1 .
Expanding the determinant along the first row:
b ⃗ × c ⃗ = i ^ [ λ ( λ 2 − 1 ) − 4 ⋅ 4 ] − j ^ [ 1 ( λ 2 − 1 ) − 4 ⋅ 2 ] + k ^ [ 1 ⋅ 4 − λ ⋅ 2 ] . \vec b\times\vec c
= \hat i\bigl[\lambda(\lambda^2-1)-4\cdot4\bigr]
-\hat j\bigl[1(\lambda^2-1)-4\cdot2\bigr]
+\hat k\bigl[1\cdot4-\lambda\cdot2\bigr]. b × c = i ^ [ λ ( λ 2 − 1 ) − 4 ⋅ 4 ] − j ^ [ 1 ( λ 2 − 1 ) − 4 ⋅ 2 ] + k ^ [ 1 ⋅ 4 − λ ⋅ 2 ] .
Now we simplify each component.
λ ( λ 2 − 1 ) − 16 = λ 3 − λ − 16 , \lambda(\lambda^2-1)-16=\lambda^3-\lambda-16, λ ( λ 2 − 1 ) − 16 = λ 3 − λ − 16 ,
1 ( λ 2 − 1 ) − 8 = λ 2 − 1 − 8 = λ 2 − 9 , 1(\lambda^2-1)-8=\lambda^2-1-8=\lambda^2-9, 1 ( λ 2 − 1 ) − 8 = λ 2 − 1 − 8 = λ 2 − 9 ,
4 − 2 λ . 4-2\lambda. 4 − 2 λ .
Thus
b ⃗ × c ⃗ = ( λ 3 − λ − 16 ) i ^ − ( λ 2 − 9 ) j ^ + ( 4 − 2 λ ) k ^ . \vec b\times\vec c=(\lambda^3-\lambda-16)\hat i-(\lambda^2-9)\hat j+(4-2\lambda)\hat k. b × c = ( λ 3 − λ − 16 ) i ^ − ( λ 2 − 9 ) j ^ + ( 4 − 2 λ ) k ^ .
Next we take the dot product of this with a ⃗ = 1 i ^ + 2 j ^ + 4 k ^ . \vec a=1\hat i+2\hat j+4\hat k. a = 1 i ^ + 2 j ^ + 4 k ^ .
a ⃗ ⋅ ( b ⃗ × c ⃗ ) = 1 ⋅ ( λ 3 − λ − 16 ) + 2 ⋅ [ − ( λ 2 − 9 ) ] + 4 ⋅ ( 4 − 2 λ ) . \vec a\cdot(\vec b\times\vec c)
=1\cdot(\lambda^3-\lambda-16)
+2\cdot\bigl[-(\lambda^2-9)\bigr]
+4\cdot(4-2\lambda). a ⋅ ( b × c ) = 1 ⋅ ( λ 3 − λ − 16 ) + 2 ⋅ [ − ( λ 2 − 9 ) ] + 4 ⋅ ( 4 − 2 λ ) .
Carrying out the multiplication term by term:
1 ( λ 3 − λ − 16 ) = λ 3 − λ − 16 , 1(\lambda^3-\lambda-16)=\lambda^3-\lambda-16, 1 ( λ 3 − λ − 16 ) = λ 3 − λ − 16 ,
2 [ − ( λ 2 − 9 ) ] = − 2 λ 2 + 18 , 2\bigl[-(\lambda^2-9)\bigr]=-2\lambda^2+18, 2 [ − ( λ 2 − 9 ) ] = − 2 λ 2 + 18 ,
4 ( 4 − 2 λ ) = 16 − 8 λ . 4(4-2\lambda)=16-8\lambda. 4 ( 4 − 2 λ ) = 16 − 8 λ .
Adding all these contributions gives
λ 3 − λ − 16 − 2 λ 2 + 18 + 16 − 8 λ = λ 3 − 2 λ 2 − 9 λ + 18. \lambda^3-\lambda-16-2\lambda^2+18+16-8\lambda
=\lambda^3-2\lambda^2-9\lambda+18. λ 3 − λ − 16 − 2 λ 2 + 18 + 16 − 8 λ = λ 3 − 2 λ 2 − 9 λ + 18.
For coplanarity this expression must be zero, so
λ 3 − 2 λ 2 − 9 λ + 18 = 0. \lambda^3-2\lambda^2-9\lambda+18=0. λ 3 − 2 λ 2 − 9 λ + 18 = 0.
We factor the cubic. Trying simple factors, λ = 3 \lambda=3 λ = 3 satisfies the equation, so we divide by ( λ − 3 ) (\lambda-3) ( λ − 3 ) :
λ 3 − 2 λ 2 − 9 λ + 18 = ( λ − 3 ) ( λ 2 + λ − 6 ) . \lambda^3-2\lambda^2-9\lambda+18=(\lambda-3)(\lambda^2+\lambda-6). λ 3 − 2 λ 2 − 9 λ + 18 = ( λ − 3 ) ( λ 2 + λ − 6 ) .
Further factoring λ 2 + λ − 6 = ( λ + 3 ) ( λ − 2 ) . \lambda^2+\lambda-6=(\lambda+3)(\lambda-2). λ 2 + λ − 6 = ( λ + 3 ) ( λ − 2 ) .
Hence
λ 3 − 2 λ 2 − 9 λ + 18 = ( λ − 3 ) ( λ + 3 ) ( λ − 2 ) = 0 , \lambda^3-2\lambda^2-9\lambda+18=(\lambda-3)(\lambda+3)(\lambda-2)=0, λ 3 − 2 λ 2 − 9 λ + 18 = ( λ − 3 ) ( λ + 3 ) ( λ − 2 ) = 0 ,
giving three possible values
λ = 3 , λ = − 3 , λ = 2. \lambda=3,\qquad\lambda=-3,\qquad\lambda=2. λ = 3 , λ = − 3 , λ = 2.
We now turn to the required cross product a ⃗ × c ⃗ . \vec a\times\vec c. a × c . Again we use the determinant:
a ⃗ × c ⃗ = ∣ i ^ j ^ k ^ 1 2 4 2 4 λ 2 − 1 ∣ . \vec a\times\vec c=\begin{vmatrix}
\hat i & \hat j & \hat k\\
1 & 2 & 4\\
2 & 4 & \lambda^2-1
\end{vmatrix}. a × c = i ^ 1 2 j ^ 2 4 k ^ 4 λ 2 − 1 .
Expanding along the first row:
a ⃗ × c ⃗ = i ^ [ 2 ( λ 2 − 1 ) − 4 ⋅ 4 ] − j ^ [ 1 ( λ 2 − 1 ) − 4 ⋅ 2 ] + k ^ [ 1 ⋅ 4 − 2 ⋅ 2 ] . \vec a\times\vec c
= \hat i\bigl[2(\lambda^2-1)-4\cdot4\bigr]
-\hat j\bigl[1(\lambda^2-1)-4\cdot2\bigr]
+\hat k\bigl[1\cdot4-2\cdot2\bigr]. a × c = i ^ [ 2 ( λ 2 − 1 ) − 4 ⋅ 4 ] − j ^ [ 1 ( λ 2 − 1 ) − 4 ⋅ 2 ] + k ^ [ 1 ⋅ 4 − 2 ⋅ 2 ] .
We simplify each bracket:
2 ( λ 2 − 1 ) − 16 = 2 λ 2 − 2 − 16 = 2 λ 2 − 18 , 2(\lambda^2-1)-16=2\lambda^2-2-16=2\lambda^2-18, 2 ( λ 2 − 1 ) − 16 = 2 λ 2 − 2 − 16 = 2 λ 2 − 18 ,
1 ( λ 2 − 1 ) − 8 = λ 2 − 9 , 1(\lambda^2-1)-8=\lambda^2-9, 1 ( λ 2 − 1 ) − 8 = λ 2 − 9 ,
4 − 4 = 0. 4-4=0. 4 − 4 = 0.
Therefore
a ⃗ × c ⃗ = ( 2 λ 2 − 18 ) i ^ − ( λ 2 − 9 ) j ^ + 0 k ^ . \vec a\times\vec c=(2\lambda^2-18)\hat i-(\lambda^2-9)\hat j+0\hat k. a × c = ( 2 λ 2 − 18 ) i ^ − ( λ 2 − 9 ) j ^ + 0 k ^ .
Factoring out the common term ( λ 2 − 9 ) (\lambda^2-9) ( λ 2 − 9 ) :
a ⃗ × c ⃗ = ( λ 2 − 9 ) ( 2 i ^ − j ^ ) . \vec a\times\vec c=(\lambda^2-9)\bigl(2\hat i-\hat j\bigr). a × c = ( λ 2 − 9 ) ( 2 i ^ − j ^ ) .
This result shows that if λ 2 − 9 = 0 \lambda^2-9=0 λ 2 − 9 = 0 (i.e., λ = ± 3 \lambda=\pm3 λ = ± 3 ) the cross product vanishes, but the question asks for a non-zero vector. Hence we must choose the remaining admissible value λ = 2. \lambda=2. λ = 2.
Substituting λ = 2 \lambda=2 λ = 2 gives
λ 2 − 9 = 4 − 9 = − 5 , \lambda^2-9=4-9=-5, λ 2 − 9 = 4 − 9 = − 5 ,
and hence
a ⃗ × c ⃗ = − 5 ( 2 i ^ − j ^ ) = − 10 i ^ + 5 j ^ . \vec a\times\vec c=-5\,(2\hat i-\hat j)=-10\hat i+5\hat j. a × c = − 5 ( 2 i ^ − j ^ ) = − 10 i ^ + 5 j ^ .
This matches the vector in Option D.
Hence, the correct answer is Option D.