For three vectors to be mutually orthogonal, the dot product of every pair must vanish.
First verify that a and b are orthogonal:
a⋅b=(1)(2)+(−1)(4)+(2)(1)=2−4+2=0
Hence a⊥b as required.
Let c=λi^+j^+μk^.
Orthogonality of c with the other two vectors gives two linear equations.
1. a⋅c=0:
(i^−j^+2k^)⋅(λi^+j^+μk^)=0
⇒λ−1+2μ=0 −(1)
2. b⋅c=0:
(2i^+4j^+k^)⋅(λi^+j^+μk^)=0
⇒2λ+4+μ=0 −(2)
Solve the simultaneous equations (1) and (2).
From (1): λ+2μ=1 −(3)
From (2): 2λ+μ=−4 −(4)
Multiply (3) by 2 and subtract (4):
2λ+4μ=2
2λ+4μ=2<−(2λ+μ=−4)
3μ=6⇒μ=2
Substitute μ=2 into (3):
λ+2(2)=1⇒λ=−3
Therefore (λ,μ)=(−3,2).
Option D which is: (−3,2)