The given vectors are
a=−i^+j^+3k^,b=i^+3j^+k^.
Let c=λa+μb where λ,μ∈R.
First write c in component form:
c=λ(−1,1,3)+μ(1,3,1)=(−λ+μ,λ+3μ,3λ+μ).
The two scalar-product conditions are
1. c⋅(3i^−6j^+2k^)=10,
2. c⋅(i^+j^+k^)=−2.
Condition 1:
(−λ+μ,λ+3μ,3λ+μ)⋅(3,−6,2)=10.
Compute the dot product:
3(−λ+μ)+(−6)(λ+3μ)+2(3λ+μ)=10.
(−3λ+3μ)+(−6λ−18μ)+(6λ+2μ)=10.
Combine like terms:
(−3λ−6λ+6λ)+(3μ−18μ+2μ)=10⟹−3λ−13μ=10.
⇒−(3λ+13μ)=10−(1)
Condition 2:
(−λ+μ,λ+3μ,3λ+μ)⋅(1,1,1)=−2.
Compute the dot product:
(−λ+μ)+(λ+3μ)+(3λ+μ)=−2.
Simplify:
(−λ+λ+3λ)+(μ+3μ+μ)=−2⟹3λ+5μ=−2−(2)
We now have the simultaneous linear equations
−3λ−13μ=10−(1)
3λ+5μ=−2−(2)
Add equations (1)+(2):
(−3λ+3λ)+(−13μ+5μ)=10−2⟹−8μ=8⟹μ=−1.
Substitute μ=−1 in (2):
3λ+5(−1)=−2⟹3λ−5=−2⟹3λ=3⟹λ=1.
Hence λ=1,μ=−1.
Therefore
c=1⋅a+(−1)⋅b=a−b.
Compute c explicitly:
c=(−1,1,3)−(1,3,1)=(−2,−2,2).
The square of its magnitude is
∣c∣2=(−2)2+(−2)2+22=4+4+4=12.
Hence, ∣c∣2=12.
Option B which is: 12