Let the three given vectors be
v1=ai^+aj^+ck^=(a,a,c),
v2=i^+k^=(1,0,1),
v3=ci^+cj^+bk^=(c,c,b).
For v1,v2,v3 to lie in the same plane, their scalar triple product must vanish:
[v1v2v3]=0.
Write the scalar triple product as a determinant:
a1ca0cc1b=0.
Evaluate the determinant:
a1ca0cc1b=a0c1b−a1c1b+c1c0c=a(0⋅b−1⋅c)−a(1⋅b−1⋅c)+c(1⋅c−0⋅c)=a(−c)−a(b−c)+c2=−ac−ab+ac+c2=c2−ab.$
The coplanarity condition therefore gives
c2−ab=0⟹c2=ab.\
Since a,b,c are non-negative and distinct, we take the positive square root:
c=ab,
which is exactly the geometric mean of a and b.
Hence the correct option is:
Option A which is: the Geometric Mean of a and b.