Since the circumcentre of △ABC is at the origin O, we have ∣OA∣=∣OB∣=∣OC∣=R (circumradius). The lines OA, OB, OC have slopes tanα, tanβ, tanγ respectively, so we can write A=R(cosα,sinα), B=R(cosβ,sinβ), C=R(cosγ,sinγ).
The orthocentre of a triangle inscribed in a circle of radius R centred at the origin is H=A+B+C=R(cosα+cosβ+cosγ,sinα+sinβ+sinγ).
Since the orthocentre lies on the y-axis, its x-coordinate is zero: cosα+cosβ+cosγ=0.
Now we evaluate the given expression. Using the identity cos3θ=4cos3θ−3cosθ, we get cos3α+cos3β+cos3γ=4(cos3α+cos3β+cos3γ)−3(cosα+cosβ+cosγ).
Since cosα+cosβ+cosγ=0, the second term vanishes. For the first term, we use the identity: if p+q+r=0, then p3+q3+r3=3pqr. With p=cosα, q=cosβ, r=cosγ, we get cos3α+cos3β+cos3γ=3cosαcosβcosγ.
Therefore cos3α+cos3β+cos3γ=4⋅3cosαcosβcosγ=12cosαcosβcosγ.
The required expression is cosα⋅cosβ⋅cosγcos3α+cos3β+cos3γ=cosαcosβcosγ12cosαcosβcosγ=12.
Therefore (cosα⋅cosβ⋅cosγcos3α+cos3β+cos3γ)2=122=144.