We have f(x)=6x−α5x+3. We need (f∘f)(x)=x for all x∈R∖{6α}.
Computing f(f(x)) directly: let u=f(x)=6x−α5x+3, then
f(u)=6u−α5u+3=6⋅6x−α5x+3−α5⋅6x−α5x+3+3.
Multiplying numerator and denominator by (6x−α):
f(f(x))=6(5x+3)−α(6x−α)5(5x+3)+3(6x−α)=30x+18−6αx+α225x+15+18x−3α.
For this to equal x for all valid x, we need (30−6α)x+(18+α2)(25+18)x+(15−3α)=x, i.e.,
(43)x+(15−3α)=x[(30−6α)x+(18+α2)].
For this to hold as an identity, the coefficient of x2 on the right must be zero: 30−6α=0, but this gives α=5. Let us verify: with α=5, the numerator becomes (25+18)x+(15−15)=43x and the denominator becomes (30−30)x+(18+25)=43, so f(f(x))=4343x=x. ✓
Therefore α=5.