For infinitely many solutions, D=D1=D2=D3=0.
Coefficient matrix determinant:
D=2211−52−1λ−5
=2(25−2λ)−1(−10−λ)+(−1)(4+5)
=50−4λ+10+λ−9=51−3λ
For D=0: λ=17
Now find μ using D1=0:
D1=5μ71−52−117−5
=5(25−34)−1(−5μ−119)+(−1)(2μ+35)
=5(−9)+5μ+119−2μ−35
=−45+3μ+84=3μ+39
D1=0: μ=−13
(λ+μ)2+(λ−μ)2=(17−13)2+(17+13)2=16+900=916
This matches option 2: 916.