We have A=[i−i−ii]. First, we compute A2=A⋅A=[i2+i2−i2−i2−i2−i2i2+i2]=[−222−2].
Let B=[1−1−11], so A2=−2B. We note that B2=[2−2−22]=2B. Therefore A4=(A2)2=4B2=8B and A8=(A4)2=64B2=128B=[128−128−128128].
The system A8[xy]=[864] becomes 128(x−y)=8 and 128(−x+y)=64, which gives x−y=161 and −x+y=21.
Adding these two equations: 0=161+21=169, which is a contradiction. Therefore, the system has no solution.