A is a square matrix with AAT=I, i.e., A is orthogonal, so AT=A−1.
Find 21A[(A+AT)2+(A−AT)2].
Expand the squares.
(A+AT)2=A2+AAT+ATA+(AT)2
(A−AT)2=A2−AAT−ATA+(AT)2
Add the two expressions.
(A+AT)2+(A−AT)2=2A2+2(AT)2
Multiply by 21A.
21A⋅[2A2+2(AT)2]=A⋅[A2+(AT)2]=A3+A(AT)2
Simplify A(AT)2.
A(AT)2=(AAT)⋅AT=I⋅AT=AT
Final result.
21A[(A+AT)2+(A−AT)2]=A3+AT
The correct answer is Option 4: A3+AT.