We recall a basic property of determinants: for any two square matrices of the same order we have the formula det(AB)=det(A)det(B). We also know that det(AT)=det(A) and det(A−1)=det(A)1. All matrices here are of order 3×3, so these facts apply directly.
First, we are told that
det(ABAT)=8.
Using the determinant product rule and the fact that a transpose does not change a determinant, we write
det(ABAT)=det(A)det(B)det(AT)=det(A)det(B)det(A).
Simplifying, this becomes
det(ABAT)=(det(A))2det(B)=8.
Next, we are also given
det(AB−1)=8.
Again applying the product rule and the inverse‐determinant relation, we have
det(AB−1)=det(A)det(B−1)=det(A)det(B)1=8.
To keep the algebra clear, let us denote
x=det(A),y=det(B).
The two pieces of information now translate to the pair of equations
x2y=8(1)
and
yx=8(2).
From equation (2) we immediately find
y=8x.
Substituting this value of y into equation (1) gives
x2(8x)=8.
This simplifies step by step:
8x3=8,
so
x3=64,
and therefore
x=4.
Using x=4 in the relation y=8x we get
y=84=21.
Now we need the determinant
det(BA−1BT).
Applying the same determinant properties, we write
det(BA−1BT)=det(B)det(A−1)det(BT).
Since det(BT)=det(B) and det(A−1)=det(A)1, this becomes
det(BA−1BT)=y×x1×y=xy2.
Substituting x=4 and y=21, we calculate
det(BA−1BT)=4(21)2=441=161.
Hence, the correct answer is Option C.