You made a valiant effort to obtain the component representation of the third invariant of the second-order tensor. You are generally on the right track but you did not express the scalar product completely in the component form as you should have done. Since a scalar triple product is a scalar, you should be able to express it completely without any vector basis. I have decided to work it out fully for the class and my notes from now on will carry the explanation I am supplying below:
This should provide a complete answer to what you were trying to do. If you have further questions, please comment below.