Htykuut

Show that for any given $3times 3$ complex matrix $A$, there exist a $3 times 3$ unitary matrix $U$, such that
$$U^{-1}AU=
begin{pmatrix}
* & 0 & * \
* & * & 0 \
* & 0 & * \
end{pmatrix}
$$
It is a question in the Chinese Ph.d Entrance Exam, I think it is unusual because I never thought this kind of form before and don't know how to get start. If we replace the unitary with invertible, then the question is already solved by the answer which PSG give. But I am quite sure Jordan is not necessary with unitary.

I have an idea that we can use some $2times 2$ unitary matrix to adjust the first principal submatrix of $A$ in order to make the second principal submatrix of $A$ into a normal matrix by changing the center entry of $A$ and then we diagonalize the second principal submatrix of $A$ and apply Schur Lemma on the first principal submatrix of $A$ to make $A$ into the form that we want.

Let $U=pmatrix{u&v&w}$. The statements that $U$ is unitary and $U^{-1}AU$ has the zero pattern in question mean that ${u,v,w}$ is an orthonormal basis of $mathbb C^3$, $v$ is an eigenvector of $A$ and $Awperp v$ (or equivalently, $wperp A^ast v$).

Since $A$ is a complex square matrix, it always has a unit eigenvector $v$. Let $V=operatorname{span}{v}$. As $V^perp$ is two-dimensional and the orthogonal projection of $operatorname{span}{A^ast v}$ on $V^perp$ is at most one-dimensional, there always exists some nonzero vector $win V^perp$ that is orthogonal to $A^ast v$. Normalise $w$ and extend ${v,w}$ into an orthonormal basis ${u,v,w}$. Take $U=pmatrix{u&v&w}$ and you are done.

$textbf{Case 1:}$ The characteristic polynomial is $(x-alpha)(x-beta)(x-gamma)$, where $alpha,beta$ and $gamma$ are distinct. $A$ is diagonalisable.

$textbf{Case 2:}$ The characteristic polynomial is $(x-alpha)^2(x-gamma)$, where $alpha$ and $gamma$ are distinct. If the minimal polynomial is $(x-alpha)(x-gamma)$, $A$ is diagonalisable and if the minimal polynomial is $(x-alpha)^2(x-gamma)$, $A$ has Jordon form $begin{pmatrix}alpha&0&0\1&alpha&0\0&0&gammaend{pmatrix}$

$textbf{Case 3:}$ The characteristic polynomial is $(x-alpha)^3$, if the minimal polynomial is $(x-alpha)$, $A$ is diagonalisable, if the minimal polynomial is $(x-alpha)^2$,$A$ has Jordon form $begin{pmatrix}alpha&0&0\1&alpha&0\0&0&alphaend{pmatrix}$ and if the minimal polynomial is $(x-alpha)^3$, $A$ has a cyclic vector say $v$ [i.e. ${v,Av, A^2 v}$ forms a basis]. Then with respect to this basis: ,$A$ has form $begin{pmatrix}0&0&*\1&0&*\0&1&*end{pmatrix}simbegin{pmatrix}*&0&0\*&1&0\*&0&1end{pmatrix} $

I will expand on PSG's case 3.

First, by permutations of last 2 rows and cols we can search for form
$$
begin{pmatrix}
*&*&0\
*&*&0\
*&0&*\
end{pmatrix}
$$
Jordan chain of vectors $V=(v_1, v_2, v_3)$ gives us the following form:
$$
V^TAV=begin{pmatrix}
lambda&0&0\
1&lambda&0\
0&1&lambda\
end{pmatrix}
$$

When we apply standard orthonormalization we will obtain orthonormal basis $V'=(v_1', v_2', v_3')$:
$$
V'^TAV'=begin{pmatrix}
lambda&0&0\
a_0&lambda&0\
b_0&c_0&lambda\
end{pmatrix}
$$
Notably, we can always choose phases of our basis (by multiplying by right $e^{iphi}$), so all $b_0$ and $c_0$ are real.
Now we want to rotate basis $V'$ around $v'_3$ by angle $theta$, obtaining new orthonomal basis $U(theta)$:
$$
U(theta)^TAU(theta)=begin{pmatrix}
lambda_1(theta)&a'(theta)&0\
a(theta)&lambda_2(theta)&0\
b(theta)&c(theta)&lambda\
end{pmatrix}
$$
Particularly, $c(theta)=c_0costheta-b_0sintheta$. It means there is such $theta$, that $c(theta)=0$.