It works the other way around too. A pure state is rank $1$, and any channel with more than one Kraus operator will give a higher-rank Choi matrix, which can be easily seen from the definition.
You can also work it out on a different condition for pure states:
For any pure state $\rho$ we have $\rho^{2} = \rho.$
$$
(J(\mathcal{E}))^{2} = \sum_{ij}\sum_{kl} \mathcal{E}(|i\rangle \langle j|)\mathcal{E}(|k\rangle \langle l|) \otimes |i\rangle \langle j|k\rangle\langle l| = \sum_{ijl}\mathcal{E}(|i\rangle \langle j|)\mathcal{E}(|j\rangle \langle l|) \otimes |i\rangle \langle l|
$$
so if $J(\mathcal{E})$ is pure then:
$$
\sum_{ij}\mathcal{E}(|i\rangle \langle j|) \otimes |i\rangle \langle j| = \sum_{ijl}\mathcal{E}(|i\rangle \langle j|)\mathcal{E}(|j\rangle \langle l|) \otimes |i\rangle \langle l|
$$
which, when relabeling $j <-> l$ on the right hand side, leads to:
$$
\sum_{ij}\mathcal{E}(|i\rangle \langle j|) \otimes |i\rangle \langle j| = \sum_{l}\sum_{ij}\mathcal{E}(|i\rangle \langle l|)\mathcal{E}(|l\rangle \langle j|) \otimes |i\rangle \langle j|
$$
Since all different $|i\rangle\langle j |$ are orthogonal, this needs to hold term-by-term:
$$
\mathcal{E}(|i\rangle \langle j|) = \sum_{l}\mathcal{E}(|i\rangle \langle l|)\mathcal{E}(|l\rangle \langle j|).
$$
Writing $\mathcal{E}$ in it's Kraus decomposition $\{A_{k}\}$ sheds some extra light:
$$
\sum_{k} A_{k}|i\rangle \langle j | A_{k}^{\dagger} = \sum_{l} \sum_{k'}\sum_{k''} A_{k'}|i\rangle \langle l | A_{k'}^{\dagger} A_{k''}|l\rangle \langle j | A_{k''}^{\dagger}
$$
noting that $\sum_{l} \langle l| A^{\dagger}_{k'} A_{k''}|l\rangle = \mathrm{tr}[A^{\dagger}_{k'} A_{k''}]$, we get:
$$
\sum_{k} A_{k}|i\rangle \langle j | A_{k}^{\dagger} = \sum_{k'}\sum_{k''}\mathrm{tr}[A^{\dagger}_{k'} A_{k''}] A_{k'}|i\rangle \langle j | A_{k''}^{\dagger}
$$
and taking the trace and using its cyclic property on either side we get:
$$
\sum_{k'k''}\delta_{k'k''} \langle j | A_{k''}^{\dagger}A_{k'}|i\rangle = \sum_{k'}\sum_{k''}\mathrm{tr}[A^{\dagger}_{k'} A_{k''}] \langle j | A_{k''}^{\dagger}A_{k'}|i\rangle
$$
Importantly, this works for every $|i\rangle, | j \rangle$, so the above equation can only hold if $\delta_{k'k''} = \mathrm{tr}[A^{\dagger}_{k'} A_{k''}]$, which is evidently only true if the Kraus operators are orthogonal and of unit length. But then they are unitary, which means there is only a single Kraus operator, necessarily unitary.