For the linear sets of equations, you would first write {n1[t+1], n2[t+1]} as the matrix product of a matrix, M, and the vector {n1[t], n2[t]}. You then want to find Mt, which is easiest to find by determining the eigenvalues (placing these along the diagonal of a matrix D) and eigenvectors (placing these in the columns of a matrix A), and then determining the matrix product: ADtA-1. This product equals Mt and, when multiplied by {n1[0], n2[0]}, describes the position of the population at time t, {n1[t], n2[t]}. By analysing what happens over time, you can get a sense of the dynamics of the system and can determine the equilibria.
For non-linear sets of equations, there is no general transition matrix that does not contain n1[t] and/or n2[t]. However, if you find the equilibria of the system, you can approximate the dynamical equations near the equilibria by finding the local stability matrix (the matrix of partial derivatives evaluated at the equilibrium). If this local stability matrix has any eigenvalue greater than one in magnitude, the equilibrium is unstable. If all eigenvalues are less than one, the equilibrium is stable. If the leading eigenvalues are complex, then you would observe cycling around the equilibrium (these cycles may spiral in or out from the equilibrium depending on the eigenvalues). Although this analysis identifies the equilibria and their stability, it does not tell you what happens when the system starts very far from the equilibrium. It is generally good to simulate the dynamics away from equilibrium to infer the global behavior of the system.