Let's zoom in on the Nyquist diagrams corresponding to different gains using the zoom command.

K = 4.5

sys=tf(50,[1 9 30 40]);
k=4.5;
nyquist(k*sys)

Note that the Nyquist diagram does not encircle the -1 point.

We will now look at our step response. Remember that we must look at the closed-loop step response.

sys_cl = feedback(k*sys,1);
step(sys_cl)

The system is stable.

K = 4.6

k=4.6;
nyquist(k*sys)

Note that the Nyquist diagram goes right through the -1 point.

We will now look at our step response. Remember that we must look at the closed-loop step response.

sys_cl = feedback(k*sys,1);
step(sys_cl)

The system is marginally stable.

K= 4.7

k=4.7;
nyquist(k*sys)

Note that the Nyquist diagram encircles the -1 point.

We will now look at our step response. Remember that we must look at the closed-loop step response.

sys_cl = feedback(k*sys,1);
t=0:0.1:150;
step(sys_cl,t)

The system is unstable.