SEO is what makes the difference in my opinion. If you get it right, sometimes you can get a few hundred views on the first day (even 1K occasionally). Then the external views help boost the algorithm that starts to work to recommend the story internally, too.
But I agree with you on followers. Although, I’d say you need over 20K of them to see some sort of positive change. I mean, if I compare the views when I had 1000 followers vs. now with almost 2000 – there’s not much of a difference.
Of course, you need to count with the amount of fake followers, too.
I’d trade half of my followers for a SEO strategy that consistently works to be honest.