Each area has its own skillset that takes time to learn and perfect. If you want to be the best at something, do that. I mean.. unless you want to be "okay" at all of them.
One major advantage I've found, besides accelerating my skillset, is that specializing has freed me from a lot of distractions. I've had opportunities to shoot other things.. family portraits, couples photos, babies, weddings...etc. I've done it a few times because of who asked me and it's usually a favor but, far more often, I've walked away. I'm a headshot photographer. But I'm happy to refer you to other great photographers I know who do exceptional work in what you're looking for. Do you want family photos? I know someone who does really great family photos. Do you want fitness photos? I know someone who is outstanding with fitness photos. (Incidentally, I talked to him about specializing almost two years ago. And he's really, really good at fitness photos.) If you want headshots, I know a guy who's working on being the best in the world. So do you.
I struggled with what I wanted to do until, through a series of events that I won't go into now, I found that headshots were my answer. At that point, ALL I wanted to do were headshots. They kind of consumed me.. and still do. But I don't want to just do headshots. I want to be the BOSS! The King of Headshots. I want to be the go-to guy... in the world. I have a long way to go in my world conquest but I've also come a long way in a short time. And, if all I do is headshots, I can spend the rest of my life doing so and there's enough to keep me always learning and always improving. I want to be the best. And that's all there is to it.
If you think about other fields and the people who are at the top of their game, it aligns. Photography isn't usually looked at in the same way. Michael Jordan spent his time practicing and playing basketball. It possible that he might have liked ice skating now and then.. but, if so, it was a break and distraction from the hoop. His focus.. being the best basketball player. The greatest actors don't just act but what they advertise and where they put most of their efforts.. acting. I'm sure the man had a life and a ton of things he did well, but what is Robert DeNiro known for? (Jimmie Dean is an exception. I like sausage. There are exceptions to "the specialization rule". There are exceptions to every rule. But they are the exceptions.)
In photography, the line can blur between what we do professionally and what we do as a hobby, especially if our hobby is also what we do as a business. But I don't confuse the two and, at the end of the day, I shoot headshots. That's what I specialize in. And I will be the best. There is no Plan B.