Specialize in one domain at the expense of others?

I know focus (specialization) is important to obtain a high degree of skill and knowledge in a given domain.

However, I have a number of hobbies that are in a tangential domain to my specialization with little overlap. I always feel bad when I spend time doing them when I could be doing more work in my specialization. How do you overcome this? Is it something you can overcome, or just something you deal with?