There are a lot of issues with calories. First of all, you can have 2 people who weigh exactly the same, and are of the same age and sex, and they can both eat the exact same meals in a day. One will lose fat, one will gain fat. Everyone knows someone who can eat whatever s/he likes and not gain an ounce. Then, we also know other people who can just look at food and gain weight.
So calories aren't reliable and aren't a good way to go.
Furthermore, body fat and muscle is seriously controlled by hormones. I have a video on my YouTube channel that explains it well. First focus on hormonal optimization, digestive optimization, etc.
Unless you're a competitive bodybuilder or figure competitor, I wouldn't worry too much about calories.