When you say this, do you mean to make someone feel smaller? Or is that just an accident?
It's a powerful method for making folks who create stuff feel bad. We're socialized to see working as good so this statement weaponizes the concept of free-time. Instead of recognizing that people have different levels of interest in different things, this places a clear border between 'useful' and 'screwing around' like they're objectively measurable things.
I build stuff, I create thing, I follow my passions and when someone suggests that I have 'too much free time' because I'm doing something different than them, I've got to disagree. No, we all have the same amount of fixed time, I'm just organizing how I use mine differently.
"Well, I don't mean it that way-" stop. For the sake of communication, your intention is not going to change how plenty of folks will hear it. Consider the possibility that you're unknowingly causing hurt with an innocent sounding phrase and that whether you intended to or not doesn't change that fact. Someone who slaves over a drawing or spends hours assembling a model or puts their heart and soul into an elaborate cosplay setup and hears an implication that it was just messing around because it's not Useful Work can't read your mind.
My request to you: stop accidentally shaming people (because 'too much free time' implies laziness) for making things. If you want to train people not to show you their creations and to make them feel bad about themselves, I guess this is a good way to do it. If, however, you'd rather _not_ be the person who punishes those around them for choosing to open themselves to a little vulnerability by showing you something into which they invested their hearts and souls, consider deleting this phrase from your vocabulary.
If you have the time, that is.