Is it creepy for a grown man to hang out in the kiddie pool?

I have two kids and when I take them to the gym we go to the kiddie pool. So the other day I went to the pool by myself and out of habit I sat in the kiddie pool and watched the kids play. Someone alerted the life guard and I got kicked out of the pool! I wasn't doing anything, I'm not a pedophile. She said it was "creepy" for a guy wearing banana yellow speedos to be sitting in the kiddie pool by himself watching the other kids play. I disagree - it's completely innocent especially when I go there all the time with my own kids!