I agree with you 100% of the way. He's much smarter than he likes to act, and I'm pretty positive that he's not as much of a lecher as he wants people to think. When he finds out that the girl in the dungeon from volume one is a middleschooler, his willingness to help her out and let her have a good time doesn't change one bit. If he were truly such a playboy, I expect his reaction would have been more like "Damn it!" and less like "Where are you going? We want to help you."
I wish Pai could see that...

EDIT: ...Sixth time in three days that I've knocked the thread over onto the next page. I'm gonna stop counting now, but suffice it to say that it REALLY drives me nuts when I'm the one who does that. ><