TL;DR: When your application becomes an intimidating framework.
From the application developer's point of view, that's when customization would allow the application to break in such ways that support becomes impossible, either because the user is unable to report how he set up the application or because various settings interactions become too tortuous to make heads or tails. Think the customization system thoroughly and allow for information to get back to you in a meaningful way.
From the application user's point of view, that's when the user feels like setting up the application is daunting, often because it's akin to programming, for a loose definition of "programming" (This includes GUI-oriented programming or Blinkenswitches).
Yes the line is a blur.
Yes sometimes a good code or GUI (re)design can make the application switch board even with the same customizable feature set.
Create a learning curve between "casual", "advanced" and "expert" settings. It could go all the way to providing an API and/or scripting. All users are not starting on an equal foot: a tiered system will make each one feel at home. It can also create a sense of progress and achievement when a beginner switches from "curated" to "advanced".
Good examples in various areas include Firefox (preferences, about:config, userchrome.css&al.), Chrome (basic settings vs "Under the hood"), Mac OS X (pref panes, "defaults(1)", applescript/automator), or even Vim's vimrc. Bad examples include any application whose settings pane feels like a maze. I'm sure you could name half a dozen from the top of your head (unless they traumatized you into forgetting them).