This common phrase is believed to have originated from the literary fairy tale The Ugly Duckling, by Hans Christian Anderson.
From the looks of it, the term began being used in a figurative sense and unrelated to the story sometime later. An example of that is seen in the Stevens Point Daily Journal, 1883:
"She gazed me at through her gold-rimmed glasses then remarked to Mrs. Maria, as though I were
not present: 'If this is the ugly duckling, 'pon my honor she does not strike me as so plain as she has
Read about it all here.