In comparison to a lot of idioms this one is rather new. It appeared in the 1970's and the first use of the phrase found is from the Maryland newspaper The Capital, January 1973, in a piece headed 'How To Be Liberated':
The major goal of each of these role changes is to give a woman time to herself, Ms. Burton explained."A woman's right and responsibility is to be self fulfilling," she said. She gives "quality time" rather than "quantity time" to each task, whether it be writing, cleaning the house or tending the children.
The phrase today has moved away from being possibly task related and is generally meant in a relationship capacity - i.e. giving undivided attention to a person.
And just so we're clear, this is what it is not...