After Gloria Vanderbilt's father died, she was left in the "care" of her mother. However, Gloria Morgan Vanderbilt was extremely absent in and appeared to care nothing about the life of her daughter. Vanderbilt would frequently come home drunk after a night of partying, if she even returned home at all. Vanderbilt's daughter often said how she thought of her grandmother, Naney, and her governess, Dodo as her real parents because they were the ones who took care of her and loved her.
In The Rainbow Comes and Goes, Vanderbilt writes, "In my young eyes, they were a couple: Mother Dodo and Father Naney. They were my parents, wrapping me with love as though with swaddling cloth, while my so-called real mother and father left for a ten-month European vacation soon after I was born."
While Vanderbilt, her mother, Naney, and Dodo were all living together in Europe, Naney and Dodo began to plan out how they could separate Gloria Morgan Vanderbilt from her neglected daughter. They came up with the solution that Vanderbilt could live with her late father's sister, Gertrude Vanderbilt Whitney. Her mother would not give up custody of Vanderbilt because she did not want her reputation tarnished. Gloria Morgan Vanderbilt then lost custody of her daughter to Gertrude in what the newspapers called the "Matter of Vanderbilt" or "Trial of the Century." The most prominent factor that led to Gertrude gaining custody of Vanderbilt was Vanderbilt herself. Naney and Dodo had instructed her to say very specific things about her mother during the trial that persuaded the judge to rule in favor of Gertrude. Although the judge ruled in favor of Gertrude, the Aunt began to ignore Vanderbilt after the custody battle, and Vanderbilt was sent to see her mother in Manhattan on the weekends.