Carter was born in Ferndale, Michigan. He started to play cello at the age of 10, but when his family moved to Detroit, he ran into difficulties regarding the racial stereotyping of classical musicians and instead moved to bass. He attended the historic Cass Technical High School in Detroit, and, later, the Eastman School of Music in Rochester, New York, where he played in its Philharmonic Orchestra. He gained his bachelor’s degree at Eastman in 1959, and in 1961 a master’s degree in double bass performance from the Manhattan School of Music in New York City.
His first jobs as a jazz musician were with Jaki Byard and Chico Hamilton. His first records were made with Eric Dolphy (another former member of Hamilton’s group) and Don Ellis, in 1960. His own first date as leader, Where?, with Eric Dolphy, Charlie Persip, Mal Waldron, George Duvivier, and a date also with Dolphy called Out There with George Duvivier and Roy Haynes and Carter on cello; its advanced harmonies and concepts were in step with the third stream movement.