Ivankov was born on January 2, 1940 in Georgia to ethnically Russian parents, Olga Gostasvits and Bernard Royal-Ivankov, at the time part of the Soviet Union and grew up in Moscow. He was an amateur wrestler in his youth and served his first prison time for his participation in a bar fight, in which he claimed he was defending the honor of a woman. After his release he began to move up in the criminal world, selling goods on the black market.
Later Ivankov became involved in gang activity. His gang used forged police documents to enter houses and then burglarize them. In 1982 authorities had finally caught up with him and he was arrested on firearms, forgery and drug-trafficking charges. Though he was sentenced to fourteen years he was released in 1991, reportedly thanks to the intervention of a powerful politician and a bribed judge of the Russian supreme court. It was during these years of imprisonment that he was initiated as a vor v zakone.