Serena Williams returns the ball to Sofia Kenin during their women's singles third round match in Paris on Saturday. AFP
It was 37-year-old Serena's earliest exit at the Slams since a third round loss to Alize Cornet at Wimbledon in 2014.
Kenin, the 20-year-old world number 35, will face eighth seed Ashleigh Barty of Australia for a place in the quarter-finals.
Three-time Roland Garros champion Williams was attempting to equal Margaret Court's record of 24 majors.
"She played really well, especially in the first set where she hit pretty much within an inch of the line all the time," said Williams.
Sofia Kenin reacts during her third round match against Serena Williams. Reuters
The America veteran, who played just one clay court match in Rome before suffering a knee injury in the run up to the French Open, said she had expected to progress further in Paris.
"I would have expected to have gotten past the the third round. If someone said I would only get this far, I'd have said they were a liar. "But it is what it is."
"I haven't been on the court as much as I would have wanted to and it's been a gruelling season for me." Kenin wasn't even born when Williams made her Paris debut in 1998. "There's a lot of emotions now," said the Russian-born player.
Sofia Kenin and Serena Williams shake hands at the end of their women's singles third round match. AFP
"Serena is such a great champion and I have all the respect for her. I had to fight for every point."
Williams, who has still to add to her majors collection since she returned to the tour after giving birth to her daughter, fired 30 winners and 34 unforced errors.
Kenin committed only half as many. Kenin broke twice in the first set, celebrating every mini-triumph with a 'c'mon'.
She went 3-1 ahead in the second set before Williams, the 2002, 2013 and 2015 champion, clawed her way back to 3-3.
But Kenin grabbed what proved to be the crucial break in the 11th game and held her nerve to take victory on a second match point when Williams fired a backhand long.
Tennis star Serena Williams has revealed that she has apologised to Japan's Naomi Osaka who defeated her in the finals of the US Open last year.
London: Andy Murray’s management team confirmed on Tuesday that the three-times Grand Slam singles champion will be joining forces with Serena Williams in the mixed doubles at Wimbledon this year. The two-time Wimbledon champion’s partnership with seven-time winner Williams make them
Eight-time Wimbledon champion Roger Federer powered his way into the fourth round for the 17th time of the championships on Saturday.
Naomi Osaka staged a second successive Roland Garros great escape on Thursday as fellow top seed Novak Djokovic and Serena Williams strolled into the last 32. World number one Osaka, bidding to add the French Open to her US and Australian Open titles, came back from a set and 2-4 down
Atletico Madrid failed to win for a third straight game in all competitions as an inspired performance from Ruben Blanco secured Celta Vigo an unexpected 0-0 draw at the Wanda Metropolitano.
Leipzig retained top spot in the German League with a facile 3-0 win over Werder Bremen on Saturday. While Willi Orban opened Leipzig’s account in the 13th minute, Marcel Sabitzer in the 35th and Marcelo Saracchi in the 83rd minute were the other scorers for the winners on the night.
Defending champions New Zealand put the rugby World Cup on notice with an impressive 23-13 win over South Africa on Saturday after France and Australia both survived major scares.