Serena Williams returns the ball to Sofia Kenin during their women's singles third round match in Paris on Saturday. AFP
It was 37-year-old Serena's earliest exit at the Slams since a third round loss to Alize Cornet at Wimbledon in 2014.
Kenin, the 20-year-old world number 35, will face eighth seed Ashleigh Barty of Australia for a place in the quarter-finals.
Three-time Roland Garros champion Williams was attempting to equal Margaret Court's record of 24 majors.
"She played really well, especially in the first set where she hit pretty much within an inch of the line all the time," said Williams.
Sofia Kenin reacts during her third round match against Serena Williams. Reuters
The America veteran, who played just one clay court match in Rome before suffering a knee injury in the run up to the French Open, said she had expected to progress further in Paris.
"I would have expected to have gotten past the the third round. If someone said I would only get this far, I'd have said they were a liar. "But it is what it is."
"I haven't been on the court as much as I would have wanted to and it's been a gruelling season for me." Kenin wasn't even born when Williams made her Paris debut in 1998. "There's a lot of emotions now," said the Russian-born player.
Sofia Kenin and Serena Williams shake hands at the end of their women's singles third round match. AFP
"Serena is such a great champion and I have all the respect for her. I had to fight for every point."
Williams, who has still to add to her majors collection since she returned to the tour after giving birth to her daughter, fired 30 winners and 34 unforced errors.
Kenin committed only half as many. Kenin broke twice in the first set, celebrating every mini-triumph with a 'c'mon'.
She went 3-1 ahead in the second set before Williams, the 2002, 2013 and 2015 champion, clawed her way back to 3-3.
But Kenin grabbed what proved to be the crucial break in the 11th game and held her nerve to take victory on a second match point when Williams fired a backhand long.
Tennis star Serena Williams has revealed that she has apologised to Japan's Naomi Osaka who defeated her in the finals of the US Open last year.
London: Andy Murray’s management team confirmed on Tuesday that the three-times Grand Slam singles champion will be joining forces with Serena Williams in the mixed doubles at Wimbledon this year. The two-time Wimbledon champion’s partnership with seven-time winner Williams make them
Eight-time Wimbledon champion Roger Federer powered his way into the fourth round for the 17th time of the championships on Saturday.
Naomi Osaka staged a second successive Roland Garros great escape on Thursday as fellow top seed Novak Djokovic and Serena Williams strolled into the last 32. World number one Osaka, bidding to add the French Open to her US and Australian Open titles, came back from a set and 2-4 down
Former Pakistan captain Waqar Younis submitted a request to the Pakistan Cricket Board (PCB) to consider his candidature for the post of the team's bowling coach. According to Geo News, Waqar is the most high profile applicant for the post.
Skipper Dimuth Karunaratne moved from his overnight 49 to score 65 and provided the only bit of resistance in the morning session, which saw 29.3 overs bowled. Boult took two early wickets in one over after Sri Lanka resumed the day at 85 for two. Dhananjaya de Silva on 32 and Dilruwan Perera on five were at the crease.
Fast bowler Josh Hazlewood took five wickets for 30 runs, with England's collapse leaving them 112 runs behind after Australia made 179 in their first innings.