Charith Asalanka (L) and Chamika Karunaratne (R) leaves the field after defeating Australia in Colombo on Sunday. AP
Set a tricky 292 to win after a Travis Head 70, the hosts won comfortably with nine balls to spare in Colombo, in their second straight victory in the five-match series.
Maxwell bowled Niroshan Dickwella for 25 but Nissanka and Kusal Mendis ground down the injury-depleted Australian attack of Josh Hazlewood, Jhye Richardson and Matthew Kuhnemann.
Chamika Karunaratne (C) celebrates with his teammate Charith Asalanka after their win. AFP
Mendis retired hurt unbeaten on 87 and was replaced by Dhananjaya de Silva who made 25, but Nissanka clung on until almost the end, caught off Richardson for 137 off 147 balls.
Richardson then bowled captain Dasun Shanaka for a duck, but Charith AsaLanka — with an ice-cold six in the penultimate over off Hazlewood — and Chamika Karunaratne saw Sri Lanka home.
Earlier Australia braved a spin onslaught by Sri Lanka to reach 291-6 in their 50 overs after winning the toss and opting to bat first.
Travis Head plays a shot during the third ODI match at R. Premadasa International Cricket Stadium. AFP
Opener David Warner went for nine in the third over, top-edging a Dushmantha Chameera bouncer to Maheesh Theekshana at fine leg.
Back from injury, Mitchell Marsh struggled to get going and departed for 10 off 23 deliveries, caught at cover by Mendis after a slower delivery by Dunith Wellalage.
Marnus Labuschagne came in at number four ahead of Head and he and Aaron Finch added 69 before Labuschagne was stumped by Niroshan Dickwella off Jeffrey Vandersay for 29.
Two overs later, Vandersay bagged Finch too, with the Australian captain edging the spinner to a diving de Silva at slip having made 62 off 85.
Cricket fans cheer in support of Sri Lanka during the ODI match. AFP
Head and Alex Carey added 72 runs before de Silva trapped Carey leg-before on the front pad trying to sweep, having made a quick-fire 49 off 52 deliveries.
Maxwell did some damage, smashing 33 off 18 before giving Vandersay his third wicket, miscuing the spinner into the hands of Wellalage scurrying back from midwicket.
But Head stayed firm, reaching his half-century and smashing Wellalage for three sixes in the penultimate over and ending 70 not out off 65 deliveries, with Cameron Green unbeaten on 15. Vandersay ended with figures of 3 for 49.
Already missing Marcus Stoinis, Kane Richardson and Mitchell Starc, Australia left out Steve Smith after the batsman injured his left quad batting in Thursday's 26-run defeat in Pallekele.
Glenn Maxwell plays a shot during the third ODI. AFP
Marsh, however, returned to the side, as did Green and Richardson, while Pat Cummins and Mitchell Swepson were left out. Spinner Kuhnemann was again in for Ashton Agar.
The hosts were once again without key all-rounder Wanindu Hasaranga, who injured his groin in the first match while Dickwella replaced Danushka Gunathilaka.
Australia: Aaron Finch (captain), David Warner, Mitchell Marsh, Marnus Labuschagne, Alex Carey, Travis Head, Glenn Maxwell, Matthew Kuhnemann, Jhye Richardson, Cameron Green, Josh Hazlewood.
Sri Lanka: Niroshan Dickwella, Pathum Nissanka, Charith Asalanka, Kusal Mendis, Dhananjaya de Silva, Dasun Shanaka (captain), Dunith Wellalage, Jeffrey Vandersay, Chamika Karunaratne, Dushmantha Chameera, Maheesh Theekshana.
Australia captain Aaron Finch equalled his highest one-day international score of 153 as he powered the World Cup holders to 334-7 against sloppy Sri Lanka on Saturday.
Minister for Home Affairs Peter Dutton will meet Sri Lanka's President Maithripala Sirisena and Prime Minister Ranil Wickremesinghe to discuss Australia's policy of returning boats carrying asylum seekers to their origin point, the minister's spokeswoman said.
Australia’s military intercepted a Sri Lankan boat carrying 13 asylum-seekers last month, Minister for Home Affairs Peter Dutton said on Monday, as Canberra sought to defend
The "soul-destroying" decision made about 15 percent of staff redundant and was aimed at saving Aus$40 million in costs, Cricket Australia chairman Earl Eddings said. The body said it is concerned about further financial shocks this year.
In another dominant performance, UAE defeated hosts Malaysia by five wickets in the final of the ACC Women’s T20 Championship on Saturday.
Petra Kvitova warmed up for Wimbledon by winning the Eastbourne singles title for the first time with a 6-3, 6-2 rout of Jelena Ostapenko on Saturday.
A 13.2km time trial will get the 109th edition of the Tour De France underway, before the riders eventually roll into the mountains and push their bodies and physical capabilities to the absolute limit.