New Zealand's Martin Guptill (C-R) embraces India's Jasprit Bumrah after winning the first semi-final at Old Trafford on Wednesday. AFP
It was a superb display by the underdog Black Caps, who came into the match on the back of three straight defeats by Pakistan, Australia and England and only qualified for the semi-finals on net run-rate, while India topped the 10-team group stage.
Earlier, New Zealand added 28 runs in the remaining 23 balls of their innings to finish on 239-8 after rain had stopped play on Tuesday.
Ross Taylor, who resumed on 67 not out, top-scored with 74 after Williamson had made a masterly 67 in awkward conditions.
New Zealand will face the winners of Thursday's second semi-final between Australia — the team that beat them in the 2015 final — and hosts England in Sunday's title match at Lord's.
Two-time world champions India lost key batsmen Rohit Sharma, Virat Kohli and KL Rahul for just one run each.
They slumped to five for three in pursuit of a target of 240 and were 24 for four when Dinesh Karthik was out.
New Zealand pacemen Matt Henry (three for 37) and Trent Boult (two for 42) did the damage with the new ball.
India, one of the pre-tournament favourites, were all but beaten at 92 for six when Jadeja came in to bat alongside veteran former captain MS Dhoni.
The pair gave India hope during a World Cup record seventh-wicket stand of 116, all-rounder Jadeja boldly leading the way with 77.
Jadeja hit well-struck sixes off both Jimmy Neesham and left-arm spinner Mitchell Santner during a 39-ball fifty.
With 10 overs left, India were 150 for six and needing a further 90 runs off the final 60 balls of the innings.
Jadeja got them closer with a superb straight six off fast bowler Lockie Ferguson and, with five overs left, the target was down to 52.
But with India in sight of an amazing win, left-hander Jadeja's 59-ball innings, featuring four fours and four sixes, ended when he skyed Boult to New Zealand captain Kane Williamson.
It was a fine, if ultimately futile, effort from Jadeja, dismissed as a "bits and pieces player" by former India batsman Sanjay Manjrekar, after he had also taken one for 34 with his left-arm spin and been brilliant in the field.
With India needing 31 off two overs, Dhoni uppercut Ferguson for six to the delight of the massed ranks of India fans in the crowd.
But going for a second run to keep the strike, the 38-year-old Dhoni, India's 2011 World Cup-winning captain, was just run out by Martin Guptill's direct hit for 50 off 72 balls.
With 23 required off the last six balls, tailender Yuzvendra Chahal glanced Neesham for four. But the match finished with three balls to spare when Chahal edged to wicketkeeper Tom Latham, a review confirming his dismissal and New Zealand's shock victory.
Sharma, who had already made a five hundreds at this tournament — a record for a single World Cup, began the procession of top-order dismissals when he was caught behind off Henry.
Kohli fell next, lbw to a superb inswinger from left-armer Boult.
Kohli reviewed but replays upheld English official Richard Illingworth's original out decision on umpire's call.
India were then five for three when Rahul was brilliantly caught by a diving Latham after edging Henry.
At that stage, India had lost three wickets for one run in 10 balls.
Neesham then held a brilliant superb one-handed catch at backward point to dismiss Karthik off Henry.
Rishabh Pant holed out for 32 when he was caught at deep midwicket by Colin de Grandhomme following a lofted sweep off Santner.
India suffered another setback when Hardik Pandya, slogging across the line, skyed Santner to Williamson at midwicket as he fell for 32.
Paceman Tim Southee missed out due to a calf injury as the Kiwis -- runners-up in the 2015 edition -- went in with two all-rounders in their starting line-up.
The two-time champions, who are unbeaten in their six outings so far, including a washed-out game, fielded an unchanged side after their narrow victory against Afghanistan.
Left-arm seamer Shaheen Afridi (3-28) had helped reduce the Black Caps to 46-4 and 83-5, when Kane Williamson was out for 41.
The 33-year-old was in virtuoso form to charge past the 27th seed 6-1, 6-2, 6-4 in hot sunshine on Rod Laver Arena and make the round of 16 at Melbourne Park for a 13th time.
Worknesh Degefa made it a Dubai double with victory in the 2020 Standard Chartered Dubai Marathon on Friday as debutant Olika Adugna won a dramatic sprint
Tiger Woods, seeking an unprecedented 83rd career US PGA Tour victory, fired a three-under par 69 that left him three off the first-round lead shared by Sebastian