New Zealand's Martin Guptill (C-R) embraces India's Jasprit Bumrah after winning the first semi-final at Old Trafford on Wednesday. AFP
It was a superb display by the underdog Black Caps, who came into the match on the back of three straight defeats by Pakistan, Australia and England and only qualified for the semi-finals on net run-rate, while India topped the 10-team group stage.
Earlier, New Zealand added 28 runs in the remaining 23 balls of their innings to finish on 239-8 after rain had stopped play on Tuesday.
Ross Taylor, who resumed on 67 not out, top-scored with 74 after Williamson had made a masterly 67 in awkward conditions.
New Zealand will face the winners of Thursday's second semi-final between Australia — the team that beat them in the 2015 final — and hosts England in Sunday's title match at Lord's.
Two-time world champions India lost key batsmen Rohit Sharma, Virat Kohli and KL Rahul for just one run each.
They slumped to five for three in pursuit of a target of 240 and were 24 for four when Dinesh Karthik was out.
New Zealand pacemen Matt Henry (three for 37) and Trent Boult (two for 42) did the damage with the new ball.
India, one of the pre-tournament favourites, were all but beaten at 92 for six when Jadeja came in to bat alongside veteran former captain MS Dhoni.
The pair gave India hope during a World Cup record seventh-wicket stand of 116, all-rounder Jadeja boldly leading the way with 77.
Jadeja hit well-struck sixes off both Jimmy Neesham and left-arm spinner Mitchell Santner during a 39-ball fifty.
With 10 overs left, India were 150 for six and needing a further 90 runs off the final 60 balls of the innings.
Jadeja got them closer with a superb straight six off fast bowler Lockie Ferguson and, with five overs left, the target was down to 52.
But with India in sight of an amazing win, left-hander Jadeja's 59-ball innings, featuring four fours and four sixes, ended when he skyed Boult to New Zealand captain Kane Williamson.
It was a fine, if ultimately futile, effort from Jadeja, dismissed as a "bits and pieces player" by former India batsman Sanjay Manjrekar, after he had also taken one for 34 with his left-arm spin and been brilliant in the field.
With India needing 31 off two overs, Dhoni uppercut Ferguson for six to the delight of the massed ranks of India fans in the crowd.
But going for a second run to keep the strike, the 38-year-old Dhoni, India's 2011 World Cup-winning captain, was just run out by Martin Guptill's direct hit for 50 off 72 balls.
With 23 required off the last six balls, tailender Yuzvendra Chahal glanced Neesham for four. But the match finished with three balls to spare when Chahal edged to wicketkeeper Tom Latham, a review confirming his dismissal and New Zealand's shock victory.
Sharma, who had already made a five hundreds at this tournament — a record for a single World Cup, began the procession of top-order dismissals when he was caught behind off Henry.
Kohli fell next, lbw to a superb inswinger from left-armer Boult.
Kohli reviewed but replays upheld English official Richard Illingworth's original out decision on umpire's call.
India were then five for three when Rahul was brilliantly caught by a diving Latham after edging Henry.
At that stage, India had lost three wickets for one run in 10 balls.
Neesham then held a brilliant superb one-handed catch at backward point to dismiss Karthik off Henry.
Rishabh Pant holed out for 32 when he was caught at deep midwicket by Colin de Grandhomme following a lofted sweep off Santner.
India suffered another setback when Hardik Pandya, slogging across the line, skyed Santner to Williamson at midwicket as he fell for 32.
Paceman Tim Southee missed out due to a calf injury as the Kiwis -- runners-up in the 2015 edition -- went in with two all-rounders in their starting line-up.
The two-time champions, who are unbeaten in their six outings so far, including a washed-out game, fielded an unchanged side after their narrow victory against Afghanistan.
Left-arm seamer Shaheen Afridi (3-28) had helped reduce the Black Caps to 46-4 and 83-5, when Kane Williamson was out for 41.
The Blackcaps bowled out India for just 170 in their second innings, with the match extended into Wednesday's reserve sixth day after two days were washed out completely.
Peshawar Zalmi put the United to bat first, who scored 174 for the loss of nine wickets and set a 175-run target for the Wahab Riaz-led side. Hasan Ali’s start with the ball was not as effective as his batting in the first innings — where he hit 45 off 16 balls.
Tim Southee removed both India openers late on Tuesday to give New Zealand hope of an unlikely victory in the World Test Championship final as the showpiece match headed into the reserve sixth day.
Novak Djokovic is the targeting a record-equalling 20th major and sixth Wimbledon title from next week, edging him closer to becoming only the third man to complete a calendar Grand Slam. The world number one has already won a ninth Australian Open