O.J. Mayo had Denver beat with a tie-breaking 3-pointer late in regulation.
The mistake-prone Dallas guard gave the Nuggets another chance to set a franchise record in wins with a turnover in the final seconds, so he had to hit another go-ahead shot from long range in overtime.
Dirk Nowitzki scored 22, Mayo overcame his huge mistake with his first 20-point game in more than a month, and the Mavericks denied the Nuggets their 55th win with a 108-105 overtime victory on Friday night.
The Mavericks were up by two when they inbounded to Mayo -- their turnover leader this year -- with 10 seconds left in regulation. He tried to dribble through traffic but lost the ball to Andre Miller, who found Corey Brewer for a tying layup with 3.9 seconds remaining.
"You've just got to understand there's still more game left," said Mayo, who surprisingly got all the way to rim after the bucket by Brewer but missed a layup at the buzzer. "You can't really sit there and dwell on turning the ball over. They made the play at the end, and I thought we made more plays in overtime to get the win."
Mayo scored the first five Dallas points in the extra period, including a 3-pointer from the left wing for a 104-101 Dallas lead with 2:40 remaining. The Mavericks, playing two nights after getting eliminated from the playoff race for the first time since 2000, then held on in a frantic final minute.
Mayo was 5 of 7 from long range and finished with 20 points, his most since scoring 22 against Detroit on March 8. Vince Carter had 22 points and a season-high 12 rebounds.
Brewer, Wilson Chandler and Andre Iguodala led the Nuggets with 18 points apiece, and Kenneth Faried had 11 points and 10 rebounds.
The loss kept Denver at 54 wins, which the Nuggets have reached twice previously. The last time was with Carmelo Anthony leading the way in 2008-09, when the Nuggets beat the Mavericks in five games in the second round of the playoffs.
The Nuggets lost their last chance to finish as the No. 2 seed and will remain in a battle for home court in the first round with Memphis and the Los Angeles Clippers over their final three games.
"I'm not going to overreact to anything other than we have three games left," Denver coach George Karl said. "I think we all want third. But if we get fourth or fifth, we've had a helluva year and we'll get ready to go play in the playoffs."
While the Mavericks will miss the postseason for the first time since Nowitzki's second year, they can get back to .500 for the first time since December with a win Sunday at New Orleans. That would also mean the bearded German and several Dallas veterans can shave after vowing two months ago not to do it until they reached the break-even mark.
"Honestly, I try to play the end of the season out strong and win the next game for coach," said Nowitzki, who scored 16 in the first half. "Once I go out there, it's fun. I try to win and go out there and compete."
Dallas was nursing a 106-105 lead when Brewer was called for a foul on Nowitzki with 0.5 seconds on the shot clock and 6.1 seconds left in the game. The Mavericks threw a long inbound pass to Darren Collison, who avoided being fouled until there were just 1.9 seconds left.
Collison hit both free throws and Iguodala was short on a desperation 3-pointer at the buzzer.
"It's amazing," Karl said after a long pause when asked about the foul call against Brewer. "There are a lot of things about that game. We didn't get a foul call that whole third quarter. We were driving to the basket on almost every possession. Not one foul."
Denver point guard Ty Lawson returned after missing five games with a right plantar fasciitis tear, but didn't play half of the fourth quarter and overtime because he was limited to 20 minutes. He scored the first seven points of the fourth quarter and finished 13.
"I got three weeks of rest, but I've got to get back into game shape," Lawson said. "I didn't feel like I was tired out there or anything, but I need to get back to teammates getting used to me. They've been playing a different style since I've been out."
After Mayo put Dallas ahead 99-96 with a 3-pointer in the final minute of regulation, Miller was fouled on a drive and the referees ruled goaltending against Brandan Wright before overturning the call on video review.
Miller missed the second free throw, and Mayo found Wright, who was fouled as he barely missed a layup. Wright missed both free throws, but Iguodala, who hit a game-winning layup to beat the Mavericks in Denver and was coming off a triple-double against San Antonio, missed a contested jumper before Mayo's turnover gave Denver another chance to tie.
NOTES: Nowitzki finished 10 points shy of 25,000 for his career. He should become the 17th NBA player to reach the milestone Sunday at New Orleans. Jerry West is 16th at 25,192 points. Nowitzki is third among active players behind Kobe Bryant and Kevin Garnett. ... The Nuggets won all five games that Lawson, their leading scorer, missed with his foot problem. That included almost three full games without Danilo Gallinari, who tore the anterior cruciate ligament in his left knee early in a home game against the Mavericks last week. Gallinari had a career-high 39 points at Dallas in December. Mavericks C Elton Brand returned after missing four games with a strained right calf.