The Iron needed to win to stand any chance of pulling off a great escape but made the worst possible start when Kris Boyd put Forest ahead after just nine minutes.
Their survival hopes were left hanging by an even slender thread when Luke Chambers extended the lead midway through the first half.
Michael O'Connor halved the deficit from the penalty spot in the 36th minute but half-time substitute Paul Anderson restored the Reds' two-goal cushion just three minutes after his introduction before late goals from Chambers and Boyd completed the rout.
Scunthorpe will return to League One next season while Forest look set for a second successive assault on the play-offs.
Billy Davies' side go into the final game of the season three points in front of seventh-placed Leeds and are six goals better off on goal difference.
Source: PA
Source: PA