A stump grinder is configured to mount to a portable machine that operably supports the stump grinder during a stump grinding procedure. The stump grinder includes a cutting wheel that is rotatably driven during the stump grinding procedure. The cutting wheel has a cutting surface to grind the stump by moving the rotating cutting wheel across the stump. A wheel stabilizing assembly is disposed between the portable machine and the drive system for supporting the drive system and cutting wheel. The cutting wheel is held in a generally vertical orientation and movement of the cutting wheel is biased relative to the portable machine. Accordingly, forces applied against the cutting surface of the cutting wheel during the grinding procedure are limited by the biased movement.