An oscillating drive for a power tool includes a wobble shaft located on a drive shaft of a motor for a power tool, and a wobble member rotatably supported on the wobble shaft. The wobble member includes a wobble drive structure that defines a drive axis. A tool driver includes a tool drive structure at one end that is coupled to the wobble drive structure and a tool holder configured to releasably retain an accessory tool. Rotation of the drive shaft causes the wobble shaft to wobble about an axis of the drive shaft. The tool driver holds the wobble member against the rotation of the wobble shaft such that the wobble member wobbles without rotating which results in an oscillating motion of the drive axis of the wobble member in a plane that is parallel to the axis of the drive shaft. The oscillating motion of the drive axis causes the tool driver to oscillate.