On one surface of a piece of wooden sheet, a cloth or paper is adhered to make an adhered sheet. An approximately V-shaped notch is provided on the adhered sheet, the adhered sheet is wetted to be softened, and a lubricant which acts to keep water is impregnated thereinto. Edge portions of the notch are overlapped, and the adhered sheet is tentatively molded in an approximately horn shape by a first hot press molding. The adhered sheet is then dried, impregnated with thermosetting resin, and half-dried. The adhered sheet is subject to a second hot press molding. A moisture-proof agent is applied on the adhered sheet, a center hole for a voice coil and periphery are removed to make a speaker diaphragm having a predetermined dimension and shape.