A rubber composition based on at least one diene elastomer, containing less than 2 phr of zinc, phr meaning parts per hundred parts of rubber (elastomer), and based on at least one reinforcing filler and one sulphur-based crosslinking system, comprising a zinc carboxylate of formula: (RCOO)2Zn, in which R represents a hydrocarbon group chosen from cyclic or non-cyclic, linear or branched alkyl groups having 1 to 16 carbon atoms, aryl groups, aralkyl groups or alkaryl groups having 1 to 16 carbon atoms. Also disclosed is a process for preparing such a rubber composition which is suitable for manufacturing tires or semi-finished products made of rubber intended for these tires.