Techniques related to a low-power, low-memory multiply and accumulate (MAC) unit are described. In an example, the MAC unit performs a MAC operation that represents a multiplication of numbers. At least the bit representation of one number is compressed based on a quantization and a clustering of quantization values, whereby index bits are used instead of the actual bit representation. The index bits are loaded in an index buffer and a bit representation of another number is loaded in an input buffer. The index bits are used in a lookup to determine whether the corresponding bit representation and shift operations are applied to the input buffer based on this bit representation, followed by accumulation operations.