Domain-specific parameters may be used for tuning speech processing. A pre-trained transformer-based language model may train domain-specific parameters using domain-specific unlabeled text data. This domain-specific parameters can then be appended to candidate texts produced by a speech model on received speech data and input to the transformer-based language model to score the candidate texts. The scores of the candidate texts determined using the pre-trained transformer-based language model can then be used to select a candidate text for further speech processing.