Aspects of the disclosure relate to a payment device to detect real-time suspicious payment cards. Prior to a transaction, a payment device detects suspicious payment cards based on captured images of the payment card. An alert may be generated upon detection of any suspicious or fraudulent payment card. In some arrangements, institutions may share information regarding suspicious payment cards to prevent fraud. The payment device may utilize machine learning models or machine learning capabilities to detect suspicious payment cards. A characterization score may be generated and used to determine if a payment card is suspicious. The characterization scores may be updated based on different card issuer criteria and transaction use of each payment card.