Large Language Model-Guided Credit Scoring

Large language Models (LLMs) have revolutionized the financial analysis realm. In the field of credit scoring, where a loaner's default probability is measured, LLMs show great potential. Three issues remain in the related studying currently: 1) alignment with credit risk expertise; 2) model efficiency and vast computing power consumption; 3) model hallucination. They are challenging because of the insufficiency of annotated credit data, the huge complexity of models, and the diversity of LLMs hallucination. To address these issues and tackle the challenges, we proposed a light-weight novel LLMs credit scoring framework: Demographic and Behavioral-Credit-Large Language Model (DB-Credit-LLM). In our framework, we combined financial domain knowledge with LLMs to achieve alignment and captured demographic and transactional characteristics with newly designed linear attention mechanism. Additionally, we simplified the mitigation of hallucination with two-step prompting techniques. Besides, we improved the efficiency of LLMs through fine-tuning DeepSeek with efficient framework and Chain-ofThought removing. We proved the satisfactory zero-shot learning performance of DeepSeek. Finally, we generated a simulated dataset with LLMs, and it demonstrates the generalization ability of our framework and alleviates the label scarcity. We verified our framework on three open-source credit scoring datasets and one simulated dataset. To the best of our knowledge, it achieves state-of-the-art performance among 7B-level LLM-based methods in most cases and is one of the most efficient LLMs so far. Our code is available at https://github.com/puding26/DB-Credit-LLM.git.