GERHARDT SCRIVEN | Executive Director, Capco
TONY MOENICKE | Senior Consultant, Capco
SEBASTIAN EHRIG | Senior Consultant, Capco

The rapid development of Large Language Models (LLMs) has revolutionized software development, yet the predominance of closed-source models has restricted their extensive adoption. In this paper, we explore open-source Large Language Models as an alternative to closed-source models like ChatGPT, particularly for the use case of interpreting legacy software source code. 

We evaluate open-source models for their capacity in understanding and explaining COBOL code to a human user, a crucial task for financial institutions looking to update their legacy systems while keeping their data secure in-house.

Evaluating LLMs in this domain is challenging since there’s no simple right or wrong answer to the specific types of COBOL related questions we ask. Towards this, we have benchmarked the responses obtained from various proprietary and open-source LLMs against an expert human response. This method allows us to assess which models perform best for a specific type of question and are effective in a practical context.

This article provides insights for financial institutions looking to optimize or modernize their legacy systems using LLMs as well as offering considerations for adapting and integrating these models into their IT environments.