Being a student interested in PL design, I have had this idea floating around for a while: the gist is finding out what programming languages LLMs might be the most proficient in, to study their design choices and syntactic features with the goal of designing the perfect language for LLMs. This is, of course, gimmicky, but I entertained the idea for a while as a fun afterschool project.
The challenge is: what would be the best way to evaluate programming performance _in specific languages_? There are two main hypotheses here:
1. There are intrinsic syntactic/structural features that the transformer architecture is uniquely able to parse/reproduce/understand best, leading to higher quality code generated. For example: Lisp dialects make parsing code structure and blocks very easy, so one could assume an LLM can "understand their code better" 2. There is so much Python/JS out there that the question isn't even worth asking, and the performance in those will beat whatever other language you throw at it. This is probably not as much of a point thanks to newer transformer architectures but the question is still up.
I suspect the answer can be made somewhat interesting by considering performance relative to language popularity, but the ground question is: is there a general dataset containing different programming challenges, of varying difficulty, in multiple languages, with standard solutions? I couldn't find anything when I looked around, but I might have missed something obvious. It wouldn't be impossible to build a simple website to crowdsource, but I'm thinking that if I missed something obvious I'd rather find out early than late. Also, if you have any input on the project itself, I'd love to hear your ideas!
I would expect the reverse: lisp has no syntactic sugar, making it harder for a LLM to glue code fragments together in a way that produces valid lisp code. Even guaranteeing that parentheses are correctly nested already can be a challenge.
As to a set of programs: they aren’t exactly what you’re looking for, but I would consider https://projecteuler.net (does not contain solutions, but searching for project Euler solutions” finds some) or https://benchmarksgame-team.pages.debian.net/benchmarksgame.
nit-pick details:
Ignoring hardware differences, "performance" comparisons can be based on differences between algorithm(s) used vs. how algorithm is implimented. For a given language, "algorithm implimentation performance" can be defined as the trade-offs on how a a given algorithm is implimented in a language (compared to other programming languages, but also easy use/flexibility based on 'language generation level -> https://www.geeksforgeeks.org/generation-programming-languag... )
----------------------
1) General computation language specialty 'modules' not withstanding; "languages" are built/optimised around core algorithmic concepts / anticipated area/concentration of targeted professional environment. aka opencl (gpu), R (statistics), Lisp (engineering design), C (OS level), sql (data selection), jasper reports, cobol (business), etc. Languages tend to be 'popular' because of the ecosystem provided around/for a given language.
snarky side note -> can always write a more standard language that compiles to an esolang & provide appropriate emacs/vim/sed/spacemacs ide support.: https://esolangs.org/wiki/Main_Page
LLM's are very useful at curating information and recognizing/summarizing "statisical" relevance. aka apl is great for engineering mind set, not so good for business use cases aka cobal. LLM might recognize a language for a given user that combines commonly used 'apl' aspecs of user and commonly used 'cobal' aspecs of user and recommend a language(s) with suitable commonalities for given user.
?? how easy / hard based on 'profession' aka artist vs. software designer 20 years experience programming in scheme; environment -- NASA vs. google vs. insurance company.
?? from scratch : https://synoptek.com/insights/it-blogs/10-challenges-every-software-product-developer-faces/
?? based on industry standards ?? ; just trying to keep skills honed ??