LLMs with C and C++ - switch from HFT to AI lab

I'm a senior C++ dev recently started working with a neolab (who works with anthropic). Thought I would write some of observations i made.

* I had experimented with LLMs for a while before making the switch. LLMs fail a lot on C and C++ due to harder nature and powerful nature of language.
* Talking purely from benchmarks, languages like python and JS (the vibecoder's first language) have very hard benchmarks - think fixing actual bug that touches 3 different modules from scratch with access to tools like grep, cat and python3 executer.
* Whereas, benchmarks for C and C++ are at basic QA style questions. I have added a task from benchmark, which fable 5 could not solve.
* LLMs do not have understanding of latest ISO standards - for some reasons it switches to C++17 again and again
* LLMs are trash at template metaprogramming. Try debugging CRTP type of errors.

Looking at the efforts and progress, I am still not sure if we will see LLMs writing MRs to linux kernel. The approach they used for vibecoding languages do not care about memory safety, thread safety and performance much. It would be interesting to see the space evolve.

PS: a example from benchmark

PS2: i'm not associated with benchmark. they say the code is taken from real github issues.

Observe the following faulty CPP code snippet and error type list. Your task is to select the error type of the code based on the error list provided.
You only need to answer error type. Do not write anything else in your response.
For example, if the code snippet is missing a semicolon, Your output should be 'missing_colons'.
faulty code:
```cpp
#include <bits/stdc++.h>

int countPermutations(int n, int k, int qq[])
{
const int N = 505, P = 998244353;
int *q = new int[n + 10];
int m, dp[N][N], jc[N], f[N], ans;
memset(q, 0, sizeof(int) * (n + 1));
memset(dp, 0, sizeof(dp));
memset(jc, 0, sizeof(jc));
memset(f, 0, sizeof(f));
ans = 0;

for (int i = 1; i <= n; i++)
q[i] = qq[i - 1];
dp[0][0] = f[0] = 1;
for (int i = jc[0] = 1; i <= n; i++)
jc[i] = 1LL * jc[i - 1] * i % P;
for (int i = 1; i <= n; i++)
{
f[i] = jc[i];
for (int j = 1; j < i; j++)
f[i] = (f[i] + P - 1LL * f[j] * jc[i - j] % P) % P;
}
for (int i = 1; i <= n; i++)
{
for (int j = 0; j < i; j++)
for (int k = 1; k <= n; k++)
dp[i][k] = (dp[i][k] + dp[j][k - 1] * 1LL * f[i - j] % P) % P;
}
m = 0;
for (int i = 1; i <= n; i++)
if (q[i] > q[i + 1])
{
m = i;
break;
}
if (m == n)
{
for (int i = k; i <= n; i++)
ans = (ans + dp[n][i]) % P;
}
else
{
for (int i = m + 1; i <= n; i++)
{
if (i != m + 1 && (q[i - 1] > q[i] || q[i] < q[m]))
break;
int c = k + i - n - 1;
if (c >= 0)
ans = (ans + dp[m][c] * 1LL * jc[i - m - 1] % P) % P;
}
}
return ans;
}
```
error list:
['Delayed Execution', 'Improper HTML structure', 'Missing $', 'Missing mut', 'Misused := and =', 'Misused === and ==', 'Misused =>', 'Misused Macro Definition', 'Misused Spread Operator', 'Misused begin/end', 'Misused match', 'Misused var and val', 'Unused Variable', 'algorithm_error', 'condition_error', 'double_bug', 'faulty_indexing', 'function_error', 'html_unclosed_label', 'html_value_error', 'html_wrong_label', 'illegal_comment', 'illegal_indentation', 'illegal_keyword', 'illegal_separation', 'json_content_error', 'json_digital_leader_is_0', 'json_duplicate keys', 'json_struct_error', 'markdown_content_error', 'markdown_title_error', 'markdown_unclosed_error', 'missing_backtick', 'missing_colons', 'misused ==and=',