It's impossible not to notice that we live in an age of technological wonders, stretching back to the primitive hominids who dared to ask "Why?" but also continually accelerating and pulling everything apart while it does, in the exact same manner as the Universe at large. It is why all the hackers you know are invested so heavily in Deep Learning right now, as if someone got on a megaphone at Chaos Communication Camp and said "ACHTUNG. DID YOU KNOW THAT YOU CAN CALCULATE UNIVERSAL TRUTHS BACKWARDS ACROSS A MATRIX USING BASIC CALCULUS. VIELEN DANK UND AUF WIEDERSEHEN!". 

1. Focus on Specific Tasks and Structured Outputs: LLMs can be unreliable when given open-ended or overly broad tasks. To mitigate hallucinations and ensure accurate results, it's crucial to provide highly specific instructions and enforce structured outputs. (aka, the naive metrics people are providing are probably not useful). 
2. Manage Context Windows Effectively: While larger context windows are beneficial, strategically feeding code to the LLM in smaller, relevant chunks, like focusing on the call chain from user input to output, is key to maximizing efficiency and accuracy. They did a great job here, and this is important even if you have a huge context window to play with (aka, Gemini).
3. Leverage Existing Libraries for Code Parsing: Dynamically typed languages like Python present unique challenges for static analysis. Utilizing libraries like Jedi, which is designed for Python autocompletion, can significantly streamline the process of identifying and retrieving relevant code segments for the LLM. They recommend a rewrite here themselves using treesitter to look at C/C++, although I would probably personally have used an IDE plugin to handle Python (and give you debugger access).
4. Prompt Engineering is Essential: The way you structure prompts has a huge impact on the LLM's performance. Clear instructions, well-defined tasks, and even the use of XML tags for clarity can make a significant difference in the LLM's ability to find vulnerabilities. But of course, in my experience, the better your LLM (larger really), the less prompt engineering matters. And when you are testing against multiple LLMs you don't want to introduce prompt engineering as a variable.
5. Bypass Identification and Multi-Step Vulnerability Analysis: LLMs can be remarkably effective at identifying security bypass techniques and understanding complex, multi-step vulnerabilities that traditional static code analyzers might miss. There's a ton of work to be done in future analysis in how this happens and what the boundaries are.
6. Avoid Over-Reliance on Fine-Tuning and RAG: While seemingly promising, fine-tuning LLMs on vulnerability datasets can lead to oversensitivity and an abundance of false positives. Similarly, retrieval augmented generation (RAG) may not be sufficiently precise for pinpointing the specific code snippets required for comprehensive analysis. Knowing that everyone is having problems with these techniques actually is good because it goes against the common understanding of how you would build something like this.

At its core, vulnerability discovery is as much about understanding as it is about finding flaws. To find a vulnerability, one has to unravel the code, decipher its intent, and even the assumptions of its creator. This process mirrors the deeper question of what it means to truly understand code—of seeing beyond syntax and function to grasp the logic, intention, and potential points of failure embedded within. Like Bohr’s exploration of complementary truths in physics, understanding code vulnerabilities requires seeing both what the code does and what it could do under different conditions. In this way, the act of discovering vulnerabilities is itself a study in comprehension, one that goes beyond detection to touch on the very nature of insight.