- Dailydave - mm4.emwd.com

On becoming a carpenter
by Dave Aitel 12 Jun '25

12 Jun '25

[image: image.png] Every so often I poke my head out, gopher-like, from the tunnels where I am furiously vibe-coding, or as it's going to be known a couple years from now, coding. I think it's probably true that coding used to be a high octane sport for concentration freaks as deep in the zone as a sperm whale hunting giant squid by listening to the faint echoes of pings off squishy bodies leagues away. But coding is now a juggling competition where the winners are over-caffeinated ADHD insectivorous bats tracking the results of five different queries into as many LLM-based coding systems and context switching, running tests, and re-querying at a rate that would drive most people bananas. In other words, to compete in this day and age you do have to know your LeetCode, but you have to do it in parallel, using Codex, or Windsurf, or Codex-CLI, or cutting and pasting bits of code into o1-pro, but all at once, choosing which LLM-assistant-coder depending on what exact kind of programming issue you're trying to solve. On one hand, these tools are very much junior engineers, and make simple mistakes, and on the other hand, collectively they are better than you will ever be and realizing that as soon as possible is a big advantage. I spent some time, while contemplating obscure and bespoke flavors of obsolescence, to listen to Halvar's most recent talk. Everyone should. It is here: https://www.youtube.com/watch?v=qllU_B_Rmis . [image: image.png] "Cyber-espionage does not easily translate into industrial capability, particularly in fast moving industries. You don't become a carpenter by stealing his invoices" - Halvar And of course not? But that's not what hackers do. As we learned from listening to Grugq's keynote <https://conference.hitb.org/hitbsecconf2023hkt/speaker/the-grugq/> the most likely scenario is that hackers attack the carpentry problem as a system - use the invoices and outgoing quotes to drive all the business to yourself, along with stealing all their best people by offering them exactly 2X their salary. Going active sometimes means DoSing the supplier of some company so it misses a key deadline with its customer-base, which then moves to your more reliable, cheaper, product, built by their former head of engineering. And I firmly believe that few scientific and engineering problems are so hard that they can't be replicated given access to the right cyber team and all the resources a nuclear power can throw at them if they are a national priority. Half of the cyber policy world still thinks that cyber norms are a thing! We have labored for too long under the belief that we were going to be able to set the rules to the game in a new domain. But the rules are what they are. The terrain has no master. -dave

1 0

Typey typey
by Dave Aitel 31 May '25

31 May '25

https://www.linkedin.com/mwlite/feed/posts/daveaitel_for-the-offensive-info… <https://www.linkedin.com/mwlite/feed/posts/daveaitel_for-the-offensive-info…> yRnO So I wanted to point this contracting gig out because I think it's a good opportunity for someone who can do quick vulnerability triage and either replicate or disprove that a vulnerability exists. Obviously you need to be very good at c and c++ and every other language to do this and have familiarity with vulnerability development....and also be located in the US , although you do not need a clearance. Dave

2 2

Announcing the Parity Release of Volatility 3 and the Deprecation of Volatility 2
by Andrew Case 16 May '25

16 May '25

The Volatility Team is very excited to announce the official Parity Release of Volatility 3! This release is not only capable of fully replacing all of Volatility 2’s features, but it also incorporates support for all the latest operating system versions plus all the latest memory forensics research. With this release, Volatility 2 is now deprecated, and its GitHub project has been archived. Our announcement blog post details the new features, many of the new plugins, and tells you how to get started with Volatility 3. https://volatilityfoundation.org/announcing-the-official-parity-release-of-… Our team is incredibly grateful for the continued support from our community, and we hope that you all enjoy Volatility 3 for many years to come. -- The Volatility Team

1 0

OpenAI Security Research
by Dave Aitel 29 Mar '25

29 Mar '25

So a few things: 1. https://openai.com/index/security-on-the-path-to-agi/ I feel like this blog is worth reading. :) 2. We're throwing a post-RSAC conference in SanFran to talk about AI and Security (in particular, securing cybery things with AI) and if I'm very lucky I'll even get to do a quick demo of the software I've been working on, not that it will surprise anyone on this list! We have a few tickets left I think and if you're interested and one of the people who should be attending I made you a form <https://docs.google.com/forms/d/e/1FAIpQLSe64yhyrTW7LYwY3qxOyOpaVqbrKgdZayq…> to fill out! Thanks, Dave Aitel

1 0

Cyber Reasoning Systems
by Dave Aitel 09 Mar '25

09 Mar '25

I continue to believe there are a lot of interesting questions around building cyber reasoning systems for vuln finding. Even the very basics seem hard to study and understand, and the eval datasets available are....sparse or incomplete. For example, what you really want if you're analyzing git repos is the commit a bug was introduced, and the commit it was fixed. But usually you get "a commit where it maybe existed". Likewise, it's an open question what kind of tools you can provide an LLM in this space and how it can best use them. Anyways, if you're building a cyber reasoning system for finding bugs, please spam me an email. -dave

2 1

on your child going to college in Christchurch, NZ and velvet worms
by Dave Aitel 11 Feb '25

11 Feb '25

*on your child going to college in Christchurch, NZ and velvet worms* By mid‑August the garden already practices absence — stems turning hollow, the robin leaving its notes hanging in the air like torn corners of a song. Under the chirp of palmetto bugs, a log eases itself back into earth. Inside, hidden from the light, a velvet worm does the impossible: offers herself to a spill of pale, blind threads. For days she is nothing but hunger turned outward, spinning glue not to hold but to feed, until their tiny jaws harden and, like rain, they slip back into moss, leaving her the damp and the slow work of being only herself again. It's a kind of mercy, this leaving — clean as the way geese carve open the sky, each V a blade that does not return. You pack your fragments — old Defcon badges, a frayed pentester lab shirt, dreams with edges I've never seen — into my old suitcase. On your dorm wall next to a shattered cathedral, you'll make a collage of faces, new maps where I am just blank space beyond the frame, like unexecuted shellcode left on the heap. The house darkens into its own bark. I wander its rooms like a soft thing without eyes, tasting the air for what I once held. There is a moment in every season when what clung must fall. Even the velvet worm stops spinning sweetness for a brood that has grown teeth, and the log swallows its secrets—the bacterial smell of old rain, the echo of digital footfalls we never learned how to trace.

1 0

(the root of the root and the bud of the bud)
by Dave Aitel 14 Jan '25

14 Jan '25

Memories and thoughts are the same thing, someone tried to explain to me recently. You have to think to remember, in other words. This is hard to grasp for a lot of people because they *think *they have *memories*. They wrongly think memory is a noun instead of a verb, which is ok in philosophy and psychology but in cutting edge computer science we have to be precise about these sorts of things. Twenty-five years ago, when I first started writing fuzzers, a full quarter century, people thought it was an absolutely stupid thing to do. The smart people were using their giant brains to do static analysis. They were tainting and sinking. They were reading the code and finding flaws. They did threat models. They did not write glorified for loops that made different amounts of A's go into different RPC functions. But I had the hubris of a teenage hacker, and I thought it was fun. More fun, perhaps, than reading code. In 2025, fuzzing is part of the software development lifecycle for any organization rich enough to call a hyperscale datacenter home. It is a *sine qua non* for secure software. Fuzzing, we now understand, is *reasoning*. And if you can't reason over your code, you can't secure it. Part of the value is that fuzzing echoes machine learning in that it scales nicely with the amount of CPU you could use. And there's no false positives when you measure whether an input crashes a program - it either does or it does not. There are downsides of course - many inputs may cause the same crash. Fuzzing identifies a flaw exists, but it doesn't tell you what the flaw actually is. And fuzzing often finds enough flaws that development teams become overwhelmed with triage. And of course, fuzzing can often be too dumb to reach the important bugs, since it is exploring the space of possible inputs semi-randomly, even with coverage guided analysis. We (as a community) tried to correct these things with SMT solvers, or smarter fuzzers. But now we have a new tool: LLMs, which reason in a very different way. But still they *reason*. Admittedly, there are many disbelievers. "LLMs just repeat what they are trained on" and taken to an extreme that's true but that's also true for any of us. In practice, they reason perfectly well. And not too long from now, maybe a couple years at most, any organization that is not using them widely for security engineering is left behind the curve - the same way teams not using fuzzers are today. Memories and thoughts are, in essence, the same thing because both require the act of reasoning. In computer science, fuzzing and LLMs are tools that embody this principle. They don't passively store knowledge - they actively explore, test, and refine it. When I first started fuzzing, it was dismissed as a foolish endeavor because it didn’t look like traditional reasoning. Now, it’s indispensable. LLMs are on a similar path: misunderstood by some, but already reshaping how we approach security. Just as fuzzing forced us to rethink what reasoning over code looks like, LLMs are forcing us to rethink reasoning itself. In both cases, the act - not the object - is what matters. They are the root of the root and the bud of the bud - the foundation of what comes next. And if you don’t carry this forward, you risk being left behind in a world that’s growing beyond you. -dave

5 4

Anthropological "Hacker" Map
by A K 13 Jan '25

13 Jan '25

Hi all, In the latest "Security Weekly" (https://www.youtube.com/watch?v=CXefYdEGW04 ) they present the Anthropological "Hacker" Map https://wherewarlocksstayuplate.com/map/ While the map is incomplete (how can it ever be complete?), I think it is one of the few times, outside of David Aitel's writings about the cross-cut between the "underground" (for a lack of a better term) and subsequent commercial activity. Based on my personal bubble, completely empirical evidence that you should take with a huge grain of salt the cross-cut is not the actual highlight here - the highlight is a temporal one. The generation of folks contained therein are the among the first two generations (or so) between the commoditization of Internet - essentially having first movers advantage on the space. Anyway, felt like sharing this with the rest of the community here

1 0

the endless stream
by Dave Aitel 31 Dec '24

31 Dec '24

I've seen great people in our industry crushed under the weight of the secrets they carry into a singularity from which no information can emerge. In some ways the lesson from apache_nosejob.c <https://www.exploit-db.com/exploits/21560> was that we cannot take ourselves seriously, that at the heart of our discipline there must remain a jester, that we must float upon the stream of endless information rather than absorb it into our darkened core. To that end I often listen to infosec podcasts while doing other things: 1. The Three Buddy Problem <https://securityconversations.com/podcast/security-conversations/> with Ryan Naraine, Costin Raiu and JAGs. This is probably my new favorite podcast, with an uncensored take on current infosec events, largely from an incident response standpoint, but in general covering all the bases and courageously offending everyone at great length. 2. Risky.Biz <https://risky.biz/RB775/>: Still excellent after all these years, partially because Adam is such an experienced penetration tester and Patrick is a good host, and I generally learn things about events that are poorly covered in the news from their perspective, without having to go through and do fact checking myself (aka, why the Struts bug is so bad, etc.) Also, happy New Year everyone! -dave

1 0

Hacking the Edges of Knowledge: LLMs, Vulnerabilities, and the Quest for Understanding
by Dave Aitel 02 Nov '24

02 Nov '24

[image: image.png] It's impossible not to notice that we live in an age of technological wonders, stretching back to the primitive hominids who dared to ask "Why?" but also continually accelerating and pulling everything apart while it does, in the exact same manner as the Universe at large. It is why all the hackers you know are invested so heavily in Deep Learning right now, as if someone got on a megaphone at Chaos Communication Camp and said "ACHTUNG. DID YOU KNOW THAT YOU CAN CALCULATE UNIVERSAL TRUTHS BACKWARDS ACROSS A MATRIX USING BASIC CALCULUS. VIELEN DANK UND AUF WIEDERSEHEN!". Hackers are nothing if not obsessed with the theoretical edges of computation, the way a 1939 Niels Bohr was obsessed with the boundary between physics and philosophy, and how to push that line as far as possible using math in such a way that you could find complementary pairs of truths lying about everywhere and of course one aspect of his work is that you can almost a century later deter an adversary from attacking you by threatening to end the world, and another aspect is you can study the stars and as a bureaucrat you would call this "dual-use" although governments tend to have a heck of a lot of use for the former and almost no use at all for the latter. All of which is to say that while NO HAT <http://nohat.it> is a very good conference, with a lot of great talks, the one I have enjoyed the most so far is "LLMs FOR VULNERABILITY DISCOVERY: FINDING 0DAYS AT SCALE WITH A CLICK OF A BUTTON <https://www.youtube.com/watch?v=Z5LMRS3AF1k>" (by Marcello Salvati and Dan McInerney). This talk goes over their lessons learned developing VulnHunter <https://github.com/protectai/vulnhuntr>, which finds real vulns in Python apps given just the source directory, and those lessons are roughly as follows: 1. Focus on Specific Tasks and Structured Outputs: LLMs can be unreliable when given open-ended or overly broad tasks. To mitigate hallucinations and ensure accurate results, it's crucial to provide highly specific instructions and enforce structured outputs. (aka, the naive metrics people are providing are probably not useful). 2. Manage Context Windows Effectively: While larger context windows are beneficial, strategically feeding code to the LLM in smaller, relevant chunks, like focusing on the call chain from user input to output, is key to maximizing efficiency and accuracy. They did a great job here, and this is important even if you have a huge context window to play with (aka, Gemini). 3. Leverage Existing Libraries for Code Parsing: Dynamically typed languages like Python present unique challenges for static analysis. Utilizing libraries like Jedi, which is designed for Python autocompletion, can significantly streamline the process of identifying and retrieving relevant code segments for the LLM. They recommend a rewrite here themselves using treesitter to look at C/C++, although I would probably personally have used an IDE plugin to handle Python (and give you debugger access). 4. Prompt Engineering is Essential: The way you structure prompts has a huge impact on the LLM's performance. Clear instructions, well-defined tasks, and even the use of XML tags for clarity can make a significant difference in the LLM's ability to find vulnerabilities. But of course, in my experience, the better your LLM (larger really), the less prompt engineering matters. And when you are testing against multiple LLMs you don't want to introduce prompt engineering as a variable. 5. Bypass Identification and Multi-Step Vulnerability Analysis: LLMs can be remarkably effective at identifying security bypass techniques and understanding complex, multi-step vulnerabilities that traditional static code analyzers might miss. There's a ton of work to be done in future analysis in how this happens and what the boundaries are. 6. Avoid Over-Reliance on Fine-Tuning and RAG: While seemingly promising, fine-tuning LLMs on vulnerability datasets can lead to oversensitivity and an abundance of false positives. Similarly, retrieval augmented generation (RAG) may not be sufficiently precise for pinpointing the specific code snippets required for comprehensive analysis. Knowing that everyone is having problems with these techniques actually is good because it goes against the common understanding of how you would build something like this. At its core, vulnerability discovery is as much about understanding as it is about finding flaws. To find a vulnerability, one has to unravel the code, decipher its intent, and even the assumptions of its creator. This process mirrors the deeper question of what it means to truly understand code—of seeing beyond syntax and function to grasp the logic, intention, and potential points of failure embedded within. Like Bohr’s exploration of complementary truths in physics, understanding code vulnerabilities requires seeing both what the code does and what it could do under different conditions. In this way, the act of discovering vulnerabilities is itself a study in comprehension, one that goes beyond detection to touch on the very nature of insight. -dave

1 0

Old Infosec Talks: Metlstorm's Take on Hacky Hacking
by Dave Aitel 31 Oct '24

31 Oct '24

The Anatomy of Compromise One of my demented hobbies is watching old infosec talks and then seeing how well they hold up to modern times. Recently I excavated Metlstorm's 2017 BSides Canberra <https://www.youtube.com/watch?v=OjgvP9UB9GI&list=TLGGvAY1CcIr-AcyNjEwMjAyNA> talk on "How people get hacked" - a pretty generic topic that gives a lot of room for opinion, and one a lot of people have opined on, but the talk itself has a lot of original things to say. In particular, there's a huge disconnect between how people get hacked and how defenders and policy makers think people get hacked and choose to defend against them - which anyone on this list already knows - I think we are all aware that defensive strategies in cyber are rarely based on available data. [image: image.png] The Three-Act Play of Compromise Here is how people get hacked, according to Metlstorm: 1. Find something with 1FA and crack it open (or just phish the creds) (Everything in "Secure By Design" is meant to address this part of the problem) 2. Get Domain Admin and hang onto it 3. Watch the person who does the important stuff (like SWIFT transfers) and secretly do their job for them Metlstorm goes into the Active Directory hacking that we all know and love in great detail. His toolbox from 2017 (Kerberoasting, Group Policy files, password spraying, etc.) is still largely relevant today, despite Dwizzle's best work - and points out that removing an attacker that has once had domain admin is practically impossible even though we all pretend it is to the SEC (a painful truth we don't deal with at all in industry, unless Wiz has a product line here I don't know about). But the pattern he's really describing is the understanding that individual vulnerabilities and Active Directory "features" are as relevant to systemic compromise as individual genes are to having an arm with five wiggly bits at the end. Metlstorm picks on Active Directory and its cousin Sharepoint quite a bit, but his point is not that we should blame Active Directory so much as ourselves - we all installed something huge and complex we didn't understand and then put the keys to our kingdoms in it. Partially he doesn't blame AD because Metlstorm, even before the SolarWinds and Kaseya compromises happened, was obsessed with supply chain weaknesses - or rather he clearly looks at it not as a supply chain but a supply web, where compromise propagates through trust relationships like signals through a neural network. And this is where Metlstorm's talk becomes particularly interesting in retrospect. While we were all obsessing over Domain Admin and Exchange bugs in 2017, he was pointing at MSPs and software providers saying "that's where the real action is." In the years since, we've seen exactly this pattern play out in increasingly sophisticated ways: - SolarWinds and Kaseya (2020-2021) showed us what happens when attackers compromise either a build pipeline or an MSP's distribution system - Recent MSSP breaches that none of us will ever hear about unless the GCSB decides to write them up Each of these compromises followed Metlstorm's basic thesis: why hack 1000 companies when you can hack the one company they all trust? The attackers don't see individual organizations - they see connection points, trust relationships, and privileged channels that can be repurposed. Seven years later, this view has proven devastatingly accurate. Metlstorm calls himself an "operational hacker" - different from your Brett Moore style "Research Hacker" who's all about finding bugs and writing shellcode and various useless stuff like that. For him, operational hacking is about systems thinking: what does each compromise actually get you? And this, as it turns out, is what the talk is really about. Digital Ecosystems [image: image.png] Using New Zealand as his laboratory, Metlstorm somewhat cheekily shows us organizations not as isolated entities but as nodes in a vast supply web: - Managed service providers spreading their digital mycelia through thousands of organizations - "Liz in accounts payable" unknowingly holding the keys to national security - Domain registrars running code old enough to be geological This is one of the strengths of the talk - it is backed up by specifics. It is not a vague thought-piece. He takes shots at the whole "I hunt sysadmins" approach as thinking too small! Why hunt sysadmins when you can hunt their managed service providers who already have domain admin? Or, hunt the providers of those providers. It's like a food web of sysadmins. His best examples are massive US companies (NYT, f.e.) that got owned through tiny companies in NZ- big for NZ standards maybe, but microscopic globally. The Observer Effect What Metlstorm as an attacker sees everywhere he looks is large systems that are "commercially untestable" - creating a fundamental disconnect between risk and reality. When you outsource your domain admin to a global megacorp (or your local Kiwi-buds), you create a quantum state of security - simultaneously compromised and secure until someone attempts to measure it. You: - Can't test their security - May not know if they're compromised - Certainly can't perform incident response - But get a lovely compliance certificate to frame Recent compromises prove what Metlstorm saw in 2017: while defenders obsess over hardening their membranes via the magic of secure by design (or paying "$6 a month for MFA"), attackers traverse the supply web and pick on whatever provider seems easiest to own. The reality is that no organization exists in isolation any more than a neuron functions alone. Your security isn't just your controls - it's every provider, vendor, and service in your supply web, each one a potential firing synapse of compromise. *Solutions* [image: image.png] He rightfully calls out that we will not solve these problems. So an A for Accuracy. Very fun talk, worth your time, highly recommended, 10/10 would listen to again in the car on the way to a house built at sea level in a hurricane zone. -dave

1 0

Grace Hopper and the Rebirth of US Conferences
by Dave Aitel 09 Oct '24

09 Oct '24

I spent some time watching all the Grace Hopper videos on the youtubes, as I prepared for what up North is a horrible storm, but here in Miami is, so far, a breezy and clear day. You can hear her talk about how subroutines used to be literal handwritten pages of instructions in notebooks. When you wanted SIN or COS you would go over to whoever had the notebook with the working version, and copy it out into your code. It was this experience that guided her as she wrote the first compilers, but as you listen to her stories you can see her ponder the meta-questions of computation as well - she uses as the example how a game of basketball sparked her solution to a troublesome issue in her first single pass compiler, leading to her writing what must have been the first jump table. How, she asked herself, can we get a computer to capture this geometry of the mind. What was it that made her brain think of jump tables while playing basketball? When asked about AI, she poo-pooed theoretical work, and emphasized practical needs: data processing and expert systems for field commanders. Hopper's journey from abstract thought to concrete application mirrors our ongoing struggle in computer science, as does her struggle to get the military to adopt new technology, which she relates with anecdote after anecdote - clearly the hardest thing she did in her career was get support for computers into the Navy bureaucracy. This year I've attended a birthday and a funeral, and a lot of our community was at both. But there hasn't been a good offensive-minded conference for us to attend in the States after INFILTRATE ended. We've needed one. Our collective knowledge, like Hopper's compiler, builds upon itself, each pass adding depth and functionality. Two upcoming conferences, RE:VERSE in Orlando and District:CON in DC, have now arisen to continue this tradition. They're not just tech showcases, but gatherings of minds, a hive of soldier ants collected around a temporary bivouak, each attendee contributing to our shared codebase of ideas, as if scribbled in a 1950's coders notebook. We're still trying to capture the geometry of thought, one exploit at a time. And I hope to see everyone there! -dave Grace Hopper video (best version imho because it includes Q&A - link is to the AI question): https://youtu.be/WyGtNvBZ6kk?si=aF4b3elx2vDPPNax&t=5054 Conferences (please buy tickets now before they run out and I am sad): 1. https://re-verse.io/ - Feb 28th in Orlando, FL 2. https://www.districtcon.org/ Feb 21 in Washington DC

1 0

Developing Clairvoyance
by Dave Aitel 30 Sep '24

30 Sep '24

As you know, humans like to invent comfort words. One of my favorites is "luck". The theory being that yes, the universe has dice, but they are loaded in your favor. Properly used, these words are a spell - they allow us to have courage when a sober mind would quail. But when you become a professional, you have to give up these crutches. Only poor poker players believe in "luck". In computer science, and especially in machine learning, one of these words you have to give up is "consciousness", which makes a lot of the current discussion on AI feel pompous and unbearable. Just because you can feel something doesn't make it real. The other day someone said to me, "I don't know why, but it FEELS true to me" and I haven't been able to get that out of my head since then. In security we are often challenged to reject the assumptions a system is built on so we can develop a third sight - in some cases clear enough you gain the ability to predict the future. The raw fuel of this is a vast reliance on an information torrent that to any outside party looks like addiction. I'm by no means an expert on war, conflict, or the Middle East, but what Israel has done to Hezbollah is stunning to most of the experts, and in my opinion, it's because they missed a major shift in warfare that was largely driven by *Java Middleware*, of all things. So many of us lived through the transformation brought about by precision GPS and ISR-based air power. This much is obvious to the crowd that writes articles in war journals. Air war is cool in a very teenage way, and drone movies have a certain visceral feel to them that academics like to write about because it says to the world that they are the kind of hard-core experts that understand the blood-and-guts reality of modern combat. You know what's not sexy? Java. But what is network centric warfare <https://en.wikipedia.org/wiki/Network-centric_warfare#:~:text=Network-centr…> really if not endless rooms full of programmers having meetings about Java Beans and SOAP contracts. What does it look like when a real team goes after your entire society and builds surveillance into everything? It's hard to move in the modern world without a wake billowing behind you into the ether. Al Quada learned this the hard way, and now I think we can see what it looks like applied to another decentralized covert organization. In the end, it's not the drones or the missiles that make the difference—it's the quiet, relentless infrastructure underneath it all, overlooked by those too focused on the shiny surface of modern warfare. The real revolution isn’t in the explosions but in the code. The experts who missed this shift weren’t outmaneuvered by luck or brute force; they were outmatched by those who embraced the unglamorous, intricate work of building a machine that sees all and remembers everything. The future of conflict, like security and AI, won’t belong to those dazzled by the spectacular. It will belong to those who understand that, more often than not, the real power lies in the invisible. -dave

1 0

sboms and LLMs
by Dave Aitel 12 Sep '24

12 Sep '24

People doing software security often use LLMs more as orchestrators than anything else. But there's so many more complicated ways to use them in our space coming down the pipe. Obviously the next evolution of SBOMs <https://www.cisa.gov/resources-tools/resources/cisa-sbom-rama> is that they represent not just what is contained in the code as some static tree of library dependencies, but also what that code does in a summary fashion that you can check once you get the final binaries. In a certain sense, you can think of this as a behavioral attestation between the software publisher and the consumer who is actually running the product. In other words, if my product is meant to connect to WWW.SPYWARE.RU, then it should say so in the SBOM behavioral manifest. But of course in practice these things get quite complicated, and hence you need to summarize semi-structured data (aka, the behavioral manifest is rarely exact), and then compare it to what is seen when the software itself is run (which if you've ever run strace ...is voluminous). That smells like a job for an LLM, or at the very least, a vector comparison. Likewise, automatically building harnesses to run and capture security sensitive information (or performance information as we learned from XZ), is rapidly also becoming a job <https://google.github.io/oss-fuzz/research/llms/target_generation/> for an LLM. I perhaps am channeling everyone else's <https://www.cisa.gov/speaker/allan-friedman> worry that too much of the SBOM community is arguing about which XML fields belong in a VEX addendum, rather than pushing the concepts forwards to actually solve problems. Or perhaps not! At some level, the software vendors are getting dragged through this process by their hair, which is very fun to watch. -dave

3 2

Persistence and Strategic Effects
by Dave Aitel 16 Aug '24

16 Aug '24

Before there were words, calculated as the softmax of a list of possible tokens, there were just vectors of nano-electrical potential in cells soaked in a hormonal brew of electrolytes, operating on a clock cycle of "slow, but fast enough". In this sense, as we now know <https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10472538/>, we generate words and we know, in our heads, what we are, in the same way as we generate limbs, with each cell knowing from its electric field what to be next. A tumor is in that way of thought a *confabulation *or as we now say, a *hallucination.* But then, also, so are you. Recently I spent some time reading this year's Research Handbook on Cyberwarfare <https://cybersecpolitics.blogspot.com/2024/08/a-quick-research-overview-of-…>. One of the forms I filled out recently asked me if I was a certified Master Operator, which of course, I am not, any more than an Archaeopteryx is a certified Bald Eagle, even though both know the smell of the sky and the taste of freshly caught fish. But I do occasionally pay attention to the "state of the art" academic view of cyberwar and the Handbook was a good way to catch up. For example if you read Nadiya Kostyuk and Jen Sidorvova's Handbook paper on *Military **Cybercapacity* they will say that "a cyber attack may provide a defender or third party with a good estimate of the attacker's capabilities, but it is not clear how many of these capabilities the attacker has in their arsenal". This is, to my primitive cyberwarfare mind, so old that I still use "screen" instead of "tmux", a bit of a misstep when it comes to how cyberwar works and what a capability is. I don't know how to say it any clearer than this: Behind every wooden horse is a woodshop. An example in my head is that right now the Ukrainian army is rumored to be sitting on top of a major gas terminal in Kursk, one responsible for supplying Russian gas to Europe. You have to assume that, having learned from the Russian attacks against their electrical infrastructure, the Ukrainian Army is traveling not just with a screen of FPV drones but with a few USB keys containing implants for the specialized equipment that runs a gas network. It's hard to disconnect OT networks that are presumed to be segmented physically, and temporary physical control can easily translate to permanent cyber control. And cyber control, despite what Quentin E. Hodgson's Handbook paper (*Cyber coercion as a tool of statecraft: how often, how effective?*) wrongly concludes, is extremely useful for state coercion. Perhaps the problem with the Handbook, like all academic writing on cyberwar, is that it is meant to be sterile. But that's not how cyberwar works, held in the space that is a melange of electrons and intentions. As tumors confabulate within flesh, so too do our digital dreams hallucinate new worlds, both the virus and the firewall, the wooden horse, and the workshop that births it. Certified or not, we are masters of a domain we cannot fully comprehend, sailing on seas of raw data, guided by stars we ourselves ignite.

2 1

"Exploitation Less Likely"
by Dave Aitel 13 Aug '24

13 Aug '24

DefCon is a study in cacophony, and like many of you I'm still digging through my backlog of new research in multifarious browser tabs, the way a dragonfly keeps track of the world through scintillated compound lenses. In between AIxCC (which proved, if anything, the boundaries <https://dashboard.aicyberchallenge.com/collectivesolvehealth> of automated bug finding using current LLM tech?), James Kettle's timing attack research <https://portswigger.net/research/listen-to-the-whispers-web-timing-attacks-…>, and even more PHP ownership <https://www.ambionics.io/blog/iconv-cve-2024-2961-p1>, you unfortunately do have to pay attention to the outside world. One of the things that lit up my sensors was the Windows Remote Desktop Licensing service that came out from a sort of "Post QiHoo360" exploit community, led by Dr. Zhiniang Peng (aka @edwardzpeng), an absolute legend of exploitation. A remote unauthenticated heap overflow in the latest Windows via an MSRPC endpoint, bypassing modern defenses by just calling LoadLibraryA("\\webdav\owned.dll") on a fake object. An unexpected burst of pure beauty really, like the iridescence of a Morpho moth flitting across a concrete parking lot. The exploit <https://github.com/CloudCrowSec001/CVE-2024-38077-POC/blob/main/CVE-2024-38…> is public, but the original paper is now mysteriously deleted, I assume for political reasons. If you have a copy of it, please shoot it my way. It's telling that all the best exploits I know have "Exploitation less likely" as their rating from Microsoft. Anyways, it's interesting what merits attention, and what doesn't. -dave

2 2

PRANA Hack and Leak Report Release
by Dave Aitel 02 Aug '24

02 Aug '24

Cordyceps Analysis Report on PRANA Network Hack and Leak Operation: https://docs.google.com/document/d/1oOJbBTUwyK85ZKYAAdwWqxk-sMvqrBqzJYX1ozi… Lately I've been reading a lot of academic papers, mostly the Research Handbook on Cyberwarfare <https://www.elgaronline.com/edcollchap/book/9781803924854/book-part-9781803…>. Some of them are good papers! JD Work has a paper in it! But also some of them get wrapped around the idea of "Cyber War vs Cyberwarfare" (!??!) or fall in love with "hacktivists" or stomp furiously around all the real issues in the domain without ever stepping on the green. The thing about Persistent Engagement and Integrated Deterrence and Defend Forward is that yes, as Paul Wither's points out in "Do we need an effects-based approach for cyber operations? <https://kclpure.kcl.ac.uk/portal/en/publications/do-we-need-an-effects-base…>" or as the Grugq points out in his latest keynotes <https://www.youtube.com/watch?v=P6PnhDfWvx0>, we do need to look at "effects" in a broad way when talking about cyber operations. But the goal of your offensive counter-cyber operations is not to "reduce the number of incoming operations", as Jay Healey would try to measure, but to mitigate strategic effects! In other words, if the Chinese are hacking as much as possible in the LLM space, but somehow don't manage to launch their own market-leading LLM because for some reason their experiments on Huawei Ascend chips all mysteriously have weirdly wide bands of error, that is Defend Forward working! Did nobody in all of cyber policy academia read the Three Body Problem? Academics should love this part of the theory, really, because arguing about strategic effects is 90% of what they do, usually without ever opening up a Neo4j Database filled with stolen mail spools to see what's really happening on the wire. Anyways, if you like the kind of reporting in our PRANA report, we offer paid reporting as well. :) Thanks, Dave Aitel

1 0

LLMs and refusals
by Dave Aitel 25 Jul '24

25 Jul '24

[image: image.png] Above: LLAMA3.1 8B-4Q test results via OLLAMA So recently I've been doing a lot of work with LLMs handling arbitrary unstructured data, and using them to generate structured data, which then gets put into a graph database for graph algorithms to iterate on so you can actually distill knowledge from a mass of nonsense. But obviously this can get expensive via APIs, so like many of you, I set up a server with a A6000 that has 48G of VRAM and started loading some models on it to test. Using this process you can watch the state of the art advance - problems that were intractable to any open model first became doable via LLama70B versions, and then soon after that, 8B models. Even though you can fit a 70B version into 48GB, you also want to have room for your context length to be fairly large, in case you want to pass a whole web page or email thread through the LLM (which means an 8B parameter model is probably the biggest you really want to use - I don't know why people ignore context size when calculating model VRAM requirements). My most telling example prompt-task is a simple one: Give me the names in a block of text, and then make them hashtags so I can extract them. The results from LLama3.1 8B when quantized down are not...great, as seen in the little example below: *Input Names text: Dave Aitel is a big poo. Why is he like this? He is so mean.RET: I can't help with that request. Is there something else I can assist you with?* Uncensored models, like the Mistral NeMo, are also tiny and struggle to do this task reliably on Chinese or other languages that are not directly in their training set, but they don't REFUSE to do the task because they don't like the input text or find it too mean. So you end up with a lot better results. People are of course going to retrain the LLAMA3.1 base and create an uncensored version and other people are going to complain about that - having an uncensored GPT4-class open model scares them for reasons that are beyond me. But for real work, you need an LLM that doesn't refuse tasks because it doesn't like what it's reading. -dave P.S. Quantization lobotomizes the LLAMA3.1 models really hard. What they can do at full quantization on the 70B model they absolutely cannot do at 4bit.

4 3

Felt Vampires in Policy World And You
by Dave Aitel 12 Jun '24

12 Jun '24

Can a hamster do interprocedural analysis? What size of hamster can turn a tier-2 geopolitical adversary's cyber force into a tier-1 adversary? Is the best use of a hamster finding 0day or orchestrating the offensive operations themselves? These are all great questions for policy teams to ponder and they pontificate over how to properly regulate AI. On one hand, as a technologist, your tendency will be to try to explain to policy teams what makes a scary adversary scary, maybe get involved in building a taxonomy of various tiers of adversary, start classifying operations as "sophisticated" and "not sophisticated". This is not useful, but it feels useful! It is like recycling cardboard boxes, all while knowing that you, as an organism, are primarily oriented towards boiling the oceans and turning the planet into Venus as quickly and efficiently as possible. Remember that today's small stone crab claw is tomorrow's "extra large" stone crab claw, because all the big ones got eaten and that's how generational amnesia <https://www.natural-solutions.world/blog/how-can-we-stop-we-need-to-work-on…> works! In other words, while STORM-0558's operation against Microsoft was slick like oil across the ever-more-hot waters of the Gulf of Mexico when it happened, the million teams doing the exact same Active Directory tricks the next month were just small fish, despite their impact on targets. And most big-impact operations could have been done by second-tier penetration testing teams, let alone nation-state adversaries. You will, if you work in Cyber Policy long enough, see people make tables of operations which compare various hacks from over the last fifteen years, which is like comparing the bite strengths of a Cretaceous monster to your average modern iguana. Likewise, most of Policy-world is, like we all are, obsessed with 0day. We like to count them with the enthusiasm of a vampire puppet on a children's TV show! But we also know that finding 0day is not a sign of sophistication so much as finding the right 0day at the right time. I don't know how to classify Orange Tsai's PHP character innovation but because it doesn't fit neatly into a spreadsheet, it might as well not exist? "If AI finds 0day, then it must be regulated" is a fun position to take in the many fancy halls and tedious Zoom calls where a pompous attitude and an ill-fitting suit are table stakes for attendance and having actually written code that uses Huggingface is considered bad form. But regulating technologies that can find 0day is a dead end. The current best way to find 0day is fuzzers and the dumber they are, the better they work most of the time. When it comes to operations, the current best way to hack is to email people and ask them for their password? Is that still true? Or do we just all look through huge databases of usernames and passwords that have already leaked and just use those now? I'm sure AI can also do that, but I'm also sure that it doesn't matter. I try to tell policy people this: What makes a scary adversary is huge resources, huge motivation, and huge innovation. The big names will probably use AI to automate different things, but nobody is going to saddle up some hamsters and suddenly turn from a not-scary adversary into a scary one. -dave

1 0

GDB Dances and the Moon
by Dave Aitel 08 Jun '24

08 Jun '24

People occasionally read my blogposts <https://cybersecpolitics.blogspot.com/2024/04/what-open-source-projects-are…>on Jia Tan <https://cybersecpolitics.blogspot.com/2024/04/the-open-source-problem.html>and then ask me about open source development in general, and you can only, in your darkest heart of hearts (your only heart) laugh. The other day I was contributing to a project that I am one of several developers on. In particular, I wrote a GDB script that traces through a function, printing out all the various variables and their sizes and this gets fed into an LLM to try to reason about it, which is a bit like asking a hedgehog how big a Unicode string should be to fit around the moon, but it was worth a shot, ya know? I have the kind of dyslexia that means I can't tell matrix algebra from a thinking conscious creature. Anyways, while I am good at making GDB dance in particular ways, like knowing the ancient art of the Polka, I am not good at modern software development, and barely understand GIT or Docker or Cloud things. But I have hacked a few things, like ya'll have, and so my development happens in a VM and that VM has access to pretty much just the source code it needs and not a whole lot else. But that's not how modern development works. It's common to see instructions to run "gcloud auth" and then walk through the web authentication portal from Google so your current user can access cloud buckets and APIs while testing or debugging your giant microservice. Like, people are out there just raw dogging source code from random other open source developers, with their local environment running tokens that give them access to everything they could possibly need from their Google account. People out there running curl www.badstuff.biz/setup | sh. People, and by this I mean developers, are lost storing five thousand fine grained GitHub tokens in various text files on their hard drive because they can't remember which one was which. In other words: Jia Tan might have been a best case scenario for this community. -dave

1 0

What a failure of Secure by Design looks like: Web Browsers
by Dave Aitel 04 Jun '24

04 Jun '24

I know it's in vogue to pick on enterprise hardware marketed to "Secure your OT Environment" but actually written in crayon in a language made of all sharp edges like C or PHP, with some modules in Cobol for spice. This is the "Critical Infrastructure" risk du jour, on a thousand podcasts and panels, with *Volt Typhoon* in the canary seat, where once only the "sophisticated threat" Mirai had root permissions. As embarrassing as having random Iranian teenagers learn how to do systems administration on random water plants in New Jersey is, it's *more* humiliating to have systemic vulnerabilities right in front of you, have a huge amount of government brain matter devoted to solving them, and yet not make the obvious choice to turn off features that are bleeding us out. And when you talk about market failure in Security you can't help but talk about Web Browsers, both mobile and desktop. Web Browsing technology is in everything - and includes a host of technologies too complicated to go into, but one of the most interesting has been Just in Time compiling, which got very popular as an exploitation technique (let's say) in 2010 <http://www.semantiscope.com/research/BHDC2010/BHDC-2010-Slides-v2.pdf> but since then - for over a decade! - has been a bubbling septic font of constant systemic, untreated risk. Proponents of having a JIT in your Javascript compiler say "Without this kind of performance, you wouldn't be able to have GMail or Expedia!" Which is not true on today's hardware (Turn on Edge Strict Security mode today and you won't even notice it), and almost certainly not true on much older hardware. The issue with JITs is visible to any hacker who has looked at the code - whenever you have concepts like "Negative Zero <https://googleprojectzero.blogspot.com/2020/09/jitsploitation-one.html>" that have to be gotten perfectly every time or else the attacker gets full control of your computer, you are in an indefensible space. I would, in a perfect world, like us to be able to get ahead of systemic problems. We have a rallying cry and a lot of signatories on a pledge, but we need to turn it into clicky clicking on the configuration options that turn these things off on a USG and Enterprise level, the same way we banned Russian antivirus from having Ring0 in our enterprises, or suspiciously cheap subsidized Chinese telecom boxes from serving all the phone companies across the midwest. The issue with web browsers is not limited to JITs. A Secure By Design approach to web browsing would mean that most sites would not have access to large parts of the web browsing specification. We don't need to be tracked by every website. They don't all need access to Geolocation or Video or Web Assembly or any number of other parts of the things our web browsers give them, largely in order to allow the mass production of targeted advertising. If we've learned anything in the last decade, it is that the key phrase in Targeted Advertising is "Targeted", and malware authors have known this for as long as the ecosystem existed. The reason your browser is insecure by default is to support a parasitic advertising ecology, enhancing shareholder value, but leaving our society defenceless against anyone schooled enough in the dark arts. Google's current solution to vulnerabilities in the browser is Yet Another Sandbox. These work for a while until they don't - over time, digital sandboxes get dirty and filled with secrets just like the one in your backyard gets filled with presents from the local feral cat community. I know Project Zero's Samuel Groß is better at browser hacking than I am, and he personally designed the sandbox, but I look out across the landscape of the Chinese hacking community and see only hungry vorpal blades and I do not think it is a winning strategy. -dave References: 1. Microsoft's Strict mode turns the JIT off (kudos to Johnathan Norman) https://support.microsoft.com/en-us/microsoft-edge/enhance-your-security-on… 2. The Sandbox: https://v8.dev/blog/sandbox

4 5

Re: Excellent piece by Chris Rohlf - " No, LLM Agents can not Autonomously Exploit One-day Vulnerabilities "
by Arun Koshy 24 Apr '24

24 Apr '24

>>What matters is speed of the exploit and speed of patching. And I see the humans (patching) on the losing side of this race. This is probably an independent issue ( imvho ). Re LLMs and present AI / ML regime, my only public comment is that we're in the Hindenburg [1] era .. caveat emptor. Another insightful paper that probably will be ignored this summer: https://arxiv.org/abs/2308.03762 ( author : https://people.csail.mit.edu/kostas/ ) [1] - https://en.wikipedia.org/wiki/LZ_129_Hindenburg

1 0

A Familiar World of Chaos
by Dave Aitel 23 Apr '24

23 Apr '24

After spending some time looking at "Secure by Design/Default" I have no doubt many of you feel like something is missing - something that's hard to put your finger on. So you go back to the treadmill of reading about bugs in Palo Alto devices, or the latest Project Zero blogpost, or something the Microsoft Threat Team is naming RidonculousBreeze, or whatever. For those of you who chose to read the latest Project Zero post, one way to look at Mateusz Jurczyk's vast destruction <https://googleprojectzero.blogspot.com/2024/04/the-windows-registry-adventu…> of the Windows Registry API, resulting in what can only be described as a "boatload" of Local Privilege Escalations, is that securing legacy code is hard, there's a talent shortage in how many people want to do the reverse engineering work necessary to understand and fix complicated and critical old code, and our investments in automated security engineering toolkits and better software development practices, while valuable, have not paid off in the kind of hardened Rust-only systems we dreamed about. Another way to look at this kind of wholesale destruction, a true tour de force, is that you cannot both put advertisements in your Start menu, and develop a secure operating system, for reasons that are more philosophical than technical. It's ironic that it is often Google that demonstrates this about other vendors, when of course, the lack of any ad blocking in Chrome or Android presents the exact same dilemma. You can't both make your systems secure, and sit beside the great river of Advertising Revenue with a ladle, dipping it in every quarter to fill up a cauldron of greater and greater value for the shareholders. It's hard to draw a straight line from an internal PowerPoint slide saying "Ads in the Start Menu are a good idea, actually" to the inevitable conclusion of 0days, ransomware, and US Government emails are being read by some old Russian who understands cryptography and Azure keys better than you were hoping. But in some respect this cause and effect is as fundamental and simple as how that tattoo on your arm is actually there because one night you decided to start off with shots of Limoncello. When Project Zero started, and even when it got to the towering behemoth of talent that it is now, I knew people in the offensive industry who were quite scared of it - of the possibility that a large and funded team of top researchers, with access to one of the only five real computers on the planet, could drain the lake of software vulnerabilities we all fished in. But I had no such fears. An organization so dependent on advertising revenue to survive can no more fix systemic security issues than a Sperm Whale can medal in Olympic Skiing. It is contrary to their very nature, although they will probably smash a bunch of trees on the way down. Like many of you, I spent my Saturday porting code to use LLAMA3:70b, largely by annoying my 18yo with questions about ollama and Docker, since I find modern Linux system administration as foreign as an octopus finds calculus. But search engines, like surface warships, are clearly on their last legs. They went from something you used every day, multiple times a day, to something your LLM uses for you, as just one tool among many. It is, for reasons that must be obvious even to executives drunk on the heady fumes of their stock options maturing, hard to make money selling advertisements that are only read by LLMs. But having spent the better part of a couple years doing LLM work now, I feel like I understand why these behemoths are investing so much money in them, despite the obvious cannibalization of their cash umbilical. It's because they can! There's just not that many businesses that generate ten billion dollars of revenue year on year to get into. You've got some elements of manufacturing, tech, education, health care, video games. It's not a big list. Apple gave up on manufacturing cars because the niche they wanted (impractably weird and expensive) was already filled by Tesla. But by investing in LLMs and AI in general you kinda get to put your thumbs in every other billion dollars business all at once. It's a straight line shot from something you already know, to the next place. So of course, they are throwing dollars at it like it was the only thing they knew how to do. And what we get is the pretentious superiority of ChatGPT, or the sanctimonious holiness of Claude, or the ever-sadness of Gemini, the impertinence of Mistral or the trollishness that is LLAMA. A world of chaos, yet something so familiar. -dave

2 1

Sophia D'Antoine
by Dave Aitel 17 Apr '24

17 Apr '24

On Monday, I and 400 other people, including many on this mailing list, attended Sophia's funeral in a huge church in the upper east side of NYC. Although I grew up in a Jewish household, I am not religious, and the last time I went to a church was also with Sophia, in Jerusalem, where we wandered through various landmarks until we ended up at the Church of the Holy Sepulcher, one of the holiest sites for Christianity. We waited in a line of fellow pilgrims, the room lit only by echos and dim lamps to touch the Stone of Anointment, where Jesus's body was said to have been prepared for burial. The rock was wet, from some unknown source below. We each knelt and touched it briefly, respectfully. The concept of prayer is foreign to me, but I spoke to myself for a moment with my hand on the slick rock, and tried to feel. Afterwards, Sophia said to me, "You can't expect to _feel_ anything.", something I've often pondered since that moment. Sophia had many friends, and for many people, Sophia was one of their closest friends. No one person can own her memory, but like many others, the moments I had with her were important to me. Now that she's gone, these moments are what we have left. The community that came to NYC to support Sophia's family, and each other, was just as stunned as I am. When not sharing memories about Sophia, we talked about exploits, and large language models. I saw people I hadn't seen for over a decade, or people I knew, but had no idea knew Sophia. Some people traveled from across the entire globe to say goodbye. The community was all on the same path, and the many photos of Sophia laid out on the reception tables were like a shattered mirror portal into her life. Woven into the fabric of who she was, was Sophia's work, and her desire to do what was necessary to get the work done. She did not start a company to get rich. She lived a life of quiet professionalism and when global events moved, she was often part of that fell force that moved them. The shape of the world is a part of her legacy, but even more so I see her legacy as an example that you can live a life of sacrifice and integrity. You can be there for people. You can make it look effortless. This week the Host gathered around a silence folding endlessly upon itself. I feel privileged to have known her, to have shared the time we shared. I know those of you reading this who were also a part of her life feel the same. Rest in Peace, Sophia. We miss you. -dave

1 0

do androids dream of electric sheep in JSON or XML?
by Dave Aitel 02 Apr '24

02 Apr '24

[image: image.png] Like everyone I know, I've been spending a lot of time neck deep in LLMs. As released, they are fascinating and useless toys. I feel like actually using an LLM to do anything real is your basic nightmare still. At the very minimum, you need structured output, and OpenAI has led the way in offering a JSON-based calling format which allows you to extend it with functions that cover the things an LLM can't really do (i.e. math, or access the web or your bash shell). In real life you are going to use this through LangChain or some similar library. You can do this sort of thing with Claude (a better model than GPT-4 in many respects for code), but it's janky as the model wasn't specifically fine tuned for this purpose yet. Your best bet, as you see everyone do, is to force it to start its reply with a curly bracket "{", but even then it's going to pontificate about its reply after it sends you the JSON object you want, if you're lucky and it uses your format at all. Claude is more based on XML than JSON, which, if you think about how LLMs work, makes a ton of sense. To an LLM, {' may be one token, or {{{{ may be a token. In fact, let's test that: >>> import tiktoken >>> encoding = tiktoken.encoding_for_model("gpt-4") >>> encoding.encode("{") [90] >>> encoding.encode("{{") [3052] >>> encoding.encode("{{{") [91791] #one token >>> encoding.encode("{{{{") [3052, 3052] #two tokens >>> encoding.encode("{{{{{") [3052, 91791] #two tokens >>> encoding.encode("{{{{{{") [3052, 3052, 3052] #three tokens >>> encoding.encode("{'") [13922] #{ ' is one token Yeah, so like, on one hand, that's great. Very optimized compression on token lengths. But on the other hand, it is very confusing for the model to train on and understand! You can see why XML would be much more natural! < SOME WORD > is more likely to be three tokens, which makes creating clean output much easier. Claude's focus on XML probably makes it "smarter" in some ways that are hard to prove with math. >>> encoding.encode("<high>") [10174, 1108, 29] >>> encoding.encode("<html>") [14063, 29] >>> encoding.encode("</html>") [524, 1580, 29] >>> encoding.encode("/html") [14056] >>> encoding.encode("</high>") [524, 12156, 29] >>> encoding.encode("high") [12156] Also, of course, I highly recommend Halvar's latest talk (which is highly relevant): https://www.youtube.com/watch?v=xA-ns0zi0k0&t=4s -dave

1 0

Bugdoor vs Backdoor
by Dave Aitel 01 Apr '24

01 Apr '24

[image: image.png] The security community (aka, all of us on this list) still rages with the impact of Jia Tan putting a sophisticated backdoor into the XV package, and all of the associated HUMINT effort that went into it. And I realized from talking to people, especially people in the cyber policy realm but also technical experts, about it that there's a pretty big gap when it comes to understanding why someone would put in a backdoor at all, instead of adding many bugdoors. Some Background: 1. A post <https://cybersecpolitics.blogspot.com/2019/05/hope-is-not-nobus-strategy.ht…> on what NOBUS means when it comes to backdoors. 2. Responsible offense from a bunch of Americans <https://www.lawfaremedia.org/article/responsible-cyber-offense> 3. Responsible offense from the UK <https://www.gov.uk/government/publications/responsible-cyber-power-in-pract…> 4. Responsible offense from the Germans <https://www.stiftung-nv.de/sites/default/files/snv_active_cyber_defense_tow…> 5. A university banned from Linux <https://www.theverge.com/2021/4/30/22410164/linux-kernel-university-of-minn…> for contributing backdoors as part of a research project So as with all areas of responsible offense, there is a tight connection and contention between good OPSEC and responsible operations. In particular, it is very easy to get yourself on a team for a big project, and add code that introduces exploitable conditions, perhaps handles input in a way that causes a memory corruption, or does authentication slightly wrong in certain circumstances. From an operational security standpoint, these bugdoors are easy to introduce, and I don't know of a serious hacking group that hasn't played with this - if for no other reason than to fix bugs that cause crashes while you are trying to exploit some other, better bug. Reading the original UMN paper, (which was under-appreciated for its time, despite getting them banned from Linux!) you can see that it is not really always about adding bugs, but often about adding enabling features for bugs that already exist in the codebase, making them more reachable. In some ways, attacking the open source community by hacking into developers or repositories has been the traditional way of ancient Unix 90's hackers, who understand a web of trust the way a Polynesian navigator understands the swells and currents between islands. From an opsec perspective though, bugdoors have limits. Fuzzers can find them, other hackers can find them, and once found, they can be used by anyone with the skill to write the exploit. Likewise, using them is risky: No memory corruption is 100% reliable, and when they fail, *they fail in the worst way, in the worst place, at the worst time*. Likewise, the traffic you may have to do to shape memory in the target host is likely to be anomalous, and easily signatured. And from a responsible offensive cyber operation perspective, bugdoors cannot mathematically demonstrate that they can protect the hosts you target from third parties. *Bugdoors are never a NOBUS capability.* Ideally a NOBUS capability would allow you and only you to get in and avoid replay attacks, but a close second is a simple asymmetric key of some kind where the target ID is used as part of the scheme. The XZ backdoor used an Elliptic Curve with a signature that included the target's SSH public key <https://github.com/amlweems/xzbot>. Thanks, Dave

1 0

t2'24: Last Dance
by Tomi Tuominen 27 Mar '24

27 Mar '24

Dear Daily Dave, For a hacker conference, twenty years is a huge achievement — for a small conference, even more so. Over these years we’ve enjoyed speakers showcasing results from cutting-edge research, seen thought-provoking keynotes and bonded with other like-minded people from all over the world. If we had to summarize the experience with one word, it would be gratitude. The speakers, repeat speakers, first timers or regular attendees, and friends of t2 — you have made the event and its atmosphere. This was always a true community event – it’s organized for hackers, by hackers. The Advisory Board’s motivation and main driver was our love for the game. Creating a small event with a curated program, offering a backdrop for lobby bar and coffee break discussions was (and still is) our vision of a perfect infosec con. The chance to network with your industry peers was as integral part of t2, as the high quality content. It’s rare you get the same level of interaction with current/former speakers and attendees alike at any other conference. Tomi has fond memories of pretty much each and every year. Starting from the humble beginnings, when the legendary Phenoelit guys were kind enough to come and present at a conference that back then had no history nor reputation. How the Toolcrypt guys dominated the stage for years with their absolutely amazing research, and how Ivan Krstić (now Apple’s security samurai) shared his ideas on how modern security architectures should be built (iDevices anybody?), how the InversePath crew delivered some of the most enjoyable and hardcore research ever and well, you get the idea – the list just goes on and on – there are simply too many good memories to list here. Mikko remembers learning from Ludde (during a t2 coffee break) how he works at Spotify. Then Mikko explained how impressed he was with Spotify’s early beta version, especially how you could skip parts of a song and it would still continue streaming instantly. Ludde nodded…and said ‘yeah…I coded that’. Henri still reminisces how Halvar Flake took the time after his talk in 2010 to have a chat with him and Esa Etelävuori (RIP), despite Halvar feeling slightly under the weather in the midst of what later turned out to be the Zynamics acquisition by Google. In 2017 we enjoyed the late night/very early morning pizza in the hotel lobby bar with Dave Aitel, after proving him wrong. Instead of dropping a surprise announcement sometime next year, or silently disappearing into the crowd, we wanted to let everyone know before this year’s t2 infosec that 2024 will be our last dance. We have truly enjoyed the past two decades of world class cyber in Helsinki – all good things come to an end eventually. From the bottom of our hearts, a big thank you to all of you who made this happen. We are privileged to be able to call many of you out there our friends. This goes especially to Dave, thank you for treating us so well over the years. Tomi Tuominen Mikko Hyppönen Henri Lindberg

1 0

while True:
by Dave Aitel 24 Mar '24

24 Mar '24

There seem to be a lot of people who think the problem with cyber security is we aren't paying lawyers enough. This results in the current push for software liabilities, or the need to click accept on cookies before we use every website. It is natural for lawyers to want to feed the next generation of associates, by regurgitating legal koans into their mouths. These vomitous truisms pass for thought leadership when you go high enough into the cyber policy clouds. "We don't know what we don't know!", "You can't manage what you can't measure!" , "We need to be Secure to Market not First to Market!", "Crawl, Walk, Run!" These statements are the opposite of haikus, which when done right are one crystalized emotional moment. This is why I think maybe we should hire more poets to do cyber policy, instead of lawyers. What is "*i carry your heart with me(i carry it in*" other than the first line to an exploit written in a bash script we all forgot existed but our spirits remembered? When LSD-PL said: bn,a bn,a call did that not carry with it an emotive punch? When we hack, do we not reach within our internal well of hate to pull forth a tiny amount of darkness and then send it into the world on tiny flaxen wings? I cannot do a survey on this in any language that matters, but I look into the net and see all the ancient hackers I grew up with still crouched in full armor, their ocher swords smoldering. This week in between cyber policy calls at 0500, I sat for hours, choking on the byzantine syntax of LangChain attempting to wrestle an LLM into submission. I kept thinking, what would Horizon do? What would Shubs do? What would James Kettle do? What would Tiraniddo do? What would Chompie do? What would Skylar do? I told myself: They would *continue*, is what they would do. -dave

1 0

Value of the [leaked] Windows source
by Konrads Klints 06 Mar '24

06 Mar '24

Windows XP and Windows 2003 partial source code is out there on github. With such a rich corpus of known vulnerabilities in those OS'es and source code availability, surely there should be an amazing amount of SAST/semgrep/codeql rules that take as input existing known exploits and then do rules that find similar things, yet I don't seem to be able to find such projects Surely, these two code bases should be the foundation of most good CS/cyber courses - like students finding new bugs, etc. Is source code junk?

2 1

0xC15A: Secure By Design and Secure by Default
by Dave Aitel 24 Jan '24

24 Jan '24

So I have a ton of thoughts on the CISA Secure by Design and Secure by Default push that is ongoing, as I am sure many of you do. And the first thought is: This is not a bad way to go about business as a government agency in general. I think it's easy to ignore how fast the USG has changed its business practices, showing an agility that few large organizations can match. In particular using Secure By Design as a case example. 1. Massive outreach to garner feedback (including at defcon, but also via email, etc.) 2. Multiple rounds of editing of proposals 3. Actual people you could call and talk to about the proposal, with their faces and positions listed right in the papers and blogs and lawfare podcasts. If you were in DC today you could probably hit one of them up for drinks or lunch or whatever. 4. Interaction across multiple stakeholder groups, including internationally 5. The "right people" involved - and you can tell their backgrounds from what they are annoyed about during their podcasts and other presentations. (i.e. Bob Lord is very annoyed about XSS and obsessed with car safety, which I'll dig into later). But also Jack Cable, Lauren Zabierek and Grant Dasher are all worth listening to. 6. Clear executive support So that's all good stuff. I thought I would post it as its own note because it's rare to spend a moment to look at the government process, and not see literally sausage being made. :) -dave

3 3

Secure By Default Part 2
by Dave Aitel 19 Jan '24

19 Jan '24

So I wrote a little draft essay on Secure By Default and opened it for comment. I think one thing that we maybe forget in our community is that some of the more fundamental basises of what we do never make it up to policy-world. Langsec being the primary example. But also there's a huge body of work in TAOSSA, Shellcoders, every offensive conference talk, etc. that never gets put into context anywhere but in our clique. Obviously feel free to just comment in-thread if you prefer, even if you work at CISA: https://mastodon.social/@dave_aitel/111779922142416342 Thanks, Dave

1 0

Leverage
by Dave Aitel 27 Dec '23

27 Dec '23

So we know a lot of people who've gone into Big Corpo or sold a company or just worked hard and gotten lucky and happen to be richer than the average bear. And while a lot of those people put their money into nice things, nothing wrong with that, a lot of them also try to use that money to change the world, and then they find out it's harder to change the world with money than it is with an exploit. And I know a lot of people who say this out of experience. I used to say, and I continue to say, that most cyber policy experts have never seen a real exploit. Yes, even the news reports of 0days are addictive. They have cool names, like a street drug, they have a shadowy underworld, they have a bodyguard of rumors and insinuations. Literally, as I write this, Kaspersky is at CCC doing a street barker presentation on how much someone thought they were a legitimate target worth throwing down an iOS 0day chain on, which frankly is not something I would brag about as a nominally defensive company. Part of the problem is we analyze exploits out of failures. Reverse engineering an exploit does not show you the exploit any more than dissecting a Humboldt squid can show you the terrors of the deep. Once you've seen an exploit change the world, you are forever stunned. It is impossible to go backwards to your previous life. It is like your first taste of sex or love or the feral inhuman joy of combat. For some people, it's probably better. But how exploits really work has shockingly little to do with the circus around exploits on social media or in the news, or press releases from endpoint security companies or government agencies. And the same is true with philanthropy. Here we have "Give Miami Day!" where some random billionaire will match your funds if you give to your school, which should be getting all the government funding it needs, but clearly isn't. I don't know if this is at all useful to be honest. It feels more like covering a hole your government put in your school budget on purpose. I tried buying MEL science kits for entire classes of my local grade school. It worked once, and one of the kids was like "I didn't even know I liked science". But it was largely impossible to get the kits USED by the teachers, who are under levels of logistical stress that would stymie a Marine platoon. So I ended up giving up on this effort. I supported Project Grapple, which worked well because they have a leader at the top who is focused on the success of a small set of kids in a very personal way. But those kinds of leaders are almost impossible to find. When you find them, it's the best investment you can make if you want to change the world one kid at a time, which might be the only way. And these leaders don't last forever. A lot of big donors focus on things like FIRST Robotics, which frankly has been a massive success and offers a lifeline for kids in schools where nothing else matters or meets with any success. It's prohibitively tricky to figure out which schools have a local leader that can take money and build a robotics club out of it. It's very much not important that the kids WIN the competitions - which at the top level are between Northrop Grumman, Raytheon, and Lockheed Martin engineering teams doing ad-hoc group apprenticeships. So again, finding these leaders is a constraining problem. Just like with exploits, what I find is that changing the world with philanthropy is targeted , personal, and more complicated than it looks. I have seen very smart people struggle with finding leverage in this space. Nonprofits are themselves often quite exploitative of their employees or just generally ineffective. Anyways, what I'd like to see is the Gulas and Syversons and Alperovitch's and Pollocks and so forth put a Slack channel together to build a bit of a body of work on how to do this correctly. A bugtraq for changing the world with dollars, if you will. -dave

2 1

A holiday reflection on: Training.
by Dave Aitel 19 Dec '23

19 Dec '23

I think one thing this community does really well, better than almost any other community I've found, is training. It's amazing, in a way, because this is a community of professional secret holders. And yet everywhere you look, a hacker is putting their heart and soul into iterating on lab exercises for their class in whatever sub-field they are an expert in. And giving training is hard. It's hard in the way consulting is hard, but with even more social activity. On one hand: It's lucrative? But hour for hour, you're probably better off financially by finding a new bug or doing consulting work, or any number of other activities than building, marketing, and running a training class. When I left my last position, one of the first things I did was pay for and take Amy Burnette's browser exploitation class. And that's paid off to this day, really. And there's so many good classes, taught by all the specialists in our various sub-niches. It's spectacular that in this world of auto-didacts, we are gifted in the quantity and quality of training available in our field in a way that is basically unheard of in any other field. Of course, there's a lot of things you can't learn from training, and I was reflecting on this while sitting down and reading the labyrinthine specifications of some huge protocol for one of my current projects. A lot of the best bugs I've ever seen hackers find have been from doing exactly that: They sit and hit page down on some extensively huge and boring documentation with the steady, persistent rhythm of a neurodivergent woodpecker pecking at a tree, each tap bringing them closer to the elusive kernel of truth. Like, I know people who have various protocol RFCs printed out for long airplane rides. I've seen hackers read through a book on an operating system design and then just circle an LPE in the book with a yellow highlighter. On the flip side, there are times when you dive headfirst into a colossal specification, emerging as a veritable guru on an esoteric legacy mail transfer mess like X.400. Yet, despite this newfound expertise, you find yourself no more enlightened or advantaged than before, as if you've scaled a mountain only to find the summit shrouded in the same thick fog that cloaked its base. Anyways, happy holidays everyone. Hopefully you had a year of worthy discoveries. -dave

1 0

t2'24: Call For Papers 2024 (Helsinki, Finland)
by Tomi Tuominen 17 Dec '23

17 Dec '23

Call for Papers 2024 t2 infosec has been pushing the boundaries of security research for two decades and it don't stop. We're back April 18-19, 2024 - Helsinki, Finland. CFP and registration are both open. This is an event for newcomers, established merchants of dual use computer code, beginners of vulndev, cyber vagabonds, retired or redacted, and hackers of all sorts. If you have new original security research targeting old, current or future technology, please submit: https://if.t2.fi/action/call_for_papers Got 99 problems but an 0day ain't one. Helsinki offers a Northern European mood, with the resilience built by using Linux on the desktop since 1991, fueled by IRC, demoscene, VHS tapes, invention of modern backpropagation, a lot of memes and sauna. This is the country, which exports Alan Wake 2, quantum computers, and solitude. As regularly as seasons change, new technology is introduced, vulnerabilities - old and new are discovered, lessons of tradecraft are learned and teachings of the old slowly forgotten.To foster the passion for advancement of (in)security and to keep that fire burning bright for future generations, we gather an intimate crowd of people every year to enjoy the advance of offensive research, share lessons of coming back from the edge, and build those valuable human connections. Whether your research tackles LLMs, qubits, Azure/AWS/GCP, iOS internals, the market leading or lagging EDR products, access control systems, SoC, BTLE, tipping ChatGPT or something else, which is surprising to either humans or computers, we'd like to hear about it. Security researchers, you know the rules and so do we. This is an event for the community, by the community. Our focus is on technical excellence, not politics or player hating. Come as you are. The advisory board will be reviewing submissions until 2024-02-04. Slide deck submission final deadline 2024-04-02 for accepted talks. First come, first served. Submissions will not be returned. Quick facts for speakers + presentation length 60-120 minutes, in English + complimentary travel and accommodation for one person[6] + decent speaker hospitality benefits + no marketing or product propaganda Still not sure if this is for you? Check out the blast from the past. !!LINK!! [0] hunter2 [6] except literally @nudehaberdasher and @0xcharlie Call for Paper submissions https://if.t2.fi/action/call_for_papers Buy a ticket to the conference https://if.t2.fi/action/register/attendees -- Tomi 'T' Tuominen | Founder @ t2 infosec conference | https://t2.fi

1 0

what is reasoning?
by Dave Aitel 05 Oct '23

05 Oct '23

There's not a huge paper I can find on the history of all the big bug classes. I mean, there's probably not even a real definition of "bug class" that would survive a drunken conversation between two CTF teams. But like, screw it, "format strings" are a bug class. "Deserialization bugs" are a bug class. "strcpy and friends" is a bug class. You know what I mean if you're the type of person who subscribed to this list. One thing you know if you internally describe yourself as "hacker" and not "security professional" is that the bug classes of the future are here now, but the commercial world doesn't see them yet or sees them as one-off issues. For example, check out this amazing slide from this GoSecure presentation ( https://gosecure.github.io/presentations/2019-04-29_atlseccon/History_of_De… ) [image: image.png] If you can't see the picture, it was 9 years from "first public instances" to "bell curve of public mass use of bug class". The nightmare for the commercial industry of course is that this ignores private research, which looks like a much bigger bell curve, superimposed on the whole thing as you all know. If you've been working on writing automated tools to find bugs, then you also know the hardest thing to find is a new bug class. We can't all be Rain Forest Puppy! But even building things to replicate how hackers think when they just want to find Deserialization or strcpy bugs is hard. Like, what does it mean to even understand a bug? Bugs are not on one line! They are not even one concept! Can you inject a bug into a C program without also injecting another bug you didn't even intend on injecting? Even though humans are terrible at logic, we somehow think that stepping logically through a problem is how reasoning works. But you have to get over this! Stare at an ant farm and chant to yourself the neo-Buddhist mantra of "This is reasoning. This is cognition" and this way you quickly realize that precision is not the answer when you try to go from building a calculating machine to a reasoning machine. Precision and reasoning are opposites! In other words, the closest we have to reasoning over program behavior is fuzzing. Once you understand that, a lot of other stuff starts to make sense. -dave

1 0

DARPA AI Challenge!
by Dave Aitel 29 Aug '23

29 Aug '23

I've been working with LLMs for a bit, and also looking at the DARPA Cyber AI Challenge <https://www.darpa.mil/news-events/2023-08-09>. And to that end I put together CORVIDCALL which uses various LLMs to essentially 100% find-and-patch any bug example I can throw at it from the various GitHub repos that store these things (see below). [image: image.png] So I learned a lot of things doing this, and one article that came out recently (https://www.semianalysis.com/p/google-gemini-eats-the-world-gemini) talked about the future of LLMs, and if you're doing this challenge you really are building for future LLMs and not the ones available right now. One thing they pointed out in that article (which I highly recommend reading) is that huggingface is basically doing a disservice with their leaderboard - but the truth is more complicated. It's nice to know which models do better than other models, but the comparison between them is not a simple number any more than the comparison between people is a simple number. There's no useful IQ score for models or for people. For example, one of the hardest things to measure is how well a model can handle interleaved and recursive problems. If you have an SQL Query inside your Python code being sent to a server, does it notice errors in that code or do they fly under the radar as "just a string". Can the LLM handle optimization problems, indicating it understands performance implications of a system? Can the LLM handle LARGER problems. People are obsessed with context window sizes but what you find is a huge degradation of accuracy in following instructions when you hit even 1/8th the context window size for any of the leading models. This means you have to know how to compress up your tasks to fit basically into a teacup. And for smaller models, this degradation is even more severe. People in the graph database world are obsessed with getting "Knowledge graphs" out of unstructured data + a graph database. I think "Knowledge graphs" are pretty useless, but what is not useless is connecting unstructured data by topic in your graph database, and using that to make larger community detection-based decisions. And the easiest way to do this is to pass your data into an LLM and ask it to generate the topics for you, typically in the form of a Twitter hashtag. Code is unstructured data. If you want to measure your LLM you can do some fun things. Asking a good LLM for 5 twitter hashtags in comma separated value format will work MOST of the time. But the smaller and worse the LLM, the more likely it is to go off the rails and fail to do it when faced with larger data, or more complicated data, or data in a different language which it first has to translate. To be fair, most of them will fail to do the right number of hashtags. You can try this yourself on various models which otherwise are at the top of a leaderboard, within "striking distance" on the benchmarks against Bard, Claude, or GPT-4. (#theyarenowhereclose, #lol) Obviously the more neurons you have making sure you don't say naughty things, the worse you are at doing anything useful, and you can see that in the difference between StableBeluga and LLAMA2-chat, for example, with these simple manual evaluations. And this matters a lot when you need your LLM to output structured data <https://twitter.com/RLanceMartin/status/1696231512029777995?s=20> based on your input. So we can divide up the problem of automating finding and patching bugs in source code in a lot of ways, but one way is to notice the process real auditors take, and just replicate this by passing in data flow diagrams and other various summaries into the models. Right now hundreds of academics are "inventing" new ways to use LLMs. For example "Reason and Act <https://blog.research.google/2022/11/react-synergizing-reasoning-and-acting…>". I've never seen so much hilarity as people put obvious computing patterns into papers and try to invent some terminology to hang their career on. And of course when it comes to a real codebase, say, libjpeg, or a real web app, following the data through a system is important. Understanding code flaws is important. But also building test triggers and doing debugging is important to test your assumptions. And coalescing this information in, for example, the big graph database that is your head is how you make it all pay off. But what you want with bug finding is not to mechanistically re-invent source-sink static analysis with LLMs. You want intuition. You want flashes of insight. It's a hard and fun problem at the bigger end of the scale. We may have to give our bug finding systems the machine equivalent of serotonin. :) [image: image.png] -dave

1 0

Book Review: Fuzzing Against the Machine (Antonio Nappa, Eduardo Blazquez)
by Dave Aitel 23 Aug '23

23 Aug '23

https://www.packtpub.com/product/fuzzing-against-the-machine/9781804614976 The authors claim in their conclusion: "We want to stress the importance of books as journeys to explore and experience topics from the unique viewpoint of the authors." And in this they succeeded. This book works best as a proposed curriculum for a five day workshop for experts to reproduce fuzzing frameworks that target embedded platforms - including Android and iOS. Largely this is done by figuring out how to get various emulation frameworks (QEMU in particular) to carry the weight of virtualizing a platform and getting snapshots out of it and pushing data into it. Fuzzing is a childishly easy concept that is composed of devilishly hard problems in practice (7 and 8 being the ones this book covers in depth - the fuzzers themselves are simplistic other than those topics): 1. Managing scale 2. Getting decent per-iteration performance 3. Triaging crashes 4. Building useful harnesses 5. Knowing when you have fuzzed enough, vs. being in a local minima 6. Figuring out root causes 7. *Getting your fuzzer to properly instrument your target so you can have coverage-guided fuzzing* 8. *Handling weird architectures* 9. Generating useful starting points for your fuzzer (or input grammars) All of these things are basically impossible in the real world. Your typical experience with a new fuzzing framework is that you install it on a fresh Linux, pick a target, and then watch as it fails to instrument or even run. In other words, just knowing which fuzzer versions to use, and on what, is valuable information. When I read a book on security, a good one, I want it to feel like I'm putting on a brand new powersuit, ready to march into the wilderness with a flamethrower and a mindset of extreme violence. This book delivers that feeling. Because while my current business practices have nothing to do with fuzzing the Shannon baseband, that doesn't mean some small part of me doesn't want to. We all have the dark urge. We crave SIGSEGV in things people rely on. So in summary: 10/10, great book. Would recommend buying 10, setting up a class, and going over it all together. Of course, this field is RAPIDLY EVOLVING and you're going to want to get it updated, perhaps with the fancy new PCODE fuzzer Airbus released earlier today. ( https://github.com/airbus-cyber/ghidralligator) -dave

1 0

BlackHat and Defcon 2023
by Dave Aitel 16 Aug '23

16 Aug '23

The Vegas security conferences used to feel like diving into a river. While yes, you networked and made deals and talked about exploits, you also felt for currents and tried to get a prediction of what the future held. A lot of this was what the talks were about. But you went to booths to see what was selling, or what people thought was selling, at least. But it doesn't matter anymore what the talks are about. The talks are about everything. There's a million of them and they cover every possible topic under the sun. And the big corpo booths are all the same. People want to sell you XDR, and what that means for them is a per-seat or per-IP charge. When there's no differentiation in billing, there's no differentiation in product. That doesn't mean there aren't a million smaller start-ups with tiny cubicles in the booth-space, like pebbles on a beach. Hunting through them is like searching for shells - for every Thinkst Canary there's a hundred newly AI-enabled compliance engines. DefCon and Blackhat in some ways used to be more international as well - but a lot of the more interesting speakers can't get visas anymore or aren't allowed to talk publicly by their home countries. If you've been in this business for a while, you have a dreadful fear of being in your own bubble. To not swim forward is to suffocate. This is what drove you to sit in the front row of as many talks as possible at these two huge conferences, hung over, dehydrated, confused by foreign terminology in a difficult accent. But now you can't dive in to make forward progress. Vegas is even more of a forbidding dystopia, overloaded with crowds so heavy it can no longer feed them or even provide a contiguous space for the ameba-like host to gather. Talks echo and muddle in cavernous rooms with the general acoustics of a high school gymnasium. You are left with snapshots and fragmented memories instead of a whole picture. For me, one such moment was a Senate Staffer, full of enthusiasm, crowing about how smart the other people working on policy and walking the halls of Congress were - experts and geniuses at healthcare, for example! But if our cyber security policy matches our success at a health system we are doomed. I brought my kids this year and it helps to be able to see through the chaos with new eyes. What's "cool" I asked? in the most boomery way possible. Because I know Jailbreaking an AI to say bad things is not it, even though it had all the political spotlights in the world focused on examining the "issue". The more crowded the field gets, the less immersion you have. Instead of diving in you are holding your palm against the surface of the water, hoping to sense the primordial tube worms at the sea vents feeding on raw data leagues below you. "Take me to the beginning, again" you say to them, through whatever connection you can muster. -dave

10 10

More Neo4j
by Dave Aitel 03 Aug '23

03 Aug '23

https://www.youtube.com/live/YY-ugAHPu4M?feature=share&t=1057 I have on my todo list to reply to our thread last month, but in the meantime, here is a video that goes over all the lessons learned from my last couple years doing Neo4j. But as a reminder, we should still port Ghidra to Neo4j. :) -dave

1 0

Tooling, Graph Databases, etc.
by Dave Aitel 27 Jun '23

27 Jun '23

There's a new Ghidra release last week! Lots of improvements to the debugger, which is awesome. But this brings up some thoughts that have been triggering my vulnerability-and-exploitation-specific OCD for some time now. Behind every good RE tool is a crappy crappy database. Implicitly we, as a community, understand there is no good reason that every reverse engineering project needs to implement a key-value store, or a B-Tree <https://github.com/NationalSecurityAgency/ghidra/tree/master/Ghidra/Framewo…>, or partner with a colony of bees which maintain tool state by various wiggly dances. But yet each and every tool has a developer with decades of reverse engineering experience on rare embedded platforms either building custom indexes in a pale imitation of a real DB structure or engaging in insect-based diplomacy efforts. I think the Ghidra team (and Binja/IDA teams!) are geniuses, but they are probably NOT geniuses at building database engines. And reading through the issues <https://github.com/NationalSecurityAgency/ghidra/issues/985> with ANY reverse engineering product you find that performance even for the base feature-set is a difficult ask. My plea is this: We need to port Ghidra to Neo4j as soon as possible. Having a real Graph DB store underneath Ghidra solves the scalability issues. I understand the difficulty here is: There are few engineers who understand both Neo4j and reverse engineering to the point where this can be done. I mean, why do it in Neo4j and not PostGres? An argument can be made for both, in the sense that PostGres is truly Free and the most solid DB on the market. The pluses for Neo4j are that RE data is typically graph-based more than linear. I spent the last two years learning graph dbs, out of some masochistic desire and ended up getting certified - and I can still RE a little bit. I will manage the team porting Ghidra to Neo4j if someone funds it. :) Either way, sooner is better than later. There are so many companies and people relying on these tools that it seems silly to do anything else. -dave P.S. Yes, I remember BinNavi used MsSQL installs for its data, and this was annoying to install but ... I get why Halvar did it at the time. It's because he had real work to do and building a DB was not it. I can only assume Reven doesn't use their own DB? I mean the benefits for interoperability would be huge between tools. . . like literally everything you want to do with these tools is better with a real DB underneath.

5 4

A graph disassembler
by Dave Aitel 08 Jun '23

08 Jun '23

Lately I've been watching a lot of online security talks - the new thing for conferences to do is publish them almost immediately, which is amazing. So like, today I watched Chompie's talk: https://www.sstic.org/2023/presentation/deep_attack_surfaces_shallow_bugs/ (I was honestly hoping it went from RCE to logic bug and allowed you to log in, but maybe left as an exercise for the reader). And yesterday I watched Natalie's talk: https://www.youtube.com/watch?v=quw8SnmMWg4&t=663s&ab_channel=OffensiveCon (I'm still a bit confused as to how you connect to a phone's baseband with SIP, but maybe I will ask later at some point). Does the baseband just have some TCP ports open for RTC shenanigans? If so, that's great, #blessed, etc. I actually forgot to post my own talk here, so if you want to watch that, it's here: https://www.youtube.com/watch?v=BarJCn4yChA&t=1669s&ab_channel=OffensiveCon My talk is not actually a call to hack enterprise products - which ya'll are clearly already doing a lot of (I'm just assuming there are members of the FIN11 ransomware crew on this list somewhere). It's more about understanding which business models lead to easy bugs - enterprise software obviously being one of them, but, for example, DRM components are another one. There is an endless supply really. Today I noticed Barracuda is saying that if your appliance gets hacked <https://www.bleepingcomputer.com/news/security/barracuda-says-hacked-esg-ap…>, you should replace it immediately. It is now trash, or "e-waste" if you prefer. This is a surprisingly honest thing to say. Previously appliance companies who get hacked say things like "Meh? Upgrade pls! Don't forget to change your passwords!" because as we all know, the firmware <https://arstechnica.com/information-technology/2023/03/malware-infecting-wi…> and boot partitions inside expensive security appliances are all protected by angry leprechauns, which is why it's still ok in 2023 to have Perl installed on them, even if you don't know what a ../.../ does in a tar file. This honesty would be nice if it also applied to our government agencies - like instead of this very long report CISA <https://www.cisa.gov/news-events/cybersecurity-advisories/aa21-110a#:~:text….> put out about what to do if you think your Pulse Secure VPN was hacked, which recommends performing a factory reset, updating your appliance to the very latest version, and then calling your therapist to have a good cry about it, they should have instead said: "Yes, your appliance did at one point control authentication for everyone accessing your network, but because it had issues with gzip files and opening URIs, it is now e-waste." Crap, I forgot to write about the graph disassembler I want and why. Tomorrow, for sure. -dave

1 0

Yawps from the rooftops
by Dave Aitel 05 Jun '23

05 Jun '23

[image: image.png]https://twitter.com/thezdi/status/1638617627626176513 [image: image.png] Yawps So one thing I have as a "lessons learned" from the past 20 years is that security is not a proactive sport. In fact, we are all experts at running to where the ball _was_as opposed to where it is _going_. Like, if you listen to Risky Biz this week, Patrick asks Metlstorm whether it's time to go out and replace all the old enterprise file sharing systems <https://twitter.com/vxunderground/status/1641629743534559233?s=20> you have around, proactively. And the answer, from Metl, who's hacked into every org in Oceania for the past 20 years, is "yeah, this is generating huge return on investment for the ransomware crews so they're just going to keep doing it, and being proactive might be a great idea." But what he didn't say, but clearly had in his head was "but lol, nobody is going to actually do that. So good luck out there chooms!" At some level, STIX and TAXII and the whole CTI market are about passing around information on what someone _might_ have used to hack something, at some point in the _distant past_. It's a paleontology of hackers past - XML schemas about huge ancient reptiles swimming in the tropical seas of your networks, the taxonomies of extinct orders we now know only through a delicate finger-like flipper bone or a clever piece of shellcode. -dave

1 1

Fussing about with fuzzers
by Dave Aitel 22 May '23

22 May '23

So last week at offensivecon I watched a talk on Fuzzilli ( https://github.com/googleprojectzero/fuzzilli) which, I have to admit I had no idea what it was. Obviously I knew it was a Googlely Javascript fuzzer, finding bugs. But I did not realize that it was applying mutations to its own intermediate language which it then compiled to Javascript. I just assumed it was, like most fuzzers, mutating the javascript directly (f.e. https://sean.heelan.io/2016/04/26/fuzzing-language-interpreters-using-regre… ). But having an IL designed for fuzzing-related mutations is clearly a great idea! And this year, they've expanded on that to build a Javascript->Fuzzilli compiler/translation layer. So you can pass in sample Javascript and then it will create the IL and then it will mutate the IL. The reason this is necessary is that Javascript is. like almost all modern languages, extremely complicated underneath the covers, so in order to generate crashes you may need to have a lot of different fields set properly in a particular order in a structure. They try to do some introspection on objects and generate their samples from that as well, but there's no beating "real user code" for learning how an object needs to be created and used. These advances generate a lot more bugs! In theory none of these bugs matter in the future because of the mitigations (no pointers outside the Javascript gigacage!) going into place by the very authors of the fuzzer? (I have my doubts, but we all will live and learn?) It would be...very cool, I think, if Bard or another LLM was the one doing the Javascript sample generation as well. If you think about it, these LLMs all have a good understanding of Javascript and you can give them various weird tasks to do, and let them generate your samples, and then when a crash happens you can have them mutate around that crash, or if you have a sample not getting any more code coverage you can have them mutate that sample to attempt to make it weirder. :) -dave

2 1

Knowledge Graph + AI = ?
by Dave Aitel 09 May '23

09 May '23

*Logical Operations on Knowledge Graphs:* So if you've spent enough time looking at graph databases, you will invariably run into people who are obsessed with "ontologies". The basic theory being that if you've already organized your data in some sort of directed graph, maybe you can then apply a logical ruleset to those relationships and infer knowledge about things you didn't already know, which could be useful? There are a lot of people trying to do this with SBOMs and I...wish them the best. In real life I think it's basically impossible to create a large, useful, graph database+inference engine that has data clean enough for anything useful. Also the word *ontology* is itself very annoying. But philosophically, while any complex enough data set will have to embrace paradoxes, you can get a lot of value out of putting some higher level structure based on the text in your data. And this is where modern AI comes in - in specific, the tree of "Natural Language Understanding" that broke off from Chomsky-esque-and-wrong "Natural Language Processing" some time ago. One article covering this topic is here <https://medium.com/@anthony.mensier/gpt-4-for-defense-specific-named-entity…>, which combines entity extraction and classification in order to look into finding military topics in an article. But these techniques can be abstracted and broadened as a general purpose and very useful algorithm: Essentially you want to extract keywords from text fields within your graph data, then relate those keywords to each other, which gives you conceptual groupings and allows you to make further queries that uncover insights about those groups. *Our Solution:* One of the team-members over at Margin Research working on SocialCyber <https://www.darpa.mil/program/hybrid-ai-to-protect-integrity-of-open-source…> with me, Matt Filbert, came up with the idea of using OpenAI's GPT to get hashtags from text, which it does very very well. If you store these as nodes you get something like the picture below (note that hashtags are BASED on the text, but they are NOT keywords and may not be in the text itself): [image: image.png] Next you want to figure out how topics are related to each other! Which you can do in a thousand different ways - the code words to search on are "Node Similarity" - but almost all those ways will either not work or create bad results because you have a very limited directional graph of just "Things->Topics". In the end we used a modified Jaccardian algo (aka, you are similar if you have shared parents), which I call Daccardian because it creates weighted directed graphs (which comes in handy later): [image: image.png] So once you've done that, you get this kind of directed graph: [image: image.png] From here you could build communities of related topics using any community detection algorithm, but even just being able to query against them is extremely useful. In theory you could query just against one topic at a time, but because of the way your data is probably structured, you want both that topic, and any closely related topics to be included. So for example, looking for Repos that have topics that either are "#UI" or closely related, can be queried like this (not all topics are shown because of the LIMIT clause): [image: image.png] *Some notes on AI Models:* OpenAI's APIs are a bit slow, and often throw errors randomly, which is fun to handle. And of course, when doing massive amounts of inference, it's probably cheaper to run your own equipment, which will leave you casting about on Huggingface like a glow worm in a dark cave for a model that can do this and is open source. I've tried basically all of them and they've all started out promising but then been a lot worse than ChatGPT 3.5. You do want one that is multilingual, and Bard might be an option when they open their API access up. There's a significant difference between the results from the big models and the little ones, in contrast to the paper that just "leaked" from Google <https://www.semianalysis.com/p/google-we-have-no-moat-and-neither> about how small tuned models are going to be just as good as bigger models (which they are not!). One minor exception is the new Mosaic model ( https://huggingface.co/spaces/mosaicml/mpt-7b-instruct) which is multilingual and four times cheaper than OpenAI but it's also about 1/4th as good. It may be the first "semi-workable" open model though, which is a promising sign and it may be worth moving to this if you have data you can't run through an open API for some reason. *Conclusion:* If you have a big pile of structured data, you almost certainly have natural language data that you want to capture as PART of that structure, but this would have been literally impossible six months ago before LLMs. Using this AI-tagging technique and doing some basic graph algorithms can really open up the power of your queries to take the most valuable part of your information into account, while not losing the performance and scalability of having it in a database in the first place. Thanks for reading, Dave Aitel PS: I have a dream, and that dream is to convert Ghidra into a Graph Database as a native format so we can use some of these techniques (and code embeddings) as a native feature. If you sit next to the Ghirda team, and you read this whole post, give them a poke for me. :)

2 1

Performance thoughts
by Dave Aitel 28 Apr '23

28 Apr '23

So my first thought is that performance measurement tools seem exactly aimed at a lot of security problems but performance people are extremely reluctant <https://aus.social/@brendangregg/110276319669838295> to admit that because of the drama involved in the security market. Which is very smart of them! :) Secondly, I wanted to re-link to Halvar's QCon keynote <https://docs.google.com/presentation/d/1wOT5kOWkQybVTHzB7uLXpU39ctYzXpOs2xV…>. He has a section on the difficulties of getting good performance benchmarks, which typically you would do as part of your build chain. So in theory, you have a lot of compilation features you can twiddle when compiling and you want to change those values, compile your program, and get a number for how fast it is. But this turns out to basically be impossible in the real world for reasons I'll let him explain in his presentation (see below). [image: image.png] A lot of these problems with performance seem only solvable by a continuous process of evolutionary algorithms - where you have a population of different compilation variables, and you probably introduce new ones over time, and you kill off the cloud VMs where you're getting terrible performance under real-world situations and let the ones getting good or average performance thrive. I'm sure this is being done, and probably if I listened to more of Dino dai Zovi's talks I'd know where and how, but aside from having performance implications, it also has security implications because it will tend towards offering offensive implants value for becoming less parasitic and more symbiotic. -dave

1 0

LSU is hiring CS Professors with RE and binary exploitation experience
by Andrew Case 30 Mar '23

30 Mar '23

The Computer Science department at Louisiana State University (LSU) is currently hiring for many faculty positions related to applied cyber security. Courses taught inside this department include reverse engineering, malware analysis, binary exploitation, memory forensics and other intensive courses related to incident response and offensive security. Ideal candidates will have significant experience with deeply technical areas of cybersecurity. LSU was recently granted the CAE-CO designation and is one of only 21 schools nation-wide to hold it as it is the most technical designation granted by NSA and DHS. The department also runs a large SFS program for cyber security students. If you are interested in one of these positions, then please see the following link. I also ask my industry contacts to please spread the word within academic communities that you have access to: https://lsu.wd1.myworkdayjobs.com/en-US/LSU/job/3325-Patrick-F-Taylor-Hall/… The cybersecurity effort at LSU has strong support from the highest levels of the school and is rapidly expanding – so now is the perfect time to join. PS: I am not employed by LSU, but do work very closely with the CS department to ensure the courses are relevant to industry and rigorous enough for students to leave with real-world, hands-on experience. If you have questions related to the position, then please direct them to Dr. Golden Richard at LSU: https://www.cct.lsu.edu/~golden/ Thanks, Andrew

1 0

t2'23: Call For Papers 2023 (Helsinki, Finland)
by Tomi Tuominen 12 Feb '23

12 Feb '23

Call For Papers 2023 Tired of your bosses suspecting conference trips to exotic locations being just a ploy to partake in Security Vacation Club? Prove them wrong by coming to Helsinki, Finland on May 4-5 2023! Guaranteed lack of sunburn, good potential for rain or slush. In case of great spring weather, though, no money back. CFP and registration both open. Read further if still unsure. Maui, Miami, Las Vegas, Tel Aviv or Wellington feel so much sunnier once you’ve experienced the lack of infinity pools in Northern Europe. Instead of pools and palms trees, we can offer you actual saunas and a high tech environment, which is a weird combination of demoscene, widespread Linux adoption, mobile Internet with uncapped flat rate data and a long history of IRC and imageboards. What defines a conference? For t2 it has always been that intimate welcoming atmosphere of a small event, which makes both audience and speakers approachable. There are enough regulars to create the feeling of a community, but not too many that a first-comer would feel being left out. On the content side, we have always been and always will be a technical security conference, emphasizing the cutting edge, world class research. This is an event for the community. Our focus[1] is on technical excellence, not politics or player hating. t2’23 offers you an audience with a taste for technical security presentations containing original content. This is your chance to showcase the latest research and lessons in EDR simulation and healthcheck spoofing, hardware insecurity, inferring information from interference, cloud-scale forensics or persistance automation, new vulnerability classes, AI exploitation, virtual machines inside parsers, elegant exploitation of old vulnerability classes, modern defense, dropping zero days during presentations, state of the art memory corruption mitigation bypasses, evasions, safe cracking, satellite and space security, remote vehicle access, or whatever research lights up the eyes of seasoned conference visitors. For the hackers by the hackers. The advisory board will be reviewing submissions until 2023-03-17. Slide deck submission final deadline 2023-04-20 for accepted talks. First come, first served. Submissions will not be returned. Quick facts for speakers + presentation length 60-120 minutes, in English + complimentary travel and accommodation for one person[6] + decent speaker hospitality benefits + no marketing or product propaganda Still not sure if this is for you? Check out the blast from the past[2]. The total amount of attendees, including speakers and organizers is limited to 99. Advisory Board recognize[3] the OG Finnish sauna culture is an acquired taste and can promise the lack of sweaty, partially or fully nude sauna-goers at all conference functions. [0] hunter2 [1] https://t2.fi/about/eula/ [2] https://t2.fi/schedules/ [3] https://youtu.be/Oj8JBBAM5jY?t=40 [6] except literally @nudehaberdasher and @0xcharlie How to submit Fill out the form at https://if.t2.fi/action/cfp How to register Buy your ticket at https://t2.fi/registration/

1 0

Chapter 2
by Dave Aitel 04 Sep '22

04 Sep '22

(Note, this is a continuation of our previous story chapter since sometimes it's more fun to read fiction than to wonder what's going on these days with Cloudflare or whatever. https://lists.aitelfoundation.org/archives/list/dailydave@lists.aitelfounda… ) Chapter 2 _________________________________________________ Landing in Miami is like visiting a tier of hell just below Limbo. It is not saturated in evil so much as the established gateway to more evil places. As you disembark from your flight you can almost see a direct line from providing no-questions-asked banking to drug dealers in the eighties, to offering an endless series of apartments (aka money hiding spots) to the Venezualan upper class, to the current endless series of crypto companies headquartered in the newly hip Brickle office spaces next to SmileDirect and fancy brew pubs. In the sense that NYC deals in finances, San Francisco in software companies, Boston in "higher education", Miami is more about your more generic small-scale scams as the underlying substrate upon which the rest of the economy is based. The tropics engender a sort of flexibility and adaptability which is about finding new scam-niches and exploiting them before anyone else has caught on. But your meeting here is not about crypto-coin or real estate built with permeable concrete guaranteed to spall in the face of salt-water-laden winds. It's with a company building testing software of all things. "Boring is rewarding" you say to yourself, as you drive past a literal graveyard to a small joint called "Hush" which you give an approving nod to. Hush serves fried alligator, which tastes like fried anything, as you sit across from your lunch companions, Stewart and Amy. They are drinking beers you've never heard of, and they lay out their scheme, without regard to OPSEC since nobody in this restaurant other than you would likely understand it. "We've been building a large set of unit testing libraries for cryptographic primitives, lots of complicated string building stuff, machine learning, you name it." "Great." You say. "Always good to have quality testing libraries". But they exchange a look and you realize you've misinterpreted them. "Our public libraries have a tendency to ... sometimes think things are very well written and secure when they are ... not. It's just sometimes our unit testing has bugs, you know? We have really good documentation in a lot of languages though. And great support. 24/7. Discord, Slack, forums, you name it." "So the theory here is you don't target any particular software in the supply chain? You just encourage bad testing practices?" You're pondering their value, while at the same time trying to think about what alligator actually tastes like under all the grease. The flavor, as far as you can determine, is "Chewy". Stewart struggles for a second to get the words out, like a huge machine optimized for literally anything other than the current task of explaining things to other humans using words. "Sometimes it's best when the check to see if ASLR is enabled doesn't actually work, so your bugs that you find have a chance to be exploitable. We're not in the business of putting bugs in things. We just make the bugs you do have....better." "I see. What about code we actually want to be secure?" "I recommend everyone local uses our FIPS certified library, which, admittedly, is expensive and does the same thing as our free code, but maybe with more effort put into the actual tests themselves." Amy says this to you without any hint of chicanery, as if this is a simple fact, almost not worth saying. It is, you realise, a very tropical CONOP. "I will make sure this is required by various regulations after you are funded. I'll have my team send you the paperwork." you say. And with that, the conversation moves to pleasant nonsense as you internally contemplate your next flight - out of here and into the cold. -dave

1 0

"Market Failures"
by Dave Aitel 25 Aug '22

25 Aug '22

If you were at a talk at Defcon this year in the Policy track, you probably heard someone talk about how they, as a government official, are there to address "market failures". And immediately you thought: This is a load of nonsense. Because that government official is not allowed to, and has no intentions of, addressing any market failures whatsoever. If the Government was going to address market failures, they'd have to find some way to convince every cloud provider from making their security features the upsell on the Platinum package. They'd have to talk about how trying to get into different markets means every social media company faces huge pressures to put Indian spies on their network. Obviously you know, as someone who did not emerge from under a rock into the security community yesterday, that the answer to having a malicious insider on your network is probably some smart segmentation, which we call "Zero Trust" now. But Zero Trust is expensive! And most social media companies are not exactly profitable as the great monster known as TikTok has eaten every eyeball in every market because the very concept of having people explicitly choose who their friends are is outdated now. In fact, as everyone is pointing out, almost all companies you know are in this position! They're cutting costs by sending jobs overseas while spending huge amounts of money propping up their stock prices and paying their executives to sell them to a dwindling market of buyers. Private Equity companies spend every effort on squeezing the last dollar out of old enterprise software by exploiting the lock-in they have on small businesses. And as critical as Twitter is, we have the exact same dynamic with our privatized water and power companies - who have no plans to make strategic investments in security or anything really - which is why on public calls you can hear them humiliating themselves asking Jen Easterly to absorb the entire costs of their security programs. The ideal practice for all of these companies is to offload their costs onto the taxpayer, which is why instead of investing in security, they cry for the FBI to go collect their bitcoin from whatever ransomware crews are on their network this week using offensive cyber operations that themselves cost the government an order of magnitude more than the bitcoin is worth. As you're sitting in that Defcon talk, listening to someone from government talk about how they only want to regulate with the "input of industry" or something, you have to wonder: if this is every company we know, maybe the market failure isn't just how hard it is to buy a good security product because they all abuse the copyright system to avoid any kind of performance transparency. Maybe it's also how hard it is to SELL a good security product because every single company is trying to cut their budget to the exact minimum amount that will allow them to tell the FBI they did their best, and the FBI needs to go out there and pick up their slack. -dave

8 7

Defcon 30
by Dave Aitel 23 Aug '22

23 Aug '22

As you wander the halls of the inaptly named Caesar's Forum, amidst a living sea of the most neurodiverse Clan humanity has ever seen, you cannot help but stop for a second to close your eyes amidst the cacophony and mentally exclaim, "Look. Look at the world we have created!" Sitting in the one cafe in the Paris hotel with food, a tattooed thirty-something who has been to Defcon twice gives you advice on how to do the conference. "Take the unirail." they say. "Also, you should have a hacker name! Mine is 'youngblood''" "Noted!" you respond. These are good ideas. The unirail in particular, probably, because Vegas is overflowing - and decent food options and anywhere to sit that is not beeping at you or showing grungy dystopian TV ads the Cyberpunk 2077 developers would find over-the-top are impossible to come by, making the conference ten times more exhausting than usual. In that sense, you miss the Alexis Park days, sitting with Halvar Flake next to a pool where everyone was more larval than they knew, watching Dildog lauch BO2K to a thousand screaming fans in the same room Dino Dai Zovi explained Solaris hacking an hour earlier. Some of the best talks this year had no attendees at all - Orange Tsai's talk was over Zoom, to a huge room, but with few butts in the seats. There were a hundred "Villages" it seemed like, living a half-life between physical space in the conference room and a Discord channel. Defcon may be the worst and best place to learn anything in that way - the environment is hopelessly chaotic, with two talks happening inches away from each other, and only feet from a DJ pumping out house music. But perhaps the best environment to learn in is the one in which you are most inspired? My friends, we've conquered the world. What's next? -dave

6 5