WAF Metrics
So I'm making a video on metrics, of all things, and I wanted to post both this
question https://twitter.com/daveaitel/status/1281629327776522242?s=20and
the best answer so far to the list to see if anyone had any other ideas or
followups.
-dave
[image: image.png]
[image: image.png]
So, I recently did an integration for a company that took their web app
scanner results and mapped those to existing WAF rules. I can think of 2
metrics based off that
- How many real-world vulns have a corresponding check in the WAF? and
- Once the WAF rules have been put in place to protect actually-vulnerable
endpoints, how many attacks were actually stifled?
John
On Sat, Jul 11, 2020 at 12:51 PM Dave Aitel via Dailydave <
dailydave@lists.aitelfoundation.org> wrote:
**** CAUTION: This email was sent from an EXTERNAL source. Think before
clicking links or opening attachments. ****
So I'm making a video on metrics, of all things, and I wanted to post both this
question https://twitter.com/daveaitel/status/1281629327776522242?s=20and
the best answer so far to the list to see if anyone had any other ideas or
followups.
-dave
[image: image.png]
[image: image.png]
Dailydave mailing list -- dailydave@lists.aitelfoundation.org
To unsubscribe send an email to dailydave-leave@lists.aitelfoundation.org
I guess some of us who grew up mapping ports and protocols into their neat
buckets will need to live with that fact that everything will eventually
ride over a multiplexed 443 socket, just something to think about before
the rant.
TL;DR - The answer to your question about measurement and effectiveness is
going to come down: "how long before you can see what I'm doing".
WAF's are a rather complex beast, but I guess they do deserve a look from
time to time as I am not sure who is actually doing the hard work of
judging their effectiveness. If you are doing that work, resist the urge to
just push everything to Graphana as if a fancy graph will make us happy.
Quickly, you may find that the complexity of testing any number of WAF's is
to find what is good represented battery of tests are. This is what makes
something like MITRE ATT&CK interesting because you are framing a series of
techniques which can be modified and changed somewhat dynamically. Testing
a WAF then may become a more simplified affair, instead of having to
construct hundreds of potential scenarios. Your favorite WAF may miss the
injection technique du-jour because it may be running all of its
normalizations through an old IDN library and misses your attack string.
Either way, for anyone else's consideration; from what I gather most
commercial WAFs have taken one of three approaches. Firstly, they are using
a very simple blacklist approach typically its that of using a simple
ModSecurity CRS with over the top signatures. You can probably guess which
ones do that, and there are a few commercial WAFs that still do this and it
does prevent some simple attacks. Some of these WAFs are trying to be even
fancier by 'dynamically creating signatures based on attack patterns'
which they may call M/L. The second approach is that of a training WAF,
this approach has the other classic problem that is, I will use some static
blacklists like User-Agent SQLMAP == bad, but I will raise you 'learning
what is good in your app'. This approach has been interesting because what
if you just learned badness? Or what if your application is just terrible,
such as an application I used a few months ago in which, hilariously,
constructed Java functions in the browser as a response and sent into the
web app. I don't know why, I guess so I can { java.lang.math } while I fill
out this web form. The final approach and maybe the more sane approach in
some respects(?) is to treat the WAF as a library in the application to
introspect things. This idea is probably not the most novel as companies
like newrelic seem to do this fairly well for performance. Maybe in this
way you can try and build enough telemetry to make better decisions about
the state of good or bad. In addition we can now 'share data' amongst a
larger number of people and in this way make global decisions based on
large datasets. Either way, as it comes to metrics and testing, having a
solid test bed and strategy always seems to be the hardest part of the
equation not just metrics.
What is even more interesting is probably what the WAFs real threat model
is becoming today. While most ops folks maybe worried about a ransomware
payload or a crypto miner payload, the real driver many times are bots.
Folks that are coming in and scraping web data in order to resell, replace,
come in cheaper, or what have you. This is in theory not illegal which is
even more interesting because it comes down to mining and abusing pseudo
disclosed datasets.
-M
On Sat, Jul 11, 2020 at 12:44 PM Dave Aitel via Dailydave <
dailydave@lists.aitelfoundation.org> wrote:
So I'm making a video on metrics, of all things, and I wanted to post both this
question https://twitter.com/daveaitel/status/1281629327776522242?s=20and
the best answer so far to the list to see if anyone had any other ideas or
followups.
-dave
[image: image.png]
[image: image.png]
Dailydave mailing list -- dailydave@lists.aitelfoundation.org
To unsubscribe send an email to dailydave-leave@lists.aitelfoundation.org
John,
Can you expand on #2? How do you measure the number of attacks stifled?
_--
Rafal
_Mobile: (404) 606-6056
_Email: Rafal.Los@Seventy7.Consultingmailto:Rafal.Los@Seventy7.Consulting
From: John Lampe via Dailydave dailydave@lists.aitelfoundation.org
Reply-To: John Lampe jlampe@tenable.com
Date: Saturday, July 11, 2020 at 9:52 PM
To: Dave Aitel dave.aitel@gmail.com
Cc: "dailydave@lists.aitelfoundation.org" dailydave@lists.aitelfoundation.org
Subject: [Dailydave] Re: [EXTERNAL] WAF Metrics
So, I recently did an integration for a company that took their web app scanner results and mapped those to existing WAF rules. I can think of 2 metrics based off that
- How many real-world vulns have a corresponding check in the WAF? and
- Once the WAF rules have been put in place to protect actually-vulnerable endpoints, how many attacks were actually stifled?
John
On Sat, Jul 11, 2020 at 12:51 PM Dave Aitel via Dailydave <dailydave@lists.aitelfoundation.orgmailto:dailydave@lists.aitelfoundation.org> wrote:
*** CAUTION: This email was sent from an EXTERNAL source. Think before clicking links or opening attachments. ***
So I'm making a video on metrics, of all things, and I wanted to post both this question https://twitter.com/daveaitel/status/1281629327776522242?s=20 and the best answer so far to the list to see if anyone had any other ideas or followups.
-dave
[cid:image001.png@01D65902.CB3B5C00]
[cid:image002.png@01D65902.CB3B5C00]
Dailydave mailing list -- dailydave@lists.aitelfoundation.orgmailto:dailydave@lists.aitelfoundation.org
To unsubscribe send an email to dailydave-leave@lists.aitelfoundation.orgmailto:dailydave-leave@lists.aitelfoundation.org
We’ve released a mid-pandemic product that is designed to test production deployed WAF’s by doing exactly what @ranger_cha is describing.
It will run tests that include both known/existing attacks that a WAF should stop and common patterns that all WAF’s should recognize and stop. Separately and clearly, so the use can see the impact of stopping both sets of assessments separately.
https://www.ixiacom.com/products/threat-simulator
The intent of this product is to move out of the realm of lab testing firewalls/WAFs/Load Balancers and into the production network space.
-chuck
From: Dave Aitel via Dailydave dailydave@lists.aitelfoundation.org
Reply-To: Dave Aitel dave.aitel@gmail.com
Date: Saturday, July 11, 2020 at 12:42 PM
To: "dailydave@lists.aitelfoundation.org" dailydave@lists.aitelfoundation.org
Subject: [Dailydave] WAF Metrics
[EXTERNAL]
So I'm making a video on metrics, of all things, and I wanted to post both this question https://urldefense.com/v3/__https:/twitter.com/daveaitel/status/1281629327776522242?s=20__;!!I5pVk4LIGAfnvw!z6kkVfeJEkYBNKEYbXzCN3lUbXnmBTnoqdbT0yC2ouioivlD5GYxZrzY2Uqfn6Pg$ and the best answer so far to the list to see if anyone had any other ideas or followups.
-dave
Yeah, I guess the way I would envision it going would be:
- web app scanner sees XSS vuln on /path/to/foo.php
- my integration ties that web app scan into a format to pass to WAF
- WAF sets up anti-xss rules on /path/to/foo.php (we had to actually
create a static mapping for this step) - measure how many hits the waf blocks to that endpoint for the XSS
John
On Mon, Jul 13, 2020 at 10:46 AM Rafal Los Rafal@ishackingyou.com wrote:
**** CAUTION: This email was sent from an EXTERNAL source. Think before
clicking links or opening attachments. ****
John,
Can you expand on #2? How do you measure the number of attacks stifled?
_
*-- *Rafal
_Mobile:
- (404) 606-6056 _Email: Rafal.Los@Seventy7.Consulting
Rafal.Los@Seventy7.Consulting*
*From: *John Lampe via Dailydave dailydave@lists.aitelfoundation.org
*Reply-To: *John Lampe jlampe@tenable.com
*Date: *Saturday, July 11, 2020 at 9:52 PM
*To: *Dave Aitel dave.aitel@gmail.com
*Cc: *"dailydave@lists.aitelfoundation.org" <
dailydave@lists.aitelfoundation.org>
*Subject: *[Dailydave] Re: [EXTERNAL] WAF Metrics
So, I recently did an integration for a company that took their web app
scanner results and mapped those to existing WAF rules. I can think of 2
metrics based off that
-
How many real-world vulns have a corresponding check in the WAF? and
-
Once the WAF rules have been put in place to protect
actually-vulnerable endpoints, how many attacks were actually stifled?
John
On Sat, Jul 11, 2020 at 12:51 PM Dave Aitel via Dailydave <
dailydave@lists.aitelfoundation.org> wrote:
*** CAUTION: This email was sent from an EXTERNAL source. Think before
clicking links or opening attachments. ***
So I'm making a video on metrics, of all things, and I wanted to post both this
question https://twitter.com/daveaitel/status/1281629327776522242?s=20and
the best answer so far to the list to see if anyone had any other ideas or
followups.
-dave
Dailydave mailing list -- dailydave@lists.aitelfoundation.org
To unsubscribe send an email to dailydave-leave@lists.aitelfoundation.org
This isn’t directly related to John’s observation below, but it got me motivated to further clarify some of the challenges involved in testing WAFs.
I’ve seen many implementations over the years that try to determine the decision making process of an IPS, WAF, or similar device by simply interrogating it from the client side only. The realities of test of measurement is that it requires the user to implement both a client and server side process of whatever it is you are testing to validate that not only was malicious content allowed or blocked, but that IF it was allowed, it was allowed through without a modification that would impact the intent of the vulnerability.
That’s a mouthful of a sentence. Let me make it briefer:
You can’t tell if the bad stuff got through unless you are wrapped around the bad stuff blocking thing.
This gets even harder when you are testing WAF’s. They very often interrupt the connection, marshal the request, then issue a new HTTP request to the server. This means that while your malicious request might have altered headers, or Content-Transfer-Encoding changes the actual little nugget of maliciousness is left alone.
“Fine”, you say, “I’ll just examine the URL request, because I’m only interested in testing items that are impacted there.” And then CVE-2017-5638 comes along and you need to accommodate for deserialization vulnerabilities in other headers. The WAF catches it. But your current test implementation can’t.
And that’s an easy example.
You can have a chain like this:
Client POST multipart http2 -> proxy -> TLS 1_1 -> Internet -> IPS Performing TLS inspection -> Different handshake TLS 1_0 -> WAF/LB -> http1.1 cleartext/chunked -> server.
I just came up with that off the top of my head, so please don’t challenge me on the technical limitations of said example :).
The point is that testing in isolation is very different from testing a deployed system under test and for each new technology deployed, the number of permutations increases dramatically. This list has discussed many times the merits and problems with “defense in depth” strategies, so I won’t belabor that point. Only to say “it’s hard to get an accurate read on if you are secure or not.” Lunchtime doubly so.
-chuck
From: John Lampe via Dailydave dailydave@lists.aitelfoundation.org
Reply-To: John Lampe jlampe@tenable.com
Date: Monday, July 13, 2020 at 8:07 PM
To: Rafal Los Rafal@ishackingyou.com
Cc: "dailydave@lists.aitelfoundation.org" dailydave@lists.aitelfoundation.org
Subject: [Dailydave] Re: [EXTERNAL] WAF Metrics
[EXTERNAL]
Yeah, I guess the way I would envision it going would be:
- web app scanner sees XSS vuln on /path/to/foo.php
- my integration ties that web app scan into a format to pass to WAF
- WAF sets up anti-xss rules on /path/to/foo.php (we had to actually create a static mapping for this step)
- measure how many hits the waf blocks to that endpoint for the XSS
John
On Mon, Jul 13, 2020 at 10:46 AM Rafal Los <Rafal@ishackingyou.commailto:Rafal@ishackingyou.com> wrote:
*** CAUTION: This email was sent from an EXTERNAL source. Think before clicking links or opening attachments. ***
John,
Can you expand on #2? How do you measure the number of attacks stifled?
_--
Rafal
_Mobile: (404) 606-6056
_Email: Rafal.Los@Seventy7.Consultingmailto:Rafal.Los@Seventy7.Consulting
From: John Lampe via Dailydave <dailydave@lists.aitelfoundation.orgmailto:dailydave@lists.aitelfoundation.org>
Reply-To: John Lampe <jlampe@tenable.commailto:jlampe@tenable.com>
Date: Saturday, July 11, 2020 at 9:52 PM
To: Dave Aitel <dave.aitel@gmail.commailto:dave.aitel@gmail.com>
Cc: "dailydave@lists.aitelfoundation.orgmailto:dailydave@lists.aitelfoundation.org" <dailydave@lists.aitelfoundation.orgmailto:dailydave@lists.aitelfoundation.org>
Subject: [Dailydave] Re: [EXTERNAL] WAF Metrics
So, I recently did an integration for a company that took their web app scanner results and mapped those to existing WAF rules. I can think of 2 metrics based off that
- How many real-world vulns have a corresponding check in the WAF? and
- Once the WAF rules have been put in place to protect actually-vulnerable endpoints, how many attacks were actually stifled?
John
On Sat, Jul 11, 2020 at 12:51 PM Dave Aitel via Dailydave <dailydave@lists.aitelfoundation.orgmailto:dailydave@lists.aitelfoundation.org> wrote:
*** CAUTION: This email was sent from an EXTERNAL source. Think before clicking links or opening attachments. ***
So I'm making a video on metrics, of all things, and I wanted to post both this question https://urldefense.com/v3/__https:/twitter.com/daveaitel/status/1281629327776522242?s=20__;!!I5pVk4LIGAfnvw!1DZZL1viGJTRH-H2akN1jntqUUjdEe6Oa7-HctTc9IePgQzC3DN13JryFgb8Id0i$ and the best answer so far to the list to see if anyone had any other ideas or followups.
-dave
[cid:image001.png@01D659C7.81D98AD0]
[cid:image002.png@01D659C7.81D98AD0]
Dailydave mailing list -- dailydave@lists.aitelfoundation.orgmailto:dailydave@lists.aitelfoundation.org
To unsubscribe send an email to dailydave-leave@lists.aitelfoundation.orgmailto:dailydave-leave@lists.aitelfoundation.org
So far, this conversation focuses on how effectively WAFs block malicious HTTP requests. I'd argue that this is both a red herring and an abuse of WAF technology. A WAF only protects the enterprise when it blocks a request that would trigger an actual bug. If there's no bug present, all that's really happening is that likely malicious requests are being logged at a much higher costs than if it were simply allowed to sit in the web logs. Being attacked constantly is a side-effect of being on the Internet. You should expect to be constantly scanned and probed. Ideally, action should only be necessary if an attack is actually practical, otherwise you're simply reacting to the background chatter of the Internet.
If there is a bug present and the WAF blocks the attempt, then it is a legitimate protective action has occurred. The WAF is simply a stopgap, however. If the bug is present, then the WAF will eventually be evaded allowing an attack to succeed. As a result, I'd argue that success metrics for a WAF are effectively twofold:
- How clearly does it signal that there's an actual bug in the software behind it?
- How much time can it provide for a fix to that software?
The problems you are protecting against aren't in the WAF or it's tuning -- it's in the underlying software. A WAF is simply a stopgap that buys time while code is corrected or vendors ship a patch. Relying on a WAF as anything more than a defense-in-depth mechanism is a losing proposition over time. Metrics need to reflect that.
From: Chuck McAuley via Dailydave dailydave@lists.aitelfoundation.org
Sent: Tuesday, July 14, 2020 7:14 AM
To: John Lampe jlampe@tenable.com; Rafal Los Rafal@ishackingyou.com
Cc: dailydave@lists.aitelfoundation.org dailydave@lists.aitelfoundation.org
Subject: [Dailydave] Re: [EXTERNAL] WAF Metrics
This isn’t directly related to John’s observation below, but it got me motivated to further clarify some of the challenges involved in testing WAFs.
I’ve seen many implementations over the years that try to determine the decision making process of an IPS, WAF, or similar device by simply interrogating it from the client side only. The realities of test of measurement is that it requires the user to implement both a client and server side process of whatever it is you are testing to validate that not only was malicious content allowed or blocked, but that IF it was allowed, it was allowed through without a modification that would impact the intent of the vulnerability.
That’s a mouthful of a sentence. Let me make it briefer:
You can’t tell if the bad stuff got through unless you are wrapped around the bad stuff blocking thing.
This gets even harder when you are testing WAF’s. They very often interrupt the connection, marshal the request, then issue a new HTTP request to the server. This means that while your malicious request might have altered headers, or Content-Transfer-Encoding changes the actual little nugget of maliciousness is left alone.
“Fine”, you say, “I’ll just examine the URL request, because I’m only interested in testing items that are impacted there.” And then CVE-2017-5638 comes along and you need to accommodate for deserialization vulnerabilities in other headers. The WAF catches it. But your current test implementation can’t.
And that’s an easy example.
You can have a chain like this:
Client POST multipart http2 -> proxy -> TLS 1_1 -> Internet -> IPS Performing TLS inspection -> Different handshake TLS 1_0 -> WAF/LB -> http1.1 cleartext/chunked -> server.
I just came up with that off the top of my head, so please don’t challenge me on the technical limitations of said example :).
The point is that testing in isolation is very different from testing a deployed system under test and for each new technology deployed, the number of permutations increases dramatically. This list has discussed many times the merits and problems with “defense in depth” strategies, so I won’t belabor that point. Only to say “it’s hard to get an accurate read on if you are secure or not.” Lunchtime doubly so.
-chuck
From: John Lampe via Dailydave dailydave@lists.aitelfoundation.org
Reply-To: John Lampe jlampe@tenable.com
Date: Monday, July 13, 2020 at 8:07 PM
To: Rafal Los Rafal@ishackingyou.com
Cc: "dailydave@lists.aitelfoundation.org" dailydave@lists.aitelfoundation.org
Subject: [Dailydave] Re: [EXTERNAL] WAF Metrics
[EXTERNAL]
Yeah, I guess the way I would envision it going would be:
-
web app scanner sees XSS vuln on /path/to/foo.php
-
my integration ties that web app scan into a format to pass to WAF
-
WAF sets up anti-xss rules on /path/to/foo.php (we had to actually create a static mapping for this step)
-
measure how many hits the waf blocks to that endpoint for the XSS
John
On Mon, Jul 13, 2020 at 10:46 AM Rafal Los <Rafal@ishackingyou.commailto:Rafal@ishackingyou.com> wrote:
*** CAUTION: This email was sent from an EXTERNAL source. Think before clicking links or opening attachments. ***
John,
Can you expand on #2? How do you measure the number of attacks stifled?
_--
Rafal
_Mobile: (404) 606-6056
_Email: Rafal.Los@Seventy7.Consultingmailto:Rafal.Los@Seventy7.Consulting
From: John Lampe via Dailydave <dailydave@lists.aitelfoundation.orgmailto:dailydave@lists.aitelfoundation.org>
Reply-To: John Lampe <jlampe@tenable.commailto:jlampe@tenable.com>
Date: Saturday, July 11, 2020 at 9:52 PM
To: Dave Aitel <dave.aitel@gmail.commailto:dave.aitel@gmail.com>
Cc: "dailydave@lists.aitelfoundation.orgmailto:dailydave@lists.aitelfoundation.org" <dailydave@lists.aitelfoundation.orgmailto:dailydave@lists.aitelfoundation.org>
Subject: [Dailydave] Re: [EXTERNAL] WAF Metrics
So, I recently did an integration for a company that took their web app scanner results and mapped those to existing WAF rules. I can think of 2 metrics based off that
-
How many real-world vulns have a corresponding check in the WAF? and
-
Once the WAF rules have been put in place to protect actually-vulnerable endpoints, how many attacks were actually stifled?
John
On Sat, Jul 11, 2020 at 12:51 PM Dave Aitel via Dailydave <dailydave@lists.aitelfoundation.orgmailto:dailydave@lists.aitelfoundation.org> wrote:
*** CAUTION: This email was sent from an EXTERNAL source. Think before clicking links or opening attachments. ***
So I'm making a video on metrics, of all things, and I wanted to post both this question https://urldefense.com/v3/__https:/twitter.com/daveaitel/status/1281629327776522242?s=20__;!!I5pVk4LIGAfnvw!1DZZL1viGJTRH-H2akN1jntqUUjdEe6Oa7-HctTc9IePgQzC3DN13JryFgb8Id0i$ and the best answer so far to the list to see if anyone had any other ideas or followups.
-dave
[cid:image001.png@01D659C7.81D98AD0]
[cid:image002.png@01D659C7.81D98AD0]
Dailydave mailing list -- dailydave@lists.aitelfoundation.orgmailto:dailydave@lists.aitelfoundation.org
To unsubscribe send an email to dailydave-leave@lists.aitelfoundation.orgmailto:dailydave-leave@lists.aitelfoundation.org
I'm not parsing your argument. If you knew the bug was there, you would fix
the bug. The WAF is there to mitigate the bugs that you are not aware of.
Further, web accesses that are out of scope of your intended functionality
but do not trigger a bug may be information gathering attacks that you
would, in hindsight, have wished your WAF had blocked. I would argue that
the WAF is not a stop-gap at all--it is an integral part of your site's
functionality.
A questionfor the masses. You note that "Being attacked constantly is a
side-effect of being on the Internet." Do you find it disturbing that that
is the case? Would you be interested in a p2p protocol stack that blocked
(e.g.,) port-scanning? I find it odd that we all put up with the constant
presence of malicious behavior. It can be stopped--why do we not invest in
the technology to stop it?
On Wed, Jul 15, 2020 at 6:14 PM Don Ankney via Dailydave <
dailydave@lists.aitelfoundation.org> wrote:
So far, this conversation focuses on how effectively WAFs block malicious
HTTP requests. I'd argue that this is both a red herring and an abuse of
WAF technology. A WAF only protects the enterprise when it blocks a
request that would trigger an actual bug. If there's no bug present, all
that's really happening is that likely malicious requests are being logged
at a much higher costs than if it were simply allowed to sit in the web
logs. Being attacked constantly is a side-effect of being on the Internet.
You should expect to be constantly scanned and probed. Ideally, action
should only be necessary if an attack is actually practical, otherwise
you're simply reacting to the background chatter of the Internet.
If there is a bug present and the WAF blocks the attempt, then it is a
legitimate protective action has occurred. The WAF is simply a stopgap,
however. If the bug is present, then the WAF will eventually be evaded
allowing an attack to succeed. As a result, I'd argue that success metrics
for a WAF are effectively twofold:
- How clearly does it signal that there's an actual bug in the software
behind it? - How much time can it provide for a fix to that software?
The problems you are protecting against aren't in the WAF or it's tuning
-- it's in the underlying software. A WAF is simply a stopgap that buys
time while code is corrected or vendors ship a patch. Relying on a WAF as
anything more than a defense-in-depth mechanism is a losing proposition
over time. Metrics need to reflect that.
From: Chuck McAuley via Dailydave dailydave@lists.aitelfoundation.org
Sent: Tuesday, July 14, 2020 7:14 AM
To: John Lampe jlampe@tenable.com; Rafal Los Rafal@ishackingyou.com
Cc: dailydave@lists.aitelfoundation.org <
dailydave@lists.aitelfoundation.org>
Subject: [Dailydave] Re: [EXTERNAL] WAF Metrics
This isn’t directly related to John’s observation below, but it got me
motivated to further clarify some of the challenges involved in testing
WAFs.
I’ve seen many implementations over the years that try to determine the
decision making process of an IPS, WAF, or similar device by simply
interrogating it from the client side only. The realities of test of
measurement is that it requires the user to implement both a client and
server side process of whatever it is you are testing to validate that not
only was malicious content allowed or blocked, but that IF it was allowed,
it was allowed through without a modification that would impact the intent
of the vulnerability.
That’s a mouthful of a sentence. Let me make it briefer:
You can’t tell if the bad stuff got through unless you are wrapped around
the bad stuff blocking thing.
This gets even harder when you are testing WAF’s. They very often
interrupt the connection, marshal the request, then issue a new HTTP
request to the server. This means that while your malicious request might
have altered headers, or Content-Transfer-Encoding changes the actual
little nugget of maliciousness is left alone.
“Fine”, you say, “I’ll just examine the URL request, because I’m only
interested in testing items that are impacted there.” And then
CVE-2017-5638 comes along and you need to accommodate for deserialization
vulnerabilities in other headers. The WAF catches it. But your current test
implementation can’t.
And that’s an easy example.
You can have a chain like this:
Client POST multipart http2 -> proxy -> TLS 1_1 -> Internet -> IPS
Performing TLS inspection -> Different handshake TLS 1_0 -> WAF/LB ->
http1.1 cleartext/chunked -> server.
I just came up with that off the top of my head, so please don’t challenge
me on the technical limitations of said example :).
The point is that testing in isolation is very different from testing a
deployed system under test and for each new technology deployed, the number
of permutations increases dramatically. This list has discussed many times
the merits and problems with “defense in depth” strategies, so I won’t
belabor that point. Only to say “it’s hard to get an accurate read on if
you are secure or not.” Lunchtime doubly so.
-chuck
*From: *John Lampe via Dailydave dailydave@lists.aitelfoundation.org
*Reply-To: *John Lampe jlampe@tenable.com
*Date: *Monday, July 13, 2020 at 8:07 PM
*To: *Rafal Los Rafal@ishackingyou.com
*Cc: *"dailydave@lists.aitelfoundation.org" <
dailydave@lists.aitelfoundation.org>
*Subject: *[Dailydave] Re: [EXTERNAL] WAF Metrics
[EXTERNAL]
Yeah, I guess the way I would envision it going would be:
-
web app scanner sees XSS vuln on /path/to/foo.php
-
my integration ties that web app scan into a format to pass to WAF
-
WAF sets up anti-xss rules on /path/to/foo.php (we had to actually
create a static mapping for this step) -
measure how many hits the waf blocks to that endpoint for the XSS
John
On Mon, Jul 13, 2020 at 10:46 AM Rafal Los Rafal@ishackingyou.com wrote:
*** CAUTION: This email was sent from an EXTERNAL source. Think before
clicking links or opening attachments. ***
John,
Can you expand on #2? How do you measure the number of attacks stifled?
_
*-- *Rafal
_Mobile:
- (404) 606-6056 _Email: Rafal.Los@Seventy7.Consulting
Rafal.Los@Seventy7.Consulting*
*From: *John Lampe via Dailydave dailydave@lists.aitelfoundation.org
*Reply-To: *John Lampe jlampe@tenable.com
*Date: *Saturday, July 11, 2020 at 9:52 PM
*To: *Dave Aitel dave.aitel@gmail.com
*Cc: *"dailydave@lists.aitelfoundation.org" <
dailydave@lists.aitelfoundation.org>
*Subject: *[Dailydave] Re: [EXTERNAL] WAF Metrics
So, I recently did an integration for a company that took their web app
scanner results and mapped those to existing WAF rules. I can think of 2
metrics based off that
-
How many real-world vulns have a corresponding check in the WAF? and
-
Once the WAF rules have been put in place to protect
actually-vulnerable endpoints, how many attacks were actually stifled?
John
On Sat, Jul 11, 2020 at 12:51 PM Dave Aitel via Dailydave <
dailydave@lists.aitelfoundation.org> wrote:
*** CAUTION: This email was sent from an EXTERNAL source. Think before
clicking links or opening attachments. ***
So I'm making a video on metrics, of all things, and I wanted to post both this
question
https://urldefense.com/v3/__https:/twitter.com/daveaitel/status/1281629327776522242?s=20__;!!I5pVk4LIGAfnvw!1DZZL1viGJTRH-H2akN1jntqUUjdEe6Oa7-HctTc9IePgQzC3DN13JryFgb8Id0i$and
the best answer so far to the list to see if anyone had any other ideas or
followups.
-dave
Dailydave mailing list -- dailydave@lists.aitelfoundation.org
To unsubscribe send an email to dailydave-leave@lists.aitelfoundation.org
Dailydave mailing list -- dailydave@lists.aitelfoundation.org
To unsubscribe send an email to dailydave-leave@lists.aitelfoundation.org
Isn’t using a WAF an “investment in technology to stop constant attacks?”
-chuck
From: Greg Frazier glfrazier@alum.mit.edu
Date: Friday, July 17, 2020 at 3:46 PM
To: Don Ankney dankney@hackerco.de
Cc: John Lampe jlampe@tenable.com, Rafal Los Rafal@ishackingyou.com, Chuck McAuley chuck.mcauley@keysight.com, "dailydave@lists.aitelfoundation.org" dailydave@lists.aitelfoundation.org
Subject: Re: [Dailydave] Re: [EXTERNAL] WAF Metrics
[EXTERNAL]
I'm not parsing your argument. If you knew the bug was there, you would fix the bug. The WAF is there to mitigate the bugs that you are not aware of. Further, web accesses that are out of scope of your intended functionality but do not trigger a bug may be information gathering attacks that you would, in hindsight, have wished your WAF had blocked. I would argue that the WAF is not a stop-gap at all--it is an integral part of your site's functionality.
A questionfor the masses. You note that "Being attacked constantly is a side-effect of being on the Internet." Do you find it disturbing that that is the case? Would you be interested in a p2p protocol stack that blocked (e.g.,) port-scanning? I find it odd that we all put up with the constant presence of malicious behavior. It can be stopped--why do we not invest in the technology to stop it?
On Wed, Jul 15, 2020 at 6:14 PM Don Ankney via Dailydave <dailydave@lists.aitelfoundation.orgmailto:dailydave@lists.aitelfoundation.org> wrote:
So far, this conversation focuses on how effectively WAFs block malicious HTTP requests. I'd argue that this is both a red herring and an abuse of WAF technology. A WAF only protects the enterprise when it blocks a request that would trigger an actual bug. If there's no bug present, all that's really happening is that likely malicious requests are being logged at a much higher costs than if it were simply allowed to sit in the web logs. Being attacked constantly is a side-effect of being on the Internet. You should expect to be constantly scanned and probed. Ideally, action should only be necessary if an attack is actually practical, otherwise you're simply reacting to the background chatter of the Internet.
If there is a bug present and the WAF blocks the attempt, then it is a legitimate protective action has occurred. The WAF is simply a stopgap, however. If the bug is present, then the WAF will eventually be evaded allowing an attack to succeed. As a result, I'd argue that success metrics for a WAF are effectively twofold:
- How clearly does it signal that there's an actual bug in the software behind it?
- How much time can it provide for a fix to that software?
The problems you are protecting against aren't in the WAF or it's tuning -- it's in the underlying software. A WAF is simply a stopgap that buys time while code is corrected or vendors ship a patch. Relying on a WAF as anything more than a defense-in-depth mechanism is a losing proposition over time. Metrics need to reflect that.
From: Chuck McAuley via Dailydave <dailydave@lists.aitelfoundation.orgmailto:dailydave@lists.aitelfoundation.org>
Sent: Tuesday, July 14, 2020 7:14 AM
To: John Lampe <jlampe@tenable.commailto:jlampe@tenable.com>; Rafal Los <Rafal@ishackingyou.commailto:Rafal@ishackingyou.com>
Cc: dailydave@lists.aitelfoundation.orgmailto:dailydave@lists.aitelfoundation.org <dailydave@lists.aitelfoundation.orgmailto:dailydave@lists.aitelfoundation.org>
Subject: [Dailydave] Re: [EXTERNAL] WAF Metrics
This isn’t directly related to John’s observation below, but it got me motivated to further clarify some of the challenges involved in testing WAFs.
I’ve seen many implementations over the years that try to determine the decision making process of an IPS, WAF, or similar device by simply interrogating it from the client side only. The realities of test of measurement is that it requires the user to implement both a client and server side process of whatever it is you are testing to validate that not only was malicious content allowed or blocked, but that IF it was allowed, it was allowed through without a modification that would impact the intent of the vulnerability.
That’s a mouthful of a sentence. Let me make it briefer:
You can’t tell if the bad stuff got through unless you are wrapped around the bad stuff blocking thing.
This gets even harder when you are testing WAF’s. They very often interrupt the connection, marshal the request, then issue a new HTTP request to the server. This means that while your malicious request might have altered headers, or Content-Transfer-Encoding changes the actual little nugget of maliciousness is left alone.
“Fine”, you say, “I’ll just examine the URL request, because I’m only interested in testing items that are impacted there.” And then CVE-2017-5638 comes along and you need to accommodate for deserialization vulnerabilities in other headers. The WAF catches it. But your current test implementation can’t.
And that’s an easy example.
You can have a chain like this:
Client POST multipart http2 -> proxy -> TLS 1_1 -> Internet -> IPS Performing TLS inspection -> Different handshake TLS 1_0 -> WAF/LB -> http1.1 cleartext/chunked -> server.
I just came up with that off the top of my head, so please don’t challenge me on the technical limitations of said example :).
The point is that testing in isolation is very different from testing a deployed system under test and for each new technology deployed, the number of permutations increases dramatically. This list has discussed many times the merits and problems with “defense in depth” strategies, so I won’t belabor that point. Only to say “it’s hard to get an accurate read on if you are secure or not.” Lunchtime doubly so.
-chuck
From: John Lampe via Dailydave <dailydave@lists.aitelfoundation.orgmailto:dailydave@lists.aitelfoundation.org>
Reply-To: John Lampe <jlampe@tenable.commailto:jlampe@tenable.com>
Date: Monday, July 13, 2020 at 8:07 PM
To: Rafal Los <Rafal@ishackingyou.commailto:Rafal@ishackingyou.com>
Cc: "dailydave@lists.aitelfoundation.orgmailto:dailydave@lists.aitelfoundation.org" <dailydave@lists.aitelfoundation.orgmailto:dailydave@lists.aitelfoundation.org>
Subject: [Dailydave] Re: [EXTERNAL] WAF Metrics
[EXTERNAL]
Yeah, I guess the way I would envision it going would be:
-
web app scanner sees XSS vuln on /path/to/foo.php
-
my integration ties that web app scan into a format to pass to WAF
-
WAF sets up anti-xss rules on /path/to/foo.php (we had to actually create a static mapping for this step)
-
measure how many hits the waf blocks to that endpoint for the XSS
John
On Mon, Jul 13, 2020 at 10:46 AM Rafal Los <Rafal@ishackingyou.commailto:Rafal@ishackingyou.com> wrote:
*** CAUTION: This email was sent from an EXTERNAL source. Think before clicking links or opening attachments. ***
John,
Can you expand on #2? How do you measure the number of attacks stifled?
_--
Rafal
_Mobile: (404) 606-6056
_Email: Rafal.Los@Seventy7.Consultingmailto:Rafal.Los@Seventy7.Consulting
From: John Lampe via Dailydave <dailydave@lists.aitelfoundation.orgmailto:dailydave@lists.aitelfoundation.org>
Reply-To: John Lampe <jlampe@tenable.commailto:jlampe@tenable.com>
Date: Saturday, July 11, 2020 at 9:52 PM
To: Dave Aitel <dave.aitel@gmail.commailto:dave.aitel@gmail.com>
Cc: "dailydave@lists.aitelfoundation.orgmailto:dailydave@lists.aitelfoundation.org" <dailydave@lists.aitelfoundation.orgmailto:dailydave@lists.aitelfoundation.org>
Subject: [Dailydave] Re: [EXTERNAL] WAF Metrics
So, I recently did an integration for a company that took their web app scanner results and mapped those to existing WAF rules. I can think of 2 metrics based off that
-
How many real-world vulns have a corresponding check in the WAF? and
-
Once the WAF rules have been put in place to protect actually-vulnerable endpoints, how many attacks were actually stifled?
John
On Sat, Jul 11, 2020 at 12:51 PM Dave Aitel via Dailydave <dailydave@lists.aitelfoundation.orgmailto:dailydave@lists.aitelfoundation.org> wrote:
*** CAUTION: This email was sent from an EXTERNAL source. Think before clicking links or opening attachments. ***
So I'm making a video on metrics, of all things, and I wanted to post both this question https://urldefense.com/v3/__https:/twitter.com/daveaitel/status/1281629327776522242?s=20__;!!I5pVk4LIGAfnvw!1DZZL1viGJTRH-H2akN1jntqUUjdEe6Oa7-HctTc9IePgQzC3DN13JryFgb8Id0i$ and the best answer so far to the list to see if anyone had any other ideas or followups.
-dave
[cid:image001.png@01D65C53.BC34B4A0]
[cid:image002.png@01D65C53.BC34B4A0]
Dailydave mailing list -- dailydave@lists.aitelfoundation.orgmailto:dailydave@lists.aitelfoundation.org
To unsubscribe send an email to dailydave-leave@lists.aitelfoundation.orgmailto:dailydave-leave@lists.aitelfoundation.org
Dailydave mailing list -- dailydave@lists.aitelfoundation.orgmailto:dailydave@lists.aitelfoundation.org
To unsubscribe send an email to dailydave-leave@lists.aitelfoundation.orgmailto:dailydave-leave@lists.aitelfoundation.org
