Blog

Archive for July, 2020

Building a .freq file with Public Domain Data Sources, (Fri, Jul 31st)

This diary started out as a frequency analysis of zone files for domains that expire before May 2023. Our intent was to look for frequency of random on all Generic Top-Level Domains (gTLDs). This exercise quickly turned into “create the freq file” for the analysis.

First, we create our .freq file

python3 ./freq.py —create bookmag.freq

The name will make sense as sources are revealed.

My first pass was to download a few famous books from the Gutenberg project (e.g., Sherlock Holmes, a Tall of Two Cities, War and Peace) following the example from [2] [4] Mark. The frequency analysis on that first attempt did not match up to my randomly (not true random as my brain was the random generator, read into that what you will :)).

This got me thinking, can you compile some strange sources and LOTS of data to create a better frequency analysis. More data == better right (well, not always but in this case, maybe)…

This gave me an idea, why not put all of the venerable PHRACK [8] mags in my freq file… To the “Intertubes” BatMan…

Now when you download “ALL” the TGZ files there are a few steps to get them in your new bookman.freq. The first is to uncompress them. And YES, I downloaded them by clicking on them and after about phrack15 or 16 wget -r or curl – something came to mind. Stuck with it and was polite. Thanks Phrack for leaving these rare gems up!

Once you get them unpacked, you can crawl through them pretty easily as noted below:

for fname in `ls ../phrack*/*`
do
echo $fname
python3 ./freq.py --normalfile $fname bookmag.freq
done

Now are bookmag.freq has all of Phrack as part of its analysis baseline.

Onto the Gutenberg portion cause now the curiosity of “Can I put EVEN MORE literary works in this thing” came to mind.. Go Big or Go ….

So after locating a DVD ISO of the Gutenberg project (not listed here, but it is easy to find) and going through uncompressing all the ZIP files:

NOTE: Files were transferred from the root of the ISO to a working “RAW” Directory.

for fname in `find ./ "*.ZIP"`
do
unzip $fname -d ../textdocs
done

Here is where zsh barked at me cause it was “TOOOOOO LOOOOOONG”, as apparently there is a limit to looping through an ls.

Settled into manually reducing the load on my poor ‘ls’ cmd.

For example changing the ls to ‘0d*.txt’

for fname in `ls ../textdocs/0d*.txt`
do
echo $fname
python3 ./freq.py --normalfile $fname bookmag.freq
done

../textdocs/0ddc809a.txt
../textdocs/0ddcc10.txt
../textdocs/0ddcd09.txt
../textdocs/0ddcl10.txt
../textdocs/0drvb10.txt

Now that we have a robust .freq file we can start testing.

Let us know what sources you decide to use?

References
[1] https://isc.sans.edu/diary/Detecting+Random+-+Finding+Algorithmically+chosen+DNS+names+%28DGA%29/19893
[2] https://github.com/MarkBaggett/freq
[3] http://dev.gutenberg.org/
[4] https://www.youtube.com/watch?v=FpfOzcRpzs8
[5] https://github.com/sans-blue-team/freq.py/blob/master/freq.py
[6] https://wiki.sans.blue/Tools/pdfs/freq.py.pdf
[7] https://isc.sans.edu/diary/freq.py+super+powers%3F/19903
[8] http://phrack.org/

 

(c) SANS Internet Storm Center. https://isc.sans.edu Creative Commons Attribution-Noncommercial 3.0 United States License.

Reposted from SANS. View original.

Posted in: SANS

Leave a Comment (0) →

Python Developers: Prepare!!!, (Thu, Jul 30th)

I know… tried it several times… growing up is hard. So instead, you decided to become a “Red Teamer” (aka Pentesters…). You got the hoodie, and you acquired a taste for highly caffeinated energy drinks. Now the only thing left: Learning to write a script. So like all the other “kids,” you learn Python and start writing and publishing tools (Yes… all the world needed was DNS covert channel tool #32773… you realize you could have written that as a bash oneliner?).

So what follows comes from a reluctant occasional Python coder.

So instead of learning a real language like Perl, you figured Python is it. Lately, in my ongoing quest to avoid growing up, I jumped from Perl on the Python bandwagon to also be one of the cool kids. Like most developers, I heavily lean on Stackoverflow and other tools like Github to find sample code that I am sure millions of others have reviewed (isn’t that the point of Open Sources?) and made sure it is well written secure code.

But as I learn more about Python, I noticed a dangerous trend in Python: People all for sudden forgot that preparing SQL statements is a thing. I think the problem is in part that Python makes it so easy to mix up prepared/not-prepared.

Compare these two snippets:

sql = “””SELECT count(*) FROM users where id=%s”””
vars = (‘642063 OR 1=1’,)
cursor.execute(sql , vars)
count=cursor.fetchone()
print(count[0])

Returns: 1

sql = “””SELECT count(*) FROM users where id=%s”””
vars = (‘642063 OR 1=1’,)
cursor.execute(sql % vars)
count=cursor.fetchone()
print(count[0])

Returns: 123237

The difference is a single character in the line highlighted in red: a “,” vs. a “%.”

A “,” will identify the variables as a second parameter. A ‘%’ is a format string like operators, and it will alter the string, which is the only parameter in this case. So really no better than concatenating the string. Yes, the ‘{}’ notation is not much better. You can specify more specific formats, but that falls apart quickly if you are using arbitrary strings.

Unlike with Perl’s amazing PDO library, Python is a bit inconsistent between different databases. So double-check this if you are using SQLite or Postgresql. 

And if you would like to read about this by someone who appears to know how to code in Python: I found this blog post that I thought was pretty good and went into more details: https://www.btelligent.com/en/blog/best-practice-for-sql-statements-in-python/


Johannes B. Ullrich, Ph.D., Dean of Research, SANS Technology Institute
Twitter|

(c) SANS Internet Storm Center. https://isc.sans.edu Creative Commons Attribution-Noncommercial 3.0 United States License.

Reposted from SANS. View original.

Posted in: SANS

Leave a Comment (0) →

Consumer VPNs: You May Be Fine Without, (Wed, Jul 29th)

If you are watching YouTube videos, you may have noticed how many of them include sales pitches for various VPN providers. Here at the Internet Storm Center, we do regularly receive offers for paid articles, or requests for podcast pitches, from VPN providers or “VPN Review Guides”. VPNs have certainly become a new hot business.

In more generic terms, VPNs offer a more secure and private way to use the internet. VPNs do deliver on some of these claims. But I think they also overpromise.

Let’s start with some threat scenarios. I think it does not make much sense to talk about security without a clear threat in mind that you are defending against.

1. Coffee Shop Wifi

So you are using Wifi at a local coffee shop. A VPN will in almost all cases sufficiently protect your traffic. There are a handful of VPN providers that were caught in the past not encrypting at all. Those are the exceptions and if you are going with one of the heavily advertised name brands, you should be good in this case. The content of your traffic as well as the nature of the sites you visit should be difficult to detect for anybody else in the coffee shop (well.. maybe if they are setting up an IPv6 router and your VPN doesn’t protect you from the IPv6 leak attack.. )

On the other hand, even without a VPN you may be in pretty good shape. With 80+ % of websites using HTTPS, and browsers/operating systems implementing DNS over HTTPS, you pretty much already have all the protection a VPN would give you. It is actually more secure than you may think to use coffee house wifi for online banking.

An alternative solution of course: Tether to your phone. LTE will also avoid a lot of the layer 2 issues that you may run into. But as a (former?) frequent traveler, I can attest to the spotty performance of this option and I have often been “pushed” to insecure Wifi as a result.

2. Private Home Browsing

Many high-speed internet connections come with a more or less static IP address. This may allow an adversary (advertiser) to track you. Using a VPN will allow you to obtain a more dynamic IP address in a location not associated with you (this of course can also be used to bypass some content restrictions). Overall, this works well. But be aware that some sites will just outright block connections via VPNs. In addition, most “tracking” doesn’t use IP addresses. Instead features like cookies are used and VPNs will not affect that tracking at all unless they inspect content which opens up other problems. 

You will also conceal your traffic from your ISPs. While ISPs have been seen inspecting and even modifying traffic, all you do is replace your ISP for your VPN provider. VPN providers are not immune to the same commercial pressures as ISPs.

Some VPNs promise faster connection speeds. The argument is that some ISPs will slow certain traffic like for example BitTorrent, or traffic to certain streaming sites. A VPN will conceal the nature of the traffic and makes it more difficult to apply these types of rules. In my experience, the opposite is true. ISPs (at least in my experience) currently do very little filtering like this. A VPN will almost always lead to a lower-performing connection due to the overhead added by the VPN. You may also run into bottlenecks in the VPN infrastructure.

3. Nation-State / Government Attacks

All countries I am aware of have some form of legal intercept legislation requiring telecom providers like ISPs to provide access to customer traffic. Encryption is, of course, one primary defense against this kind of data collection. In most cases, VPN providers are subject to the exact same laws. While some offer “no logging” (which has been shown to be often a false claim), after they receive a respective order they will likely not be able to say “no”, unless they are located in a country that makes these orders difficult to obtain. In this scenario, you will need a VPN provider and endpoint in a different country than yours. Some countries are known to outright block VPN connections for that reason, making setting up a well-performing and reliable VPN connection difficult. If you are afraid of traffic interception while traveling to a foreign country: You are almost always better off connecting to a VPN server with a custom configuration at home vs. using commercial VPN providers. Commercial VPN providers use well-known servers that are easily blocked. Modern HTTPS/DoH may again provide almost the same level of security as a VPN.

What Do VPNs not Do?

VPNs will not protect you from malware or visiting malicious sites unless the VPN provider is inspecting your traffic. Even then, a decent up to date endpoint protection suite is likely a lot better than what the VPN provider is doing. And inspecting traffic is exactly what you are usually trying to avoid. In most of the scenarios discussed above, the attacker is more likely going to attack you via the endpoint vs. bothering to intercept your traffic.

The same is true for tracking. As mentioned above, you can still be tracked via good old cookies and similar techniques. In most cases, VPNs do not make tracking significantly more difficult.

You are replacing one choke point, your ISP, with another, the VPN provider. VPN providers are far less regulated and monitored than ISPs. As recent leaks have shown, VPN providers promising no-logging have actually been logging customer traffic. VPN providers are subject to the laws of the countries they operate in and have to collect and make available logs according to those laws.

What Alternatives Do I have?

As mentioned above: An up to date system with a browser configured to use DNS over HTTPs will give you 90% of what a VPN will get you (of course, you may have similar trust issues with the DoH endpoint you select).

You could set up a VPN server at home. This is a great solution if you are mostly concerned about privacy while traveling. It will also give you secure access to systems in your house and is not terribly difficult to set up. As an alternative to a home system, a VPN server on a cheap cloud provider may work as well. The price of a simple cloud server is comparable to some VPN providers. Or better: Set up a complete virtual system in the cloud and use its browser. This will probably give you the best protection from malware and tracking if you just “reset” the system often.

Having a VPN available from a reputable company is a good option if you would like to occasionally watch a movie only available in a different country, or if you would like to add additional privacy when browsing. But understand the limitations and it will not prevent you from ransomware if you keep turning off your anti-malware solution and if you run Excel macros that arrive in various emails. 

I will not recommend a VPN provider. Do your homework, search recent news for VPN providers who leaked data (or didn’t encrypt traffic at all). If you are a VPN provider and are looking for a guest appearance on my daily podcast: The only guests I feature are SANS.edu students who score As on their research paper. So have your marketing person sign up for the SANS.edu masters program, and I will feature them once they submit a research paper and receive an A grade.

Additional Reading:

Study about trackers in Android VPN Software: https://research.csiro.au/isp/wp-content/uploads/sites/106/2016/08/paper-1.pdf
Recent VPN Data Leaks: https://www.welivesecurity.com/2020/07/20/seven-vpn-services-leaked-data-20million-users-report


Johannes B. Ullrich, Ph.D., Dean of Research, SANS Technology Institute
Twitter|

(c) SANS Internet Storm Center. https://isc.sans.edu Creative Commons Attribution-Noncommercial 3.0 United States License.

Reposted from SANS. View original.

Posted in: SANS

Leave a Comment (0) →

All I want this Tuesday: More Data, (Tue, Jul 28th)

We are making more and more data available via our API.  Couple things just added:

  • A combined “Threatintel” feed that includes basic categories of notable IPs
  • A list of subnets used by prominent cloud providers

First a few reminders about our API:

You may use data from our API for free. This includes the use in commercial networks. We only ask you to talk to us about licensing if you are reselling the data as part of a product. For example, if you include this data in your own data feeds and you are charging money for the feed.

At this point, we are not asking for any kind of authentication. We are doing minimal tracking and do not care who is downloading the data. However, we may block users who we feel abuse the data. We will try to contact you if you include contact information in your user agent and it is possible that we will block certain “generic” user agents. Best to customize the user agent somehow.

What I do ask for in exchange for using these feeds:

  • provide feedback. Let us know how you use these feeds, and please let us know about problems (see our contact page)
  • Consider contributing data via our honeypot.

The use case I am envisioning for the data is to include it in a SIEM or other log monitoring products to add “color” to the IP address. It may be useful to know that the IP attacking you is just “yet another infected bot”.

WE DO NOT RECOMMEND THE USE OF OUR DATA AS A SIMPLE BLOCK LIST

Our data includes false positives. I see it as a feature as false positives is also something we continuously learn from. For example when it comes to DoS attacks, or artifacts of firewalls blocking “odd” packets. Best case: If you are blocking based on our “Top 100” list, you are blocking a bunch of bots that scan for vulnerabilities you are hopefully not susceptible to. And if you are vulnerable to any of them: There are bot number 101-123183849 that will still get you.

Our API offers data in different formats. Just add the format specifier to the URL. For example “?json” for JSON which is probably now the preferred output format.

Back to the new datafeeds I added:

  1. “Intelfeed” https://isc.sans.edu/api/intelfeed

A lot of organizations like to ingest random feeds of “threatintel” data. This feed is trying to extract some notable data from across our different collections. It includes for port scanners detected by out DShield sensors, hosts scanning for web vulnerabilities and ssh brute force bots reported by our honeypots and data from various other feeds we are collection. A quick snippet:

   {
    “ip”: “1.119.147.51”,
    “description”: “DShield Ports: 65529,16379,6379,22,1433”
  },
  {
    “ip”: “1.119.195.58”,
    “description”: “dshieldssh”
  },
  {
    “ip”: “1.160.6.79”,
    “description”: “talos”
  },
  {
    “ip”: “5.11.11.10”,
    “description”: “tldns”
  },

  • The first IP is a host scanning various ports based on our DShield data (I only include hosts that hit several target IPs to limit the size of the feed).
  • The second IP scans for SSH servers
  • The third IP is included in the Talos IP blocklist
  • Finally, 5.11.11.10 is a name server for a top level domain (don’t block these! 😉 )

The number of categories is likely going to increase.

2. Cloud IPs https://isc.sans.edu/cloudips

This is a simple feed including prefixes used by major cloud providers. Right now, it includes AWS, Azure, Google and Oracle with more to come. This one is pretty straight forward.

These feeds are only as good as you make them. Feedback is very welcome. Please use our Contact Page for feedback.

 


Johannes B. Ullrich, Ph.D. , Dean of Research, SANS Technology Institute
Twitter|

(c) SANS Internet Storm Center. https://isc.sans.edu Creative Commons Attribution-Noncommercial 3.0 United States License.

Reposted from SANS. View original.

Posted in: SANS

Leave a Comment (0) →
Page 1 of 7 12345...»