When we want to gather information about something, we search it on Google. We look for it on the first search results page and sometimes on the second page. And, when we do not find it, we tend to quit it, right? What we all lack to understand is the presence of information in hundreds of search results pages. Moreover, it is not always possible to look into every page and gather information. Hence, in today’s article, we will talk about OSINT tools that will help you gather information about anything lucidly.
But, before we start talking about the top 10 OSINT tools, let’s clear our concepts about OSINT.
What is OSINT?
Open Source Intelligence or OSINT is the custom of collecting information and data from either published or publicly available sources. A myriad of professionals practice OSINT operations. Starting from IT security pros, state-sanctioned intelligence operatives to malicious hackers. These professionals look into information haystack and then finally find out the particular needle. The professionals learn information which many do not realize is public.
OSINT in quite a few days is a mirage of OPSEC or operational security. This is a security process organizations use to protect their public information and data when analyzed; if revealed, it can be very damaging. Organizations’ IT security departments are extensively tasked with performing OSINT operations to shore up OPSEC.
According to DoD, OSINT is fabricated from publicly accessible information that is exploited, collected, and disseminated effectively. This is done in a well-timed manner for addressing an intelligence requirement to an audience.
The ever-expanding and exploding internet world are letting and making people share their thoughts and views on various platforms. This is when they purchase various services and goods and share their thoughts and opinions via personal blogs.
Hence, this generates information or data in various forms like images, videos, audio, and texts. These are available to every person on the internet unless restricted by a particular organization.
Read: Integration Platform as a Service Basics Explained
OSINT has six primary information sources. They are categorized and mentioned below.
- Media: These include magazines, television, radio, and print newspapers from across the world and between countries.
- Internet: Online publications, discussion groups, blogs, citizen media (i.e., user-created content and smartphone videos). YouTube, Instagram, Facebook, and other social media handles.
- Public: Public government reports, hearings, budgets, press conferences, speeches, websites, and government data. This source is an official source though information and data are publicly accessible. This is subjectable to use freely and openly.
- Professional: Information from journals, symposia, dissertations, academic publications, and theses.
- Commercial imagery, commercial data, databases, and industrial and financial assessments.
- Grey literature, patents, preprints, business documents, newsletters, technical reports, and unpublished works.
Therefore, to collect this massive data and information on a particular topic, we need tools to make our work easier. OSINT tools reduce our analysis time. Hence, making our work relatively easier. Below we have listed some of the best as well as free OSINT tools. These tools are used by social engineers, Penetration testers, and security researchers for various subjects and topics.
OSINT History
During the 1980s, intelligence services and the military started shifting their covert information-gathering activities to discover hidden secrets. The covert activities included tapping phones and or reading an adversary’s email. So, they started looking for free and useful intelligence. The intelligence might also be officially published.
The world was changing at that time as well. Even though the internet was not in the scene, there were other sources where various information was available. Information was available in public databases and newspapers. Moreover, they contained exciting and useful facts as well. This information was beneficial if someone knew how to connect the dots. Hence, OSINT was initially referred to as this type of spycraft.
These same techniques are applied to cybersecurity. So, most organizations now have public-facing, vast infrastructures spanning over many networks, hosting services, technologies, and namespaces. Hence, important information is easily hidden on an employer’s desktop, over cloud services, webcams, employee-owned BYOD devices, and legacy on-prem servers. Moreover, data is also hidden inactive programs’ and active apps’ source codes.
In reality, IT technical staff do not know about all the information and asset of their organization, public or not. Adding to the fact, many organizations do control or own additional assets and information indirectly. When fallen into the wrong hands, many potential important information and secrets could be hazardous and harmful.
Therefore, IT professionals use OSINT tools to know the different and essential aspects of their organization and other relevant matters. So, we have listed the best tools to keep track and know crucial information about a topic.
Why is OSINT important?
OSINT is essential in keeping tabs on significant chaos. OSINT has three crucial tasks that IT professionals have to fulfill. So, to complete this task, several OSINT tools have been built. Most tools serve all three functions.
Search public-facing assets
OSINT tools’ primary function is to help IT professionals discover unknown public assets and information and mapping what data they possess that could be very harmful during an attack surface. Their primary function is to detect and record what information an average person can publicly find out. This is done without hacking or resorting.
Detecting important information outside organizations
Another OSINT tools’ essential function is to discover and detect assets outside the organization. These assets or information is available in social media posts or locations and domains. These are mostly outside of the securely defined network. Your organization’s information may be open and stored in third-party applications and SaaS services. Moreover, mergers and acquisitions as well bring in a lot of problems. So, OSINT is very helpful during mergers and acquisition attacks. The popularity and growth of social media is a significant advantage. Hence, one can quickly look for information on social media sites outside the organization’s perimeters.
Group all discovered information into actionable form
The final function of OSINT tools is to group and collate all the critical discovered information into actionable form. When you run an OSINT scan for a prominent organization, this can lead to thousands of results. So, one has to pierce through all that data and simply stick to the most severe problems. The open-source intelligence tool will help you to deal with issues and address the important ones at first.
Now that we know about OSINT, let’s review the best OSINT tools.
Best OSINT Tools
1. Maltego
Maltego experts uncover relationships among companies, people, publicly accessible information, and domains on the internet. It’s also well-known for taking discovered information enormous amount, sometimes. And then plotting it all out in easy-to-read graphs and charts. The graphs do a great job of extracting raw intelligence and then making it actionable. Moreover, each graph contains up to 10,000 data points.
Maltego’s program works by mechanizing the different public data sources’ searching. Therefore, users easily click on one button and then successfully execute multiple queries. Maltego’s program calls a search plan a “transform action.” Moreover, Maltego comes with quite a few by default, including familiar public information sources like whois records, search engines, DNS records, and social networks. As the program uses public interfaces for performing its searches, it is compatible with almost all sources of public interface information. Therefore, adding more searches or making a whole new one to a transform action is very lucid.
Once the information is successfully gathered, Maltego makes connections, unmasking the hidden relationships. Relationships between email addresses, names, aliases, affiliations, websites, document owners, companies, and other information might prove useful in an investigation. Or, it can also be to look for potential future problems. The program itself runs in Java, so it works with Mac, Linux, and Windows platforms.
There is a free version with limited features called Maltego CE. Desktop versions of Maltego XL run at $1,999 per instance. Moreover, if you want server installations for large-scale commercial use, then it starts at $40,000. Also, it comes with a complete training program. Therefore, undoubtedly, Maltego is one of the best free OSINT tools.
2. Recon-ng
Developers working in Python have easy access to a powerful tool in Recon-ng. It is written in that language. Recon-ng’s interface looks very identical to the popular Metasploit Framework. Hence, reducing the learning curve for those who already have experience with it. Moreover, it comes with an interactive help function that many Python modules lack, making developers quickly pick it up.
Recon-ng quickly automates time-consuming OSINT activities, like pasting and cutting. Recon-ng doesn’t claim about all OSINT gathering being conducted by its tool. But, it is used to automate the most popular kinds of harvesting. Hence, leaving much more time for the things that still must be done manually.
Recon-g is so beautifully designed, even the most junior Python developers can create searches of publicly available data. Hence, returning useful results. It has a very modular framework with a lot of built-in functionality. Everyday tasks like interacting with databases, managing API keys, standardizing output, and making web requests are all part of the interface. Instead of programming Recon-ng to perform searches, developers choose the functions they want it to perform. Therefore, building an automated module in just a few minutes.
Recon-ng is free, open-source software. Recon-ng’s available wiki includes comprehensive information for getting started with the tool. Moreover, it also has best practices for using it.
3. theHarvester
theHarvester is one of the most straightforward ONINT tools on this list. It is skillfully designed for capturing public information existing outside an organization’s owned network. Moreover, it also finds incidental things on internal networks as well. But, the majority of tools focus on outward-facing. theHarvester is quite useful as an investigating step before penetration testing.
theHarvester uses sources, including popular search engines like Google and Bing. Moreover, it also uses the lesser-known ones like DNSdumpster, dogpile, and the Exalead metadata engine. It also uses the AlienVault Open Threat Exchange and Netcraft Data Mining. It even taps the search engine Shodan for discovering open ports on discovered hosts. In general, theHarvester gathers names, IPs, emails, subdomains, and URLs.
TheHarvester easily accesses most public sources without any such special preparations. However, few sources used to require an API key. Moreover, you must also have Python 3.6 or even better in your environment.
You can quickly obtain theHarvester on GitHub. It is highly recommended that you use a virtualenv to create an isolated Python environment when cloning it from there.
4. Shodan
Shodan is a dedicated search engine for finding intelligence about devices that are not often searchable. But these devices happen to be everywhere these days. These devices are like the millions that make up the IoT or the internet of things. It is also used to find stuff like vulnerabilities and open ports on targeted systems. Some other OSINT tools, like theHarvester, uses Shodan as a data source. But, remember Sho9dan’s deep interaction requires a paid account.
The many places Shodan monitors and searches as part of an OSINT effort is awe-inspiring. It’s one of the few engines capable of examining OT or Operational Technology. This includes the kind used in industrial control systems at places like manufacturing facilities and power plants. Any industries’ OSINT gathering effort deploying both OT and information technology will miss a considerable part of that infrastructure without having a Shodan tool.
In addition to IoT devices like building sensors, cameras, and security devices, Shodan equally turns to look upon databases to check if any information is publicly accessible through various paths other than the main interface. It can even work with videogames, discovering things like Counter-Strike: Global Offensive or Minecraft servers, hiding on corporate networks. The games should not be hiding on corporate networks. Hence, Shodan works with video games and discovers what vulnerabilities they generate.
Hence, you can purchase a Freelancer license and use Shodan. You can quickly scan up to 5,120 IP addresses per month, with returns of up to a million results. That costs $59 per month. Serious users like IT professionals can get a Corporate license, providing unlimited results and scanning up to 300,000 IPs monthly. The Corporate version also includes premium support and vulnerability search filter and costs $899 per month.
5. Metagoofil
Another one of the best free OSINT tools available on GitHub is Metagoofil. It is optimized for extracting metadata from public documents. Metagoofil investigates almost all kinds of documents, reaching through public channels including .doc, .pfd, .ppt, .xls, and many others.
The quantity of exciting data, Metagoofil gathers is very impressive. Searches results include the usernames associated with discovered documents, as well as real names if available. Moreover, it also successfully maps the paths of getting those documents, which provides things like shared resources, server names, and directory tree information related to the host organization.
Everything that Metagoofil finds out is beneficial for a hacker, who uses it to do things like phishing emails and launching brute-force password attacks. So, organizations wanting to protect themselves must take the same OSINT gathered information. Therefore, protecting or hiding it before a malicious actor takes the initiative.
6. search code
Search code is an excellent option for people who need to go deep into the OSINT gathering’s complex matrixes. It is a very specialized search engine looking for useful intelligence inside source codes. This powerful engine is surprisingly a single developer’s work.
As a repository of code is first added to the program before becoming searchable, search code straddles the line between an OSINT tool and one designed to find things other than public information. However, search code is still considered an OSINT tool because developers use it to discover problems associated, having sensitive information accessible inside code on either running apps. It is also accessible on that are still in development. In the latter case, the problems are fixed before deployment into a production environment.
Although matters involving code will require much more knowledge than a simple Google search, search code significantly does a great job of making its interface as lucid to use as possible. So, users type in their search fields, and then the search code returns relevant search results with search terms highlighted in the code lines. Suggested searches include usernames, security flaws, unique characters that can launch code injection attacks, and unwanted active functions like re. compile.
Most of the time, the results by search code are self-explanatory. However, it’s possible to click through those results to find matching problems or more in-depth information if needed.
7. SpiderFoot
This meta-OSINT tool is like OSINT’s Metasploit. So, set SpiderFoot to lose on the fragrance of a domain name, IP address, username, email address, subnet, or ASN, specify some of the modules or all of the modules to use, and SpiderFoot will bring you back all the things.
SpiderFoot is a freemium tool. It is available for download, and you can easily use it as an open-source product at no cost or a SaaS offering called SpiderFoot HX.
The tool easily integrates with nearly all OSINT data feeds currently available, i.e., equal to 200 according to their website, including HaveIBeenPwned, AlienVault, Shodan SecurityTrails. Moreover, their website promises, integrating new OSINT data feeds within “days, not weeks.” This makes SpiderFoot perfect for monitoring publicly-exposed information about your organization, as well as your competitors.
SpiderFoot offers data visualization tools to make sense of the data collected. You can also export the data collected as CSV, JSON, or GEXF to analyze it in the comforts of your offline database.
8. Babel X
Relevant information is not always in English. Only a quarter of internet users speak English as their primary language, according to Statista. A lot of sources suggest as much as 55% of internet content is in English. The information you need might be in Chinese, French, Spanish, Malayali, or Tamil.
Babel X from Babel Street is a multilingual search tool for the public internet. This also includes blogs, social media, news sites, and message boards. It also searches the dark web, including Onion sites, and some in-depth web content that Babel X can access through licensing or agreements from the content owners. The product is skilfully able to geo-locate the information’s source it finds. Moreover, it also performs text analysis for identifying relevant results. Babel X is presently capable of searching in more than 200 languages.
Use cases where a multilingual search is useful to include searching global news for situational awareness, let’s say, knowing trends in targeting for ransomware attacks. Babel X is also used to spot a company’s intellectual property for sale on a foreign website. Moreover, it also spots information showing a key partner has been compromised. Customers have also used Babel X for finding user handles of malicious and suspected attackers on non-English message boards.
The main Babel X product is cloud-based. It also allows customers customization by adding their data sources to search. Babel Box is an on-premises version. But, do note that it lacks certain Babel X‘s features, such as accessing deep web data sources. Babel Channels, the lowest cost option, is data sources’ curated collection. A mobile app is available for all the possibilities.
9. Google Dorks
While investigating people or organizations, many IT security professionals forget the importance of using traditional search engines for intel gathering and recon. In such a case, Google Dorks is your best friend. They are here since 2002 and helps professionals a lot in their intel reconnaissance. Google Dorks are simple ways for querying Google against certain information that is useful for security investigations. Search engines index a lot of information and data about almost everything on the internet, including individual, organizations, and their data.
Some popular operators used to perform Google Dorking
- Filetype: One can use this dork to find any kind of filetype.
- Inurl: This will look out for mentioned words inside the URL of any website.
- Intitle: This will search for any specific words inside the page title.
- Ext: It helps you to find files with specific extensions (eg. .txt, .log, etc).
- Intext: It performs queries that help to search for specific text inside any page.
10. Creepy
Creepy is an open-source Geolocation intelligence tool, gathering information about Geolocation by using many different images hosting services and social networking platforms that are formerly distributed somewhere else.
Generally, Creepy is classified into two primary tabs: the ‘map view’ and ‘Targets’ tab. It shows the maps’ descriptions by applying a search filter based on the exact date and location. And just not that, even all these reports are accessible in KML and format as well. Moreover, it is written in python language. It also comes with a packaged binary for Linux distributions like Debian, Ubuntu, Backtrack, and even Microsoft Windows.
Hence, these were the top 10 best OSINT tools.
11. OSINT Framework
While the tools mentioned above offer a wealth of OSINT data, you will find many other techniques and tools available to help you completely understand your organization’s public footprint. An excellent resource to discover more tools is the OSINT framework. The OSINT Framework offers a web-based interface, breaking down a myriad of topic areas to OSINT researchers. Thereby connecting you to the tools that will help you sniff out the information you need.
The tools that the OSINT Framework will point you to are all free of cost. However, some may require registration or also have more fully-featured paid versions available. Some are tools that simply help to construct advanced Google searches that to yield a surprising amount of information. Justine Nordine maintains the OSINT Framework.
Is OSINT illegal?
While malicious hackers frequently use OSINT techniques as an examination before they launch an illegal attack. For the top part, the methods and tools themselves are perfectly legal; they are designed to help people home in on published data. This information might also be in the public view. Moreover, government agencies are very encouraged to use OSINT techniques for finding out holes in their cybersecurity defenses.
Following the trail opened by the OSINT queries might get you into legal grey areas. Media Sonar has good advice on how to stay on the law’s right side. For instance, it’s not that illegal accessing the dark web’s public areas, and it is essential to do so if you are trying to determine if your companies’ data has been breached or stolen. Moreover, you must not attempt to buy collections of stolen data as part of your research. You must also not impersonate or ask a law enforcement officer to take information out of shady characters.
In general, it’s essential for developing a code of conduct before guiding your employees’ behavior on these researches, and also to document everything you do to demonstrate that you’re sticking to those guidelines and haven’t broken any regulations or laws.
Final Thoughts!
Hence, these were some of the best OSINT tools. You can use them to get information about anything on the internet. Moreover, you can also use it to find loopholes in your organization and system. Therefore, get a hand in any one of the free OSINT tools, and start exploring.
