PacketPig : Big Data Security Analytics Platform
Labels:
Forensics,
Reverse Engineering,
Security Tools
PacketPig is an open source “big data” platform for analyzing full packet captures. "The idea of full packet capture has been likened to network TiVo, and largely discounted," said Michael Baker, CTO at Packetloop.
This open source tool was presented at this years BlackHat EU held in March. PacketPig is called so, because it has been created by a company called PacketLoop and Pig is the primary language used. It can be considered as a one stop shop for network security monitoring.
Baker said one challenge with analyzing large sets of data traditionally has been that as pcaps age, tools tend to aggregate the data, meaning that a minute-by-minute look at packets becomes averaged into an hour, and hours later rolled up into days or even months. "The key point is to not lose fidelity," he said.
Packetpig is built on Pig, which is a platform--programmed in a language called Pig Latin--for creating MapReduce jobs, a concept Google outlined in a 2004 research paper. The jobs spread problems involving large amounts of data across multiple nodes. In particular, Packetpig is a series of data-analysis jobs that run on Hadoop, an open-source implementation of MapReduce, to handle the replication of data across multiple nodes. These nodes could be anything from spare servers or compute time scrounged by the information security group to Amazon Simple Storage Service (Amazon S3).
List of PacketPig Pig Loaders:
- PacketLoader() – opens packet captures and provides access to TCP, UDP and IP headers e.g. Source IP Address, Source Port, Destination IP Address, Destination Port.
- SnortLoader() – wraps the Snort Intrusion Detection application allowing packet captures to be analysed across a Hadoop Cluster. The loader analyses packets and returns signature, priority, message, protocol and Source IP/Port, Destination IP/Port. It also produces a record for each alert triggered.
- ConversationLoader() – links packets to their conversations or flows. The conversation start and end, the way the conversation ended, the number of packets, their size and delay can all be extracted through this loader.
- DNSConversationLoader() – provides additional functionality for the deep packet inspection of DNS conversations.
- HTTPConversationLoader() – provides additional functionality for the deep packet inspection of HTTP conversations.
- ConversationFileLoader() – allows file metadata and files themselves to be extracted from conversations. The file name, extension, libmagic information as well as MD5, SHA1 and SHA256 hashes are returned through this loader. In addition the actual files themselves can be extracted and dumped.
- FingerprintLoader() – a wrapper for p0f that allows it to operate across a Hadoop Cluster, giving you an operating system for each packet in a pcap.
- PacketNgramLoader() – extracts data from each packet in a conversation and breaks it into an N-Gram. Unigram, Bigram and Trigrams are most commonly used however any integer can be passed to the loader.
Subscribe to our Daily Newsletter via email - Be First to know about Security and Penetration testing tools. or Join our Huge Hackers Community on Facebook , Google+ and Twitter .
Subscribe Top Hacker Stories via email
The Hacker News - Daily Updates