TCPDUMP from NIC to Network Layer

NIC - Network Interface Card

• NIC or network interface card or network adapter, LAN adapter is an important hardware requirement for connectivity . Actually this interface card provides us the 'gateway' through which data from one computer A can be transferred to another computer B .
• A network interface card (NIC) is a circuit board or card that is installed in a computer so that it can be connected to a network. A network interface card provides the computer with a dedicated, full-time connection to a network.
• NIC is OSI layer 2 device (Data Link Layer) and tcpdump should be obtained data packet from the network card driver / network interface card (NIC).

TCPDUMP

• Some times during troubleshooting this tool proves to be very helpful. With the help of this tool you can analyze the packet before it reaches the application stack. And sometimes detect why the server is not responding to a ping request, why an application is not responding to a certain machine etc etc.
• Its mainly used to debug the protocol of the captured packet which in turn reveals the network traffic characteristics.
• tcpdump is a most powerful and widely used command-line packets sniffer or package analyzer tool which is used to capture or filter TCP/IP packets that received or transferred over a network on a specific interface.
• Tcpdump is a very powerful  because of its strength in capturing packets based on different parameters given.  It operates on network layer, so will be able to capture all the packets in and out of the machine. You can use tcpdump to capture and save the packets to a file to analyze it later.
• TCPDUMP uses Libpcap(a c/c++ library that's used for packet capturing.) and It is available under most of the Linux/Unix based operating systems.
• There are other tools out there which does the same job of packet capture/analyzing like wireshark, but tcpdump keeps all the captures raw. Which means its shows us the raw data it captures as it is.
• A network packet header consists of sender,destination,state information and other flag information's.
• TCPDUMP only captures the first 96 bytes of data from the packet by default.
• tcpdump also gives us a option to save captured packets in a file for future analysis. It saves the file in a pcap format, that can be viewed by tcpdump command or  a open source GUI based tool called Wireshark (Network Protocol Analyzier) that reads tcpdump pcap format files.

Objective

• How to monitor/track HTTP and HTTPS calls i,e traffic inflow and outflow of Apache httpd server on any specific port like port 80 or 443
• How to intercept the HTTP traffic initiated from the browser (or) SOAP UI to application server and sneak a peak into the content like Request Body like XML,JSON and Username and Password etc.
• How to tamper and read the incoming and outgoing HTTP traffic to our applications
• How to dig into the incoming (or) outgoing HTTP traffic and take a look at the concrete elements of HTTP protocol such as headers, cookies, request body as they gets transmitted.

 

Open Systems Interconnection (OSI) model  

• The Open Systems Interconnection (OSI) model defines a networking framework to implement protocols in layers, with control passed from one layer to the next.
• It conceptually divides computer network architecture into 7 layers in a logical progression. The lower layers deal with electrical signals, chunks of binary data, and routing of these data across networks. Higher levels cover network requests and responses, representation of data, and network protocols .
• The data communications between different networks are not possible if there’s no common rules for transmitting and receiving the packets of data. These rules are known as protocols.
• Transmission Control Protocol (TCP)/Internet Protocol (IP) is one of the most widely used protocols.
• OSI model acts as a reference model and is not implemented in Internet because of its late invention.
• Both models are based on the concept of layering.

Network errors and analysis tools assigned to OSI layers

Layering: 

Divide a task into sub-tasks and then solve each sub-task independently and establishing a well defined interface between layers makes porting easier.

Layering Example: Federal Express

• Letter in envelope, address on outside
• FedX guy adds addressing information,barcode.
• Local office drives to airport and delivers to hub.
• Sent via airplane to nearest city.
• Delivered to right office
• Delivered to right person

There are many mnemonics way can be used to remember the order of the OSI layer:

• All People Seem To Need Data Processing.
• All People Should Try New Diet Pepsi.
• All People Share Their Nasty Delivered Pizza.

The TCP/IP model and OSI model are both conceptual models used for description of all network communications, while TCP/IP itself is also an important protocol used in all Internet operations. Generally, when we talk about layer 2, layer 3 or layer 7 in which a network device works, we are referring to the OSI model. The TCP/IP model is used for both modelling current Internet architecture and providing a set of rules that are followed by all forms of transmission over the network. 

* Protocol data unit (PDU) being processed in different layers.

7. Application Layer – PDU (Data)

It functions as the user interface platform (including software within the system) . The application layer enables the user, whether human or software, to access the network.

User interface Examples – DNS, IMAP, POP, SMTP, SNMP, Telnet, HTTP, FTP , Browsers, Skype Messenger etc

6. Presentation Layer / Translation layer – PDU (Data) 
The presentation layer checks the data to ensure it is compatible with the communications resources. It translates the data into the form that the application level and lower levels accept i,e It provides a translation of data to the next layer (prepare the data to the Application Layer).

Translation: different encoding methods like ASCII to EBCDIC.

Encryption/ Decryption: It translates the data into another form or code. The encrypted data is known as the cipher text and the decrypted data is known as plain text . (For the data that has high security requirements, such as a text message containing your password, it will be encrypted at this layer.)

Compression: It reduces the number of bits that need to be transmitted on the network. (Video calls will be compressed during the transmission so that it can be transmitted faster, and the data will be recovered at the receiving side).

5. Session Layer - PDU (Data)

This layer identifies the established system session between different network entities. It controls dialogues (connections) between computers (3 ways handshaking). It establishes, manages, maintains and ultimately terminates the connections between the local and remote application. It handles authentication and authorization functions. It verifies the data is delivered as well. 

Dialog control: It allows two systems to start communication with each other in half-duplex or full-duplex. (In a full-duplex system, both parties can communicate with each other simultaneously. In a half-duplex system, both parties can communicate with each other, but not simultaneously; the communication is one direction at a time.)

Synchronization: The session layer allows a process to add checkpoints (synchronization points) into a stream of data. For example, if a system is sending a file of 2,000 pages, it is advisable to insert checkpoints after every 100 pages to ensure that each 100-page unit is received  and acknowledged independently. In this case, if a crash happens during the transmission of page 523, the only pages that need to be resent after system recovery are pages 501 to 523. Pages previous to 501 need not be resent.

Note:

• All the above 3 layers (including Session Layer) are integrated as a single layer in TCP/IP model as “Application Layer”.
• Implementation of these 3 layers is done by the network application itself. These are also known as Upper Layers or Software Layers.  

4. Transport Layer - PDU (Segment (TCP) / Datagram (UDP))

This layer ensures the transport/sending of data is successful. This function can include error checking operation; along with the ability to keep data message in sequence means transferring  of data sequences from a source to a destination host via one or more networks, ensure the complete End to End delivery of the data. The integrity of the data can be guaranteed via error correction and similar functions.  It also provides the acknowledgment of the successful data transmission and re-transmits the data if an error is found. It also adds Source and Destination port number in its header and forwards the segmented data to the Network Layer.

Service-point addressing: In order to deliver the message to correct process, transport layer header includes a type of address called service point address or port address. Thus by specifying this address, transport layer makes sure that the message is delivered to the correct process.

Segmentation and reassembly:  A message is divided into transmittable segments, with each segment containing a sequence number. These numbers enable the transport layer to reassemble the message correctly upon arriving at the destination and to identify and replace packets that were lost in transmission.

Connection control: The transport layer can be either connectionless or connection-oriented.
(Connection-oriented: the receiving device sends an acknowledgment, back to the source after a packet or group of packet is received, Connection less service: the receiver does not acknowledge receipt of a packet. This approach allows for much faster communication between devices.  Connection oriented Service is more reliable than connection less Service. TCP (Transmission Control Protocol) is connection oriented, whereas UDP (User Datagram Protocol) is connection-less.  This means that TCP tracks all data sent, requiring acknowledgment for each octet (generally).)

Flow control: Flow control at this layer is performed end to end rather than across a single link

Error control: Error control at this layer is performed process-to-process rather than across a single link

Note:

• Data in the Transport Layer is called as Segments.
• Transport layer is operated by the Operating System. It is a part of the OS and communicates with the Application Layer by making system calls. Transport Layer is called as Heart of OSI model.

3. Network Layer - PDU (Packet)
The network layer handles packet routing via logical addressing and switching functions. A network is a medium to which many nodes can be connected. Every node has an address. When a node needs to transfer message to other nodes, it can merely provide the content of the massage and the address of the destination node, then the network will find the way to deliver the message to the destination node, possibly routing through other nodes. If the message is too long, the network may split it into several segments at one node, sending them separately and reassembling the fragments at another node.

Routing: The network layer protocols determine which route is suitable from source to destination. This function of network layer is known as routing. Example: IP, IPX

Logical Addressing:  In order to identify each device on internetwork uniquely, network layer defines an addressing scheme. The sender & receiver’s IP address are placed in the header by network layer

Note:

• This layer provides connections between hosts on different networks.
• Segment in Network layer is referred as Packet.
• Network layer is implemented by networking devices such as routers.

2. Data Link Layer - PDU (Frame)

The data link layer is responsible for the node to node delivery of the message.  The main function of this layer is to make sure data transfer is error free from one node to another, over the physical layer. When a packet arrives in a network, it is the responsibility of DLL to transmit it to the Host using its MAC address.  

Data Link Layer is divided into two sub layers:

• Logical Link Control (LLC)
• Media Access Control (MAC)

The packet received from Network layer is further divided into frames depending on the frame size of NIC(Network Interface Card). DLL also encapsulates Sender and Receiver’s MAC address in the header.
Examples:  HDLC, 802.3/802.2 for Ethernet” and “802.11 for Wi-Fi”

Framing: Combines bits into bytes and bytes into frames i,e It divides the stream of bits received from the network layer into manageable data units called frames

Physical addressing: After creating frames, Data link layer adds physical addresses (MAC address) of sender and/or receiver in the header of each frame.

Error control: Data link layer provides the mechanism of error control in which it detects and retransmits damaged or lost frames.

Flow Control: The data rate must be constant on both sides else the data may get corrupted thus , flow control coordinates that amount of data that can be sent before receiving acknowledgement.

Access control: When two or more devices are connected to the same link, data link layer protocols are necessary to determine which device has control over the link at any given time.

Note:

• This layer provides connections between hosts on the same networks.
• Packet in Data Link layer is referred as Frame.
• Data Link layer is handled by the NIC (Network Interface Card) and device drivers of host machines.
• Switch & Bridge are Data Link Layer devices.

1. Physical Layer - PDU (Bit)

This layer describes the electrical and physical specifications for devices such as, cables, connectors, hubs, repeaters and more. It is responsible for the actual physical connection between the devices. The physical layer contains information in the form of bits.  When receiving data, this  will convert it into 0s and 1s and send them to the Data Link layer, which will put the frame back together. 

Bit synchronization: the sender and the receiver clocks must be synchronized.

Bit rate control:  It defines the transmission rate i.e. the number of bits sent per second.

Physical topologies:  It specifies the way in which the different, devices/nodes are arranged in a network i.e. bus, star or mesh topolgy.

Transmission mode: It defines the way in which the data flows between the two connected devices. The various transmission modes possible are: Simplex, half-duplex and full-duplex.

Note:

• Hub, Repeater, Modem, Cables are Physical Layer devices.
• Network Layer, Data Link Layer and Physical Layer are also known as Lower Layers or Hardware Layers.

TCP/IP Model vs. OSI Model

One of the major difference is that OSI is a conceptual model which is not practically used for communication, whereas, TCP/IP is used for establishing a connection and communicating through the network.

Similarities between OSI Reference Model and TCP/IP Reference Model :

• Both have layered architecture.
• Layers provide similar functionalities.
• Both are protocol stack.
• Both are reference models.

Difference between OSI Reference Model and TCP/IP Reference Model:

• TCP/IP is a client-server model, i.e. when the client requests for service it is provided by the server. Whereas, OSI is a conceptual model.
• TCP/IP is a standard protocol used for every network including the Internet, whereas, OSI is not a protocol but a reference model used for understanding and designing the system architecture.
• TCP/IP is a four layered model, whereas, OSI has seven layers.
• TCP/IP follows Vertical approach. On the other hand, OSI Model supports Horizontal approach.
• TCP/IP is Tangible, whereas, OSI is not.
• TCP/IP follows top to bottom approach, whereas, OSI Model follows a bottom-up approach.

The TCP/IP network model represents reality in the world, whereas the OSI mode represents an ideal.

Port Address :

• Domain Name System (DNS) - TCP/UDP Port 53
• Hypertext Transfer Protocol (HTTP) - TCP Port 80
• Simple Mail Transfer Protocol (SMTP) - TCP Port 25
• Post Office Protocol (POP) - UDP Port 110 Telnet - TCP Port 23
• Dynamic Host Configuration Protocol - UDP Port 67
• File Transfer Protocol (FTP) - TCP Ports 20 and 21

* Layer 0 is the photonics layer, more specifically, DWDM is considered as the Layer 0 of the OSI layers.

DWDM – Dense Wavelength Division Multiplexing is an Optical Transmission technology.

• DWDM allows you to transmit multiple wavelengths on a single fiber, as the name suggests it utilizes wave division multiplexing.
• DWDM can be deployed in ultra-long haul, long-haul, regional, and metro area’s.
• DWDM also uses amplification to reach long distances and can be an expensive technology to deploy.
• On the other hand CWDM which is a metro technology multiplexing a maximum of 8 wavelengths per fiber and does not require amplification i.e. much cheaper than DWDM
• WDM is used to transport SDH/Ethernet/IP between regions or cities or to aggregate traffic where large bandwidth is required

OSI Data Encapsulation and Decapsulation

In OSI Model ,Data flows in two ways i,e one is DOWN(data encapsulation) and reverse way is UP(data decapsulation)

Data Encapsulation

The process of converting Data into Bits and Bytes , is called Encapsulation.

• The computer in the above picture needs to send some data to another computer. The Application layer is where the user interface exists, here the user interacts with the application he or she is using, then this data is passed to the Presentation layer and then to the Session layer. These three layer add some extra information to the original data that came from the user and then passes it to the Transport layer. Here the data is broken into smaller pieces (one piece at a time transmitted) and the TCP header is a added. At this point, the data at the Transport layer is called a segment.
• Each segment is sequenced so the data stream can be put back together on the receiving side exactly as transmitted. Each segment is then handed to the Network layer for network addressing (logical addressing) and routing through the internet network. At the Network layer, we call the data (which includes at this point the transport header and the upper layer information) a packet.
• The Network layer add its IP header and then sends it off to the Datalink layer. Here we call the data (which includes the Network layer header, Transport layer header and upper layer information) a frame. The Datalink layer is responsible for taking packets from the Network layer and placing them on the network medium (cable). The Datalink layer encapsulates each packet in a frame which contains the hardware address (MAC) of the source and destination computer (host) and the LLC information which identifies to which protocol in the prevoius layer (Network layer) the packet should be passed when it arrives to its destination. Also, at the end, you will notice the FCS field which is the Frame Check Sequence. This is used for error checking and is also added at the end by the Datalink layer.
• If the destination computer is on a remote network, then the frame is sent to the router or gateway to be routed to the desination. To put this frame on the network, it must be put into a digital signal. Since a frame is really a logical group of 1's and 0's, the Physical layer is responsible for encapsulating these digits into a digital signal which is read by devices on the same local network.
• There are also a few 1's and 0's put at the begining of the frame, only so the receiving end can synchronize with the digital signal it will be receiving.

Data Decapsulation

The process of converting Bits and Bytes into Data , is called Decapsulation.

• The receiving computer will firstly synchronize with the digital signal by reading the few extra 1's and 0's as mentioned above. Once the synchonization is complete and it receives the whole frame and passes it to the layer above it which is the Datalink layer.
• The Datalink layer will do a Cyclic Redundancy Check (CRC) on the frame. This is a computation which the comupter does and if the result it gets matches the value in the FCS field, then it assumes that the frame has been received without any errors. Once that's out of the way, the Datalink layer will strip off any information or header which was put on by the remote system's Datalink layer and pass the rest (now we are moving from the Datalink layer to the Network layer, so we call the data a packet) to the above layer which is the Network layer.
• At the Network layer the IP address is checked and if it matches (with the machine's own IP address) then the Network layer header, or IP header if you like, is stripped off from the packet and the rest is passed to the above layer which is the Transport layer. Here the rest of the data is now called a segment.
• The segment is processed at the Transport layer, which rebuilds the data stream (at this level on the sender's computer it was actually split into pieces so they can be transferred) and acknowledges to the transmitting computer that it received each piece. It is obvious that since we are sending an ACK back to the sender from this layer that we are using TCP and not UDP. It then happily hands the data stream to the upper-layer application.
• You will find that when analysing the way data travels from one computer to another most people never analyse in detail any layers above the Transport layer. This is because the whole process of getting data from one computer to another involves usually layers 1 to 4 (Physical to Transport) or layer 5 (Session) at the most, depending on the type of data.

List Network Interfaces Using ip Command on Linux :

[root@myvm ~]#  ip link show

1: lo:  mtu 16436 qdisc noqueue state UNKNOWN

    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00

2: eth0:  mtu 1500 qdisc mq state UP qlen 1000

    link/ether b8:ac:6f:65:31:e5 brd ff:ff:ff:ff:ff:ff

3: wlan0:  mtu 1500 qdisc mq state DOWN qlen 1000

    link/ether 00:21:6a:ca:9b:10 brd ff:ff:ff:ff:ff:ff

4: vboxnet0:  mtu 1500 qdisc noop state DOWN qlen 1000

    link/ether 0a:00:27:00:00:00 brd ff:ff:ff:ff:ff:ff

5: pan0:  mtu 1500 qdisc noop state DOWN

    link/ether c2:10:fa:55:8e:32 brd ff:ff:ff:ff:ff:ff

6: vmnet1:  mtu 1500 qdisc pfifo_fast state UNKNOWN qlen 1000

    link/ether 00:50:56:c0:00:01 brd ff:ff:ff:ff:ff:ff

7: vmnet8:  mtu 1500 qdisc pfifo_fast state UNKNOWN qlen 1000

    link/ether 00:50:56:c0:00:08 brd ff:ff:ff:ff:ff:ff

11: ppp0:  mtu 1496 qdisc pfifo_fast state UNKNOWN qlen 3

    link/ppp

where

• lo – Loopback interface.
• eth0 – My first Ethernet network interface on Linux.
• wlan0 – Wireless network interface in Linux.
• ppp0 – Point to Point Protocol network interface which can be used by dial up modem, PPTP vpn connection, or 3G wireless USB modem.
• vboxnet0, vmnet1, vmnet8 – Virtual machine interface working in bridge mode or NAT mode on Linux.

Example 1: Monitor http connection (3 way handshaking (SYN, SYN-ACK, ACK))

Three-way-handshake process to startup a connection:

1. The requesting Host sends a synchronization flag (SYN) in a TCP segment to create a connection.
2. The receiving Host 172.217.166.179 receives the SYN flag and returns an acknowledgment flag (ACK).
3. The requesting Host 10.190.10.70 receives the SYN flag and returns it's own ACK flag.

To terminate a connection, a segment containing a FIN flag is sent from host 10.190.10.70 back to the host with the open session.  Refer below screenshot how a connection has established and terminated.

TCP flag bits used for 3 way handshaking : 

• S: SYN (Synchronize sequence numbers - Connection establishment)
• F: FIN (Ending of sending by sender - Connection termination)
• R: RST (Reset connection)
• P: PSH (Push data)
• .: (No flag is set)

For more details on how a client and server share the tcp flag, sequences and acknowledgement , please go through http://www.java2depth.com/2018/06/clientserver-socket-interaction-data.html

Example 2:  How to capture all HTTP traffic including request and response headers and message body for a port:

To Get the interface name of your IP which you need to specify it in the tcpdump command.  you can execut the command ifconfig (or) ip a

[root@myvm ~]# ip a

System will tell tcpdump to capture packets that's coming towards a particular interface.

tcpdump is only listening on loopback interface for packets :

[root@myvm ~]#  tcpdump -i lo

tcpdump: verbose output suppressed, use -v or -vv for full protocol decode

listening on lo, link-type EN10MB (Ethernet), capture size 96 bytes

tcpdump is only listening on eth0 interface for packets :

[root@myvm ~]#  tcpdump -i eth0

tcpdump: verbose output suppressed, use -v or -vv for full protocol decode

listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes

Display packets for a particular port

[root@myvm ~]#  tcpdump -i eth0 port 22

show packets towards a particular host

[root@myvm ~]# tcpdump -i eth0 host 10.190.10.70

Show packets from source with tcpdump

[root@myvm ~]# tcpdump -i eth0 src host 10.190.10.70

Show packets for destination with tcpdump

[root@myvm ~]# tcpdump -i eth0 dst host 10.190.10.70

Show plaintext credentials sniffinf with tcpdump and grep for common unsafe protocols

[root@myvm ~]#  tcpdump port http or port ftp or port smtp or port imap or port pop3 -l -A | egrep -i ‘pass=|pwd=|log=|login=|user=|username=|pw=|passw=|passwd=|password=|pass:|user:|username:|password:|login:|pass |user ‘ –color=auto –line-buffered -B20

See what traffic is mostly hitting you

[root@myvm ~]# tcpdump -i eth0 -n  | head

Capture TCP packets from local host to local host

[root@myvm ~]# tcpdump -i lo

How to capture All incoming  HTTP GET  traffic (or) requests

[root@myvm ~]# tcpdump -s 0 -i eth3 -A -vv 'tcp[((tcp[12:1] & 0xf0 ) >> 2 ):4] = 0x47455420'

How to capture All incoming HTTP POST requests

[root@myvm ~]#  tcpdump -s 0 -i eth3 -A -vv 'tcp[((tcp[12:1] & 0xf0 ) >> 2 ):4] = 0x504f5354'

How to capture all HTTP traffic including request and response headers and message body for a port:

[root@myvm ~]#  tcpdump -A -s 0 'tcp port 8080 and (((ip[2:2] - ((ip[0]&0xf)<<2)) - ((tcp[12]&0xf0)>>2)) != 0)'

How to capture all HTTP traffic including request and response headers and message body for a port and source IP:

[root@myvm ~]#  tcpdump -A -s 0 'src 10.190.10.70 and tcp port 11222 and (((ip[2:2] - ((ip[0]&0xf)<<2)) - ((tcp[12]&0xf0)>>2)) != 0)'

How to capture all HTTP traffic including request and response headers and message body for local host to local host

[root@myvm ~]#  tcpdump -A -s 0 'tcp port 8080 and (((ip[2:2] - ((ip[0]&0xf)<<2)) - ((tcp[12]&0xf0)>>2)) != 0)' -i lo

How to capture all HTTP traffic including request and message body only  for a port:

[root@myvm ~]#   tcpdump -A -s 0 'tcp dst port 8080 and (((ip[2:2] - ((ip[0]&0xf)<<2)) - ((tcp[12]&0xf0)>>2)) != 0)'

How to capture all HTTP traffic including request and message body for a port and source IP:

[root@myvm ~]#   tcpdump -A -s 0 'src 10.190.10.70 and tcp dst port 11222 and (((ip[2:2] - ((ip[0]&0xf)<<2)) - ((tcp[12]&0xf0)>>2)) != 0)'

How to capture all HTTP traffic including request and message body for a local host to local host:

[root@myvm ~]#   tcpdump -A -s 0 'tcp dst port 8080 and (((ip[2:2] - ((ip[0]&0xf)<<2)) - ((tcp[12]&0xf0)>>2)) != 0)' -i lo

See what traffic is mostly hitting you

[root@myvm ~]# tcpdump -i eth0 -n  | head


[root@myvm ~]# tcpdump -A -s 0 'src 192.100.101.146 and (((ip[2:2] - ((ip[0]&0xf)<<2)) - ((tcp[12]&0xf0)>>2)) != 0)'
[root@myvm ~]# tcpdump -A -s 0 'dst vwuatltapp01 and (((ip[2:2] - ((ip[0]&0xf)<<2)) - ((tcp[12]&0xf0)>>2)) != 0)'
[root@myvm ~]# tcpdump -A -s 0 '(src 192.100.101.146 and dst vwuatltapp01 )  and (((ip[2:2] - ((ip[0]&0xf)<<2)) - ((tcp[12]&0xf0)>>2)) != 0)'
[root@myvm ~]# tcpdump -A -s 0 '(src 192.100.101.146 and dst vwuatltapp01 ) or (src vwuatltapp01 and dst 192.100.101.146 ) and (((ip[2:2] - ((ip[0]&0xf)<<2)) - ((tcp[12]&0xf0)>>2)) != 0)'
[root@myvm ~]# tcpdump -A -s 0 '(src 192.100.101.146 or dst 192.100.101.146) and (((ip[2:2] - ((ip[0]&0xf)<<2)) - ((tcp[12]&0xf0)>>2)) != 0)'

[root@myvm ~]# tcpdump -A -s 0 'tcp port 6802 and (((ip[2:2] - ((ip[0]&0xf)<<2)) - ((tcp[12]&0xf0)>>2)) != 0)'

Read HTTP request from tcpdump pcap file

[root@myvm ~]# tcpdump -r tcpdump.pcap -A -s 0 'host 192.168.171.91 and port 60689 and (((ip[2:2] - ((ip[0]&0xf)<<2)) - ((tcp[12]&0xf0)>>2)) != 0)'

TCPDUMP Quick Reference :-

WinDump: The tcpdump tool for Windows

WinDump, the Windows version of tcpdump, can help you analyze network traffic to look for signs of active malware. 

WinDump comes in two parts. 

• The first is a set of network capture drivers called WinPcap, which WinDump uses to obtain packet-level access to network interfaces in the computer. 
• The second part is the program itself, windump, which is invoked from the command line after you've installed the WinPcap library.

The first option you'll want to use when you run windump is -D, which lists all available network interfaces in the current system. By default, the program listens on the first available interface, but in Windows, it is typically the software dial-up adapter, not a physical network adapter. The results from -D usually look something like this:

• \Device\NPF_GenericDialupAdapter (Generic dialup adapter)
• \Device\NPF_{707E0236-BEE4-4097-93B1-56DEC35564AA} (Intel DC21140 PCI Fast Ethernet Adapter (Microsoft's Packet Scheduler) )

To use a specific adapter, run the program with the -i switch in conjunction with an adapter number. For instance, if you wanted to use the Ethernet adapter listed above, use windump -i 2. This is a lot easier than referring to the adapter by its GUID, but bear in mind the adapter number may not remain consistent if new hardware or software adapters are added.

Windump has the ability to filter captured input by specific criteria -- specific protocols, hosts or ports -- usually specified on the command line. The syntax for this is fairly complicated.

Captures all traffic from interface #2 via port 80.
windump -i 2 port 80

Captures all traffic from interface #2 to or from the host im-chat.com.
windump -i 2 host im-chat.com

Captures all traffic from interface #1 to or from the subnet 127.x.x.x.
windump -i 1 net 127

For more details go through the link https://www.winpcap.org/windump/docs/manual.htm

Micro Benchmarking with JMH - "Measure, don’t guess"

Java Micro Benchmark with JMH

• Java Microbenchmark Harness (JMH) is a Java toolkit by OpenJDK for creating benchmarks.  
• It is an open source framework provides benchmarking for measuring the performance of your Java code.

"Measure, don’t guess" :-

We’ve all faced performance problems in our projects and were asked to tune random bits in our source code and  hoping that performance will get improved. Instead, we should set up a stable performance environment  (operating system, JVM, application server, database), measure continuously, set some performance goals then, take action when our goals are not achieved. Continuous delivery, continuous testing is one thing,  but continuous measuring is another step.

What is JMH ?

• JMH is a Java harness for building, running, and analyzing nano/micro/milli/macro benchmarks written in Java and other languages targeting the JVM.

When to use Micro Benchmarking ?

• You identified the code segment that eats most of the resources in your application and the improvement can be tested by micro benchmarks.
• You can not identify the code segment that will eat most of the resources in an application but you suspect it.

How to do a Micro Benchmark ?

• We often do not worry about the performance requirements. We start with building functionality and concentrate on making things work and don’t focus on how. Once the software goes to production, we will be facing the inevitable. 
• It is good to have the performance objectives written down before writing the code.
  We should weigh the importance of performance with respect to the functional requirements and should come up with a balance between them.
• Just because your code runs in a certain way in an extremely isolated artificial situation does not mean it will run in the  same way inside your production code. To name but a few issues, in a real program the CPU caches will be subject to pressures from other parts of your code, any object creation will have a downstream effect on GC and the JIT may have inlined and compiled code from other parts of your code that conflict with the code you have benchmarked. 

Why are Java Microbenchmarks Hard ?

• Writing benchmarks that correctly measure the performance of a small part of a larger application is hard. There are many optimizations that the JVM or underlying hardware may apply to your component when the benchmark executes that component in isolation. These optimizations may not be possible to apply when the component is running as part of a larger application. Badly implemented microbenchmarks may thus make you believe that your component's performance is better than it will be in reality.
• Writing a correct Java microbenchmark typically entails preventung the optimizations the JVM and hardware may apply during microbenchmark execution which could not have been applied in a real production system. That is what JMH - the Java Microbenchmark Harness - is helping you do.  

Purpose

• Benchmark is the process of recording the performance of a system.
• JMH is a Java harness for building, running, and analyzing nano/micro/milli/macro benchmarks written in Java and other languages targeting the JVM.
• it takes care of warm up iterations, forking JVM processes so that benchmarks don't interfere with each other, collating results and presenting then in a uniform manner.

Micro benchmarks are generally done for two reasons.

• To compare different approaches of code which implements the same logic and choose the best one to use.
• To identify any bottlenecks in a suspected area of code during performance optimization.

Benchmarking Modes:

At a basic level, JMH has two main types of measure:  throughput and time-based.

Throughput Measuring

• Throughput is the amount of operations that can be completed per the unit of time. JMH maintains a collection of successful and failed operations as the framework increases the amount of load on the test.  
• Ensure the method or test is well isolated and dependencies like test object creation is done outside of the method or pre-test in a setup method.  
• With Throughput, the higher the value, the better as it indicates that more operations can be run per unit-time.

Time-Based Measuring

• Time-based measuring is the counter-partner to throughput. 
• The goal of time-based measuring is to identify how long a particular operation takes to run per unit-time.

JMH Commands

• i - Number of measurement iterations to do. Measurement iterations are counted towards the benchmark score.
• bs - Batch size: number of benchmark method calls per operation.
• r - Minimum time to spend at each measurement iteration.
• wi - Number of warmup iterations to do.
• wbs - Warmup batch size: number of benchmark method calls per operation.
• w - Minimum time to spend at each warmup iteration.
• to - Timeout for benchmark iteration.
• t - Number of worker threads to run with.
• bm - Benchmark mode.
• si - Should JMH synchronize iterations?
• gc - Should JMH force GC between iterations?
• foe - Should JMH fail immediately if any benchmark had experienced an unrecoverable error?
• v - Verbosity mode.
• f - How many times to fork a single benchmark. Use 0 to disable forking altogether.
• wf - How many warmup forks to make for a single benchmark. 

 There are two way to run a benchmark:

1. The recommended way is to generate a pom file and use that to create a jar. The mvn install uses the shade plugin to create a jar file so that you don't have create a main method.
2. Add the JMH maven dependencies to your pom file and then add a main method to your code using the Runner object.This is useful if you want to run in your IDE. 

 Method 1 - Using command line argument

 Generate a pom file using this mvn command.

mvn archetype:generate
          -DinteractiveMode=false
          -DarchetypeGroupId=org.openjdk.jmh
          -DarchetypeArtifactId=jmh-java-benchmark-archetype
          -DgroupId=com.jenkov
          -DartifactId=first-benchmark
          -Dversion=1.0

• This will create a project called test with an empty benchmark in it called MyBenchmark.
• To build the project just use mvn clean install. This will build a jar called benchmark.jar. 
• It is the benchmark.jar that should be run to run the benchmark not any other jars produced along the way that will be in your target folder.

To run use the command  java -jar target/benchmarks.jar JMHSample04 -wi 5 -t 1 -i 5 -f 1

 Method 2 - For running in your IDE

Add these dependencies to your Maven pom.xml file:

  org.openjdk.jmh
  jmh-core
  1.5.1


  org.openjdk.jmh
  jmh-generator-annprocess
  1.5.1

Then decide which methods you want benchmarked and add the annotation @Benchmark to them. If you need any initialisation code add it in a method which should be marked @Setup. 

The easiest way to run the benchmark is by adding by adding this implementation into your main method.

public static void main(String[] args) throws RunnerException {
        Options opt = new OptionsBuilder()
                .include(JMHSample04.class.getSimpleName()).
                warmupIterations(5).
                measurementIterations(5).
                threads(1).
                forks(1).
                build();
        new Runner(opt).run();
}

JMH benchmark Results :

As an example to see the format of a JMH benchmark, this is what my results looked like:

Example 1 : Arrays sort and Collection sort Benchmark Comparison

package org.sample;

import java.util.ArrayList;
import java.util.Arrays;
import java.util.Collections;
import java.util.List;
import java.util.Random;
import java.util.concurrent.TimeUnit;
import org.openjdk.jmh.annotations.Benchmark;
import org.openjdk.jmh.annotations.BenchmarkMode;
import org.openjdk.jmh.annotations.Level;
import org.openjdk.jmh.annotations.Mode;
import org.openjdk.jmh.annotations.OutputTimeUnit;
import org.openjdk.jmh.annotations.Scope;
import org.openjdk.jmh.annotations.Setup;
import org.openjdk.jmh.annotations.State;

@State(Scope.Thread)
public class JMHSample02 {
 List arrayList;
 int[] array;
 Random random;

 @Setup(Level.Trial)
 public void init() {
  random = new Random();
  array = new int[150];
  arrayList = new ArrayList();
  for (int i = 0; i < 150; i++) {
   int randomNumber = random.nextInt();
   array[i] = randomNumber;
   arrayList.add(new Integer(randomNumber));
  }
 }

 @Benchmark
 @BenchmarkMode(Mode.Throughput)
 @OutputTimeUnit(TimeUnit.SECONDS)
 public void arraysSort() {
  Arrays.sort(array);

 }

 @Benchmark
 @BenchmarkMode(Mode.Throughput)
 @OutputTimeUnit(TimeUnit.SECONDS)
 public void collectionsSort() {
  Collections.sort(arrayList);
 }
}

JMH benchmark Results :

Example 2 : Java 8 Stream API (Parallel Stream vs Sequential Stream vs  for-each vs iterator )

Java 8 claimed that Stream API would employ multi-core CPUs  to process data in parallel fashion which would eliminate difficulties  of  dealing with multi-thread  codes , as well as gaining performance advantages obtained from  multicored CPUs. 
 
Stream parallel one is the slowest one among four approaches of summing  integers in array.

package org.sample;

import java.util.ArrayList;
import java.util.Iterator;
import java.util.List;
import java.util.concurrent.TimeUnit;
import org.openjdk.jmh.annotations.Benchmark;
import org.openjdk.jmh.annotations.BenchmarkMode;
import org.openjdk.jmh.annotations.Mode;
import org.openjdk.jmh.annotations.OutputTimeUnit;
import org.openjdk.jmh.annotations.Scope;
import org.openjdk.jmh.annotations.Setup;
import org.openjdk.jmh.annotations.State;

@State(Scope.Benchmark)
public class JMHSample_01_HelloWorld {

 volatile int counts = 9999999;
 volatile List values = new ArrayList<>(counts);
 volatile int processors = Runtime.getRuntime().availableProcessors();

 @Setup
 public void setup() {
  populate(values);

 }

 public void populate(List list) {
  for (int i = 0; i < counts; i++) {
   if (i < counts / 2) {
    list.add(i, i);
   } else {
    list.add(i, i - counts);
   }
  }
 }

 @Benchmark
 @BenchmarkMode(Mode.Throughput)
 @OutputTimeUnit(TimeUnit.MICROSECONDS)
 public int iteratorSumIntegers() {
  int result = 0;
  Iterator ite = values.iterator();
  while (ite.hasNext()) {
   result += (int) ite.next();
  }
  return result;
 }

 @Benchmark
 @BenchmarkMode(Mode.Throughput)
 @OutputTimeUnit(TimeUnit.MICROSECONDS)
 public int fooEachSumIntegers() {
  int result = 0;
  for (Integer value : values) {
   result += value.intValue();
  }
  return result;
 }

 @Benchmark
 @BenchmarkMode(Mode.Throughput)
 @OutputTimeUnit(TimeUnit.MICROSECONDS)
 public int parallelSumIntegers() {
  int result = values.parallelStream().mapToInt(i -> i).sum();
  return result;
 }

 @Benchmark
 @BenchmarkMode(Mode.Throughput)
 @OutputTimeUnit(TimeUnit.MICROSECONDS)
 public int sequentialSumIntegers() {
  int result = values.stream().mapToInt(i -> i).sum();
  return result;
 }
}

JMH benchmark Results :

Example 3 : "Protobuf performs up to 6 times faster than JSON."

Protobuf, the binary format crafted by Google, surpasses JSON performance even on JavaScript environments like Node.js/V8 and web browsers.

Protocol buffers, or Protobuf, is a binary format created by Google to serialize data between different services. Google made this protocol open source and now it provides support, out of the box, to the most common languages, like JavaScript, Java, C#, Ruby and others. In our tests, it was demonstrated that this protocol performed up to 6 times faster than JSON.

package org.sample;

import java.io.FileInputStream;
import java.io.FileOutputStream;
import java.io.IOException;
import java.util.concurrent.TimeUnit;
import org.openjdk.jmh.annotations.Benchmark;
import org.openjdk.jmh.annotations.BenchmarkMode;
import org.openjdk.jmh.annotations.Level;
import org.openjdk.jmh.annotations.Mode;
import org.openjdk.jmh.annotations.OutputTimeUnit;
import org.openjdk.jmh.annotations.Scope;
import org.openjdk.jmh.annotations.Setup;
import org.openjdk.jmh.annotations.State;
import com.proto.StudentBookProtos;
import com.proto.StudentService;
import com.proto.model.StudentBook;

@State(Scope.Thread)
public class JMHSample04 {

 private byte[] studentBookAsJSON;
 private StudentBook studentBookObject;
 private StudentBookProtos.StudentBook.Builder studentBuilder;
 private StudentService studentService;

 @Setup(Level.Trial)
 public void init() {
  studentService = new StudentService();
  studentBookObject = studentService.getStudentBookObject();
  studentBuilder = studentService.getStudentBookProtoObject();
  studentBookAsJSON = studentService.studentBookAsJSONByte(studentBookObject);

  try (FileOutputStream output = new FileOutputStream("abc_proto.txt")) {
   studentBuilder.build().writeTo(output);
   output.close();
  } catch (IOException e) {
   e.printStackTrace();
  }

  try (FileOutputStream output = new FileOutputStream("abc_json.txt")) {
   output.write(studentBookAsJSON);
   output.close();
  } catch (IOException e) {
   e.printStackTrace();
  }

 }

 @Benchmark
 @BenchmarkMode(Mode.Throughput)
 @OutputTimeUnit(TimeUnit.SECONDS)
 public void writeProtoToFile() {
  try (FileOutputStream output = new FileOutputStream("abcd_proto.txt")) {
   studentBuilder.build().writeTo(output);
   output.close();
  } catch (IOException e) {
   e.printStackTrace();
  }
 }

 @Benchmark
 @BenchmarkMode(Mode.Throughput)
 @OutputTimeUnit(TimeUnit.SECONDS)
 public void writeJSONToFile() {
  try (FileOutputStream output = new FileOutputStream("abcd_json.txt")) {
   output.write(studentBookAsJSON);
   output.close();
  } catch (IOException e) {
   e.printStackTrace();
  }
 }

 @Benchmark
 @BenchmarkMode(Mode.Throughput)
 @OutputTimeUnit(TimeUnit.SECONDS)
 public com.proto.StudentBookProtos.StudentBook deserialize_protobuf_to_student_object() {
  com.proto.StudentBookProtos.StudentBook studentBook = null;
  try (FileInputStream inputStream = new FileInputStream("abc_proto.txt")) {
   studentBook = com.proto.StudentBookProtos.StudentBook.parseFrom(inputStream);
   inputStream.close();
  } catch (IOException e) {
   e.printStackTrace();
  }
  return studentBook;
 }

 @Benchmark
 @BenchmarkMode(Mode.Throughput)
 @OutputTimeUnit(TimeUnit.SECONDS)
 public StudentBook deserialize_json_to_student_object() {
  StudentBook studentBook = null;
  try (FileInputStream inputStream = new FileInputStream("abc_json.txt")) {
   byte fileContent[] = new byte[(int) inputStream.available()];
   inputStream.read(fileContent);
   studentBook = studentService.getStudentBookFromJSONByte(fileContent);
   inputStream.close();
  } catch (IOException e) {
   e.printStackTrace();
  }
  return studentBook;
 }
}

JMH benchmark Results :