TCPDUMP from NIC to Network Layer

NIC - Network Interface Card

• NIC or network interface card or network adapter, LAN adapter is an important hardware requirement for connectivity . Actually this interface card provides us the 'gateway' through which data from one computer A can be transferred to another computer B .
• A network interface card (NIC) is a circuit board or card that is installed in a computer so that it can be connected to a network. A network interface card provides the computer with a dedicated, full-time connection to a network.
• NIC is OSI layer 2 device (Data Link Layer) and tcpdump should be obtained data packet from the network card driver / network interface card (NIC).

TCPDUMP

• Some times during troubleshooting this tool proves to be very helpful. With the help of this tool you can analyze the packet before it reaches the application stack. And sometimes detect why the server is not responding to a ping request, why an application is not responding to a certain machine etc etc.
• Its mainly used to debug the protocol of the captured packet which in turn reveals the network traffic characteristics.
• tcpdump is a most powerful and widely used command-line packets sniffer or package analyzer tool which is used to capture or filter TCP/IP packets that received or transferred over a network on a specific interface.
• Tcpdump is a very powerful  because of its strength in capturing packets based on different parameters given.  It operates on network layer, so will be able to capture all the packets in and out of the machine. You can use tcpdump to capture and save the packets to a file to analyze it later.
• TCPDUMP uses Libpcap(a c/c++ library that's used for packet capturing.) and It is available under most of the Linux/Unix based operating systems.
• There are other tools out there which does the same job of packet capture/analyzing like wireshark, but tcpdump keeps all the captures raw. Which means its shows us the raw data it captures as it is.
• A network packet header consists of sender,destination,state information and other flag information's.
• TCPDUMP only captures the first 96 bytes of data from the packet by default.
• tcpdump also gives us a option to save captured packets in a file for future analysis. It saves the file in a pcap format, that can be viewed by tcpdump command or  a open source GUI based tool called Wireshark (Network Protocol Analyzier) that reads tcpdump pcap format files.

Objective

• How to monitor/track HTTP and HTTPS calls i,e traffic inflow and outflow of Apache httpd server on any specific port like port 80 or 443
• How to intercept the HTTP traffic initiated from the browser (or) SOAP UI to application server and sneak a peak into the content like Request Body like XML,JSON and Username and Password etc.
• How to tamper and read the incoming and outgoing HTTP traffic to our applications
• How to dig into the incoming (or) outgoing HTTP traffic and take a look at the concrete elements of HTTP protocol such as headers, cookies, request body as they gets transmitted.

 

Open Systems Interconnection (OSI) model  

• The Open Systems Interconnection (OSI) model defines a networking framework to implement protocols in layers, with control passed from one layer to the next.
• It conceptually divides computer network architecture into 7 layers in a logical progression. The lower layers deal with electrical signals, chunks of binary data, and routing of these data across networks. Higher levels cover network requests and responses, representation of data, and network protocols .
• The data communications between different networks are not possible if there’s no common rules for transmitting and receiving the packets of data. These rules are known as protocols.
• Transmission Control Protocol (TCP)/Internet Protocol (IP) is one of the most widely used protocols.
• OSI model acts as a reference model and is not implemented in Internet because of its late invention.
• Both models are based on the concept of layering.

Network errors and analysis tools assigned to OSI layers

Layering: 

Divide a task into sub-tasks and then solve each sub-task independently and establishing a well defined interface between layers makes porting easier.

Layering Example: Federal Express

• Letter in envelope, address on outside
• FedX guy adds addressing information,barcode.
• Local office drives to airport and delivers to hub.
• Sent via airplane to nearest city.
• Delivered to right office
• Delivered to right person

There are many mnemonics way can be used to remember the order of the OSI layer:

• All People Seem To Need Data Processing.
• All People Should Try New Diet Pepsi.
• All People Share Their Nasty Delivered Pizza.

The TCP/IP model and OSI model are both conceptual models used for description of all network communications, while TCP/IP itself is also an important protocol used in all Internet operations. Generally, when we talk about layer 2, layer 3 or layer 7 in which a network device works, we are referring to the OSI model. The TCP/IP model is used for both modelling current Internet architecture and providing a set of rules that are followed by all forms of transmission over the network. 

* Protocol data unit (PDU) being processed in different layers.

7. Application Layer – PDU (Data)

It functions as the user interface platform (including software within the system) . The application layer enables the user, whether human or software, to access the network.

User interface Examples – DNS, IMAP, POP, SMTP, SNMP, Telnet, HTTP, FTP , Browsers, Skype Messenger etc

6. Presentation Layer / Translation layer – PDU (Data) 
The presentation layer checks the data to ensure it is compatible with the communications resources. It translates the data into the form that the application level and lower levels accept i,e It provides a translation of data to the next layer (prepare the data to the Application Layer).

Translation: different encoding methods like ASCII to EBCDIC.

Encryption/ Decryption: It translates the data into another form or code. The encrypted data is known as the cipher text and the decrypted data is known as plain text . (For the data that has high security requirements, such as a text message containing your password, it will be encrypted at this layer.)

Compression: It reduces the number of bits that need to be transmitted on the network. (Video calls will be compressed during the transmission so that it can be transmitted faster, and the data will be recovered at the receiving side).

5. Session Layer - PDU (Data)

This layer identifies the established system session between different network entities. It controls dialogues (connections) between computers (3 ways handshaking). It establishes, manages, maintains and ultimately terminates the connections between the local and remote application. It handles authentication and authorization functions. It verifies the data is delivered as well. 

Dialog control: It allows two systems to start communication with each other in half-duplex or full-duplex. (In a full-duplex system, both parties can communicate with each other simultaneously. In a half-duplex system, both parties can communicate with each other, but not simultaneously; the communication is one direction at a time.)

Synchronization: The session layer allows a process to add checkpoints (synchronization points) into a stream of data. For example, if a system is sending a file of 2,000 pages, it is advisable to insert checkpoints after every 100 pages to ensure that each 100-page unit is received  and acknowledged independently. In this case, if a crash happens during the transmission of page 523, the only pages that need to be resent after system recovery are pages 501 to 523. Pages previous to 501 need not be resent.

Note:

• All the above 3 layers (including Session Layer) are integrated as a single layer in TCP/IP model as “Application Layer”.
• Implementation of these 3 layers is done by the network application itself. These are also known as Upper Layers or Software Layers.  

4. Transport Layer - PDU (Segment (TCP) / Datagram (UDP))

This layer ensures the transport/sending of data is successful. This function can include error checking operation; along with the ability to keep data message in sequence means transferring  of data sequences from a source to a destination host via one or more networks, ensure the complete End to End delivery of the data. The integrity of the data can be guaranteed via error correction and similar functions.  It also provides the acknowledgment of the successful data transmission and re-transmits the data if an error is found. It also adds Source and Destination port number in its header and forwards the segmented data to the Network Layer.

Service-point addressing: In order to deliver the message to correct process, transport layer header includes a type of address called service point address or port address. Thus by specifying this address, transport layer makes sure that the message is delivered to the correct process.

Segmentation and reassembly:  A message is divided into transmittable segments, with each segment containing a sequence number. These numbers enable the transport layer to reassemble the message correctly upon arriving at the destination and to identify and replace packets that were lost in transmission.

Connection control: The transport layer can be either connectionless or connection-oriented.
(Connection-oriented: the receiving device sends an acknowledgment, back to the source after a packet or group of packet is received, Connection less service: the receiver does not acknowledge receipt of a packet. This approach allows for much faster communication between devices.  Connection oriented Service is more reliable than connection less Service. TCP (Transmission Control Protocol) is connection oriented, whereas UDP (User Datagram Protocol) is connection-less.  This means that TCP tracks all data sent, requiring acknowledgment for each octet (generally).)

Flow control: Flow control at this layer is performed end to end rather than across a single link

Error control: Error control at this layer is performed process-to-process rather than across a single link

Note:

• Data in the Transport Layer is called as Segments.
• Transport layer is operated by the Operating System. It is a part of the OS and communicates with the Application Layer by making system calls. Transport Layer is called as Heart of OSI model.

3. Network Layer - PDU (Packet)
The network layer handles packet routing via logical addressing and switching functions. A network is a medium to which many nodes can be connected. Every node has an address. When a node needs to transfer message to other nodes, it can merely provide the content of the massage and the address of the destination node, then the network will find the way to deliver the message to the destination node, possibly routing through other nodes. If the message is too long, the network may split it into several segments at one node, sending them separately and reassembling the fragments at another node.

Routing: The network layer protocols determine which route is suitable from source to destination. This function of network layer is known as routing. Example: IP, IPX

Logical Addressing:  In order to identify each device on internetwork uniquely, network layer defines an addressing scheme. The sender & receiver’s IP address are placed in the header by network layer

Note:

• This layer provides connections between hosts on different networks.
• Segment in Network layer is referred as Packet.
• Network layer is implemented by networking devices such as routers.

2. Data Link Layer - PDU (Frame)

The data link layer is responsible for the node to node delivery of the message.  The main function of this layer is to make sure data transfer is error free from one node to another, over the physical layer. When a packet arrives in a network, it is the responsibility of DLL to transmit it to the Host using its MAC address.  

Data Link Layer is divided into two sub layers:

• Logical Link Control (LLC)
• Media Access Control (MAC)

The packet received from Network layer is further divided into frames depending on the frame size of NIC(Network Interface Card). DLL also encapsulates Sender and Receiver’s MAC address in the header.
Examples:  HDLC, 802.3/802.2 for Ethernet” and “802.11 for Wi-Fi”

Framing: Combines bits into bytes and bytes into frames i,e It divides the stream of bits received from the network layer into manageable data units called frames

Physical addressing: After creating frames, Data link layer adds physical addresses (MAC address) of sender and/or receiver in the header of each frame.

Error control: Data link layer provides the mechanism of error control in which it detects and retransmits damaged or lost frames.

Flow Control: The data rate must be constant on both sides else the data may get corrupted thus , flow control coordinates that amount of data that can be sent before receiving acknowledgement.

Access control: When two or more devices are connected to the same link, data link layer protocols are necessary to determine which device has control over the link at any given time.

Note:

• This layer provides connections between hosts on the same networks.
• Packet in Data Link layer is referred as Frame.
• Data Link layer is handled by the NIC (Network Interface Card) and device drivers of host machines.
• Switch & Bridge are Data Link Layer devices.

1. Physical Layer - PDU (Bit)

This layer describes the electrical and physical specifications for devices such as, cables, connectors, hubs, repeaters and more. It is responsible for the actual physical connection between the devices. The physical layer contains information in the form of bits.  When receiving data, this  will convert it into 0s and 1s and send them to the Data Link layer, which will put the frame back together. 

Bit synchronization: the sender and the receiver clocks must be synchronized.

Bit rate control:  It defines the transmission rate i.e. the number of bits sent per second.

Physical topologies:  It specifies the way in which the different, devices/nodes are arranged in a network i.e. bus, star or mesh topolgy.

Transmission mode: It defines the way in which the data flows between the two connected devices. The various transmission modes possible are: Simplex, half-duplex and full-duplex.

Note:

• Hub, Repeater, Modem, Cables are Physical Layer devices.
• Network Layer, Data Link Layer and Physical Layer are also known as Lower Layers or Hardware Layers.

TCP/IP Model vs. OSI Model

One of the major difference is that OSI is a conceptual model which is not practically used for communication, whereas, TCP/IP is used for establishing a connection and communicating through the network.

Similarities between OSI Reference Model and TCP/IP Reference Model :

• Both have layered architecture.
• Layers provide similar functionalities.
• Both are protocol stack.
• Both are reference models.

Difference between OSI Reference Model and TCP/IP Reference Model:

• TCP/IP is a client-server model, i.e. when the client requests for service it is provided by the server. Whereas, OSI is a conceptual model.
• TCP/IP is a standard protocol used for every network including the Internet, whereas, OSI is not a protocol but a reference model used for understanding and designing the system architecture.
• TCP/IP is a four layered model, whereas, OSI has seven layers.
• TCP/IP follows Vertical approach. On the other hand, OSI Model supports Horizontal approach.
• TCP/IP is Tangible, whereas, OSI is not.
• TCP/IP follows top to bottom approach, whereas, OSI Model follows a bottom-up approach.

The TCP/IP network model represents reality in the world, whereas the OSI mode represents an ideal.

Port Address :

• Domain Name System (DNS) - TCP/UDP Port 53
• Hypertext Transfer Protocol (HTTP) - TCP Port 80
• Simple Mail Transfer Protocol (SMTP) - TCP Port 25
• Post Office Protocol (POP) - UDP Port 110 Telnet - TCP Port 23
• Dynamic Host Configuration Protocol - UDP Port 67
• File Transfer Protocol (FTP) - TCP Ports 20 and 21

* Layer 0 is the photonics layer, more specifically, DWDM is considered as the Layer 0 of the OSI layers.

DWDM – Dense Wavelength Division Multiplexing is an Optical Transmission technology.

• DWDM allows you to transmit multiple wavelengths on a single fiber, as the name suggests it utilizes wave division multiplexing.
• DWDM can be deployed in ultra-long haul, long-haul, regional, and metro area’s.
• DWDM also uses amplification to reach long distances and can be an expensive technology to deploy.
• On the other hand CWDM which is a metro technology multiplexing a maximum of 8 wavelengths per fiber and does not require amplification i.e. much cheaper than DWDM
• WDM is used to transport SDH/Ethernet/IP between regions or cities or to aggregate traffic where large bandwidth is required

OSI Data Encapsulation and Decapsulation

In OSI Model ,Data flows in two ways i,e one is DOWN(data encapsulation) and reverse way is UP(data decapsulation)

Data Encapsulation

The process of converting Data into Bits and Bytes , is called Encapsulation.

• The computer in the above picture needs to send some data to another computer. The Application layer is where the user interface exists, here the user interacts with the application he or she is using, then this data is passed to the Presentation layer and then to the Session layer. These three layer add some extra information to the original data that came from the user and then passes it to the Transport layer. Here the data is broken into smaller pieces (one piece at a time transmitted) and the TCP header is a added. At this point, the data at the Transport layer is called a segment.
• Each segment is sequenced so the data stream can be put back together on the receiving side exactly as transmitted. Each segment is then handed to the Network layer for network addressing (logical addressing) and routing through the internet network. At the Network layer, we call the data (which includes at this point the transport header and the upper layer information) a packet.
• The Network layer add its IP header and then sends it off to the Datalink layer. Here we call the data (which includes the Network layer header, Transport layer header and upper layer information) a frame. The Datalink layer is responsible for taking packets from the Network layer and placing them on the network medium (cable). The Datalink layer encapsulates each packet in a frame which contains the hardware address (MAC) of the source and destination computer (host) and the LLC information which identifies to which protocol in the prevoius layer (Network layer) the packet should be passed when it arrives to its destination. Also, at the end, you will notice the FCS field which is the Frame Check Sequence. This is used for error checking and is also added at the end by the Datalink layer.
• If the destination computer is on a remote network, then the frame is sent to the router or gateway to be routed to the desination. To put this frame on the network, it must be put into a digital signal. Since a frame is really a logical group of 1's and 0's, the Physical layer is responsible for encapsulating these digits into a digital signal which is read by devices on the same local network.
• There are also a few 1's and 0's put at the begining of the frame, only so the receiving end can synchronize with the digital signal it will be receiving.

Data Decapsulation

The process of converting Bits and Bytes into Data , is called Decapsulation.

• The receiving computer will firstly synchronize with the digital signal by reading the few extra 1's and 0's as mentioned above. Once the synchonization is complete and it receives the whole frame and passes it to the layer above it which is the Datalink layer.
• The Datalink layer will do a Cyclic Redundancy Check (CRC) on the frame. This is a computation which the comupter does and if the result it gets matches the value in the FCS field, then it assumes that the frame has been received without any errors. Once that's out of the way, the Datalink layer will strip off any information or header which was put on by the remote system's Datalink layer and pass the rest (now we are moving from the Datalink layer to the Network layer, so we call the data a packet) to the above layer which is the Network layer.
• At the Network layer the IP address is checked and if it matches (with the machine's own IP address) then the Network layer header, or IP header if you like, is stripped off from the packet and the rest is passed to the above layer which is the Transport layer. Here the rest of the data is now called a segment.
• The segment is processed at the Transport layer, which rebuilds the data stream (at this level on the sender's computer it was actually split into pieces so they can be transferred) and acknowledges to the transmitting computer that it received each piece. It is obvious that since we are sending an ACK back to the sender from this layer that we are using TCP and not UDP. It then happily hands the data stream to the upper-layer application.
• You will find that when analysing the way data travels from one computer to another most people never analyse in detail any layers above the Transport layer. This is because the whole process of getting data from one computer to another involves usually layers 1 to 4 (Physical to Transport) or layer 5 (Session) at the most, depending on the type of data.

List Network Interfaces Using ip Command on Linux :

[root@myvm ~]#  ip link show

1: lo:  mtu 16436 qdisc noqueue state UNKNOWN

    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00

2: eth0:  mtu 1500 qdisc mq state UP qlen 1000

    link/ether b8:ac:6f:65:31:e5 brd ff:ff:ff:ff:ff:ff

3: wlan0:  mtu 1500 qdisc mq state DOWN qlen 1000

    link/ether 00:21:6a:ca:9b:10 brd ff:ff:ff:ff:ff:ff

4: vboxnet0:  mtu 1500 qdisc noop state DOWN qlen 1000

    link/ether 0a:00:27:00:00:00 brd ff:ff:ff:ff:ff:ff

5: pan0:  mtu 1500 qdisc noop state DOWN

    link/ether c2:10:fa:55:8e:32 brd ff:ff:ff:ff:ff:ff

6: vmnet1:  mtu 1500 qdisc pfifo_fast state UNKNOWN qlen 1000

    link/ether 00:50:56:c0:00:01 brd ff:ff:ff:ff:ff:ff

7: vmnet8:  mtu 1500 qdisc pfifo_fast state UNKNOWN qlen 1000

    link/ether 00:50:56:c0:00:08 brd ff:ff:ff:ff:ff:ff

11: ppp0:  mtu 1496 qdisc pfifo_fast state UNKNOWN qlen 3

    link/ppp

where

• lo – Loopback interface.
• eth0 – My first Ethernet network interface on Linux.
• wlan0 – Wireless network interface in Linux.
• ppp0 – Point to Point Protocol network interface which can be used by dial up modem, PPTP vpn connection, or 3G wireless USB modem.
• vboxnet0, vmnet1, vmnet8 – Virtual machine interface working in bridge mode or NAT mode on Linux.

Example 1: Monitor http connection (3 way handshaking (SYN, SYN-ACK, ACK))

Three-way-handshake process to startup a connection:

1. The requesting Host sends a synchronization flag (SYN) in a TCP segment to create a connection.
2. The receiving Host 172.217.166.179 receives the SYN flag and returns an acknowledgment flag (ACK).
3. The requesting Host 10.190.10.70 receives the SYN flag and returns it's own ACK flag.

To terminate a connection, a segment containing a FIN flag is sent from host 10.190.10.70 back to the host with the open session.  Refer below screenshot how a connection has established and terminated.

TCP flag bits used for 3 way handshaking : 

• S: SYN (Synchronize sequence numbers - Connection establishment)
• F: FIN (Ending of sending by sender - Connection termination)
• R: RST (Reset connection)
• P: PSH (Push data)
• .: (No flag is set)

For more details on how a client and server share the tcp flag, sequences and acknowledgement , please go through http://www.java2depth.com/2018/06/clientserver-socket-interaction-data.html

Example 2:  How to capture all HTTP traffic including request and response headers and message body for a port:

To Get the interface name of your IP which you need to specify it in the tcpdump command.  you can execut the command ifconfig (or) ip a

[root@myvm ~]# ip a

System will tell tcpdump to capture packets that's coming towards a particular interface.

tcpdump is only listening on loopback interface for packets :

[root@myvm ~]#  tcpdump -i lo

tcpdump: verbose output suppressed, use -v or -vv for full protocol decode

listening on lo, link-type EN10MB (Ethernet), capture size 96 bytes

tcpdump is only listening on eth0 interface for packets :

[root@myvm ~]#  tcpdump -i eth0

tcpdump: verbose output suppressed, use -v or -vv for full protocol decode

listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes

Display packets for a particular port

[root@myvm ~]#  tcpdump -i eth0 port 22

show packets towards a particular host

[root@myvm ~]# tcpdump -i eth0 host 10.190.10.70

Show packets from source with tcpdump

[root@myvm ~]# tcpdump -i eth0 src host 10.190.10.70

Show packets for destination with tcpdump

[root@myvm ~]# tcpdump -i eth0 dst host 10.190.10.70

Show plaintext credentials sniffinf with tcpdump and grep for common unsafe protocols

[root@myvm ~]#  tcpdump port http or port ftp or port smtp or port imap or port pop3 -l -A | egrep -i ‘pass=|pwd=|log=|login=|user=|username=|pw=|passw=|passwd=|password=|pass:|user:|username:|password:|login:|pass |user ‘ –color=auto –line-buffered -B20

See what traffic is mostly hitting you

[root@myvm ~]# tcpdump -i eth0 -n  | head

Capture TCP packets from local host to local host

[root@myvm ~]# tcpdump -i lo

How to capture All incoming  HTTP GET  traffic (or) requests

[root@myvm ~]# tcpdump -s 0 -i eth3 -A -vv 'tcp[((tcp[12:1] & 0xf0 ) >> 2 ):4] = 0x47455420'

How to capture All incoming HTTP POST requests

[root@myvm ~]#  tcpdump -s 0 -i eth3 -A -vv 'tcp[((tcp[12:1] & 0xf0 ) >> 2 ):4] = 0x504f5354'

How to capture all HTTP traffic including request and response headers and message body for a port:

[root@myvm ~]#  tcpdump -A -s 0 'tcp port 8080 and (((ip[2:2] - ((ip[0]&0xf)<<2)) - ((tcp[12]&0xf0)>>2)) != 0)'

How to capture all HTTP traffic including request and response headers and message body for a port and source IP:

[root@myvm ~]#  tcpdump -A -s 0 'src 10.190.10.70 and tcp port 11222 and (((ip[2:2] - ((ip[0]&0xf)<<2)) - ((tcp[12]&0xf0)>>2)) != 0)'

How to capture all HTTP traffic including request and response headers and message body for local host to local host

[root@myvm ~]#  tcpdump -A -s 0 'tcp port 8080 and (((ip[2:2] - ((ip[0]&0xf)<<2)) - ((tcp[12]&0xf0)>>2)) != 0)' -i lo

How to capture all HTTP traffic including request and message body only  for a port:

[root@myvm ~]#   tcpdump -A -s 0 'tcp dst port 8080 and (((ip[2:2] - ((ip[0]&0xf)<<2)) - ((tcp[12]&0xf0)>>2)) != 0)'

How to capture all HTTP traffic including request and message body for a port and source IP:

[root@myvm ~]#   tcpdump -A -s 0 'src 10.190.10.70 and tcp dst port 11222 and (((ip[2:2] - ((ip[0]&0xf)<<2)) - ((tcp[12]&0xf0)>>2)) != 0)'

How to capture all HTTP traffic including request and message body for a local host to local host:

[root@myvm ~]#   tcpdump -A -s 0 'tcp dst port 8080 and (((ip[2:2] - ((ip[0]&0xf)<<2)) - ((tcp[12]&0xf0)>>2)) != 0)' -i lo

See what traffic is mostly hitting you

[root@myvm ~]# tcpdump -i eth0 -n  | head


[root@myvm ~]# tcpdump -A -s 0 'src 192.100.101.146 and (((ip[2:2] - ((ip[0]&0xf)<<2)) - ((tcp[12]&0xf0)>>2)) != 0)'
[root@myvm ~]# tcpdump -A -s 0 'dst vwuatltapp01 and (((ip[2:2] - ((ip[0]&0xf)<<2)) - ((tcp[12]&0xf0)>>2)) != 0)'
[root@myvm ~]# tcpdump -A -s 0 '(src 192.100.101.146 and dst vwuatltapp01 )  and (((ip[2:2] - ((ip[0]&0xf)<<2)) - ((tcp[12]&0xf0)>>2)) != 0)'
[root@myvm ~]# tcpdump -A -s 0 '(src 192.100.101.146 and dst vwuatltapp01 ) or (src vwuatltapp01 and dst 192.100.101.146 ) and (((ip[2:2] - ((ip[0]&0xf)<<2)) - ((tcp[12]&0xf0)>>2)) != 0)'
[root@myvm ~]# tcpdump -A -s 0 '(src 192.100.101.146 or dst 192.100.101.146) and (((ip[2:2] - ((ip[0]&0xf)<<2)) - ((tcp[12]&0xf0)>>2)) != 0)'

[root@myvm ~]# tcpdump -A -s 0 'tcp port 6802 and (((ip[2:2] - ((ip[0]&0xf)<<2)) - ((tcp[12]&0xf0)>>2)) != 0)'

Read HTTP request from tcpdump pcap file

[root@myvm ~]# tcpdump -r tcpdump.pcap -A -s 0 'host 192.168.171.91 and port 60689 and (((ip[2:2] - ((ip[0]&0xf)<<2)) - ((tcp[12]&0xf0)>>2)) != 0)'

TCPDUMP Quick Reference :-

WinDump: The tcpdump tool for Windows

WinDump, the Windows version of tcpdump, can help you analyze network traffic to look for signs of active malware. 

WinDump comes in two parts. 

• The first is a set of network capture drivers called WinPcap, which WinDump uses to obtain packet-level access to network interfaces in the computer. 
• The second part is the program itself, windump, which is invoked from the command line after you've installed the WinPcap library.

The first option you'll want to use when you run windump is -D, which lists all available network interfaces in the current system. By default, the program listens on the first available interface, but in Windows, it is typically the software dial-up adapter, not a physical network adapter. The results from -D usually look something like this:

• \Device\NPF_GenericDialupAdapter (Generic dialup adapter)
• \Device\NPF_{707E0236-BEE4-4097-93B1-56DEC35564AA} (Intel DC21140 PCI Fast Ethernet Adapter (Microsoft's Packet Scheduler) )

To use a specific adapter, run the program with the -i switch in conjunction with an adapter number. For instance, if you wanted to use the Ethernet adapter listed above, use windump -i 2. This is a lot easier than referring to the adapter by its GUID, but bear in mind the adapter number may not remain consistent if new hardware or software adapters are added.

Windump has the ability to filter captured input by specific criteria -- specific protocols, hosts or ports -- usually specified on the command line. The syntax for this is fairly complicated.

Captures all traffic from interface #2 via port 80.
windump -i 2 port 80

Captures all traffic from interface #2 to or from the host im-chat.com.
windump -i 2 host im-chat.com

Captures all traffic from interface #1 to or from the subnet 127.x.x.x.
windump -i 1 net 127

For more details go through the link https://www.winpcap.org/windump/docs/manual.htm

Micro Benchmarking with JMH - "Measure, don’t guess"

Java Micro Benchmark with JMH

• Java Microbenchmark Harness (JMH) is a Java toolkit by OpenJDK for creating benchmarks.  
• It is an open source framework provides benchmarking for measuring the performance of your Java code.

"Measure, don’t guess" :-

We’ve all faced performance problems in our projects and were asked to tune random bits in our source code and  hoping that performance will get improved. Instead, we should set up a stable performance environment  (operating system, JVM, application server, database), measure continuously, set some performance goals then, take action when our goals are not achieved. Continuous delivery, continuous testing is one thing,  but continuous measuring is another step.

What is JMH ?

• JMH is a Java harness for building, running, and analyzing nano/micro/milli/macro benchmarks written in Java and other languages targeting the JVM.

When to use Micro Benchmarking ?

• You identified the code segment that eats most of the resources in your application and the improvement can be tested by micro benchmarks.
• You can not identify the code segment that will eat most of the resources in an application but you suspect it.

How to do a Micro Benchmark ?

• We often do not worry about the performance requirements. We start with building functionality and concentrate on making things work and don’t focus on how. Once the software goes to production, we will be facing the inevitable. 
• It is good to have the performance objectives written down before writing the code.
  We should weigh the importance of performance with respect to the functional requirements and should come up with a balance between them.
• Just because your code runs in a certain way in an extremely isolated artificial situation does not mean it will run in the  same way inside your production code. To name but a few issues, in a real program the CPU caches will be subject to pressures from other parts of your code, any object creation will have a downstream effect on GC and the JIT may have inlined and compiled code from other parts of your code that conflict with the code you have benchmarked. 

Why are Java Microbenchmarks Hard ?

• Writing benchmarks that correctly measure the performance of a small part of a larger application is hard. There are many optimizations that the JVM or underlying hardware may apply to your component when the benchmark executes that component in isolation. These optimizations may not be possible to apply when the component is running as part of a larger application. Badly implemented microbenchmarks may thus make you believe that your component's performance is better than it will be in reality.
• Writing a correct Java microbenchmark typically entails preventung the optimizations the JVM and hardware may apply during microbenchmark execution which could not have been applied in a real production system. That is what JMH - the Java Microbenchmark Harness - is helping you do.  

Purpose

• Benchmark is the process of recording the performance of a system.
• JMH is a Java harness for building, running, and analyzing nano/micro/milli/macro benchmarks written in Java and other languages targeting the JVM.
• it takes care of warm up iterations, forking JVM processes so that benchmarks don't interfere with each other, collating results and presenting then in a uniform manner.

Micro benchmarks are generally done for two reasons.

• To compare different approaches of code which implements the same logic and choose the best one to use.
• To identify any bottlenecks in a suspected area of code during performance optimization.

Benchmarking Modes:

At a basic level, JMH has two main types of measure:  throughput and time-based.

Throughput Measuring

• Throughput is the amount of operations that can be completed per the unit of time. JMH maintains a collection of successful and failed operations as the framework increases the amount of load on the test.  
• Ensure the method or test is well isolated and dependencies like test object creation is done outside of the method or pre-test in a setup method.  
• With Throughput, the higher the value, the better as it indicates that more operations can be run per unit-time.

Time-Based Measuring

• Time-based measuring is the counter-partner to throughput. 
• The goal of time-based measuring is to identify how long a particular operation takes to run per unit-time.

JMH Commands

• i - Number of measurement iterations to do. Measurement iterations are counted towards the benchmark score.
• bs - Batch size: number of benchmark method calls per operation.
• r - Minimum time to spend at each measurement iteration.
• wi - Number of warmup iterations to do.
• wbs - Warmup batch size: number of benchmark method calls per operation.
• w - Minimum time to spend at each warmup iteration.
• to - Timeout for benchmark iteration.
• t - Number of worker threads to run with.
• bm - Benchmark mode.
• si - Should JMH synchronize iterations?
• gc - Should JMH force GC between iterations?
• foe - Should JMH fail immediately if any benchmark had experienced an unrecoverable error?
• v - Verbosity mode.
• f - How many times to fork a single benchmark. Use 0 to disable forking altogether.
• wf - How many warmup forks to make for a single benchmark. 

 There are two way to run a benchmark:

1. The recommended way is to generate a pom file and use that to create a jar. The mvn install uses the shade plugin to create a jar file so that you don't have create a main method.
2. Add the JMH maven dependencies to your pom file and then add a main method to your code using the Runner object.This is useful if you want to run in your IDE. 

 Method 1 - Using command line argument

 Generate a pom file using this mvn command.

mvn archetype:generate
          -DinteractiveMode=false
          -DarchetypeGroupId=org.openjdk.jmh
          -DarchetypeArtifactId=jmh-java-benchmark-archetype
          -DgroupId=com.jenkov
          -DartifactId=first-benchmark
          -Dversion=1.0

• This will create a project called test with an empty benchmark in it called MyBenchmark.
• To build the project just use mvn clean install. This will build a jar called benchmark.jar. 
• It is the benchmark.jar that should be run to run the benchmark not any other jars produced along the way that will be in your target folder.

To run use the command  java -jar target/benchmarks.jar JMHSample04 -wi 5 -t 1 -i 5 -f 1

 Method 2 - For running in your IDE

Add these dependencies to your Maven pom.xml file:

  org.openjdk.jmh
  jmh-core
  1.5.1


  org.openjdk.jmh
  jmh-generator-annprocess
  1.5.1

Then decide which methods you want benchmarked and add the annotation @Benchmark to them. If you need any initialisation code add it in a method which should be marked @Setup. 

The easiest way to run the benchmark is by adding by adding this implementation into your main method.

public static void main(String[] args) throws RunnerException {
        Options opt = new OptionsBuilder()
                .include(JMHSample04.class.getSimpleName()).
                warmupIterations(5).
                measurementIterations(5).
                threads(1).
                forks(1).
                build();
        new Runner(opt).run();
}

JMH benchmark Results :

As an example to see the format of a JMH benchmark, this is what my results looked like:

Example 1 : Arrays sort and Collection sort Benchmark Comparison

package org.sample;

import java.util.ArrayList;
import java.util.Arrays;
import java.util.Collections;
import java.util.List;
import java.util.Random;
import java.util.concurrent.TimeUnit;
import org.openjdk.jmh.annotations.Benchmark;
import org.openjdk.jmh.annotations.BenchmarkMode;
import org.openjdk.jmh.annotations.Level;
import org.openjdk.jmh.annotations.Mode;
import org.openjdk.jmh.annotations.OutputTimeUnit;
import org.openjdk.jmh.annotations.Scope;
import org.openjdk.jmh.annotations.Setup;
import org.openjdk.jmh.annotations.State;

@State(Scope.Thread)
public class JMHSample02 {
 List arrayList;
 int[] array;
 Random random;

 @Setup(Level.Trial)
 public void init() {
  random = new Random();
  array = new int[150];
  arrayList = new ArrayList();
  for (int i = 0; i < 150; i++) {
   int randomNumber = random.nextInt();
   array[i] = randomNumber;
   arrayList.add(new Integer(randomNumber));
  }
 }

 @Benchmark
 @BenchmarkMode(Mode.Throughput)
 @OutputTimeUnit(TimeUnit.SECONDS)
 public void arraysSort() {
  Arrays.sort(array);

 }

 @Benchmark
 @BenchmarkMode(Mode.Throughput)
 @OutputTimeUnit(TimeUnit.SECONDS)
 public void collectionsSort() {
  Collections.sort(arrayList);
 }
}

JMH benchmark Results :

Example 2 : Java 8 Stream API (Parallel Stream vs Sequential Stream vs  for-each vs iterator )

Java 8 claimed that Stream API would employ multi-core CPUs  to process data in parallel fashion which would eliminate difficulties  of  dealing with multi-thread  codes , as well as gaining performance advantages obtained from  multicored CPUs. 
 
Stream parallel one is the slowest one among four approaches of summing  integers in array.

package org.sample;

import java.util.ArrayList;
import java.util.Iterator;
import java.util.List;
import java.util.concurrent.TimeUnit;
import org.openjdk.jmh.annotations.Benchmark;
import org.openjdk.jmh.annotations.BenchmarkMode;
import org.openjdk.jmh.annotations.Mode;
import org.openjdk.jmh.annotations.OutputTimeUnit;
import org.openjdk.jmh.annotations.Scope;
import org.openjdk.jmh.annotations.Setup;
import org.openjdk.jmh.annotations.State;

@State(Scope.Benchmark)
public class JMHSample_01_HelloWorld {

 volatile int counts = 9999999;
 volatile List values = new ArrayList<>(counts);
 volatile int processors = Runtime.getRuntime().availableProcessors();

 @Setup
 public void setup() {
  populate(values);

 }

 public void populate(List list) {
  for (int i = 0; i < counts; i++) {
   if (i < counts / 2) {
    list.add(i, i);
   } else {
    list.add(i, i - counts);
   }
  }
 }

 @Benchmark
 @BenchmarkMode(Mode.Throughput)
 @OutputTimeUnit(TimeUnit.MICROSECONDS)
 public int iteratorSumIntegers() {
  int result = 0;
  Iterator ite = values.iterator();
  while (ite.hasNext()) {
   result += (int) ite.next();
  }
  return result;
 }

 @Benchmark
 @BenchmarkMode(Mode.Throughput)
 @OutputTimeUnit(TimeUnit.MICROSECONDS)
 public int fooEachSumIntegers() {
  int result = 0;
  for (Integer value : values) {
   result += value.intValue();
  }
  return result;
 }

 @Benchmark
 @BenchmarkMode(Mode.Throughput)
 @OutputTimeUnit(TimeUnit.MICROSECONDS)
 public int parallelSumIntegers() {
  int result = values.parallelStream().mapToInt(i -> i).sum();
  return result;
 }

 @Benchmark
 @BenchmarkMode(Mode.Throughput)
 @OutputTimeUnit(TimeUnit.MICROSECONDS)
 public int sequentialSumIntegers() {
  int result = values.stream().mapToInt(i -> i).sum();
  return result;
 }
}

JMH benchmark Results :

Example 3 : "Protobuf performs up to 6 times faster than JSON."

Protobuf, the binary format crafted by Google, surpasses JSON performance even on JavaScript environments like Node.js/V8 and web browsers.

Protocol buffers, or Protobuf, is a binary format created by Google to serialize data between different services. Google made this protocol open source and now it provides support, out of the box, to the most common languages, like JavaScript, Java, C#, Ruby and others. In our tests, it was demonstrated that this protocol performed up to 6 times faster than JSON.

package org.sample;

import java.io.FileInputStream;
import java.io.FileOutputStream;
import java.io.IOException;
import java.util.concurrent.TimeUnit;
import org.openjdk.jmh.annotations.Benchmark;
import org.openjdk.jmh.annotations.BenchmarkMode;
import org.openjdk.jmh.annotations.Level;
import org.openjdk.jmh.annotations.Mode;
import org.openjdk.jmh.annotations.OutputTimeUnit;
import org.openjdk.jmh.annotations.Scope;
import org.openjdk.jmh.annotations.Setup;
import org.openjdk.jmh.annotations.State;
import com.proto.StudentBookProtos;
import com.proto.StudentService;
import com.proto.model.StudentBook;

@State(Scope.Thread)
public class JMHSample04 {

 private byte[] studentBookAsJSON;
 private StudentBook studentBookObject;
 private StudentBookProtos.StudentBook.Builder studentBuilder;
 private StudentService studentService;

 @Setup(Level.Trial)
 public void init() {
  studentService = new StudentService();
  studentBookObject = studentService.getStudentBookObject();
  studentBuilder = studentService.getStudentBookProtoObject();
  studentBookAsJSON = studentService.studentBookAsJSONByte(studentBookObject);

  try (FileOutputStream output = new FileOutputStream("abc_proto.txt")) {
   studentBuilder.build().writeTo(output);
   output.close();
  } catch (IOException e) {
   e.printStackTrace();
  }

  try (FileOutputStream output = new FileOutputStream("abc_json.txt")) {
   output.write(studentBookAsJSON);
   output.close();
  } catch (IOException e) {
   e.printStackTrace();
  }

 }

 @Benchmark
 @BenchmarkMode(Mode.Throughput)
 @OutputTimeUnit(TimeUnit.SECONDS)
 public void writeProtoToFile() {
  try (FileOutputStream output = new FileOutputStream("abcd_proto.txt")) {
   studentBuilder.build().writeTo(output);
   output.close();
  } catch (IOException e) {
   e.printStackTrace();
  }
 }

 @Benchmark
 @BenchmarkMode(Mode.Throughput)
 @OutputTimeUnit(TimeUnit.SECONDS)
 public void writeJSONToFile() {
  try (FileOutputStream output = new FileOutputStream("abcd_json.txt")) {
   output.write(studentBookAsJSON);
   output.close();
  } catch (IOException e) {
   e.printStackTrace();
  }
 }

 @Benchmark
 @BenchmarkMode(Mode.Throughput)
 @OutputTimeUnit(TimeUnit.SECONDS)
 public com.proto.StudentBookProtos.StudentBook deserialize_protobuf_to_student_object() {
  com.proto.StudentBookProtos.StudentBook studentBook = null;
  try (FileInputStream inputStream = new FileInputStream("abc_proto.txt")) {
   studentBook = com.proto.StudentBookProtos.StudentBook.parseFrom(inputStream);
   inputStream.close();
  } catch (IOException e) {
   e.printStackTrace();
  }
  return studentBook;
 }

 @Benchmark
 @BenchmarkMode(Mode.Throughput)
 @OutputTimeUnit(TimeUnit.SECONDS)
 public StudentBook deserialize_json_to_student_object() {
  StudentBook studentBook = null;
  try (FileInputStream inputStream = new FileInputStream("abc_json.txt")) {
   byte fileContent[] = new byte[(int) inputStream.available()];
   inputStream.read(fileContent);
   studentBook = studentService.getStudentBookFromJSONByte(fileContent);
   inputStream.close();
  } catch (IOException e) {
   e.printStackTrace();
  }
  return studentBook;
 }
}

JMH benchmark Results :

Building a high performance Java application

Most of the time developers expect that performance optimization is a complicated topic that requires a lot of experience and knowledge.Optimizing an application to get the best performance possible isn’t an easy task. But that doesn’t mean that you can’t do anything if you haven’t acquired that knowledge. There are several easy ways to follow recommendations and best practices which help you to create a well-performing application.

System resources like threads, database connections, socket connections, File IO streams, Socket IO streams, JNI calls are the real bottleneck of a java application. If you will reuse them in a proper way then there will be a big performance impact into an application.

Java application bottleneck (CPU, IO, Heap usage) factors 

• Excessive big data encryption/decryption with strongest algorithm 
• Excessive remote web service call and creating/closing socket connections
• Frequent File IO operations and Reading/Scanning a physical directory to lookup specific file
• Huge loggers writing into log file (writing tons and tons of data to disk)
• Regular expression and compiling regex Patterns
• JNI Java Native method calls like C/C++/.Net
• Too many application active threads
• Excessive GC cycles going on & Large war size with maximum number of jars
• Uncontrolled pooling (connections/objects/threads) &  complicated SQL query execution
• Code problems for excessive back-end calls like 'infinite loops'

How would you improve performance of a Java application

• Pool valuable system resources like threads, database connections, socket connections etc. Emphasize on reuse of threads from a pool of threads. Creating new threads and discarding them after use can adversely affect performance. Also consider using multi-threading in your single-threaded applications where possible to enhance performance. Optimize the pool sizes based on system and application specifications and requirements. Having too many threads in a pool also can result in performance and scalability problems due to consumption of memory stacks (i.e. each thread has its own stack.) and CPU context switching (i.e. switching between threads as opposed to doing real computation.)
• Minimize network overheads by retrieving several related items simultaneously in one remote invocation if possible. Remote method invocations involve a network round-trip, marshaling and unmarshaling of parameters, which can cause huge performance problems if the remote interface is poorly designed.
• Distributed cache(Infinispan  in-memory data grid platform ) for fast data access , In-memory data grids are commonly used as low-latency, highly available and elastic data storage backends, often as NoSQL solutions i,e for static data storage use 2nd level cache mechanism to avoid the network database round trips and db remote connections bottleneck.
• High-performance asynchronous messaging library, ZeroMQ is used in distributed or concurrent applications. It provides a message queue, but unlike message-oriented middleware, a ZeroMQ system can run without a dedicated message broker.Connect your code in any language, on any platform.It carries messages across inproc, IPC, TCP, TIPC, multicast ,smart patterns like pub-sub, push-pull, and router-dealer and high-speed asynchronous I/O engines, in a tiny library.
• Persistent connections (A persistent connection (HTTP persistent connection) is a network communication channel that remains open for further HTTP requests and responses rather than closing after a single exchange) , It also called HTTP keep-alive, or HTTP connection reuse, is the idea of using a single TCP connection to send and receive multiple HTTP requests/responses, as opposed to opening a new connection for every single request/response pair. The newer HTTP/2 protocol uses the same idea and takes it further to allow multiple concurrent requests/responses to be multiplexed over a single connection.
• Non blocking Asynchronous IO(Log4j2) mechanism , LMAX Disruptor technology. Asynchronous Loggers internally use the Disruptor, a lock-free inter-thread communication library, instead of queues, resulting in higher throughput and lower latency.
• Streams for IO Operation (Google Protocol Buffer for object serialization and deserialization) Protocol buffers, usually referred as Protobuf, is a protocol developed by Google to allow serialization and deserialization of structured data. this protocol even surpassed JSON with better performance, better maintainability and smaller size
• Java Design Patterns to manage the objects efficiently - Flyweight design pattern to create a pool of shared objects , static factory methods instead of constructors to recycle immutable objects , visitor pattern to avoid “instanceof” constructs in frequently accessed methods. 
• Choosing the Right Garbage Collector - However, the current generation of garbage collectors has mostly solved that issue and, with proper tuning and sizing, can lead to having no noticeable collection cycles. That being said, it does take an in-depth understanding of both GC on the JVM as a whole, but also the specific profile of the application – to get there
• JDBC Performances - JDBC Connection pooling (The creation of a new connection takes time, which you can avoid if you reuse an existing connection.) ,JDBC Batching (we handle persistence is trying to batch operations wherever possible. JDBC batching allows us to send multiple SQL statements in a single database roundtrip.),Statement Caching (Depending on the underlying JDBC Driver, you can cache PreparedStatement both on the client-side (the Driver) or databases-side.
• Architectural Improvements - If you write applications with poor architecture but performs well for the current requirements, what will happen if the requirements grow and your architecture is not flexible enough to extend and creates a maintenance nightmare where fixing a code in one area would break your code in another area. This will cause your application to be re-written.

Most applications need to retrieve data from and save/update data into one or more databases. Database calls are remote calls over the network. In general data should be lazily loaded (i.e. load only when required as opposed to pre-loading from the database with a view that it can be used later) from a database to conserve memory but there are use cases (i.e. need to make several database calls) where eagerly loading data and caching can improve performance by minimizing network trips to the database. Data can be eagerly loaded with a help of SQL scripts with complex joins or stored procedures and cached using third party frameworks or building your own framework.

How would you refresh Infinispan 2nd level remote cache?

• Timed cache strategy where the cache can be replenished periodically (i.e. every 30 minutes, every hour etc). This is a simple strategy applicable when it is acceptable to show dirty data at times and also the data in the database does not change very frequently.
• Dirty check strategy where your application is the only one which can mutate (i.e. modify) the data in the database. You can set a “isDirty” flag to true when the data is modified in the database through your application and consequently your cache can be refreshed based on the “isDirty” flag.

How would you refresh your cache(distributed data grid ) if your database is shared by more than one application ?

• Database triggers: You could use database triggers to communicate between applications sharing the same database and write pollers which polls the database periodically to determine when the cache should be refreshed.
• XML messaging (Enterprise – JMS) to communicate between other applications sharing the same database or separate databases to determine when the cache should be refreshed.
• Distributed platform : Infinispan distributing data evenly across the cluster and use it from different language i,e language-independent service accessed remotely over a variety of protocols (Hot Rod, REST, Memcached and WebSockets),The purpose of Infinispan is to expose a data structure that is distributed, highly concurrent and designed ground-up to make the most of modern multi-processor and multi-core architectures. It is often used as a distributed cache, but also as a NoSQL key/value store or object database.

Optimize your I/O operations: 

Use buffering when writing to and reading from files and/or streams. Avoid writers/readers if you are dealing with only ASCII characters. You can use streams instead, which are faster. Avoid premature flushing of buffers. Also make use of the performance and scalability enhancing features such as non-blocking and asynchronous I/O, mapping of file to memory etc offered by the NIO (New I/O).

Establish whether you have a potential memory problem and manage your objects efficiently: 

1). Remove references to the short-lived objects from long-lived objects like Java collections etc to minimize any potential memory leaks. Also reuse objects where possible. It is cheaper to recycle objects than creating new objects each time. 

2). Avoid creating extra objects unnecessarily. For example use mutable StringBuffer/StringBuilder classes instead of immutable String objects in computation expensive loops  and use static factory methods instead of constructors to recycle immutable objects. 

3). Automatic garbage collection is one of the most highly touted conveniences of Java. However, it comes at a price. Creating and destroying objects occupies a significant chunk of the JVM's time. Wherever possible, you should look for ways to minimize the number of objects created in your code:

• For complex objects that are used frequently, consider creating a pool of recyclable objects rather than always instantiating new objects. This adds additional burden on the programmer to manage the pool, but in selected cases it can represent a significant performance gain. Use flyweight design pattern to create a pool of shared objects. Flyweights are typically instantiated by a flyweight factory that creates a limited number of flyweights based on some criteria. Invoking object does not directly instantiate flyweights. It gets it from the flyweight factory, which checks to see if it has a flyweight that fits a specific criteria (e.g. with or without GST etc) in the pool (e.g. HashMap). If the flyweight exists then return the reference to the flyweight. If it does not exist, then instantiate one for the specific criteria and add it to the pool (e.g. HashMap) and then return it to the invoking object.

• If repeating code within a loop, avoid creating new objects for each iteration. Create objects before entering the loop (i.e. outside the loop) and reuse them if possible.

• Use lazy initialization when you want to distribute the load of creating large amounts of objects. Use lazy initialization only when there is merit in the design. 

Where applicable apply the following performance tips in your code: 

• Use ArrayLists, HashMap etc as opposed to Vector, Hashtable etc where possible. This is because the methods in ArrayList, HashMap etc are not synchronized. Even better is to use just arrays where possible. 

• Using StringBuilder for String Concatenation - String concatenation is a very common operation, and also an inefficient one. Simply put, the problem with using += to append Strings is that it will cause an allocation of a new String with every new operation.

• Use + to concatenate Strings in in one statement - When you implemented your first application in Java, someone probably told you that you shouldn’t concatenate Strings with +. And that’s correct if you’re concatenating Strings in your application logic. Strings are immutable, and the result of each String concatenation is stored in a new String object. That requires additional memory and slows down your application, especially if you’re concatenating multiple Strings within a loop.

• Use primitives where possible - Another quick and easy way to avoid any overhead and improve the performance of your application is to use primitive types instead of their wrapper classes. So, it’s better to use an int instead of an Integer, or a double instead of a Double. That allows your JVM to store the value in the stack instead of the heap to reduce memory consumption and overall handle it more efficiently.

• Avoid Recursion - Recursive code logic leading to StackOverFlowError is another common scenario in Java applications.If we cannot do away with recursive logic, tail recursive as an alternative is better.

• Use Regular Expressions Carefully , Regular expressions are useful in a lot of scenarios, but they do, more often than not, have a very high performance cost. It’s also important to be aware of a variety of JDK String methods, which use regular expressions, such as String.replaceAll(), or String.split().

• Avoid Creating and Destroying Too Many Threads (uses a pool of threads called the ForkJoinPool, which manages the worker threads)- Creating and disposing of threads is a common cause of performance issues on the JVM, as thread objects are relatively heavy to create and destroy.If your application uses a large number of threads, using a thread pool makes a lot of sense, to allow these expensive objects to be reused.To that end, the Java ExecutorService is the foundation here and provides a high-level API to define the semantics of the thread pool and interact with it.

• Set the initial capacity of a collection (e.g. ArrayList, HashMap etc) and StringBuffer/StringBuilder appropriately. This is because these classes must grow periodically to accommodate new elements. So, if you have a very large ArrayList or a StringBuffer, and you know the size in advance then you can speed things up by setting the initial size appropriately.

• Minimize the use of casting or runtime type checking like instanceof in frequently executed methods or in loops. The “casting” and “instanceof” checks for a class marked as final will be faster. Using “instanceof” construct is not only ugly but also unmaintainable. Look at using visitor pattern to avoid “instanceof” constructs in frequently accessed methods. 

• Do not compute constants inside a large loop. Compute them outside the loop. For applets compute it in the init() method. Avoid nested loops (i.e. a “for” loop within another “for” loop etc) where applicable and make use of a Collection class. 

• Exception creation can be expensive because it has to create the full stack trace. The stack trace is obviously useful if you are planning to log or display the exception to the user. But if you are using your exception to just control the flow, which is not recommended, then throw an exception, which is precreated. An efficient way to do this is to declare a public static final Exception in your exception class itself.

• Avoid using System.out.println and use logging frameworks like Log4J2 etc, which uses Asynchronous I/O buffers.

• Minimize calls to Date, Calendar, etc related classes.

• Minimize JNI calls in your code

When in the development process should you consider performance issues? 

Set performance requirements in the specifications, include a performance focus in the analysis and design and also create a performance test environment.

When designing your new code, what level of importance would you give to the following attributes? 

• Performance
• Maintainability
• Extendibility
• Ease of use
• Scalability

You should not compromise on architectural principles for just performance. You should make effort to write architecturally sound programs as opposed to writing only fast programs. If your architecture is sound enough then it would allow your program not only to scale better but also allows it to be optimized for performance if it is not fast enough. So you should think about extendibility (i.e. ability to evolve with additional requirements), maintainability, ease of use, performance and scalability (i.e. ability to run in multiple servers or machines) during the design phase. List all possible design alternatives and pick the one which is conducive to sound design architecturally (i.e. scalable, easy to use, maintain and extend) and will allow it to be optimized later if not fast enough.

How would you detect and minimize memory leaks in Java?

In Java, memory leaks are caused by poor program design where object references are long lived and the garbage collector is unable to reclaim those objects.

Detecting memory leaks:

• Use tools like JProbe, OptimizeIt etc to detect memory leaks.
• Use operating system process monitors like task manager on NT systems, ps, vmstat, iostat, netstat etc on UNIX systems.
• Write your own utility class with the help of totalMemory() and freeMemory() methods in the Java Runtime class. Place these calls in your code strategically for pre and post memory recording where you suspect to be causing memory leaks. An even better approach than a utility class is using dynamic proxies or Aspect Oriented Programming (AOP) for pre and post memory recording where you have the control of activating memory measurement only when needed.

Minimizing memory leaks:

In Java, typically memory leak occurs when an object of a longer lifecycle has a reference to objects of a short life cycle. This prevents the objects with short life cycle being garbage collected. The developer must remember to remove the references to the short-lived objects from the long-lived objects. Objects with the same life cycle do not cause any issues because the garbage collector is smart enough to deal with the circular references.

• Design applications with an object’s life cycle in mind, instead of relying on the clever features of the JVM. Letting go of the object’s reference in one’s own class as soon as possible can mitigate memory problems. Example: myRef = null;
• Unreachable collection objects can magnify a memory leak problem. In Java it is easy to let go of an entire collection by setting the root of the collection to null. The garbage collector will reclaim all the objects (unless some objects are needed elsewhere). 
• Use weak references if you are the only one using it. The WeakHashMap is a combination of HashMap and WeakReference. This class can be used for programming problems where you need to have a HashMap of information, but you would like that information to be garbage collected if you are the only one referencing it. 
• Free native system resources like AWT frame, files, JNI etc when finished with them. Example: Frame, Dialog, and Graphics classes require that the method dispose() be called on them when they are no longer used, to free up the system resources they reserve.

Why does the JVM crash with a core dump or a Dr.Watson error?

Any problem in pure Java code throws a Java exception or error. Java exceptions or errors will not cause a core dump (on UNIX systems) or a Dr.Watson error (on WIN32systems). Any serious Java problem will result in an OutOfMemoryError thrown by the JVM with the stack trace and consequently JVM will exit. These Java stack traces are very useful for identifying the cause for an abnormal exit of the JVM. So is there a way to know that OutOfMemoryError is about to occur? The Java J2SE 5.0 has a package called java.lang.management which has useful JMX beans that we can use to manage the JVM. One of these beans is the MemoryMXBean. 

An OutOfMemoryError can be thrown due to one of the following 4 reasons:

• JVM may have a memory leak due to a bug in its internal heap management implementation. But this is highly unlikely because JVMs are well tested for this.
• The application may not have enough heap memory allocated for its running. You can allocate more JVM heap size (with –Xmx parameter to the JVM) or decrease the amount of memory your application takes to overcome this. To increase the heap space: java -Xms1024M -Xmx1024M Care should be taken not to make the –Xmx value too large because it can slow down your application. The secret is to make the maximum heap size value the right size.
• Another not so prevalent cause is the running out of a memory area called the “perm” which sits next to the heap. All the binary code of currently running classes is archived in the “perm” area. The ‘perm’ area is important if your application or any of the third party jar files you use dynamically generate classes. For example: “perm” space is consumed when XSLT templates are dynamically compiled into classes, J2EE application servers, JasperReports, JAXB etc use Java reflection to dynamically generate classes and/or large amount of classes in your application. To increase perm space: java -XX:PermSize=256M -XX:MaxPermSize=256M
• The fourth and the most common reason is that you may have a memory leak in your application when an object of a longer lifecycle has a reference to objects of a short life cycle.

So why does the JVM crash with a core dump or Dr.Watson error ? 

Both the core dump on UNIX operating system and Dr.Watson error on WIN32 systems mean the same thing. The JVM is a process like any other and when a process crashes a core dump is created. A core dump is a memory map of a running process. This can happen due to one of the following reasons:

• Using JNI (Java Native Interface) code, which has a fatal bug in its native code. Example: using Oracle OCI drivers, which are written partially in native code or JDBC-ODBC bridge drivers, which are written in non Java code. Using 100% pure Java drivers (communicates directly with the database instead of through client software utilizing the JNI) instead of native drivers can solve this problem. We can use Oracle thin driver, which is a 100% pure Java driver.
• The operating system on which your JVM is running might require a patch or a service pack. 
• The JVM implementation you are using may have a bug in translating system resources like threads, file handles, sockets etc from the platform neutral Java byte code into platform specific operations. If this JVM’s translated native code performs an illegal operation then the operating system will instantly kill the process and mostly will generate a core dump file, which is a hexadecimal file indicating program’s state in memory at the time of error. The core dump files are generated by the operating system in response to certain signals. Operating system signals are responsible for notifying certain events to its threads and processes. The JVM can also intercept certain signals like SIGQUIT which is kill -3 < process id > from the operating system and it responds to this signal by printing out a Java stack trace and then continue to run. The JVM continues to run because the JVM has a special built-in debug routine, which will trap the signal -3. On the other hand signals like SIGSTOP (kill -23 ) and SIGKILL (kill -9 ) will cause the JVM process to stop or die. The following JVM argument will indicate JVM not to pause on SIGQUIT signal from the operating system.

          java –Xsqnopause

List of performance measurement tools in java:

No.	Tool  Name	Purpose
1	jVisualVM	JVM Profiler
2	GCeasy	Universal Garbage Collection Log Analysis
3	fastThread	Java Thread Dump Analyzer
4	HeapHero	Java Heap Dump Analyzer
5	sar	Real time monitoring Linux System Performance
6	ksar	Java based frontend tool which plots a nice easy to understand graph over a period of time
7	JMeter	A pure Java application designed to load test functional behavior and measure performance.
8	jvmtop	Provide JVM internals (e.g. memory information) of running JVMs / java processes.
9	MAT	Eclipse Memory Analyzer is a fast and feature-rich Java heap analyzer
10	jstack	Thread Dump - Java HotSpot VM to provide information about performance and resource consumption of running applications
11	jmap	Heap Dump - Take histogram and heap dump from running java process.
12	jstat	Runtime JVM statistics monitoring using command line
13	Visual GC + jstatd	Visual Garbage Collection Monitoring Tool using server side jstatd binding port
14	JMH	JMH is a Java harness for building, running, and analysing nano/micro/milli/macro benchmarks written in Java and other languages targetting the JVM

Reference URL’s:

• http://gceasy.io/
• http://fastthread.io/
• http://heaphero.io/
• https://www.eclipse.org/mat/
• https://code.google.com/archive/p/jvmtop/
• https://visualvm.github.io/
• http://openjdk.java.net/projects/code-tools/jmh/