Jumbo Frames are a LAN-side performance feature, while MTU clamping in SD-WAN is a WAN-side compatibility safeguard.

 Jumbo Frames are completely separate from standard MTU and not affected by our standard MTU clamping — unless we explicitly allow and use Jumbo Frames end-to-end. When we clamp standard MTU from 1420 to 1300, we're only affecting regular Ethernet frames (standard traffic).

Jumbo Frames (LAN-side Optimization)

• Ethernet frames larger than the standard 1500 bytes, typically up to 9000 bytes.
• Primarily used within high-speed internal networks (e.g., data centers) to reduce CPU overhead and improve throughput.
• Jumbo frames require explicit support and configuration end-to-end across all network devices — not used over the public internet or SD-WAN circuits by default.

Aspect	Standard MTU	Jumbo Frames
Typical Size	1500 (often clamped to 1300–1420)	9000 (custom, non-standard)
Used by	Most Internet and enterprise traffic	Only within high-speed internal LANs / data centers
Needs Clamping?	Yes, for tunnels/overheads (e.g., IPsec, GRE, SD-WAN)	No (unless you're running Jumbo over the same constrained path, which is rare)
Negotiated?	Yes (Path MTU Discovery, MSS)	No — must be manually configured and supported end-to-end
TSO / LSO Impact	Still applies	Jumbo frame simply increases physical wire frame size

Clamping standard MTU does not impact Jumbo Frames, because :

• Jumbo Frames work only inside a Local Area Network (LAN) or similarly controlled environments like: Data centers, Storage networks (NAS/iSCSI), High-performance computing clusters.
• Operate in separate L2 environments (e.g., within LAN, not over WAN tunnels)
• Are already constrained to trusted, known MTU paths.
• Jumbo Frames are not “clamped” in the same sense, because:
• They must be explicitly allowed and configured on the interface (NIC, switch, router).
• They don’t rely on PMTUD — they’re rejected outright if any hop doesn't support them.

Here is 3 technical concepts : 

Concept	Scope	Purpose
PMTUD (Path MTU Discovery)	🌐 WAN	Dynamically discovers smallest MTU along a path to avoid fragmentation
TSO (TCP Segmentation Offload)	🌐 WAN + LAN (mostly host NIC)	Offloads TCP segmentation to NIC to boost performance for large buffers
Jumbo Frames	🏢 LAN	Increases Ethernet frame size to reduce per-packet overhead in LAN

1. PMTUD (Path MTU Discovery)

This is our current process by default where DF (Don't Fragment) is always set by default , where we can see more than 25000 bytes are send without any fragmentation , even we never see any where with Fragmentation is enabled. As most OS supports the TCP Segmentation Offload (TSO) or Large Segment Offload (LSO).

Client  exchange the MSS during the TCP 3-way handshake. "MSS is advisory, not enforced.  But the sender is not obliged to obey."

The OS kernel or TLS layer may send larger chunks for performance or protocol reasons. The sender relies on PMTUD (with DF set) to discover if large packets are acceptable along the path.  TCP stacks may choose to initially send large segments, assuming large MTUs, unless PMTUD fails.

Path MTU Discovery (PMTUD) is a mechanism that helps a sender find the maximum IP packet size (MTU) that can traverse the network without fragmentation. Each link in a network path can have a different MTU.  PMTUD helps avoid:

• Sending packets too big, which would get dropped if DF (Don't Fragment) is set
• The overhead of IP fragmentation, which can hurt performance

How PMTUD Works :

• Sender sets the DF bit (Don't Fragment) in IP packets.
• Starts sending large packets (e.g., 1500 bytes).
• If a router encounters a too-large packet, and can't fragment:
• It drops the packet.
• Sends back an ICMP Type 3, Code 4: “Fragmentation Needed but DF Set”.
• The sender receives this ICMP and reduces packet size accordingly.

2. TCP Segmentation Offload (TSO) 

Technically A TCP segment with a payload size larger than the path MTU, constructed by the OS kernel and offloaded to the network card for segmentation. This is called as TCP Segmentation Offload (TSO)” or “Large Segment Offload (LSO)".

• TSO is a hardware-level optimization, handled by the NIC, not part of the IP protocol stack.
• If TSO is enabled on a host's network interface , it is used for all outbound TCP traffic (LAN or WAN).
• The OS creates a huge TCP segment (e.g., 64KB or more). TSO allows the OS to send large TCP buffers (e.g., 64KB) to NIC, which NIC splits it into MTU-sized packets (e.g., 1500 bytes) for transmission over the network.
• Instead of the CPU breaking the data into small packets, the NIC takes over the segmentation task, reducing CPU overhead and improving throughput.

✅ Will TSO Still Work After MTU Clamping?

Yes, TSO/LSO will still function as expected, but the size of the individual MTU-sized frames generated by the NIC will be adjusted based on the clamped MTU.  We can safely clamp MTU/MSS at the edge or adjust tunnel overhead without breaking TSO.  Just ensure : Firewalls/routers don’t apply hard limits below our clamped MTU.

TCP Segmentation Offload (TSO) — also known as Large Segment Offload (LSO) — is a network performance optimization technique used in modern operating systems and network interface cards (NICs).

With TSO enabled:

• The OS creates a large TCP segment (say 64KB).
• The NIC is responsible for breaking it into packets that match the effective MTU.
• If you've clamped the MTU to 1290 or 1383 (instead of 1500), the NIC will now segment based on the new lower MTU.
• This increases the number of packets per large segment but does not break functionality.

📦 Without TSO:

• OS creates and segments each TCP packet.
• CPU is involved in calculating checksums, headers, segmentation.
• More CPU cycles used, especially at high throughput.

📊 Visual Representation:

Layer	Without TSO	With TSO
OS (Kernel)	Segments into 1460-byte TCP segments	Sends 64KB chunk to NIC
NIC (Hardware)	Just transmits	Splits into MTU-sized packets (e.g., 1460B)
Result	More CPU work, more interrupts	Less CPU work, higher performance

✅ Benefits of TSO:

• Reduces CPU load for high-volume TCP traffic.
• Enables higher throughput without additional CPU consumption.
• Particularly useful for web servers, file transfers, high-performance applications.

 

Term	Layer	Description
PMTUD	Network	Discovers the largest IP packet that can be sent without fragmentation
MSS (TCP)	Transport	Max TCP payload advertised by each endpoint
MTU (IP)	Network	Max IP packet size supported by each link
TSO/LSO	NIC	Offloads large TCP segments to be split at NIC level
Jumbo Frames	Ethernet	Frames > 1500 bytes (e.g., 9000B) — only supported on compatible networks
Jumbo TCP Segment	TCP/OS	Large segments seen in capture due to TSO — not actual on-wire sizes

 

  Impact of Lowered MTU:

• More packets per TSO segment → slightly higher CPU/network processing overhead.
• Possible minor reduction in throughput if NIC offloading is limited or CPU is under pressure.
• But TLS handshake reliability improves significantly, which is more critical in your case.

Parameter	Before Clamping (MTU=1500)	After Clamping (e.g., MTU=1290–1383)
TSO Segment Size (e.g., 64KB)	64KB	64KB
NIC Segments per TSO Segment	Fewer (64KB ÷ 1460)	More (64KB ÷ 1250 or similar)

🚫 TSO Could Break Only If:

• TSO/LSO is explicitly disabled at OS or NIC level.
• The network path drops fragmented packets (rare with TSO since fragmentation doesn’t occur at IP layer).
• There's a misconfigured firewall/router that drops larger packet bursts or doesn’t respect the new clamped MTU.

 ✅ Recommendation:

You can safely clamp MTU/MSS at the edge or adjust tunnel overhead without breaking TSO. Just ensure:

• NIC and OS TSO settings remain enabled.
• Firewalls/routers don’t apply hard limits below your clamped MTU.
• Your monitoring considers potential rise in packet counts per request (not errors).

 

3. Jumbo Frames

Jumbo Frames are Ethernet frames that exceed the standard Ethernet MTU of 1500 bytes. They are typically configured to carry up to 9000 bytes of payload (sometimes more), allowing more data per packet on high-speed networks. They must be manually enabled and matched end-to-end — and are not suitable for WAN or tunneled traffic.

Common jumbo frame sizes are 9000 bytes, but they can range from 1500 to 9600 bytes depending on the device. 

• Standard Ethernet MTU: 1500 bytes (payload)
• Jumbo Frame MTU: Typically 9000 bytes (payload)
• Jumbo Frame Ethernet Frame Size: ~9018 bytes (includes Ethernet headers and CRC)

🎯 Why Use Jumbo Frames?

• Lower CPU overhead: Fewer packets for the same amount of data = fewer interrupts and TCP/IP stack processing per byte.
• Improved throughput: Larger payloads per packet improve the efficiency of high-speed networks (e.g. 10GbE, 40GbE).
• Better for large transfers: Especially beneficial in storage networks (iSCSI, NFS), data center replication, and video streaming.

  ⚙️ How They Interact with TSO

• When TSO and Jumbo Frames are both enabled:
• OS sends a large segment (e.g. 64 KB) to the NIC (via TSO).
• NIC splits the segment into larger packets due to the higher MTU.

Example :

Setting	Packet Size on Wire	Packets for 64 KB
MTU 1500	~1460 bytes (TCP payload)	~44 packets
MTU 9000 (Jumbo)	~8960 bytes (TCP payload)	~8 packets

So with Jumbo Frames:

• Fewer segments need to be generated.
• Less per-packet overhead.
• Faster total data transfer with less CPU load.

 When to Use Jumbo Frames

In controlled environments like data centers, HPC clusters, and backend storage networks . When all devices between sender and receiver are:

• In the same broadcast domain or VLAN
• Configured with matching Jumbo MTU (e.g., 9000)

🔬 When Not to Use Jumbo Frames

• Not all network devices support it:
  
• Switches, routers, firewalls, VPN appliances, etc. must all support and be configured for the same jumbo MTU.
• Path MTU Discovery (PMTUD) can fail if ICMP is blocked:
• Leading to black holes or silent packet drops if an intermediate hop can't handle jumbo frames.
• Debugging becomes harder if partial support exists along the path.
• Increased latency per packet in some workloads (e.g., small request/response).
• Over the Internet , Over VPNs / SD-WAN / IPsec / GRE tunnels
• Where intermediate devices may silently drop large frames
• On consumer-grade or mixed-vendor networks

How MTU Works in Jumbo Frames

• Sender checks MTU: If sender NIC is configured for Jumbo MTU (e.g., 9000), and the application generates large enough data, the OS/driver builds Jumbo Frames.
• No fragmentation needed — if the entire path (Layer 2 switches, NICs) supports the larger MTU.

📊 Jumbo Frames and TSO

Feature	Effect
Jumbo Frame	Increases wire MTU (e.g. 9000), fewer packets needed
TSO	Sends large 64KB+ buffers to NIC; NIC segments
Jumbo + TSO	NIC splits large TSO buffers into large MTU-sized packets (e.g. 9000 bytes)
Combined Result	Lower CPU use, fewer packets, better throughput

🛠️ Steps to Identify Jumbo Frames in Wireshark

✅ 1. Look at "Frame" and "Ethernet II" Lengths

Each captured packet will show you:

• Frame Length on the Wire (includes L2 headers)
• Payload length (Ethernet MTU)
  
  
• Standard Ethernet Frame: ~1514 bytes (MTU 1500 + Ethernet header 14)
• Jumbo Frame: >1518 bytes (e.g., 9014 for MTU 9000)

✅ 2. Filter for large frames

Use the Wireshark display filter, To see only Jumbo Frames.:

frame.len > 1514

 

Java ClassLoader Deep-Dive and Real-World Behavior in Payara

Java Class Loaders: Deep-Dive Reference Guide

1. What is a Java Class Loader?

A Java ClassLoader is a part of the Java Runtime Environment (JRE) responsible for dynamically loading classes into the Java Virtual Machine (JVM) during runtime.
Java applications do not need to know the physical file location of classes in advance. The classloader handles locating, loading, and linking class bytecode as needed.

Key Characteristics:

• Classes are loaded only when first referenced.
• Each class is loaded by a specific class loader instance.
• Classes loaded by different class loaders are considered different even if they are identical bytecode.
  

---

2. How Java ClassLoader Works

When a class is needed, JVM follows this flow:

• Delegation Model: Delegates class loading request to its parent before attempting to load itself.
• Namespace Isolation: Every ClassLoader has a separate namespace.
• Linking & Verification: Once a class is loaded, it's verified and linked.

Class<?> myClass = Class.forName("com.example.MyClass");

Here, Class.forName internally uses the current thread’s context class loader to locate and load the class.

When a class is referenced, the JVM asks the classloader to find it. The classloader:

• Locates the .class file.
• Loads it into memory.
• Verifies its structure and bytecode.
• Prepares memory for static fields.
• Resolves references to other classes.

Each loaded class is associated with the classloader that loaded it. Multiple versions of the same class can coexist in the JVM if loaded by different classloaders. 

Delegation Model: 

When a class loader receives a request to load a class, it first delegates the request to its parent. If the parent can't load it, only then the current loader attempts.

 📌 Output Insight: String is part of the core JDK, so even the App ClassLoader delegates it to the Bootstrap ClassLoader (hence null).

Namespace Isolation:

Each class loader maintains its own set of loaded classes. Same class name loaded by different loaders are considered different. 

 

📌 Real-world Use: This is how application servers isolate applications (WARs) using different class loaders.

Linking & Verification

After loading, the JVM links the class — verification (for bytecode safety), preparation (static fields), and resolution (symbolic -> direct references).

📌 Note: When Class.forName() is called, the JVM loads and links VerificationExample. The static block runs during class initialization.

---

3. Types of Class Loaders (Traditional)

3.1. Bootstrap ClassLoader (Primordial)

• Loads core Java classes (e.g., java.lang.*, java.util.*).
• Implemented in native code.
• Loads from JAVA_HOME/lib.

3.2. Extension (Platform) ClassLoader

• Loads classes from JAVA_HOME/lib/ext or the modules path.
• Also known as PlatformClassLoader from Java 9 onwards.
  

3.3. Application (System) ClassLoader

• Loads classes from the classpath (e.g., -cp, CLASSPATH env var).
• Default loader for user applications.

3.4. Custom ClassLoaders

• Subclasses of java.lang.ClassLoader.
• Used for module isolation, reloading classes, secure loading.
  

ClassLoader myLoader = new MyCustomClassLoader();

---

4. Modern ClassLoader Enhancements

Java 9+ Modular System

• AppClassLoader, PlatformClassLoader, and BuiltinClassLoader under jdk.internal.loader.
• Strong encapsulation in Jigsaw (Module System).
  

URLClassLoader

• Loads classes and resources from file system or remote URLs.
  

Thread Context ClassLoader

• Often used in frameworks (e.g., JNDI, JAXB, JDBC) 

---

5. ClassLoader Hierarchy

Bootstrap ClassLoader
      ↓
Platform/Extension ClassLoader
      ↓
Application ClassLoader
      ↓
(Custom or Container ClassLoaders)

Example:

If a class com.abc.MyClass is present in both /app/lib/ and JAVA_HOME/lib/ext, it will be loaded by the PlatformClassLoader.

---

6. Thumb Rules in Class Loading

• Parent-first delegation: All standard loaders follow this.
• Child-first for web apps (e.g., Tomcat/Payara WAR loaders).
• Duplicate class versions must be isolated by class loaders.
• JAR conflict leads to ClassCastException or LinkageError.
• Use -verbose:class or -Xlog:class+load=info to trace loading.

---

7. Application Server ClassLoaders (Payara / GlassFish)

Types:

• Bootstrap: JVM internal classes.
• OSGi Module ClassLoader: Payara modules.
• Domain ClassLoader: /domains/domain1/lib
• WebApp ClassLoader: /applications/<app>/WEB-INF/lib

WebApp ClassLoader Behavior:

• delegate="true" (default): Tries parent loaders first. Server classes override application classes.
• delegate="false": Tries WAR classes first, then domain/lib, then modules. Application JARs override server classes.
• Classes loaded from WEB-INF/lib are unique to the app.
• If a class is available in domain1/lib or /modules, it is shared.
• Class conflicts often occur due to overlapping libraries. 

Payara Server ClassLoader (aka Server ClassLoader)

• What it is: This is the OSGi ClassLoader used internally by Payara to load core services, admin modules, and libraries from the /modules and /modules/autostart directories.
• Also includes: Other internal framework components like Hazelcast, HK2, Jersey, Jackson (default versions), and any bundled JARs marked as OSGi bundles.

        

• Used to load: Payara services (transaction manager, resource adapters, monitoring, etc.). Admin CLI and console plugins. Any JARs Payara bundles as default runtime
• ⚠️ Important note: Classes loaded here are not visible to application code unless explicitly exported via OSGi metadata. 

Application ClassLoader (Web or EAR ClassLoader)

• What it is: This is the Web Application ClassLoader, used to load application-specific classes and libraries.

         

• Used to load: Your app’s .class files. Libraries in WEB-INF/lib/ (including your version of libraries like nimbus-jose-jwt, bcprov, jackson, etc.) 
• Parent delegation: Delegates to Domain ClassLoader → which delegates to OSGi ClassLoader. Respects delegate="true" or delegate="false" rules (configured in glassfish-web.xml)
  

Example:

<class-loader delegate="false"/>

• In this case, even if hk2-api.jar is present in /modules, it will be loaded from WEB-INF/lib if found. 

---

8. Real-World Debugging Case: Nimbus JOSE + BouncyCastle

🔍 Problem

• App failed with ClassNotFoundException: org.bouncycastle.util.encoders.Base64.
• Nimbus JAR was in domain1/lib.
• BouncyCastle JARs (bcprov, bcpkix) were also in domain1/lib.
• <class-loader delegate="false"/> was active. 

🧠 Root Cause

Nimbus loaded via domain classloader, but BouncyCastle was loaded via WEB-INF/lib. Due to different classloaders, Nimbus could not access BouncyCastle.

✅ Resolution

Moved nimbus-jose-jwt-10.0.1.jar to WEB-INF/lib — so both libraries shared the same WebApp classloader, resolving the error.

9. ClassLoader Delegation Model

The JVM follows a parent-first delegation model by default:

 AppClassLoader -> ExtClassLoader -> BootstrapClassLoader

However, application servers like Payara allow configuring child-first via:

<class-loader delegate="false"/>

 This prioritizes WEB-INF/lib before domain/lib and modules.

 

 

🔍 Summary:

• Each classloader delegates upward to its parent.
• Parents cannot access children’s classes (i.e., OSGi cannot see Domain, Domain cannot see WebApp).
• If delegate="true", WebApp CL first checks Domain → OSGi → System → Bootstrap.
• If delegate="false", WebApp CL tries to load the class itself before asking parents.

 

⚠️ Risks / Considerations:

• If your JAR tries to reference another class not in its own scope (e.g., Nimbus referencing BouncyCastle in domain1/lib), and visibility is broken, you'll get ClassNotFoundException or NoClassDefFoundError.
• May lead to duplication if classes are also loaded by parent classloaders.
• Must ensure dependencies (like BouncyCastle) are placed where they’re visible to the same loader or higher up in the hierarchy.

 

OSGi-Bundled JARs (in /modules) may load themselves first — regardless of delegation

• Some OSGi-enabled JARs (e.g., nimbus-jose-jwt, jersey-*, jackson-*) are registered and activated by Payara at server startup.
• These become active bundles managed by the OSGi ClassLoader.
• Even with delegate="false", if your application references a class that’s already loaded and cached by an OSGi bundle, your version in WEB-INF/lib might get ignored or conflict at runtime.

Class Visibility Still Respects ClassLoader Hierarchy

• Let's say your app uses Nimbus (from WEB-INF/lib), and Nimbus internally calls Class.forName("org.bouncycastle...").
• If bcprov is not in WEB-INF/lib, but only in /domain1/lib, the child classloader (WebApp CL) can delegate up to Domain CL — but not downward.
• If Nimbus was mistakenly loaded by OSGi CL (because of duplication), it cannot see bcprov in your WAR → leading to ClassNotFoundException.

OSGi Enforces Strict Bundle Isolation

• OSGi bundles have explicit Import-Package and Export-Package constraints.
• Even with child-first loading, your WAR classloader cannot override what OSGi bundles expose internally, unless:
• You completely isolate the JARs (e.g., move all to WEB-INF/lib),
• Or avoid duplicating sensitive JARs like Jersey, Jackson, Nimbus, etc. 

OSGi & Classloader Conflicts in Payara

• JARs in /domain1/lib are loaded by the Domain ClassLoader.
• JARs in /modules are loaded by the OSGi ClassLoader.
• If a class exists in both /domain1/lib and /modules, and Payara has already loaded it from modules (via OSGi), it will not load the one from domain1/lib — even though delegate="false". 
• Note : Payara pre-wires and initializes OSGi services before web apps are loaded. So those classes are already loaded by the time your app starts.
  

 

 

🔷 What is OSGi?
OSGi is a dynamic module system for Java. It allows applications to be composed of many smaller components (called bundles) that can be installed, started, stopped, updated, and uninstalled without requiring a reboot. 

 

🔷 Why OSGi?

• In traditional Java applications:
• All classes exist in a flat classpath.
• No modularity, version isolation, or runtime dynamism.

OSGi solves this by:

• Enabling true modularity (split code into independently deployable bundles).
• Enforcing explicit dependencies (via manifest).
• Allowing hot updates of individual components.
• Providing isolation and controlled class visibility between modules. 

🔷 Core Concepts 

 

🔷 Bundle Example 

MANIFEST.MF inside an OSGi bundle:

 

This declares:

• This bundle is named com.example.payment.
• It exports its api package.
• It imports packages it depends on (not full JARs!). 

🔷 OSGi Class Loading Hierarchy
OSGi uses per-bundle class loaders, unlike Java EE’s global or WAR-level class loaders.

Key rules:

• Each bundle has its own classloader.
• A bundle can only access: Its own classes. Classes from packages it explicitly imports.
• Even if two bundles contain the same class, they're different types if loaded separately. 

  

🔷 How OSGi is Used in Payara / GlassFish

• Payara modules under /modules follow an OSGi-style packaging:
• JARs include OSGi metadata in MANIFEST.MF.
• The server uses a HK2 + OSGi hybrid model to isolate internal components.
• Example: hk2-locator.jar, jersey-server.jar — all export/import packages rather than relying on flat classpath

 

🔷 Real-World Scenario: Why ClassNotFoundException Happens with OSGi
Scenario:

• You place nimbus-jose-jwt.jar in /domain1/lib, but it’s already in /modules as an OSGi bundle. Now:
• If your WAR uses <class-loader delegate="false"/>, it skips module loader.
• OSGi's bundle loader doesn't export internal packages to domain/lib or web apps.
• Hence, you must include it in WEB-INF/lib to make it available within your app’s classloader scope. 

 

 🔷 OSGi in Practice: Tools

 

 

 

🔷 Summary

• OSGi enables versioned, isolated, hot-swappable Java modules.
• It solves complex classpath issues in large applications.
• It’s widely used in Java EE servers like Payara, Eclipse IDE, and IoT devices.
• Requires rethinking application structure — avoid flat lib/, use package-level import/export. 

The ./asadmin list-modules command is used in Payara Server (or GlassFish) to list all the OSGi modules that are currently registered and available in the server’s module system.

 

You get a list of installed HK2/OSGi modules, which are essentially internal system components and libraries that are managed as modules by the server. These modules are typically loaded from the server’s /modules directory and include:

• Jersey (REST framework)
• Jackson (JSON processing)
• HK2 (dependency injection)
• Jakarta EE APIs (Servlet, JAX-RS, JAXB, etc.)
• BouncyCastle (for crypto)
• Others like Grizzly, Weld, Mojarra (JSF), etc. 

 

📍 Use Cases

• Debug classloading issues : Helps confirm whether a particular component (e.g., jersey-server, hk2-locator, etc.) is available via Payara's module system.
• Avoid duplication: Before placing a JAR in domain1/lib or WEB-INF/lib, check if it's already loaded as a module.
• Verify OSGi version: Some modules may be versioned — this command shows exact versions being loaded.
• Confirm availability of internal APIs : Especially useful when <class-loader delegate="true"/> is used and you're relying on server-provided libraries.

 

 

 

Below command ./asadmin list-modules | grep OSGiModuleImpl:: | awk -F= '{print $2}' | cut -d[ -f2 | cut -d] -f1 | sort is used to extract and list the versions of all active OSGi modules deployed in a Payara (or GlassFish) server. It starts by invoking asadmin list-modules to retrieve all registered modules, filters only those associated with OSGi using grep OSGiModuleImpl::, then uses awk and cut to isolate the version number from the module's display name (which is typically formatted like OSGiModuleImpl::module-name [version]), and finally sorts the output. This command is useful for auditing the versions of OSGi modules actively loaded by the server, helping developers avoid conflicts with application-level JARs and diagnose class-loading issues. 

 

 

Below command cat /opt/payara6/glassfish/domains/domain1/osgi-cache/felix/bundle*/bundle.info | grep "file:" is used to extract the physical file paths of all OSGi bundle JARs currently cached and loaded by the Felix OSGi container within a Payara domain. By reading all bundle.info files inside each bundle* directory of the osgi-cache, it lists entries containing the file: prefix, which indicates the actual location of the bundle JAR on disk. This is helpful for identifying which exact versions of JARs were resolved and activated by the OSGi framework, and can be used to troubleshoot classpath conflicts, verify module loading behavior, or audit the internal module state of the server. 

 

 

 

This script scans all JAR files in the current directory and checks if they are OSGi-enabled bundles by looking for the Bundle-SymbolicName header in their MANIFEST.MF files. Here's what it does step-by-step:

• Loop through all JARs: It iterates over each *.jar file in the current directory.
• Read Manifest: For each JAR, it extracts the META-INF/MANIFEST.MF using unzip -p (without extracting to disk).
• Search for OSGi identifier: It greps for the Bundle-SymbolicName: header, which is a mandatory entry for an OSGi bundle and uniquely identifies the bundle in the OSGi runtime.
• Print OSGi JARs: If found, it prints the JAR filename along with its symbolic name in a formatted way.
• Sort output: The output is then sorted alphabetically.

---

Appendix: ClassLoader Debug Commands

Run with JVM option: -verbose:class

Example output:

[30.849s][info][class,load] org.glassfish.hk2.extension.ServiceLocatorGenerator source: file:/opt/payara6/glassfish/domains/domain1/applications/auruspay/WEB-INF/lib/hk2-api-3.1.1.jar 

# Show what class loader loaded which class
java -Xlog:class+load=info -cp yourapp.jar com.example.MainClass

# In Payara:
export _JAVA_OPTIONS='-Xlog:class+load=info'

<jvm-options>-Xlog:class+load:file=${com.sun.aas.instanceRoot}/logs/loader.log</jvm-options> 

---

This document consolidates class loader internals with hands-on troubleshooting seen in Payara deployments.