# Investigating the Guest Kernel If a VM running Photon OS and an application or virtual appliance is behaving preventing you from logging in to the machine, you can troubleshoot by extracting the kernel logs from the guest's memory and analyzing them with `gdb`. This advanced troubleshooting method works when you are running Photon OS as the operating system for an application or appliance on VMware Workstation, Fusion, or ESXi. The procedure in this section assumes that the virtual machine running Photon OS is functioning normally. The process to use this troubleshooting method varies by environment. The examples in this section assume that the troublesome Photon OS virtual machine is running in VMware Workstation 12 Pro on a Microsoft Windows 8 Enterprise host. The examples also use an additional, fully functional Photon OS virtual machine running in Workstation. You can use other hosts, hypervisors, and operating systems--but you will have to adapt the example process below to them. Directory paths, file names, and other aspects might be different on other systems. - [Prerequisites](#prerequisites) - [Procedure Overview](#procedure-overview) - [Procedure](#procedure) ## Prerequisites Verify that you have the following resources: * Root access to a Linux machine other than the one you are troubleshooting. It can be another Photon OS machine, Ubuntu, or another Linux variant. * The `vmss2core` utility from VMware. It is installed by default in VMware Workstation and some other VMware products. If your system doesn't already contain it, you can download it for free from https://labs.vmware.com/flings/vmss2core. * A local copy of the Photon OS ISO of the exact same version and release number as the Photon OS machine that you are troubleshooting. ## Procedure Overview The process to apply this troubleshooting method is as follows: - On a local computer, you open a file on the Photon OS ISO that contains Linux debugging information. Then you suspend the troublesome Photon OS VM and extract the kernel memory logs from the VMware hypervisor running Photon OS. - Next, you use the vmss2core tool to convert the memory logs into core dump files. The vmss2core utility converts VMware checkpoint state files into formats that third-party debugging tools understand. It can handle both suspend (.vmss) and snapshot (.vmsn) checkpoint state files (hereafter referred to as a _vmss file_) as well as monolithic and non-monolithic (separate .vmem file) encapsulation of checkpoint state data. See [Debugging Virtual Machines with the Checkpoint to Core Tool](http://www.vmware.com/pdf/snapshot2core_technote.pdf). - Finally, you prepare to run the gdb tool by using the debug info file from the ISO to create a `.gdbinit` file, which you can then analyze with the gdb shell on your local Linux machine. All three components must be in the same directory on a Linux machine. ## Procedure 1. Obtain a local copy of the Photon OS ISO of the exact same version and release number as the Photon OS machine that you are troubleshooting and mount the ISO on a Linux machine (or open it on a Windows machine): mount /mnt/cdrom 1. Locate the following file. (If you opened the Photon OS ISO on a Windows computer, copy the following file to the root folder of a Linux machine.) /RPMS/x86_64/linux-debuginfo-4.4.8-6.ph1.x86_64.rpm 1. On a Linux machine, run the following `rpm2cpio` command to convert the RPM file to a cpio file and to extract the contents of the RPM to the current directory: ``` rpm2cpio /mnt/cdrom/RPMS/x86_64/linux-debuginfo-4.4.8-6.ph1.x86_64.rpm | cpio -idmv ``` 1. From the extracted files, copy the following file to your current directory: ``` cp usr/lib/debug/lib/modules/4.4.8/vmlinux-4.4.8.debug ``` 1. Run the following command to download the dmesg functions that will help extract the kernel log from the coredump: ``` wget https://www.kernel.org/doc/Documentation/kdump/gdbmacros.txt wget https://github.com/vmware/photon/blob/master/tools/scripts/gdbmacros-for-linux.txt ``` 1. Move the file as follows: ``` mv gdbmacros-for-linux.txt .gdbinit ``` 1. Switch to your host machine so you can get the kernel memory files from the VM. Suspend the troublesome VM and locate the `.vmss` and `.vmem` files in the virtual machine's directory on the host. Example: ``` C:\Users\tester\Documents\Virtual Machines\VMware Photon 64-bit (7)>dir Volume in drive C is Windows Directory of C:\Users\tester\Documents\Virtual Machines\VMware Photon 64-bit (7) 09/20/2016 12:22 PM