new file mode 100644

@@ -0,0 +1,1614 @@

+# Photon OS Linux Troublshooting Guide

+

+-   [Introduction](#introduction)

+    -   [Systemd and TDNF](#systemd-and-tdnf)

+    -   [The Root Account and the `sudo` and `su`

+        Commands](#the-root-account-and-the-sudo-and-su-commands)

+    -   [Checking the Version and Build

+        Number](#checking-the-version-and-build-number)

+    -   [General Best Practices](#general-best-practices)

+    -   [Logs on Photon OS](#logs-on-photon-os)

+    -   [Troubleshooting Progression](#troubleshooting-progression)

+-   [Solutions to Common Problems](#solutions-to-common-problems)

+    -   [Resetting a Lost Root

+        Password](#resetting-a-lost-root-password)

+    -   [Fixing Permissions on Network Config

+        Files](#fixing-permissions-on-network-config-files)

+    -   [Permitting Root Login with

+        SSH](#permitting-root-login-with-ssh)

+    -   [Fixing Sendmail If Installed Before an FQDN Was

+        Set](#fixing-sendmail-if-installed-before-an-fqdn-was-set)

+-   [Common Troubleshooting Tools on Photon

+    OS](#common-troubleshooting-tools-on-photon-os)

+    -   [Top](#top)

+    -   [ps](#ps)

+    -   [netstat](#netstat)

+    -   [find](#find)

+    -   [Locate](#locate)

+    -   [df](#df)

+    -   [md5sum and sha256sum](#md5sum-and-sha256sum)

+    -   [strace](#strace)

+    -   [file](#file)

+    -   [stat](#stat)

+    -   [watch](#watch)

+    -   [vmstat and fdisk](#vmstat-and-fdisk)

+    -   [lsof](#lsof)

+    -   [fuser](#fuser)

+    -   [ldd](#ldd)

+    -   [gdb](#gdb)

+    -   [Other Troubleshooting Tools Installed by

+        Default](#other-troubleshooting-tools-installed-by-default)

+    -   [Installing More Tools from

+        Repositories](#installing-more-tools-from-repositories)

+    -   [Linux Troubleshooting Tools Not on Photon

+        OS](#linux-troubleshooting-tools-not-on-photon-os)

+-   [Systemd](#systemd)

+    -   [Viewing Services](#viewing-services)

+    -   [Using Systemd Commands Instead of Init.d

+        Commands](#using-systemd-commands-instead-of-init.d-commands)

+    -   [Analyzing System Logs with

+        journalctl](#analyzing-system-logs-with-journalctl)

+    -   [Inspecting Services with

+        `systemd-analyze`](#inspecting-services-with-systemd-analyze)

+-   [Networking](#networking)

+    -   [Managing the Network

+        Configuration](#managing-the-network-configuration)

+    -   [Use `ip` and `ss` Commands Instead of `ifconfig` and

+        `netstat`](#use-ip-and-ss-commands-instead-of-ifconfig-and-netstat)

+    -   [Inspecting the Status of Network Links with

+        `networkctl`](#inspecting-the-status-of-network-links-with-networkctl)

+    -   [Turning on Network

+        Debugging](#turning-on-network-debugging)

+    -   [Installing the Packages for tcpdump and netcat with

+        tdnf](#installing-the-packages-for-tcpdump-and-netcat-with-tdnf)

+    -   [Checking Firewall Rules](#checking-firewall-rules)

+    -   [Netmgr](#netmgr)

+-   [File System](#file-system)

+    -   [Checking Disk Space](#checking-disk-space)

+    -   [Adding a Disk and Partitioning

+        It](#adding-a-disk-and-partitioning-it)

+    -   [fdisk](#fdisk)

+    -   [fsck](#fsck)

+    -   [Fixing File System Errors When fsck

+        Fails](#fixing-file-system-errors-when-fsck-fails)

+-   [Packages](#packages)

+-   [Troubleshooting Kernel Problems, Boot Problems, and Login

+    Problems](#troubleshooting-kernel-problems-boot-problems-and-login-problems)

+    -   [Kernel Overview](#kernel-overview)

+    -   [Boot Process Overview](#boot-process-overview)

+    -   [Blank Screen on Reboot](#blank-screen-on-reboot)

+    -   [Investigating Strange

+        Behavior](#investigating-strange-behavior)

+    -   [Investigating the Guest Kernel When You Cannot Log

+        On](#investigating-the-guest-kernel-when-you-cannot-log-on)

+    -   [Kernel Log Replication with

+        VProbes](#kernel-log-replication-with-vprobes)

+-   [Troubleshooting Performance

+    Issues](#troubleshooting-performance-issues)

+

+

+## Introduction 

+

+This guide describes the fundamentals of troubleshooting problems on Photon OS. An open-source minimalist Linux operating system from VMware, Photon OS is optimized for cloud computing platforms, VMware vSphere deployments, virtual appliances, and applications native to the cloud.

+

+This guide covers the basics of troubleshooting systemd, packages, network interfaces, services such as SSH and Sendmail, the file system, and the Linux kernel. The guide includes a quick tour of the tools that you can use for troubleshooting and provides examples along the way. The guide also demonstrates how to access the system's log files. 

+

+For information on how to set up and manage Photon OS, see the [Photon OS Administration Guide](https://github.com/vmware/photon/blob/master/docs/photon-admin-guide.md).

+

+### Systemd and TDNF

+

+Two characteristics of Photon OS stand out: It manages services with systemd, and it manages packages with its own open source, yum-compatible package manager called tdnf, for Tiny DNF. 

+

+By using systemd, Photon OS adopts a contemporary Linux standard to bootstrap the user space and concurrently start services--an architecture that differs from traditional Linux systems such as SUSE Linux Enterprise Server 11.

+

+[[Image:photon-logo.png|right]]

+

+A traditional Linux system contains an initialization system called SysVinit. With SLES 11, for instance, SysVinit-style init programs control how the system starts up and shuts down. Init implements system runlevels. A SysVinit runlevel defines a state in which a process or service runs. In contrast to a SysVinit system, systemd defines no such runlevels. Instead, systemd uses a dependency tree of _targets_ to determine which services to start when.

+

+Because the systemd commands differ from those of an init.d-based Linux system, a section later in this guide illustrates how to troubleshoot by using systemctl commands instead of init.d-style commands. 

+

+Tdnf keeps the operating system as small as possible while preserving yum's robust package-management capabilities. On Photon OS, tdnf is the default package manager for installing new packages. Since troubleshooting with tdnf differs from using yum, a later section of this guide describes how to solve problems with packages and repositories by using tdnf commands.

+

+### The Root Account and the `sudo` and `su` Commands

+

+This guide assumes that you are logged in to Photon OS with the root account and running commands as root. The sudo program comes with the full version of Photon OS. On the minimal version, you must install sudo with tdnf if you want to use it. As an alternative to installing sudo on the minimal version, you can switch users as needed with the `su` command to run commands that require root privileges.

+

+### Checking the Version and Build Number

+

+To check the version and build number of Photon OS, concatenate `/etc/photon-release`. Example: 

+

+	cat /etc/photon-release

+	VMware Photon Linux 1.0

+	PHOTON_BUILD_NUMBER=a6f0f63

+

+The build number in the results maps to the commit number on the VMware Photon OS GitHub [commits page](https://github.com/vmware/photon/commits/master).

+

+### General Best Practices

+

+When troubleshooting, you should follow some general best practices:

+

+* **Take a snapshot.** Before you do anything to a virtual machine running Photon OS, take a snapshot of the VM so that you can restore it if need be. 

+

+* **Make a backup copy.** Before you change a configuration file, make a copy of the original in case you need to restore it later; example: `cp /etc/tdnf/tdnf.conf /etc/tdnf/tdnf.conf.orig`

+

+* **Collect logs.** Save the log files associated with a Photon OS problem; you or others might need them later. Include not only the log files on the guest but also the `vmware.log` file on the host; `vmware.log` is in the host's directory that contains the VM.

+

+* **Know what's in your toolbox.** Glance at the man page for a tool before you use it so that you know what your options are. The options can help focus the command's output on the problem you're trying to solve.

+

+* **Understand the system.** The more you know about the operating system and how it works, the better you can troubleshoot.

+

+### Logs on Photon OS

+

+On Photon OS, all the system logs except the installation log and the cloud-init log are written into the systemd journal. The `journalctl` command queries the contents of the systemd journal.

+

+The installation log files and the cloud-init log files reside in `/var/log`. If Photon OS is running on a virtual machine in a VMware hypervisor, the log file for the VMware tools (vmware-vmsvc.log) also resides in `/var/log`. 

+

+### Troubleshooting Progression

+

+If you encounter a problem running an application or appliance on Photon OS and you suspect it involves the operating system, you can troubleshoot by proceeding as follows. 

+

+First, check the services running on Photon OS:

+

+	systemctl status

+

+Second, check your application's log files for clues. (For VMware applications, see [Location of Log Files for VMware Products](https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1021806).)

+

+Third, check the service controller or service monitor for your application or appliance. 

+

+Fourth, check the network interfaces and other aspects of the network service with `systemd-network` commands.

+

+Fifth, check the operating system's log files: 

+

+	journalctl

+

+Next, run the following commands to view all services according to the order in which they were started:

+

+	systemd-analyze critical-chain 

+

+Finally, if the previous steps have not revealed enough information to isolate the problem, turn to the troubleshooting tool that you think is most likely to help with the issue at hand. You could, for example, use `strace` to identify the location of the failure. See the list of troubleshooting tools on Photon OS in a later section. 

+

+## Solutions to Common Problems

+

+This section describes solutions to problems that you're likely to encounter.

+

+### Resetting a Lost Root Password

+

+Here's how to reset a lost root password. 

+

+First, restart the Photon OS machine or the virtual machine running Photon OS. When the Photon OS splash screen appears as it restarts, type the letter `e` to go to the GNU GRUB edit menu. Be quick about it: Because Photon OS reboots so quickly, you won't have much time to type `e`. Remember that in vSphere and Workstation, you might have to give the console focus by clicking in its window before it will register input from the keyboard. 

+

+Second, in the GNU GRUB edit menu, go to the end of the line that starts with `linux`, add a space, and then add the following code exactly as it appears below:

+

+	rw init=/bin/bash

+

+After you add this code, the GNU GRUB edit menu should look exactly like this:

+

+![The modified GNU GRUB edit menu](images/grub-edit-menu-changepw.png) 

+

+Now type `F10`.

+

+At the command prompt, type `passwd` and then type (and re-enter) a new root password that conforms to the password complexity rules of Photon OS. Remember the password. 

+

+Next, type the following command:

+

+	umount /

+

+Finally, type the following command. You must include the `-f` option to force a reboot; otherwise, the kernel enters a state of panic.

+

+	reboot -f

+

+This sequence of commands should look like this:

+

+![The series of commands to reset the root password](images/resetpw.png)

+

+After the Photon OS machine reboots, log in with the new root password. 

+

+### Fixing Permissions on Network Config Files

+

+If you, as the root user, create a new network configuration file on Photon OS, the network service might be unable to process it until you set the file's mode bits to `644`.

+

+If you query the journal with `journalctl -u systemd-networkd`, you might see the following error message along with an indication that the network service did not start: 

+

+	could not load configuration files. permission denied

+

+The permissions on the network files are the likely cause of this problem. Without the correct permissions, networkd-systemd cannot parse and apply the settings, and the network configuration that you created will not be loaded. 

+

+After you create a network configuration file with a `.network` extension, you must run the `chmod` command to set the new file's mode bits to `644`. Example: 

+

+    chmod 644 10-static-en.network

+

+For Photon OS to apply the new configuration, you must restart the `systemd-networkd` service by running the following command: 

+

+	systemctl restart systemd-networkd

+

+###	Permitting Root Login with SSH

+

+The full version of Photon OS prevents root login with SSH by default. To permit root login over SSH, open `/etc/ssh/sshd_config` with the vim text editor and set `PermitRootLogin` to `yes`. 

+

+Vim is the default text editor available in both the full and minimal versions of Photon OS. (Nano is also in the full version.) After you modify the SSH daemon's configuration file, you must restart the sshd daemon for the changes to take effect. Example: 

+

+	vim /etc/ssh/sshd_config

+

+	# override default of no subsystems

+	Subsystem       sftp    /usr/libexec/sftp-server

+

+	# Example of overriding settings on a per-user basis

+	#Match User anoncvs

+	#       X11Forwarding no

+	#       AllowTcpForwarding no

+	#       PermitTTY no

+	#       ForceCommand cvs server

+	PermitRootLogin yes

+	UsePAM yes

+

+Save your changes in vim and then restart the sshd daemon: 

+

+	systemctl restart sshd

+

+You can then connect to the Photon OS machine with the root account over SSH:

+

+	steve@ubuntu:~$ ssh root@198.51.100.131

+

+### Fixing Sendmail If Installed Before an FQDN Was Set

+

+If Sendmail is behaving improperly or if it hangs during installation, it is likely that an FQDN is not set. Take the following corrective action. 

+

+First, set an FQDN for your Photon OS machine. 

+

+Then, run the following commands in the order below: 

+

+    echo $(hostname -f) > /etc/mail/local-host-names

+    

+    cat > /etc/mail/aliases << "EOF"

+        postmaster: root

+        MAILER-DAEMON: root

+        EOF

+

+    /bin/newaliases

+

+    cd /etc/mail

+

+    m4 m4/cf.m4 sendmail.mc > sendmail.cf

+

+    chmod 700 /var/spool/clientmqueue

+

+    chown smmsp:smmsp /var/spool/clientmqueue

+

+## Common Troubleshooting Tools on Photon OS

+

+This section describes tools that can help troubleshoot problems. These tools are installed by default on the full version of Photon OS. On the minimal version of Photon OS, you may have to install a tool before you can use it. 

+

+There is a manual, or man page, on Photon OS for all the tools covered in this section. The man pages provide more information about each tool's commands, options, and output. To view a tool's man page, on the Photon OS command line, type `man` and then the name of the tool. Example: 

+

+	man strace

+

+Some of the examples in this section are marked as abridged with ellipsis (`...`).

+

+### Top

+

+Photon OS includes the Top tool to monitor system resources, workloads, and performance. It can unmask problems caused by processes or applications overconsuming CPUs, time, or RAM. 

+

+To view a textual display of resource consumption, run the `top` command: 

+

+	top

+

+In Top, you can kill a runaway or stalled process by typing `k` followed by its process ID (PID). 

+

+![Top on Photon OS](images/top-in-photon-os.png)

+

+If the percent of CPU utilization is consistently high with little idle time, there might be a runaway process overconsuming CPUs. Restarting the service might solve the problem. 

+

+A handy trick while troubleshooting an unknown issue is to run Top in the background by using batch mode to write its output to a file in order to collect data about performance:

+

+	top d 120 b >> top120second.output

+

+For a list of options that filter top output and other information, see the man page for Top.

+

+### ps

+

+The `ps` tool shows the processes running on the machine. The `ps` tool derives flexibility and power from its options, all of which are covered in the tool's Photon OS man page:

+

+	man ps

+

+Here are several popular invocations of `ps` for troubleshooting. 

+

+Show processes by user: 

+

+	ps aux

+

+Show processes and child processes by user: 

+

+	ps auxf

+

+Show processes containing the string `ssh`:

+

+	ps aux | grep ssh

+

+Show processes and the command and options with which they were started: 

+

+	ps auxww

+

+Example abridged output: 

+

+	ps auxww

+	USER        PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND

+	root          1  0.0  0.9  32724  3300 ?        Ss   07:51   0:32 /lib/systemd/systemd --switched-root --system --deserialize 22

+

+### netstat

+

+The `netstat` command can identify bottlenecks causing  performance issues. It lists network connections, listening sockets, port information, and interface statistics for different protocols. Examples: 

+

+	netstat --statistics

+	netstat --listening

+

+### find

+

+The `find` command can be a useful starting point to troubleshoot a Photon OS machine that has stopped working. The following command, for example, lists the files in the root directory that have changed in the past day: 

+

+		find / -mtime -1 

+

+See the `find` [manual](See https://www.gnu.org/software/findutils/manual/find.html). Take note of the security considerations listed in the `find` manual if you are using `find` to troubleshoot an appliance running on Photon OS. 

+

+### Locate

+

+The `locate` command is a fast way to find files and directories when all you have is a keyword. Similar to `find` and part of the same `findutils` package preinstalled on the full version of Photon OS by default, the `locate` command finds file names in the file names database. Before you can use `locate` accurately, you should update its database: 

+

+	updatedb

+

+Then you can run `locate` to quickly find a file, such as any file name containing `.network`, which can be helpful to see all the system's `.network` configuration files; abridged example: 

+

+	locate .network

+	/etc/dbus-1/system.d/org.freedesktop.network1.conf

+	/etc/systemd/network/10-dhcp-en.network

+	/usr/lib/systemd/network/80-container-host0.network

+	/usr/lib/systemd/network/80-container-ve.network

+	/usr/lib/systemd/system/busnames.target.wants/org.freedesktop.network1.busname

+	/usr/lib/systemd/system/dbus-org.freedesktop.network1.service

+	/usr/lib/systemd/system/org.freedesktop.network1.busnname

+	/usr/share/dbus-1/system-services/org.freedesktop.network1.service

+

+The `locate` command is also a quick way to see whether a troubleshooting tool is installed on Photon OS. Examples: 

+

+	locate strace

+	/usr/bin/strace

+	/usr/bin/strace-graph

+	/usr/bin/strace-log-merge

+	/usr/share/man/man1/strace.1.gz

+	/usr/share/vim/vim74/syntax/strace.vim

+

+	locate traceroute

+

+The `strace` tool is there but `traceroute` is not. You can, however, quickly install `traceroute` from the Photon OS repository: 

+

+	tdnf install traceroute

+

+

+### df

+

+The `df` command reports the disk space available on the file system. Because running out of disk space can lead an application to fail, a quick check of the available space makes sense as an early troubleshooting step: 

+

+	df -h

+

+The `-h` option prints out the available and used space in human-readable sizes. After checking the space, you should also check the number of available inodes. Too few available inodes can lead to difficult-to-diagnose problems:

+

+	df -i

+

+### md5sum and sha256sum

+

+`md5sum` calculates 128-bit MD5 hashes--a message digest, or digital signature, of a file--to uniquely identify a file and verify its integrity after file transfers, downloads, or disk errors when the security of the file is not in question. Photon OS also includes `sha256sum`, which is the preferred method of calculating the authenticity of a file to prevent tampering when security is a concern. Photon OS also includes `shasum`, `sha1sum`, `sha384sum`, and `sha512sum`. See the man pages for  `md3sum`, `sha256sum`, and the other SHA utilities. 

+

+`md5sum` can help troubleshooting installation issues by verifying that the version of Photon OS being installed matches the version on the Bintray download page. If, for instance, bytes were dropped during the download, the checksums will not match. Try downloading it again. 

+

+### strace

+

+The `strace` utility follows system calls and signals as they are executed so that you can see what an application, command, or process is doing. `strace` can trace failed commands, identify where a process obtains its configuration, monitor file activity, and find the location of a crash. 

+

+By tracing system calls, `strace` can help troubleshoot a broad range of problems, including issues with input-output, memory, interprocess communication, network usage, and application performance. 

+

+For troubleshooting a problem that gives off few or no clues, the following command displays every system call: 

+

+	strace ls -al

+

+With strace commands, you can route the output to a file to make it easier to analyze: 

+

+	strace -o output.txt ls -al

+

+`strace` can reveal the files that an application is trying to open with the `-eopen` option. This combination can help troubleshoot an application that is failing because it is missing files or being denied access to a file it needs. If, for example, you see "No such file or directory" in the results of `strace -eopen`, something might be wrong: 

+

+	strace -eopen sshd

+	open("/usr/lib/x86_64/libpam.so.0", O_RDONLY|O_CLOEXEC) = -1 ENOENT (No such file or directory)

+	open("/usr/lib/libpam.so.0", O_RDONLY|O_CLOEXEC) = 3

+

+In the results above, it's OK that the first file is missing because it is found in the next line. In other cases, the application might be unable to open one of its configuration files or reading the wrong one. If the results say "permission denied" for one of the files, check the permissions of the file with `ls -l` or `stat`.   

+

+When troubleshooting with `strace`, you can include the process ID in its commands. Here's an example of how to find a process ID: 

+

+	ps -ef | grep apache

+

+And you can then use `strace` to examine the file a process is working with: 

+

+	strace -e trace=file -p 1719

+

+A similar command can trace network traffic: 

+

+	strace -p 812 -e trace=network

+

+If an application is crashing, use `strace` to trace the application and then analyze what happens right before the application crashes.

+

+You can also trace the child processes that an application spawns with the fork system call, and you can do so with systemctl commands that start a process to identify why an application crashes immediately or fails to start: 

+

+	strace -f -o output.txt systemctl start httpd

+

+Here's another example. If journalctl is showing that networkd is failing, you can run strace to help determine why: 

+

+	strace -o output.txt systemctl restart systemd-networkd

+

+And then grep inside the results for something, such as _exit_ or _error_: 

+

+	grep exit output.txt

+

+Maybe the results indicate systemd-resolved is going wrong, and you can then strace it, too: 

+

+	strace -f -o output.txt systemctl restart systemd-resolved

+

+### file

+

+The `file` command determines the file type, which can help troubleshoot problems when an application mistakes one type of file for another, leading it to misbehave. Example: 

+

+	file /usr/sbin/sshd

+	/usr/sbin/sshd: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 2.6.32, stripped

+

+### stat

+

+The `stat` command can help troubleshoot problems with files or the file system by showing the last date it was modified and other information. Example:  

+

+	stat /dev/sda1

+	File: '/dev/sda1'

+	Size: 0               Blocks: 0          IO Block: 4096   block special file

+	Device: 6h/6d   Inode: 6614        Links: 1     Device type: 8,1

+	Access: (0660/brw-rw----)  Uid: (    0/    root)   Gid: (    8/    disk)

+	Access: 2016-09-02 12:23:56.135999936 +0000

+	Modify: 2016-09-02 12:23:52.879999981 +0000

+	Change: 2016-09-02 12:23:52.879999981 +0000

+	Birth: -

+

+On Photon OS, `stat` is handy to show permissions for a file or directory in both their absolute octal notation and their read-write-execute abbreviation; truncated example: 

+

+	chmod 777 tester.md

+	stat tester.md

+	  File: 'tester.md'

+	  Size: 0               Blocks: 0          IO Block: 4096   regular empty file

+	Device: 801h/2049d      Inode: 316385      Links: 1

+	Access: (0777/-rwxrwxrwx)  Uid: (    0/    root)   Gid: (    0/    root)

+

+### watch

+

+The `watch` utility runs a command at regular intervals so you can observe how its output changes over time. `watch` can help dynamically monitor network links, routes, and other information when you are troubleshooting networking or performance issues. Examples: 

+

+	watch -n0 --differences ss

+	watch -n1 --differences ip route

+	

+Here's another example with a screenshot of the command's output. This command monitors the traffic on your network links. The highlighted numbers are updated every second so you can see the traffic fluctuating: 

+

+	watch -n1 --differences ip -s link show up

+

+![The dynamic output of the watch utility](images/watchcmd.png)  

+

+### vmstat and fdisk

+

+The `vmstat` tool displays statistics about virtual memory, processes, block input-output, disks, and CPU activity. This tool can help diagnose performance problems, especially system bottlenecks.  

+

+Its output on a Photon OS virtual machine running in VMware Workstation 12 Pro without a heavy load looks like this: 

+

+	vmstat

+	procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----

+	 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st

+	 0  0      0   5980  72084 172488    0    0    27    44  106  294  1  0 98  1  0

+

+What do all these codes mean? They are explained in the vmstat man page. 

+

+If `r`, the number of runnable processes, is higher than 10, the machine is under stress; consider intervening to reduce the number of processes or to distribute some of the processes to other machines. In other words, the machine has a bottleneck in executing processes.

+

+If `cs`, the number of context switches per second, is really high, there may be too many jobs running on the machine. 

+

+If `in`, the number of interrupts per second, is relatively high, there might be a bottleneck for network or disk IO. 

+

+You can investigate disk IO further by using vmstat's `-d` option to report disk statistics; abridged example on a machine with little load: 

+

+	vmstat -d

+	disk- ------------reads------------ ------------writes----------- -----IO------

+	       total merged sectors      ms  total merged sectors      ms    cur    sec

+	ram0       0      0       0       0      0      0       0       0      0      0

+	ram1       0      0       0       0      0      0       0       0      0      0

+	loop0      0      0       0       0      0      0       0       0      0      0

+	loop1      0      0       0       0      0      0       0       0      0      0

+	sr0        0      0       0       0      0      0       0       0      0      0

+	sda    22744    676  470604   12908  72888  24949  805224  127692      0    130

+

+The `-D` option summarizes disk statistics:

+

+	vmstat -D

+	           26 disks

+	            2 partitions

+	        22744 total reads

+	          676 merged reads

+	       470604 read sectors

+	        12908 milli reading

+	        73040 writes

+	        25001 merged writes

+	       806872 written sectors

+	       127808 milli writing

+	            0 inprogress IO

+	          130 milli spent IO

+

+You can also get statistics about a partition. First, run the `fdisk -l` command to list the machine's devices. Then run `vmstat -p` with the name of a device to view its stats: 

+

+

+	fdisk -l

+	Disk /dev/ram0: 4 MiB, 4194304 bytes, 8192 sectors

+	Units: sectors of 1 * 512 = 512 bytes

+	Sector size (logical/physical): 512 bytes / 4096 bytes

+	I/O size (minimum/optimal): 4096 bytes / 4096 bytes

+	...

+	Device        Start      End  Sectors Size Type

+	/dev/sda1      2048 16771071 16769024   8G Linux filesystem

+	/dev/sda2  16771072 16777182     6111   3M BIOS boot

+

+	vmstat -p /dev/sda1

+	sda1          reads   read sectors  writes    requested writes

+	               22579     473306      78510     866088

+

+See the vmstat man page for more options. 

+

+### lsof

+

+The `lsof` command lists open files. And this tool's definition of an open file is quite broad--directories, libraries, streams, domain sockets, and Internet sockets are all considered files, making `lsof` broadly applicable as a mid-level troubleshooting tool to identify the files a process is using. Because a Linux system like Photon OS uses files to do its work, you can run `lsof` as root to see how the system is using them and to see how an application works. 

+

+If, for example, you cannot unmount a disk because it is in use, you can run `lsof` to identify the files on the disk that are being used. Here's an example showing what's using the root directory: 

+

+	lsof /root

+	COMMAND    PID USER   FD   TYPE DEVICE SIZE/OFF   NODE NAME

+	bash       879 root  cwd    DIR    8,1     4096 262159 /root

+	bash      1265 root  cwd    DIR    8,1     4096 262159 /root

+	sftp-serv 1326 root  cwd    DIR    8,1     4096 262159 /root

+	gdb       1351 root  cwd    DIR    8,1     4096 262159 /root

+	bash      1395 root  cwd    DIR    8,1     4096 262159 /root

+	lsof      1730 root  cwd    DIR    8,1     4096 262159 /root

+

+You can do the same with an application or virtual appliance by running `lsof` with the user name or process ID of the app. Here's an example that lists the open files used by the Apache HTTP Server:  

+

+	lsof -u apache

+

+Running the command with the `-i` option lists all the open network and Internet files, which can help troubleshoot network problems: 

+

+	lsof -i

+

+See the Unix socket addresses of a user like _zookeeper_: 

+

+	lsof -u zookeeper -U

+

+And here's an example that shows the processes running on Ports 1 through 80:

+

+	lsof -i TCP:1-80

+	COMMAND  PID   USER   FD   TYPE DEVICE SIZE/OFF NODE NAME

+	httpd    403   root    3u  IPv6  10733      0t0  TCP *:http (LISTEN)

+	httpd    407 apache    3u  IPv6  10733      0t0  TCP *:http (LISTEN)

+	httpd    408 apache    3u  IPv6  10733      0t0  TCP *:http (LISTEN)

+	httpd    409 apache    3u  IPv6  10733      0t0  TCP *:http (LISTEN)

+	sshd     820   root    3u  IPv4  11336      0t0  TCP *:ssh (LISTEN)

+	sshd     820   root    4u  IPv6  11343      0t0  TCP *:ssh (LISTEN)

+	sshd    1258   root    3u  IPv4  48040      0t0  TCP 198.51.100.143:ssh->198.51.100.1:49759 (ESTABLISHED)

+	sshd    1319   root    3u  IPv4  50866      0t0  TCP 198.51.100.143:ssh->198.51.100.1:51054 (ESTABLISHED)

+	sshd    1388   root    3u  IPv4  56438      0t0  TCP 198.51.100.143:ssh->198.51.100.1:60335 (ESTABLISHED)

+

+You can also inspect the files opened by a process ID. Here's a truncated example that queries the files open by the systemd network service: 

+

+	lsof -p 1917

+	COMMAND    PID            USER   FD      TYPE             DEVICE SIZE/OFF   NODE NAME

+	systemd-n 1917 systemd-network  cwd       DIR                8,1     4096      2 /

+	systemd-n 1917 systemd-network  txt       REG                8,1   887896 272389 /usr/lib/systemd/systemd-networkd

+	systemd-n 1917 systemd-network  mem       REG                8,1   270680 262267 /usr/lib/libnss_files-2.22.so

+	systemd-n 1917 systemd-network    0r      CHR                1,3      0t0   5959 /dev/null

+	systemd-n 1917 systemd-network    1u     unix 0x0000000000000000      0t0  45734 type=STREAM

+	systemd-n 1917 systemd-network    3u  netlink                         0t0   6867 ROUTE

+	systemd-n 1917 systemd-network    4u     unix 0x0000000000000000      0t0  45744 type=DGRAM

+	systemd-n 1917 systemd-network    9u  netlink                         0t0  45754 KOBJECT_UEVENT

+	systemd-n 1917 systemd-network   12u  a_inode               0,11        0   5955 [timerfd]

+	systemd-n 1917 systemd-network   13u     IPv4             104292      0t0    UDP 198.51.100.143:bootpc

+

+### fuser

+

+The `fuser` command identifies the process IDs of processes using files or sockets. The term _process_ is, in this case, synonymous with _user_. To identify the process ID of a process using a socket, run `fuser` with its namespace option and specify `tcp` or `udp` and the name of the process or port. Examples: 

+

+	fuser -n tcp ssh

+	ssh/tcp:               940  1308

+	fuser -n tcp http

+	http/tcp:              592   594   595   596

+	fuser -n tcp 80

+	80/tcp:                592   594   595   596

+

+

+### ldd

+

+By revealing the shared libraries that a program depends on, `ldd` can help troubleshoot an application that is missing a library or finding the wrong one.

+

+If, for example, you find output that says "file not found," check the path to the library.  

+

+	ldd /usr/sbin/sshd

+    linux-vdso.so.1 (0x00007ffc0e3e3000)

+    libpam.so.0 => (file not found)

+    libcrypto.so.1.0.0 => /usr/lib/libcrypto.so.1.0.0 (0x00007f624e570000)

+

+You can also use the `objdump` command to show dependencies for a program's object files; example:

+

+	objdump -p /usr/sbin/sshd | grep NEEDED

+

+### gdb

+

+The gdb tool is the GNU debugger. It lets you peer inside a program while it executes or when it crashes so that you can catch bugs on the fly. The gdb tool is typically used to debug programs written in C and C++. On Photon OS, gdb can help you determine why an application crashed. See the man page for gdb for instructions on how to run it. For an extensive example on how to use gdb to troubleshoot Photon OS running on a VM when you cannot login to Photon OS, see the section on troubleshooting boot and logon problems. 

+

+### Other Troubleshooting Tools Installed by Default

+

+The following troubleshooting tools are included in the full version of Photon OS: 

+

+* `grep` searches files for patterns. 

+* `ping` tests network connectivity. 

+* `strings` displays the characters in a file to identify its contents.

+* `lsmod` lists loaded modules.

+* `ipcs` shows data about the inter-process communication (IPC) resources to which a process has read access--typically, shared memory segments, message queues, and semaphore arrays.

+* `nm` lists symbols from object files. 

+* `diff` compares files side by side. Useful to compare two configuration files when one version works and the other doesn't. 

+

+### Installing More Tools from Repositories

+

+You can install several troubleshooting tools from the Photon OS repositories by using the default package management system, `tdnf`. 

+

+If a tool you need is not installed, the first thing you should do is search the repositories to see whether it's available. The traceroute tool, for example, is not installed by default. Here's how to search for it in the repositories:  

+

+	tdnf search traceroute

+	traceroute : Traces the route taken by packets over an IPv4/IPv6 network

+

+The results of the above command show that traceroute exists in the repository. You install it with `tdnf`: 

+

+	tdnf install traceroute

+

+Additional tools are not installed by default but are in the repository for instant installation with `tdnf`: 

+

+* `net-tools`: networking tools.

+* `ltrace`: tool for intercepting and recording dynamic library calls. It can identify the function an application was calling when it crashed, making it useful for debugging.

+* `nfs-utils`: client tools for the kernel Network File System, or NFS, including showmount; installed by default in the full version of Photon OS but not in the minimal version. 

+* `pcstat`: A tool that inspects which pages of a file or files are being cached by the Linux kernel.

+* `sysstat` and `sar`: Utilities to monitor system performance and usage activity. Installing sysstat also installs sar.

+* `systemtap` and `crash`: The systemtap utility is a programmable instrumentation system for diagnosing problems of performance or function. Installing systemtap also installs crash, which is a kernel crash analysis utility for live systems and dump files.

+* `dstat`: versatile tool for viewing and analyzing statistics about system resources.

+

+The `dstat` tool, for example, can help troubleshoot system performance. The tool shows a live, running list of statistics about system resources: 

+

+	dstat

+	You did not select any stats, using -cdngy by default.

+	----total-cpu-usage---- -dsk/total- -net/total- ---paging-- ---system--

+	usr sys idl wai hiq siq| read  writ| recv  send|  in   out | int   csw

+	  1   0  98   1   0   0|4036B   42k|   0     0 |   0     0 |  95   276

+	  1   0  98   1   0   0|   0    64k|  60B  940B|   0     0 | 142   320

+	  1   1  98   0   0   0|   0    52k|  60B  476B|   0     0 | 149   385

+

+

+### Linux Troubleshooting Tools Not on Photon OS

+

+The following Linux troubleshoot tools are neither installed on Photon OS by default nor available in the Photon OS repositories: 

+

+* iostat

+* telnet (use SSH instead)

+* Iprm

+* hdparm

+* syslog (use journalctl instead)

+* ddd

+* ksysmoops

+* xev

+* GUI tools (because Photon OS has no GUI)

+

+## Systemd

+

+Photon OS manages services with systemd and its command-line utility for inspecting and controlling the system, `systemctl`, not the deprecated commands of init.d. For example, instead of running the /etc/init.d/ssh script to stop and start the OpenSSH server on a init.d-based Linux system, you control the service by running the following systemctl commands on Photon OS: 

+

+	systemctl stop sshd

+	systemctl start sshd

+

+For an overview of systemd, see [systemd System and Service Manager](https://www.freedesktop.org/wiki/Software/systemd/) and the [man page for systemd](https://www.freedesktop.org/software/systemd/man/systemd.html). The systemd man pages are listed at [https://www.freedesktop.org/software/systemd/man/](https://www.freedesktop.org/software/systemd/man/).

+

+### Viewing Services 

+

+To view a description of all the active, loaded units, execute the systemctl command without any options or arguments: 

+

+	systemctl

+

+To see all the loaded, active, and inactive units and their description, run this command: 

+

+	systemctl --all

+

+To see all the unit files and their current status but no description, run this command: 

+

+	systemctl list-unit-files

+

+The `grep` command filters the services by a search term, a helpful tactic to recall the exact name of a unit file without looking through a long list of names. Example: 

+

+	systemctl list-unit-files | grep network

+	org.freedesktop.network1.busname           static

+	dbus-org.freedesktop.network1.service      enabled

+	systemd-networkd-wait-online.service       enabled

+	systemd-networkd.service                   enabled

+	systemd-networkd.socket                    enabled

+	network-online.target                      static

+	network-pre.target                         static

+	network.target  

+

+### Using Systemd Commands Instead of Init.d Commands

+

+Basic system administration commands on Photon OS differ from those on operating systems that use SysVinit. Since Photon OS uses systemd instead of SysVinit, you must use systemd commands to manage services. 

+

+For example, to list all the services that you can manage on Photon OS, you run the following command instead of `ls /etc/rc.d/init.d/`: 

+

+	systemctl list-unit-files --type=service

+

+Similarly, to check whether the `sshd` service is enabled, on Photon OS you run the following command instead of `chkconfig sshd`:

+

+	systemctl is-enabled sshd

+

+The `chkconfig --list` command that shows which services are enabled for which runlevel on a SysVinit computer becomes substantially different on Photon OS because there are no runlevels, only targets: 

+

+	ls /etc/systemd/system/*.wants

+

+You can also display similar information with the following command: 

+

+	systemctl list-unit-files --type=service

+

+Here is a list of some of the systemd commands that take the place of SysVinit commands on Photon OS: 

+

+	USE THIS SYSTEMD COMMAND 	INSTEAD OF THIS SYSVINIT COMMAND

+	systemctl start sshd 		service sshd start

+	systemctl stop sshd 		service sshd stop

+	systemctl restart sshd 		service sshd restart

+	systemctl reload sshd 		service sshd reload

+	systemctl condrestart sshd 	service sshd condrestart

+	systemctl status sshd 		service sshd status

+	systemctl enable sshd 		chkconfig sshd on

+	systemctl disable sshd 		chkconfig sshd off

+	systemctl daemon-reload		chkconfig sshd --add

+

+### Analyzing System Logs with journalctl

+

+The journalctl tool queries the contents of the systemd journal. On Photon OS, all the system logs except the installation log and the cloud-init log are written into the systemd journal. 

+

+If called without parameters, the `journalctl` command shows all the contents of the journal, beginning with the oldest entry. To display the output in reverse order with new entries first, include the `-r` option in the command:

+

+	journalctl -r

+

+The `journalctl` command includes many options to filter its output. For help troubleshooting systemd, two journalctl queries are particularly useful: showing the log entries for the last boot and showing the log entries for a systemd service unit. This command displays the messages that systemd generated during the last time the machine started: 

+

+	journalctl -b

+

+This command reveals the messages for only the systemd service unit specified by the `-u` option, which in the following example is the auditing service: 

+

+	journalctl -u auditd

+

+You can look at the messages for systemd itself or for the network service:

+

+	journalctl -u systemd

+	journalctl -u systemd-networkd

+

+Example:  

+

+	root@photon-1a0375a0392e [ ~ ]# journalctl -u systemd-networkd

+	-- Logs begin at Tue 2016-08-23 14:35:50 UTC, end at Tue 2016-08-23 23:45:44 UTC. --

+	Aug 23 14:35:52 photon-1a0375a0392e systemd[1]: Starting Network Service...

+	Aug 23 14:35:52 photon-1a0375a0392e systemd-networkd[458]: Enumeration completed

+	Aug 23 14:35:52 photon-1a0375a0392e systemd[1]: Started Network Service.

+	Aug 23 14:35:52 photon-1a0375a0392e systemd-networkd[458]: eth0: Gained carrier

+	Aug 23 14:35:53 photon-1a0375a0392e systemd-networkd[458]: eth0: DHCPv4 address 198.51.100.1

+	Aug 23 14:35:54 photon-1a0375a0392e systemd-networkd[458]: eth0: Gained IPv6LL

+	Aug 23 14:35:54 photon-1a0375a0392e systemd-networkd[458]: eth0: Configured

+

+

+For more information, see [journalctl](https://www.freedesktop.org/software/systemd/man/journalctl.html) or the journalctl man page by running this command: `man journalctl`

+

+### Inspecting Services with `systemd-analyze`

+

+The `systemd-analyze` command reveals performance statistics for boot times, traces system services, and verifies unit files. It can help troubleshoot slow system boots and incorrect unit files. See the man page for a list of options. Examples:

+

+	systemd-analyze blame

+

+	systemd-analyze dump

+

+## Networking

+

+### Managing the Network Configuration

+

+The network service, which is enabled by default, starts when the system boots. You manage the network service by using systemd commands, such as systemd-networkd, systemd-resolvd, and networkctl. You can check its status of the network service by running the following command: 

+

+	systemctl status systemd-networkd

+

+Here is a healthy result of the command: 

+

+	* systemd-networkd.service - Network Service

+	   Loaded: loaded (/usr/lib/systemd/system/systemd-networkd.service; enabled; vendor preset: enabled)

+	   Active: active (running) since Fri 2016-04-29 15:08:51 UTC; 6 days ago

+	     Docs: man:systemd-networkd.service(8)

+	 Main PID: 291 (systemd-network)

+	   Status: "Processing requests..."

+	   CGroup: /system.slice/systemd-networkd.service

+	           `-291 /lib/systemd/systemd-networkd

+

+Because Photon OS relies on systemd to manage services, you should employ the systemd suite of commands, not deprecated init.d commands or other deprecated commands, to manage networking. 

+

+### Use `ip` and `ss` Commands Instead of `ifconfig` and `netstat`

+

+Although the `ifconfig` command and the `netstat` command work on Photon OS, VMware recommends that you use the `ip` or `ss` commands. The `ifconfig` and `netstat` commands are deprecated. 

+

+For example, instead of running `netstat` to display a list of network interfaces, consider running the `ss` command. Similarly, to display information for IP addresses, instead of running `ifconfig -a`, run the `ip addr` command. Examples:

+

+	USE THIS IPROUTE COMMAND 	INSTEAD OF THIS NET-TOOL COMMAND

+	ip addr 					ifconfig -a

+	ss 							netstat

+	ip route 					route

+	ip maddr 					netstat -g

+	ip link set eth0 up 		ifconfig eth0 up

+	ip -s neigh					arp -v

+	ip link set eth0 mtu 9000	ifconfig eth0 mtu 9000

+

+Using the `ip route` version of a command instead of the net-tools version often provides more complete, accurate information on Photon OS, as the following example demonstrates: 

+

+	ip neigh

+	198.51.100.2 dev eth0 lladdr 00:50:56:e2:02:0f STALE

+	198.51.100.254 dev eth0 lladdr 00:50:56:e7:13:d9 STALE

+	198.51.100.1 dev eth0 lladdr 00:50:56:c0:00:08 DELAY

+

+	arp -a

+	? (198.51.100.2) at 00:50:56:e2:02:0f [ether] on eth0

+	? (198.51.100.254) at 00:50:56:e7:13:d9 [ether] on eth0

+	? (198.51.100.1) at 00:50:56:c0:00:08 [ether] on eth0

+

+**Important:** If you modify an IPv6 configuration or add an IPv6 interface, you must restart `systemd-networkd`. Traditional methods of using `ifconfig` commands will be inadequate to register the changes. Run the following command instead: 

+

+	systemctl restart systemd-networkd

+

+

+### Inspecting the Status of Network Links with `networkctl`

+

+The `networkctl` command shows information about network connections that helps you configure networking services and troubleshoot networking problems. You can, for example, progressively add options and arguments to the `networkctl` command to move from general information about network connections to specific information about a network connection. 

+

+Running `networkctl` without options defaults to the list command:  

+

+	networkctl