@@ -232,10 +232,11 @@

         - [Permitting Root Login with SSH](photon_troubleshoot/permitting-root-login-with-ssh.md)

         - [Fixing Sendmail](photon_troubleshoot/fixing-sendmail.md)

     -   [Photon OS General Troubleshooting] (photon_troubleshoot/Photon_OS_General_Troubleshooting.md)

-        - [Package Management](photon_troubleshoot/Troubleshooting_PackageManagement.md)

-        - [Network Configurations] (photon_troubleshoot/Troubleshooting_Networkconfiguration.md)

-        - [cloud-init] (photon_troubleshoot/Troubleshooting-cloudinit.md)

-        - [vmtoolsd] (photon_troubleshoot/Troubleshooting-vmtoolsd)

+        - [Photon Code](Troubleshooting_photoncode.md)

+        - [Package Management](Troubleshooting_PackageManagement.md)

+        - [Network Configurations] (Troubleshooting_Networkconfiguration.md)

+        - [cloud-init] (Troubleshooting-cloudinit.md)

+        - [vmtoolsd] (Troubleshooting-vmtoolsd.md)

     - [Troubleshooting Tools on Photon OS](photon_troubleshoot/troubleshooting-tools.md)

         - [Common Tools](photon_troubleshoot/common-tools.md)

         - [Troubleshooting Tools Installed by Default](photon_troubleshoot/default-tools.md)

@@ -269,6 +270,7 @@

         - [Investigating Unexpected Behavior](photon_troubleshoot/investigating-strange-behavior.md)

         - [Investigating the Guest Kernel](photon_troubleshoot/investigating-the-guest-kernel.md)

         - [Kernel Log Replication with VProbes](photon_troubleshoot/kernel-log-replication-with-vprobes.md)

+        - [Troubleshooting Linux Kernel](Troubleshooting-linuxkernel.md)

     -   [Performance Issues](photon_troubleshoot/performance-issues.md)

         - [General Guidelines](photon_troubleshoot/general_performance_guidelines.md)

         - [Throughput Performance](photon_troubleshoot/throughput_performance.md)

@@ -2,6 +2,7 @@

 The section includes general troubleshooting instruction for Photon OS.

+- [Photon Code](Troubleshooting_photoncode.md)

 - [Package Management](Troubleshooting_PackageManagement.md)

 - [Network Configurations] (Troubleshooting_Networkconfiguration.md)

 - [cloud-init] (Troubleshooting-cloudinit.md)

@@ -17,10 +17,11 @@

         - [Permitting Root Login with SSH](permitting-root-login-with-ssh.md)

         - [Fixing Sendmail](fixing-sendmail.md)

     -   [Photon OS General Troubleshooting] (Photon_OS_General_Troubleshooting.md)

+        - [Photon Code](Troubleshooting_photoncode.md)

         - [Package Management](Troubleshooting_PackageManagement.md)

         - [Network Configurations] (Troubleshooting_Networkconfiguration.md)

         - [cloud-init] (Troubleshooting-cloudinit.md)

-        - [vmtoolsd] (Troubleshooting-vmtoolsd)

+        - [vmtoolsd] (Troubleshooting-vmtoolsd.md))

     -   [Troubleshooting Tools on Photon OS](troubleshooting-tools.md)

         - [Common Tools](common-tools.md)

         - [Troubleshooting Tools Installed by Default](default-tools.md)

@@ -54,6 +55,7 @@

         - [Investigating Unexpected Behavior](investigating-strange-behavior.md)

         - [Investigating the Guest Kernel](investigating-the-guest-kernel.md)

         - [Kernel Log Replication with VProbes](kernel-log-replication-with-vprobes.md)

+        - [Troubleshooting Linux Kernel](Troubleshooting-linuxkernel.md)

     -   [Performance Issues](performance-issues.md)

         - [General Guidelines](general_performance_guidelines.md)

         - [Throughput Performance](throughput_performance.md)

@@ -7,4 +7,5 @@ Photon OS includes commands to troubleshoot kernel problems and boot and login e

 - [Blank Screen on Reboot](blank-screen-on-reboot.md)

 - [Investigating Unexpected Behavior](investigating-strange-behavior.md)

 - [Investigating the Guest Kernel](investigating-the-guest-kernel.md)

-- [Kernel Log Replication with VProbes](kernel-log-replication-with-vprobes.md)

\ No newline at end of file

+- [Kernel Log Replication with VProbes](kernel-log-replication-with-vprobes.md)

+- [Troubleshooting Linux Kernel](Troubleshooting-linuxkernel.md)

\ No newline at end of file

@@ -2,4 +2,149 @@

 On Photon OS, all the system logs except the installation logs and the cloud-init logs are written into the systemd journal. The `journalctl` command queries the contents of the systemd journal.

-The installation log files and the cloud-init log files reside in `/var/log`. If Photon OS is running on a virtual machine in a VMware hypervisor, the log file for the VMware tools, `vmware-vmsvc.log`, also resides in `/var/log`. 

\ No newline at end of file

+The installation log files and the cloud-init log files reside in `/var/log`. If Photon OS is running on a virtual machine in a VMware hypervisor, the log file for the VMware tools, `vmware-vmsvc.log`, also resides in `/var/log`. 

+

+##Journalctl

+Journalctl is a utility to query and display logs from journald and systemd’s logging service. Since journald stores log data in a binary format instead of a plain text format, `journalctl` is the standard way of reading log messages processed by journald.

+

+Journald is a service provided by systemd. To see the staus of the daemon, run the following commands:

+```

+# systemctl status systemd-journald

+● systemd-journald.service - Journal Service

+Loaded: loaded (/lib/systemd/system/systemd-journald.service; static; vendor preset: enabled)

+Active: active (running) since Tue 2020-04-07 14:33:41 CST; 2 days ago

+Docs: man:systemd-journald.service(8)

+man:journald.conf(5)

+Main PID: 943 (systemd-journal)

+Status: "Processing requests..."

+Tasks: 1 (limit: 4915)

+Memory: 18.0M

+CGroup: /system.slice/systemd-journald.service

+└─943 /lib/systemd/systemd-journald

+

+

+

+Apr 07 14:33:41 photon-4a0e7f2307d4 systemd-journald[943]: Journal started

+Apr 07 14:33:41 photon-4a0e7f2307d4 systemd-journald[943]: Runtime journal (/run/log/journal/b8cebc61a6cb446a968ee1d4c5bbbbd5) is 8.0M, max 1.5G, 1.5G free.

+Apr 07 14:33:41 photon-4a0e7f2307d4 systemd-journald[943]: Time spent on flushing to /var is 88.263ms for 1455 entries.

+Apr 07 14:33:41 photon-4a0e7f2307d4 systemd-journald[943]: System journal (/var/log/journal/b8cebc61a6cb446a968ee1d4c5bbbbd5) is 40.0M, max 4.0G, 3.9G free.

+root@photon-4a0e7f2307d4 [ ~ ]#

+```

+

+The following command are related to `journalctl`:

+

+- `journalctl` : This command displays all the logs after the system has booted up. `journalctl` splits the results into pages, similar to the `less` command in Linux. You can navigate using the arrow keys, the Page Up, Page Down keys or the Space bar. To quit navigation, press the **q** key.

+- `journalctl -b` : This command displays the logs for the current boot.

+

+The following commands pull logs based on a time range:

+

+- `journalctl --since "1 hour ago"` : This command displays the journal logs from the past 1 hour.

+- `journalctl --since "2 days ago"` : This command displays the logs generated in the past 2 days.

+- `journalctl --since "2020-03-25 00:00:00" --until "2020-04-09 00:00:00"` : This command displays the logs generated between the mentioned time frame.

+

+To traverse for logs in the reverse order, run the following command:

+

+- `journalctl -r` : This command displays the logs in reverse order.

+

+**Note**: If you add `-r` at the end of a command, the logs are displayed in the reverse order. For example: `journalctl -u unit.service -r`

+

+To pull logs related to a particular daemon, run the following command:

+

+- `journalctl -u unit.service` : This command displays logs for a specific service. mention the name of the service instead of `unit`. This command helps when a service is not behaving properly or when there are crash/core dumps. 

+

+To see Journal logs by their priority, run the following command:

+

+- `journalctl -p "emerg".."crit` : This command displays logs **emerg** to **critical**. For example: core dumps.

+

+Journalctl can print log messages to the console as they are added, like the Linux `tail` command. Add the `-f` switch to follow a specific service or daemon.

+```

+journalctl -u unit.service -f

+```

+

+To list the boots of the system, run the following command:

+```

+journalctl --list-boots

+```

+

+You can maintain the journalctl logs manually, by running the following `vacuum` commands:

+

+- `journalctl --vacuum-time=2d` : This command retains the logs from the last 2 days.

+- `journalctl --vacuum-size=500M` : This command helps retain logs with a maximum size of 500 MB.

+

+You can configure Journald using the conf file located at **/etc/systemd/journald.conf**. Run the following command to configure the file:

+```

+# cat /etc/systemd/journald.conf

+# This file is part of systemd.

+#

+# systemd is free software; you can redistribute it and/or modify it

+# under the terms of the GNU Lesser General Public License as published by

+# the Free Software Foundation; either version 2.1 of the License, or

+# (at your option) any later version.

+#

+# Entries in this file show the compile time defaults.

+# You can change settings by editing this file.

+# Defaults can be restored by simply deleting this file.

+#

+# See journald.conf(5) for details.

+

+[Journal]

+#Storage=auto

+#Compress=yes

+#Seal=yes

+#SplitMode=uid

+#SyncIntervalSec=5m

+#RateLimitIntervalSec=30s

+#RateLimitBurst=10000

+#SystemMaxUse=

+#SystemKeepFree=

+#SystemMaxFileSize=

+#SystemMaxFiles=100

+#RuntimeMaxUse=

+#RuntimeKeepFree=

+#RuntimeMaxFileSize=

+#RuntimeMaxFiles=100

+#MaxRetentionSec=

+#MaxFileSec=1month

+#ForwardToSyslog=no

+#ForwardToKMsg=no

+#ForwardToConsole=no

+#ForwardToWall=yes

+#TTYPath=/dev/console

+#MaxLevelStore=debug

+#MaxLevelSyslog=debug

+#MaxLevelKMsg=notice

+#MaxLevelConsole=info

+#MaxLevelWall=emerg

+#LineMax=48K

+root@photon-4a0e7f2307d4 [ ~ ]#

+```

+By default `rotate` is disable in Photon. Once the changes are made to the conf file, for the changes to take effect you must restart the systemd-journald by running the `systemctl restart systemd-journald` command.

+

+##Cloud-init Logs

+Cloud-init is the industry standard multi-distribution method for cross-platform cloud instance initialisation. 

+

+If there are with the Cloud-init behaviour, we can debug them by looking at the logs. Run the following command to look at Cloud-init logs:

+```

+journalctl -u cloud-init

+```

+

+For better understanding/debugging, You can also look at logs from the following locations:

+

+- **/var/log/cloud-init.log** : This log contains information from each stage of Cloud-init.

+- **/var/log/cloud-init-output.log** : This log contains errors, warnings, etc..

+

+

+##Syslog

+Syslog is the general standard for logging system and program messages in the Linux environment.

+

+Photon provides the following two packages to support syslog:

+

+- **syslog-ng** : syslog-ng is syslog with some advanced next gen features. It supports TLS encryption, TCP for transport with other existing features. Configurations can be added to the **/etc/syslog-ng/syslog-ng.conf** file.

+- **rsyslog** : The official RSYSLOG website defines the utility as "the rocket-fast system for log processing". rsyslog supports some advanced features like relp, imfile, omfile, gnutls protocols. Configurations can be added to the **/etc/rsyslog.conf** file. You can configure the required TLS certificates by editing the conf file.

+

+##Logs for RPMS on Photon

+Logs for a particular RPM can be checked in the following ways:

+

+- If the RPM provides a daemon, we can see the status of daemon by running `systemctl` command and check logs using `journactl -u <service name>` command.

+- For additional logs, check if a conf file is provided by the RPM by running the `rpm -ql <rpm name> | grep conf` command and find the file path of the log file. You can also check the **/var/log** folder.

+