1. devstack
  2. Commit 59ce1d902e2137bb7346a0d1f223e0ce1cb83216
Browse code

Export all journal logs

Currently we only export the devstack@ services, and then separately
export the kernel & sudo logs to syslog.txt.

This leaves a lot of logs potentially behind in the journal for
various daemons. Just export the whole lot.

Using this output is currently very opaque and makes use of systemd
export tools that are very un-discoverable. Add a README that will
appear alongside the journal explaining how to actually use it. This
is a template as it would be nice to put into things like the list of
services that are in the journal, or maybe other magic.

Also make sure we export the logs since the start timestamp; currently
during a full run we drop the initial logs.

Change-Id: Id2626f9113d82c6d524039acda8a8ec74afb2081

Ian Wienand authored on 2019/02/11 10:27:10
Showing 3 changed files
roles/export-devstack-journal/README.rst
History View file @ 59ce1d9
... ... 
@@ -1,11 +1,15 @@
1 1 
 Export journal files from devstack services
2 2
3 
-Export the systemd journal for every devstack service in native
4 
-journal format as well as text.  Also, export a syslog-style file with
5 
-kernal and sudo messages.
3 
+This performs a number of logging collection services
6 4
7 
-Writes the output to the ``logs/`` subdirectory of
8 
-``stage_dir``.
5 
+* Export the systemd journal in native format
6 
+* For every devstack service, export logs to text in a file named
7 
+  ``screen-*`` to maintain legacy compatability when devstack services
8 
+  used to run in a screen session and were logged separately.
9 
+* Export a syslog-style file with kernel and sudo messages for legacy
10 
+  compatability.
11 
+
12 
+Writes the output to the ``logs/`` subdirectory of ``stage_dir``.
9 13
10 14 
 **Role Variables**
11 15
roles/export-devstack-journal/tasks/main.yaml
History View file @ 59ce1d9
... ... 
@@ -6,32 +6,49 @@
6 6 
     state: directory
7 7 
     owner: "{{ ansible_user }}"
8 8
9 
-# TODO: convert this to ansible
10 
-- name: Export journal files
9 
+- name: Export legacy stack screen log files
11 10 
   become: true
12 11 
   shell:
13 12 
     cmd: |
14 13 
       u=""
15 14 
       name=""
16 
-      for u in `systemctl list-unit-files | grep devstack | awk '{print $1}'`; do
15 
+      for u in $(systemctl list-unit-files | grep devstack | awk '{print $1}'); do
17 16 
         name=$(echo $u | sed 's/devstack@/screen-/' | sed 's/\.service//')
18 17 
         journalctl -o short-precise --unit $u | gzip - > {{ stage_dir }}/logs/$name.txt.gz
19 18 
       done
20 19
21 
-      # Export the journal in export format to make it downloadable
22 
-      # for later searching. It can then be rewritten to a journal native
23 
-      # format locally using systemd-journal-remote. This makes a class of
24 
-      # debugging much easier. We don't do the native conversion here as
25 
-      # some distros do not package that tooling.
26 
-      journalctl -u 'devstack@*' -o export | \
27 
-          xz --threads=0 - > {{ stage_dir }}/logs/devstack.journal.xz
28 
-
29 
-      # The journal contains everything running under systemd, we'll
30 
-      # build an old school version of the syslog with just the
31 
-      # kernel and sudo messages.
20 
+- name: Export legacy syslog.txt
21 
+  become: true
22 
+  shell:
23 
+    # The journal contains everything running under systemd, we'll
24 
+    # build an old school version of the syslog with just the
25 
+    # kernel and sudo messages.
26 
+    cmd: |
32 27 
       journalctl \
33 28 
           -t kernel \
34 29 
           -t sudo \
35 30 
           --no-pager \
36 31 
           --since="$(cat {{ devstack_base_dir }}/log-start-timestamp.txt)" \
37 32 
         | gzip - > {{ stage_dir }}/logs/syslog.txt.gz
33 
+
34 
+# TODO: convert this to ansible
35 
+#  - make a list of the above units
36 
+#  - iterate the list here
37 
+- name: Export journal
38 
+  become: true
39 
+  shell:
40 
+    # Export the journal in export format to make it downloadable
41 
+    # for later searching. It can then be rewritten to a journal native
42 
+    # format locally using systemd-journal-remote. This makes a class of
43 
+    # debugging much easier. We don't do the native conversion here as
44 
+    # some distros do not package that tooling.
45 
+    cmd: |
46 
+      journalctl -o export \
47 
+          --since="$(cat {{ devstack_base_dir }}/log-start-timestamp.txt)" \
48 
+        | xz --threads=0 - > {{ stage_dir }}/logs/devstack.journal.xz
49 
+
50 
+- name: Save journal README
51 
+  become: true
52 
+  template:
53 
+    src: devstack.journal.README.txt.j2
54 
+    dest: '{{ stage_dir }}/logs/devstack.journal.README.txt'
roles/export-devstack-journal/templates/devstack.journal.README.txt.j2
History View file @ 59ce1d9
38 55 
new file mode 100644
... ... 
@@ -0,0 +1,33 @@
0 
+Devstack systemd journal
1 
+========================
2 
+
3 
+The devstack.journal file is a copy of the systemd journal during the
4 
+devstack run.
5 
+
6 
+To use it, you will need to convert it so journalctl can read it
7 
+locally.  After downloading the file:
8 
+
9 
+ $ /lib/systemd/systemd-journal-remote <(xzcat ./devstack.journal.xz) -o output.journal
10 
+
11 
+Note this binary is not in the regular path.  On Debian/Ubuntu
12 
+platforms, you will need to have the "sytemd-journal-remote" package
13 
+installed.
14 
+
15 
+It should result in something like:
16 
+
17 
+ Finishing after writing <large number> entries
18 
+
19 
+You can then use journalctl to examine this file.  For example, to see
20 
+all devstack services try:
21 
+
22 
+ $ journalctl --file ./output.journal -u 'devstack@*'
23 
+
24 
+To see just cinder API server logs restrict the match with
25 
+
26 
+ $ journalctl --file ./output.journal -u 'devstack@c-api'
27 
+
28 
+There may be many types of logs available in the journal, a command like
29 
+
30 
+ $ journalctl --file ./output.journal --output=json-pretty | grep "_SYSTEMD_UNIT" | sort -u
31 
+
32 
+can help you find interesting things to filter on.
0 33 
\ No newline at end of file