Export Journald Logs

The Systemd Journal is a database file that contains all the system logs. In certain conditions it may be useful to be able to access a file dump of the journal content, for example for debugging purposes.
To support this use case, two utilities are made available:

Eth_journalctl Utility

The eth_journalctl is a utility script that is installed by ESF and available in the /usr/bin folder of the gateways.
It wraps the system journalctl utility providing more functionalities, mainly ESF oriented, in order to have a log representation similar to what usually observed in the kura.log file.

A description of the usage can be obtained invoking the eth_journalctl -help command:

root@boltgate2031-64:~# eth_journalctl -help

Usage: /usr/bin/eth_journalctl [-e] [-k] [-d destination file] [-u unit] [-o output format] [-f FIELD1,..,FIELDN] [-x extra-vars]
	-k enables continuous printing of new entries as they are appended to the journal. (Optional)
	-d the destination file to save the journald logs.
	-u the optional systemd unit or pattern to be used to filter the journal entries. Set to - to select all units.
	-o the output format. Supported options are: short, short-full, short-iso, short-iso-precise, short-precise, short-monotonic, short-unix, verbose, export, json, json-pretty, json-sse, cat, with-unit. Default: short
	-f a comma separated list of the fields which should be included in the output. This has an effect only for the output modes which would normally show all fields. (Optional)
	-e Selects suitable options for displaying ESF logs. This allows to print log4j logger name and exception stacktraces. Appends the "SYSLOG_IDENTIFIER,PRIORITY,ESF_LOGGER,MESSAGE,STACKTRACE" fields to current selection (see -f option), changes the output format to "verbose" and the unit to "kura" if these parameters are not explicitly set (see the -o and -u options) (Optional)
	-x extra parameters (optional)
	-h show help

Following an example of usage:

root@boltgate2031-64:~# eth_journalctl -e

Tue 2020-08-04 18:52:28.247333 UTC [s=f4476e38d4ef4b0fa693f0d30f381734;i=9f9;b=b8ecf16d6831435795af1c259e68ac23;m=e4c1aff;t=5ac11c1bb39a7;x=e2167a62199f2c9b]
    PRIORITY=4
    SYSLOG_IDENTIFIER=ESF
    LOG4J_LOGGER=org.eclipse.kura.core.data.transport.mqtt.MqttDataTransport
    MESSAGE=xxxxx  Connect failed. Forcing disconnect. xxxxx
Tue 2020-08-04 18:52:28.255242 UTC [s=f4476e38d4ef4b0fa693f0d30f381734;i=9fa;b=b8ecf16d6831435795af1c259e68ac23;m=e4c39ff;t=5ac11c1bb58a5;x=c356e0af8284bc5b]
    PRIORITY=6
    SYSLOG_IDENTIFIER=ESF
    LOG4J_LOGGER=org.eclipse.kura.core.data.transport.mqtt.MqttDataTransport
    MESSAGE=Closing client...
Tue 2020-08-04 18:52:28.263201 UTC [s=f4476e38d4ef4b0fa693f0d30f381734;i=9fb;b=b8ecf16d6831435795af1c259e68ac23;m=e4c5919;t=5ac11c1bb77bd;x=44ec1c660d284dcd]
    PRIORITY=6
    SYSLOG_IDENTIFIER=ESF
    LOG4J_LOGGER=org.eclipse.kura.core.data.transport.mqtt.MqttDataTransport
    MESSAGE=Closed
Tue 2020-08-04 18:52:28.301470 UTC [s=f4476e38d4ef4b0fa693f0d30f381734;i=9fc;b=b8ecf16d6831435795af1c259e68ac23;m=e4cefac;t=5ac11c1bc0e5e;x=cd597b041c336854]
    PRIORITY=4
    SYSLOG_IDENTIFIER=ESF
    LOG4J_LOGGER=org.eclipse.kura.core.data.DataServiceImpl
    MESSAGE=Connect failed
    STACKTRACE=org.eclipse.kura.KuraConnectException: "Connection failed. Cannot connect"
               	at org.eclipse.kura.core.data.transport.mqtt.MqttDataTransport.connect(MqttDataTransport.java:341)
               	at org.eclipse.kura.core.data.DataServiceImpl$2.run(DataServiceImpl.java:605)
               	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
               	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
               	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
               	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
               	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
               	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
               	at java.lang.Thread.run(Thread.java:748)
               Caused by: Unable to connect to server (32103) - java.net.ConnectException: Connection refused (Connection refused)
               	at org.eclipse.paho.client.mqttv3.internal.TCPNetworkModule.start(TCPNetworkModule.java:80)
               	at org.eclipse.paho.client.mqttv3.internal.ClientComms$ConnectBG.run(ClientComms.java:722)
               	... 1 more
               Caused by: java.net.ConnectException: Connection refused (Connection refused)
               	at java.net.PlainSocketImpl.socketConnect(Native Method)
               	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
               	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
               	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
               	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
               	at java.net.Socket.connect(Socket.java:589)
               	at org.eclipse.paho.client.mqttv3.internal.TCPNetworkModule.start(TCPNetworkModule.java:74)
               	... 2 more
Tue 2020-08-04 18:52:28.386296 UTC [s=f4476e38d4ef4b0fa693f0d30f381734;i=9fd;b=b8ecf16d6831435795af1c259e68ac23;m=e4e39dd;t=5ac11c1bd5884;x=6a92830a669cbb52]
    PRIORITY=6
    SYSLOG_IDENTIFIER=ESF
    LOG4J_LOGGER=AuditLogger
    MESSAGE=UI CloudConnection - Success - Successfully listed stack pids by factory for user: admin, session node0sias7bdsmkorxf41hb7o9ymq0
Tue 2020-08-04 18:52:28.634363 UTC [s=f4476e38d4ef4b0fa693f0d30f381734;i=9fe;b=b8ecf16d6831435795af1c259e68ac23;m=e52041f;t=5ac11c1c122d0;x=aaf76d431a20d4ce]
    PRIORITY=6
    SYSLOG_IDENTIFIER=ESF
    LOG4J_LOGGER=AuditLogger
    MESSAGE=UI Component - Success - Successfully listed component configs for user: admin, session node0sias7bdsmkorxf41hb7o9ymq0

Esf-journal-tools Docker Container

The esf-journal-tools is a docker container that provides the eth_journalctl capabilities to the developer workstation.
It allows to convert a journal database file retrieved from a gateway to a parseable text file even in non Linux systems like Mac or Windows.

The usage is the following:

  • the developer needs to download to his machine the gateway journal log file, for example using SSH
  • change his working directory to the one where the downloaded journal file is located
  • execute the following command:
docker run -v "$(pwd):/tmp/journal" --rm esf-journal-tools:latest

This will produce a resulting text file will all the logs in the original journal database file.