ChangelogDownloads
HomeGuidesChangelog
ChangelogDownloadsSupportMulti-service IoT Edge GatewaysEUROTECH
Guides
SupportMulti-service IoT Edge GatewaysEUROTECH

Usage Examples

Suggest Edits

This section provides a list of possible use cases and some notes on suggested configurations for the Log Analytics components.

Framework Configuration Change

A change in configuration is usually associated to a set of messages printed by the framework in the logs. An example for a configuration change triggered by a cloud request:

Aug 21 15:32:01 reliagate-10-12 ESF[2059]: Message arrived on topic: $EDC/ethdev/F8:36:9B:E8:BD:2B/CONF-V1/PUT/configurations/org.eclipse.kura.clock.ClockService
Aug 21 15:32:01 reliagate-10-12 ESF[2059]: Received new Configuration
Aug 21 15:32:02 reliagate-10-12 ESF[2059]: Storing message on topic: $EDC/#account-name/MQTT-2088719019/CONF-V1/REPLY/1598023920721-7729789662080656611, priority: 1
Aug 21 15:32:02 reliagate-10-12 ESF[2059]: Stored message on topic: $EDC/#account-name/MQTT-2088719019/CONF-V1/REPLY/1598023920721-7729789662080656611, priority: 1
Aug 21 15:32:02 reliagate-10-12 ESF[2059]: Publishing message on topic: $EDC/ethdev/MQTT-2088719019/CONF-V1/REPLY/1598023920721-7729789662080656611 with QoS: 0
Aug 21 15:32:03 reliagate-10-12 ESF[2059]: Updating configurations
Aug 21 15:32:04 reliagate-10-12 ESF[2059]: Message arrived on topic: $EDC/ethdev/F8:36:9B:E8:BD:2B/CONF-V1/GET/configurations
Aug 21 15:32:06 reliagate-10-12 ESF[2059]: The ppp0 interface is not supported
Aug 21 15:32:06 reliagate-10-12 ESF[2059]: WPA in client mode is not configured
Aug 21 15:32:06 reliagate-10-12 ESF[2059]: getWifiHostConfig() :: hostapd-wlan1.conf file doesn't exist, will generate default wifiConfig
Aug 21 15:32:06 reliagate-10-12 ESF[2059]: Storing message on topic: $EDC/#account-name/MQTT-2088719019/CONF-V1/REPLY/1598023923840--4910622246701157415, priority: 1
Aug 21 15:32:07 reliagate-10-12 ESF[2059]: Stored message on topic: $EDC/#account-name/MQTT-2088719019/CONF-V1/REPLY/1598023923840--4910622246701157415, priority: 1
Aug 21 15:32:07 reliagate-10-12 ESF[2059]: Publishing message on topic: $EDC/ethdev/MQTT-2088719019/CONF-V1/REPLY/1598023923840--4910622246701157415 with QoS: 0
Aug 21 15:32:07 reliagate-10-12 ESF[2059]: Loading init configurations from: 1598023821925...
Aug 21 15:32:09 reliagate-10-12 ESF[2059]: Merging configuration for pid: org.eclipse.kura.clock.ClockService
Aug 21 15:32:09 reliagate-10-12 ESF[2059]: Updating Configuration of ConfigurableComponent org.eclipse.kura.clock.ClockService ... Done.
Aug 21 15:32:09 reliagate-10-12 ESF[2059]: Updated...
Aug 21 15:32:10 reliagate-10-12 ESF[2059]: Writing snapshot - Saving /opt/eurotech/esf/user/snapshots/snapshot_1598023929298.xml...
Aug 21 15:32:10 reliagate-10-12 ESF[2059]: Writing snapshot - Saving /opt/eurotech/esf/user/snapshots/snapshot_1598023929298.xml... Done.

Similar messages are reported by the framework when the configuration change for a service is requested locally from the ESF UI or from a snapshot update (local or remote) triggered by the Upload and Apply feature.
To be able to identify and report to the cloud platform the messages associated to such change applied to the framework, all the log messages produced by the ESF configuration service need to be parsed and published. To achieve this, the instance of the LogManager needs to be configured with key ESF_LOGGER and value org.eclipse.kura.core.configuration.*
Following the snapshot that wraps such configuration:

<?xml version="1.0" encoding="UTF-8"?><esf:configurations xmlns:esf="http://eurotech.com/esf/2.0" xmlns:ocd="http://www.osgi.org/xmlns/metatype/v1.2.0">
    <esf:configuration pid="ExampleLogManager">
        <esf:properties>
            <esf:property array="false" encrypted="false" name="rate.limit.enabled" type="Boolean">
                <esf:value>true</esf:value>
            </esf:property>
            <esf:property array="false" encrypted="false" name="CloudPublisher.target" type="String">
                <esf:value>(kura.service.pid=default.log.publisher)</esf:value>
            </esf:property>
            <esf:property array="false" encrypted="false" name="service.factoryPid" type="String">
                <esf:value>com.eurotech.framework.log.manager.LogManager</esf:value>
            </esf:property>
            <esf:property array="true" encrypted="false" name="key.regex" type="String">
                <esf:value>ESF_LOGGER</esf:value>
                <esf:value/>
                <esf:value/>
                <esf:value/>
                <esf:value/>
            </esf:property>
            <esf:property array="false" encrypted="false" name="kura.service.pid" type="String">
                <esf:value>LogManagerDefault</esf:value>
            </esf:property>
            <esf:property array="false" encrypted="false" name="service.pid" type="String">
                <esf:value>com.eurotech.framework.log.manager.LogManager-1605263882392-8</esf:value>
            </esf:property>
            <esf:property array="false" encrypted="false" name="rate.limit.time.unit" type="String">
                <esf:value>MINUTES</esf:value>
            </esf:property>
            <esf:property array="false" encrypted="false" name="rate.limit.average" type="Long">
                <esf:value>10</esf:value>
            </esf:property>
            <esf:property array="false" encrypted="false" name="LogReader.service.pid" type="String">
                <esf:value>LogReaderJournald</esf:value>
            </esf:property>
            <esf:property array="false" encrypted="false" name="context.cache.size" type="Integer">
                <esf:value>0</esf:value>
            </esf:property>
            <esf:property array="false" encrypted="false" name="rate.limit.burst.size" type="Integer">
                <esf:value>200</esf:value>
            </esf:property>
            <esf:property array="true" encrypted="false" name="value.regex" type="String">
                <esf:value>org.eclipse.kura.core.configuration.*</esf:value>
                <esf:value/>
                <esf:value/>
                <esf:value/>
                <esf:value/>
            </esf:property>
            <esf:property array="false" encrypted="false" name="duplicate.filter.enabled" type="Boolean">
                <esf:value>true</esf:value>
            </esf:property>
        </esf:properties>
    </esf:configuration>
</esf:configurations>

If the customer is just interested in getting log messages related to the configuration change of a single component running in the framework, a possible alternative to the previous configuration is the following: 

<?xml version="1.0" encoding="UTF-8"?><esf:configurations xmlns:esf="http://eurotech.com/esf/2.0" xmlns:ocd="http://www.osgi.org/xmlns/metatype/v1.2.0">
    <esf:configuration pid="ExampleLogManager">
        <esf:properties>
            <esf:property array="false" encrypted="false" name="rate.limit.enabled" type="Boolean">
                <esf:value>true</esf:value>
            </esf:property>
            <esf:property array="false" encrypted="false" name="CloudPublisher.target" type="String">
                <esf:value>(kura.service.pid=default.log.publisher)</esf:value>
            </esf:property>
            <esf:property array="false" encrypted="false" name="service.factoryPid" type="String">
                <esf:value>com.eurotech.framework.log.manager.LogManager</esf:value>
            </esf:property>
            <esf:property array="true" encrypted="false" name="key.regex" type="String">
                <esf:value>ESF_LOGGER</esf:value>
                <esf:value>MESSAGE</esf:value>
                <esf:value/>
                <esf:value/>
                <esf:value/>
            </esf:property>
            <esf:property array="false" encrypted="false" name="kura.service.pid" type="String">
                <esf:value>LogManagerDefault</esf:value>
            </esf:property>
            <esf:property array="false" encrypted="false" name="service.pid" type="String">
                <esf:value>com.eurotech.framework.log.manager.LogManager-1605263882392-8</esf:value>
            </esf:property>
            <esf:property array="false" encrypted="false" name="rate.limit.time.unit" type="String">
                <esf:value>MINUTES</esf:value>
            </esf:property>
            <esf:property array="false" encrypted="false" name="rate.limit.average" type="Long">
                <esf:value>10</esf:value>
            </esf:property>
            <esf:property array="false" encrypted="false" name="LogReader.service.pid" type="String">
                <esf:value>LogReaderJournald</esf:value>
            </esf:property>
            <esf:property array="false" encrypted="false" name="context.cache.size" type="Integer">
                <esf:value>0</esf:value>
            </esf:property>
            <esf:property array="false" encrypted="false" name="rate.limit.burst.size" type="Integer">
                <esf:value>200</esf:value>
            </esf:property>
            <esf:property array="true" encrypted="false" name="value.regex" type="String">
                <esf:value>org.eclipse.kura.linux.clock.ClockServiceImpl</esf:value>
                <esf:value>Updated</esf:value>
                <esf:value/>
                <esf:value/>
                <esf:value/>
            </esf:property>
            <esf:property array="false" encrypted="false" name="duplicate.filter.enabled" type="Boolean">
                <esf:value>true</esf:value>
            </esf:property>
        </esf:properties>
    </esf:configuration>
</esf:configurations>

In this example, the LogManager will publish to the cloud only the LogEntries that are produced by the Clock Service and contain in the LogEntry message the word Updated.

Single Message to Notify a Configuration Change

In ESF 6.X.Y there is not a single message that can allow to identify if a configuration change has been requested, for example, from cloud or locally.

From cloud requests only, the user can define a LogManager instance that filters using key MESSAGE and value Updating configurations. This will provide to the cloud a log message every time a configuration update is requested from a cloud request.

At the opposite, the user can define a LogManager that filters using key MESSAGE and value Updating Configuration of ConfigurableComponent*.
This will trigger every time a component configuration is triggered locally or remotely. but it will also be triggered at framework boot.

🚧

This rule will also be triggered at framework boot. This can produce messages that identify a configuration change, but not triggered by a human request.

Linux User Access

In order to identify and report a user access to the system via serial port, SSH via local interface or SSH via VPN, the following configuration can be applied to a LogManager instance: 

<?xml version="1.0" encoding="UTF-8"?><esf:configurations xmlns:esf="http://eurotech.com/esf/2.0" xmlns:ocd="http://www.osgi.org/xmlns/metatype/v1.2.0">
    <esf:configuration pid="LogManagerAuth">
        <esf:properties>
            <esf:property array="false" encrypted="false" name="rate.limit.enabled" type="Boolean">
                <esf:value>true</esf:value>
            </esf:property>
            <esf:property array="false" encrypted="false" name="CloudPublisher.target" type="String">
                <esf:value>(kura.service.pid=default.log.publisher)</esf:value>
            </esf:property>
            <esf:property array="false" encrypted="false" name="service.factoryPid" type="String">
                <esf:value>com.eurotech.framework.log.manager.LogManager</esf:value>
            </esf:property>
            <esf:property array="true" encrypted="false" name="key.regex" type="String">
                <esf:value>SYSLOG_FACILITY</esf:value>
                <esf:value>_AUDIT_TYPE_NAME</esf:value>
                <esf:value/>
                <esf:value/>
                <esf:value/>
            </esf:property>
            <esf:property array="false" encrypted="false" name="kura.service.pid" type="String">
                <esf:value>LogManagerAuth</esf:value>
            </esf:property>
            <esf:property array="false" encrypted="false" name="service.pid" type="String">
                <esf:value>com.eurotech.framework.log.manager.LogManager-1605263882491-9</esf:value>
            </esf:property>
            <esf:property array="false" encrypted="false" name="rate.limit.time.unit" type="String">
                <esf:value>MINUTES</esf:value>
            </esf:property>
            <esf:property array="false" encrypted="false" name="rate.limit.average" type="Long">
                <esf:value>10</esf:value>
            </esf:property>
            <esf:property array="false" encrypted="false" name="LogReader.service.pid" type="String">
                <esf:value>LogReaderJournald</esf:value>
            </esf:property>
            <esf:property array="false" encrypted="false" name="context.cache.size" type="Integer">
                <esf:value>0</esf:value>
            </esf:property>
            <esf:property array="false" encrypted="false" name="rate.limit.burst.size" type="Integer">
                <esf:value>200</esf:value>
            </esf:property>
            <esf:property array="true" encrypted="false" name="value.regex" type="String">
                <esf:value>^4</esf:value>
                <esf:value>USER_LOGIN</esf:value>
                <esf:value/>
                <esf:value/>
                <esf:value/>
            </esf:property>
            <esf:property array="false" encrypted="false" name="duplicate.filter.enabled" type="Boolean">
                <esf:value>false</esf:value>
            </esf:property>
        </esf:properties>
    </esf:configuration>
</esf:configurations>

The proposed configuration uses the following parameters for the key-value regex filter:

  • Key Regex expressions:
    • SYSLOG_FACILITY
    • _AUDIT_TYPE_NAME
  • Value Regex expressions:
    • ^4
    • USER_LOGIN

The proposed configuration allows to identify successful and failed accesses to the system.

Update Requests

This section will provide information on how to configure a LogManager to identify (and publish) LogEntries related to Update Requests.

DEPLOY-V2

DEPLOY-V2 updates involve bundle updates or shell script updates. In both cases the interesting log messages should fall in the following two categories:

  • messages logged by the framework (by the DEPLOY-V2 cloudlet implementation).
    • messages logged during the download phase
    • messages logged during deployment package installation
    • messages logged when the verifier script is executed
  • messages logged by the update script or by the updated bundles

To capture messages logged by the DEPLOY-V2 cloudlet implementation, a LogManager instance should be configured with:

  • Key Regex expressions: ESF_LOGGER
  • Value Regex expressions: org[.]eclipse[.]kura[.]core[.]deployment.*

Detecting a DEPLOY-V2 update

In order to publish a single log entry that reports that an update related DEPLOY-V2 request has been received the following filter can be used:

  • Key Regex expressions: MESSAGE
  • Value Regex expressions: Message arrived on topic: \$EDC/[^/]*/[^/]*/DEPLOY-V2/EXEC/(?:download)|(?:install)|(?:uninstall)

CMD-V1

It is possible to capture messages logged by the CMD-V1 cloudlet with the following LogManager configuration:

  • Key Regex expressions: ESF_LOGGER
  • Value Regex expressions: org[.]eclipse[.]kura[.]cloud[.]app[.]command.*

This will publish a log entry when the request is received and another one in case the command exits with a non-zero status. In case the script completes successfully, only the first message will be logged.

Detecting a CMD-V1 update

In order to publish a single log entry that reports that a CMD-V1 request has been received the following filter can be used:

  • Key Regex expressions: MESSAGE
  • Value Regex expressions: Message arrived on topic: \$EDC/[^/]*/[^/]*/CMD-V1/EXEC/command

or

  • Key Regex expressions: MESSAGE
  • Value Regex expressions: EXECuting resource: command

Installing/Upgrading a dp through web UI

Performing this operation produces the same logs described in the Messages logged by a dp installation/upgrade section below plus the messages below by FileServlet

Wed 2020-08-26 14:15:50.746548 UTC [s=ce1453b2d05040ceb41274751f9c2682;i=1140;b=4e95e25d7e8043e6bd15a474cc1f7c5a;m=569663f07;t=5adc874f3b7ef;x=d17bfd85edda7231]
    PRIORITY=6
    SYSLOG_IDENTIFIER=ESF
    MESSAGE=Copying uploaded package file to file: /tmp/com.eurotech.framework.mbus.driver.armv7hf.feature_1.0.0.dp
    ESF_LOGGER=org.eclipse.kura.web.server.servlet.FileServlet
Wed 2020-08-26 14:15:50.769711 UTC [s=ce1453b2d05040ceb41274751f9c2682;i=1141;b=4e95e25d7e8043e6bd15a474cc1f7c5a;m=5696697fb;t=5adc874f410d4;x=6eb7977b89d614fb]
    PRIORITY=6
    SYSLOG_IDENTIFIER=ESF
    ESF_LOGGER=org.eclipse.kura.web.server.servlet.FileServlet
    MESSAGE=Installing package...

Detecting a dp install through the Web UI

Publishing all messages originated by FileServlet might include spurious messages. In order to publish a single log entry that reports that a CMD-V1 request has been received the following filter can be used:

  • Key Regex expressions:
    • MESSAGE
    • ESF_LOGGER
  • Value Regex expressions:
    • Installing package...
    • org[.]eclipse[.]kura[.]web[.]server[.]servlet[.]FileServlet

or

  • Key Regex expressions:
    • MESSAGE
    • ESF_LOGGER
  • Value Regex expressions:
    • Copying uploaded package file to file: .*
    • org[.]eclipse[.]kura[.]web[.]server[.]servlet[.]FileServlet

Detecting a dp install from Eclipse Marketplace

In order to publish a single log entry that reports that an install operation from Eclipse Marketplace has started, the following filter can be used:

  • Key Regex expressions: ESF_LOGGER
  • Value Regex expressions: Installing deployment package from Eclipse Marketplace, URL .*[.][.][.]$

Executing a command through the Web UI

Executing commands from the Web UI from the Device -> Command section do not seem to produce log entries generated by the framework.

Update Script

This section describes how to forward to journald messages produced by an update script triggered by CMD-V1 or DEPLOY-V2 and how to configure a LogManager instance to publish such messages to the cloud.

A way for simplifying this process can be executing the update script as a systemd unit with a well known name. The well known name can be used to configure a LogManager, for example if the update unit is called my-update.service, the following LogManager configuration should work:

  • Key Regex expressions: _SYSTEMD_UNIT
  • Value Regex expressions: my-update.service

Executing the update as a systemd unit has the following advantages regarding log collection:

  • allows to collect the log published by all child processes of the update script
  • allows to collect messages delivered to journald with different transports:
    • stdout
    • sd-journal APIs
      -syslog (assuming that syslog messages reach journald)

The systemd-run command line tool can be used as a simple way to run an update as systemd unit. This system has the advantage that it allows to spawn a process as a transient systemd unit, without the need of creating configuration files.

If the same unit name is used for all update script executions, a single LogManager instance is enough for collecting the generated log entries.

🚧

Note that systemd allows at most a single unit with a given name to exist at a time. This implies that using systemd-run will fail if an execution is started when a previous execution with the same unit name is still running.

🚧

If the --unit argument is not passed to `systemd-run, it will start a unit with an automatically generated name.

As an alternative, instead of relying a well known unit name it is possible to run the update scripts inside a well-known systemd slice.
This can be done by passing the --slice= argument to systemd-run, without specifying the `--unit argument.

If this is done the LogManager filter can be changed as follows, assuming that the `system-ethupdate.slice slice name is used:

  • Key Regex expressions: _SYSTEMD_SLICE
  • Value Regex expressions: system-ethupdate[.]slice

🚧

Note that dashes `- in slice name express parent-child relationships.

This change would allow to have multiple update scripts executing concurrently, the log entries generated by them can be reported by the single LogManager instance shown above

DP installation/upgrade

The messages generated by the framework in case of dp installation/upgrade can be captured with the following LogManager configuration:

  • Key Regex expressions: ESF_LOGGER
  • Value Regex expressions: org[.]eclipse[.]kura[.]deployment[.]agent.*

🚧

This will likely cause the publishing of log entries during system startup, when the dps are installed

ESF UI User Access

In ESF 6.X.Y there is no specific message in logs that identifies a new access by a user to the ESF Web UI.
When the ESF page is loaded the first time or reloaded, the following messages are registered in the log:

Sep 14 12:02:07 reliagate-10-12 ESF[2147]: Resource File /skin.js does not exist
Sep 14 12:02:10 reliagate-10-12 ESF[2147]: The 1-1.3 interface is not supported
Sep 14 12:02:10 reliagate-10-12 ESF[2147]: WPA in client mode is not configured
Sep 14 12:02:10 reliagate-10-12 ESF[2147]: WPA in client mode is not configured
Sep 14 12:02:10 reliagate-10-12 ESF[2147]: getWifiHostConfig() :: hostapd-wlan1.conf file doesn't exist, will generate default wifiConfig
Sep 14 12:02:10 reliagate-10-12 ESF[2147]: getModemConfig() :: PPPD peer file does not exist - /etc/ppp/peers/LE910_1-1.3

The only clear UI message is the first one, while the others are triggered by the various resources loaded and triggered by the UI.

Considering this, a possible filter in ESF 6.X.Y (if the customer has not applied a custom skin) is to filter by message with something like:

  • Key Regex expressions: MESSAGE
  • Value Regex expressions: Resource File /skin.js does not exist

Updated about 4 years ago