.. highlight:: rst .. _ql2sc: ##### ql2sc ##### **QuakeLink (gempa GmbH) to SeisComP3 event parameter exchange** Description =========== ql2sc manages the import of SC3 objects from one or several QuakeLink servers into a SeisComP3 system in real time. Like :ref:`scimex` but contrary to :ref:`scimport` the exchange of the SC3 objects is event based. This means no messages will be exchanged until the exporting system has produced an event. The user may control at various levels which information to import. Whenever possible server-side filters should be preferred to reduce both the network bandwidth consumption as well as the CPU and memory utilization on the local machine. .. _ql2sc_event_filter: Server-side event filter ======================== QuakeLink provides a filter syntax similar to SQL-WHERE clauses which may be used to filter interesting events on the server side: .. code-block:: none clause := condition[ AND|OR [(]clause[)]] condition := MAG|DEPTH|LAT|LON|PHASES|OTIME|UPDATED [op float|time]|[IS [NOT] NULL] op := =|>|>=|<|<=|eq|gt|ge|lt|ge time := %Y,%m,%d[,%H,%M,%S,%f] E.g. the following filter string would select only those events with a minimum magnitude of 6, detected by at least 10 stations and which are shallower than 100km: .. code-block:: sql MAG >= 6.0 AND PHASES >= 10 AND DEPTH < 100 .. _ql2sc_object_filter: Server-side object filter ========================= QuakeLink provides a coarse object filter for the most relevant SC3 objects: ============ ============================================================== Option Impact ============ ============================================================== picks include picks amplitudes include amplitudes arrivals include origin arrivals staMags include origin station magnitudes staMts include moment tensor station contributions and phase settings preferred include only preferred origin and magnitude information ============ ============================================================== .. _routing: Local object filter and routing =============================== Subsequent to the server-side filters a routing table defines which objects to import and to which message group to send them. Depending on the SC3 modules listening to the specified message groups an object may be further processed. Typically no modules (other than :ref:`scmaster`) is connected to the ``IMPORT_GROUP`` so that objects send to this group are just stored to the database. If an object should be discarded the special group identifier ``NULL`` may be used. The routing table is defined as a comma-separated list of ``object name:group name`` pairs. Also the routing rules are inherited recursively within the SC3 object tree. If no explicit rule exists for an object the routing of its parent is evaluated up to the ``EventParameters`` root node. Examples: .. code-block:: none EventParameters:IMPORT_GROUP Imports everything .. code-block:: none EventParameters:IMPORT_GROUP,Comment:NULL Imports everything except comments .. code-block:: none Origin:LOCATION,StationMagnitude:MAGNITUDE,Magnitude:MAGNITUDE Sends origins and it's children arrival, origin uncertainty to the ``LOCATION`` group but the magnitude children to the ``MAGNITUDE`` group. Skips picks, amplitudes, focal mechanisms and events. .. _agency_filter: Agency list filter ================== In addition to the local object filter the user may choose to accept only those objects originating from a set of trusted agencies. If at least one agency is defined in the ``processing.whitelist.agencies`` or ``processing.blacklist.agencies`` configuration option, then the ``creationInfo.agencyID`` of amplitudes, arrivals, comments, events, focal mechanisms, magnitudes, moment tensors, origins, picks and station magnitudes is evaluated. Objects with unmatched or unset agency information are filtered out. If objects with unset agency id should match then empty string ``""`` has to be added to the white list. The agency filter is applied on remote as well as local objects. In this way remote objects may be excluded from import and local objects my be protected from overriding or removing. Also the filter is applied recursively. If parent object (e.g. an origin) is filtered out all of its children (e.g. magnitudes) are also skipped even if they carry a different agency id. .. note:: The agency white list filter might be essential to avoid circular event updates between cross-connected SC3 systems. Workflow ======== Each event update received from a QuakeLink host is parsed and analyzed for differences to the local database. The comparison starts at the level of the top-level elements in the following order: picks, amplitudes, origins, focal mechanisms, events. For each top-level element the object tree is traversed in a depth-first search order. Objects on the same level are processed in the order of their appearance. The differences are collected as a list of notifier objects with the following operation types: ====== =========== Type Description ====== =========== ADD The object does not exist locally UPDATE The object does exist locally but differs from the remote one REMOVE The object exist locally but not remotely ====== =========== The ``ADD`` and ``REMOVE`` operation always generates notifies of the same type for all children of the current object. ``ADD`` notifiers are collected top-down, ``REMOVE`` notifiers are collected bottom-up. Because the order of child objects is arbitrary, e.g. the arrivals of an origin, each object on the remote side has to be found in the set of local objects. For public objects (e.g. origins, magnitudes, magnitudes..), the ``publicID`` property is used for comparison. All other objects are compared by looking at their index properties. For e.g. arrivals this is the ``pickID`` property, for comments the ``id`` property. Ones all notifiers are collected they are send to the local messaging system. For performance reasons and because of the processing logic of listening SC3 modules ql2sc tries to batch as many notifiers as possible into one notifier message. A separate notifier message is created if the target message group changes between successive notifiers or if the configurable :ref:`batchSize` limit is reached. .. note:: Care must be taken when configuring the ``batchSize`` limit. If the value is to big the overall message size limit (default: 1MB) may be exceeded resulting in an undeliverable message. On the other hand a much to small value will create unwanted results in the SC3 processing chain. If for instances picks are routed to the ``PICK`` group and the pick set is split into several notifier messages the local :ref:`scautoloc` might create locations based on an incomplete dataset. Caveats ======= Specific combinations of remote and local object filters may result in the loss of data. If for instance origins are imported from system ``A`` to ``B`` and additional magnitudes for the received origins are calculated on ``B`` care must be taken. Without protection a new event update containing the same origin will ``REMOVE`` all newly calculated magnitudes on ``B`` since they are not included in the magnitude set sent by ``A``. To avoid losing these local magnitudes one may decide to block magnitudes from import by routing them to ``NULL``. If magnitudes from ``A`` and from ``B`` should be available an :ref:`agency filter` may be defined. Make sure ``A`` and ``B`` uses distinct agency IDs and add the agency ID of ``B`` to ``processing.blacklist.agencies``. Configuration ============= | :file:`etc/defaults/global.cfg` | :file:`etc/defaults/ql2sc.cfg` | :file:`etc/global.cfg` | :file:`etc/ql2sc.cfg` | :file:`~/.seiscomp3/global.cfg` | :file:`~/.seiscomp3/ql2sc.cfg` ql2sc inherits :ref:`global options`. .. confval:: backLog Type: *int* Number of seconds to fetch missed updates on start up. Default is ``1800``. .. confval:: cacheSize Type: *int* Number of public objects to cache. Default is ``5000``. .. confval:: batchSize Type: *int* Maximum number of notifiers to batch in one message. If set to 0 no size limit is enforced. Default is ``200``. .. confval:: hosts Type: *list:string* Registration of QuakeLink hosts. .. note:: **host.\$name.\*** *For each host profile a connection to a QuakeLink* *server is established.* \$name is a placeholder for the name to be used and needs to be added to :confval:`hosts` to become active. .. code-block:: sh hosts = a,b host.a.value1 = ... host.b.value1 = ... # c is not active because it has not been added # to the list of hosts host.c.value1 = ... .. confval:: host.\$name.url Type: *string* URL of the QuakeLink service, the scheme 'qls' enables SSL. Format: [ql[s]:\/\/][user:pwd\@][host][:port]. If set to an empty string the application will run without any QuakeLink connection attempt. Default is ``ql://localhost:18010``. .. confval:: host.\$name.gzip Type: *boolean* Enables\/disables GZip \(GNU zip\) compression. Default is ``false``. .. confval:: host.\$name.native Type: *boolean* Request native data instead of XML format. Native data export may be disabled on some hosts. Default is ``false``. .. confval:: host.\$name.keepAlive Type: *boolean* Request server to send keep alive message every 30s to prevent connection reset by firewalls on long idle periods. If activated the client will reset the connection if no alive message is received within 60s. Default is ``false``. .. confval:: host.\$name.filter Type: *string* SQL like WHERE clause to filter the result set. clause :\= condition[ AND\|OR [\(]clause[\)]] __ condition :\= MAG\|DEPTH\|LAT\|LON\|PHASES\|OTIME\|UPDATED [op float\|time]\|[IS [NOT] NULL] __ op :\= \=\|>\|>\=\|<\|<\=\|eq\|gt\|ge\|lt\|ge __ time :\= %Y,%m,%d[,%H,%M,%S,%f] .. confval:: host.\$name.routingTable Type: *list:string* Maps datamodel class names to messaging groups. For unmapped objects the mapping of their parent objects is evaluated recursively. Objects may be excluded by mapping them to 'NULL'. Default is ``Pick:IMPORT_GROUP,Amplitude:IMPORT_GROUP,Origin:LOCATION,StationMagnitude:MAGNITUDE,Magnitude:MAGNITUDE``. .. note:: **host.\$name.data.\*** *QuakeLink options specifying which XML components to fetch.* *Note: These options are not used if 'native' data is requested.* .. confval:: host.\$name.data.picks Type: *boolean* Include picks Default is ``true``. .. confval:: host.\$name.data.amplitudes Type: *boolean* Include amplitudes Default is ``true``. .. confval:: host.\$name.data.arrivals Type: *boolean* Include origin arrivals Default is ``true``. .. confval:: host.\$name.data.staMags Type: *boolean* Include origin station magnitudes Default is ``true``. .. confval:: host.\$name.data.staMts Type: *boolean* Include moment tensor station contributions and phase settings Default is ``true``. .. confval:: host.\$name.data.preferred Type: *boolean* Include only preferred origin and magnitude information Default is ``true``. Command-line ============ .. program:: ql2sc Generic ------- .. option:: -h, --help show help message. .. option:: -V, --version show version information .. option:: --config-file arg Use alternative configuration file. When this option is used the loading of all stages is disabled. Only the given configuration file is parsed and used. To use another name for the configuration create a symbolic link of the application or copy it, eg scautopick \-> scautopick2. .. option:: --plugins arg Load given plugins. .. option:: -D, --daemon Run as daemon. This means the application will fork itself and doesn't need to be started with \&. Verbosity --------- .. option:: --verbosity arg Verbosity level [0..4]. 0:quiet, 1:error, 2:warning, 3:info, 4:debug .. option:: -v, --v Increase verbosity level \(may be repeated, eg. \-vv\) .. option:: -q, --quiet Quiet mode: no logging output .. option:: --print-component arg For each log entry print the component right after the log level. By default the component output is enabled for file output but disabled for console output. .. option:: --component arg Limits the logging to a certain component. This option can be given more than once. .. option:: -s, --syslog Use syslog logging back end. The output usually goes to \/var\/lib\/messages. .. option:: -l, --lockfile arg Path to lock file. .. option:: --console arg Send log output to stdout. .. option:: --debug Debug mode: \-\-verbosity\=4 \-\-console\=1 .. option:: --trace Trace mode: \-\-verbosity\=4 \-\-console\=1 \-\-print\-component\=1 \-\-print\-context\=1 .. option:: --log-file arg Use alternative log file. Messaging --------- .. option:: -u, --user arg Overrides configuration parameter :confval:`connection.username`. .. option:: -H, --host arg Overrides configuration parameter :confval:`connection.server`. .. option:: -t, --timeout arg Overrides configuration parameter :confval:`connection.timeout`. .. option:: -g, --primary-group arg Overrides configuration parameter :confval:`connection.primaryGroup`. .. option:: -S, --subscribe-group arg A group to subscribe to. This option can be given more than once. .. option:: --encoding arg Overrides configuration parameter :confval:`connection.encoding`. .. option:: --start-stop-msg arg Sets sending of a start\- and a stop message. Database -------- .. option:: --db-driver-list List all supported database drivers. .. option:: -d, --database arg The database connection string, format: service:\/\/user:pwd\@host\/database. \"service\" is the name of the database driver which can be queried with \"\-\-db\-driver\-list\".