This page refers to the old SeisComP 3 version, and may no longer be accurate. See seiscomp.de for the latest version.

Trouble Shooting

Processing

SeedLink

If SeedLink does not function correctly, it is recommended to ﬁrst check the log ﬁles in the “log” directory, /var/log/messages and /var/log/seedlink. In order to ﬁnd the error quickly, it is necessary to understand that the data goes through the following major components of the system:

plug-in,
SeedLink’s plug-in interface,
SeedLink’s StreamProcessor,
SeedLink’s I/O system,
SeedLink’s StreamMonitor,
clients (slarchive, etc.).

Often the malfunction of data acquisition is caused by generic operating system errors such as disk corruption, therefore it is very important to check /var/log/messages if the system behaves strangely. In the following, we will take a look at errors that can happen in each of the components.

Plug-in

A plug-in is just a normal program that sends data to ﬁle descriptor 63 (opened by SeedLink before it executes the plug-in). Have a look at a plug-in deﬁnition in seedlink.ini; it is similar to the following:

plugin edata_EDD cmd="/home/sysop/bin/serial_plugin -v -f /home/sysop/config
timeout = 600
start_retry = 60
shutdown_wait = 10

Using the command line shown, you can run the plug-in as standalone by the following (note that the plug-in name is appended to the command line):

serial_plugin -v -f /home/sysop/config/plugins.ini edata_EDD63>data.dat

Before you do it, shut down the aquisition to make sure no other plug-in instances are running. Here we asked the shell to re-open the ﬁle descriptor 63 and re-direct it to the ﬁle data.dat (this syntax does not work with C-shell, only for bash). If the plug-in works, then the data (in SeedLink’s internal format) should appear in the ﬁle data.dat. Log messages are sent to standard output. If no data arrives, check if baudrate and other settings are correct in plugins.ini. It is possible to increase verbosity by -v ﬂag or by editing plugins.ini. Some plug-ins print a help message if --help ﬂag is used.

SeedLink’s Plug-in Interface

In SeedLink, there is no one-to-one relation between stations and plug-in instances. For example, seismic channels and environmental (state of health) channels can be digitized by different digitizers which are handled by different plug-ins. On the other hand, often a single plug-in instance collects data of many stations. This is the case for the chain plug-in, as well as for other plug-ins that connect to data acquisition systems. Therefore each data packet sent by a plug-in to the ﬁle descriptor 63 contains the station ID, which tells SeedLink to which station this data belongs. If SeedLink does not know this station, the data is ignored and a warning is written to seedlink log ﬁle, which looks similar to the following:

Jun 26 03:47:22 st27 seedlink: [chain] station AMZI is not configured

SeedLink’s StreamProcessor

While Mini-SEED streams go straight to the I/O System, raw streams pass the StreamProcessor module, which is probably the most problematic part of SeedLink for inexperienced users. The StreamProcessor is enabled by pointing the parameters “streams” and “ﬁlters” in seedlink.ini to respective conﬁguration ﬁles. If StreamProcessor is not enabled, then raw data streams are ignored and you can see warning in seedlink log ﬁle that looks like this:

Feb 19 13:24:23 st27 seedlink: KRIS raw data ignored

The next possible error is mismatch between names or sample rates of raw channels and stream processing scheme used. When you look at a station deﬁnition in seedlink.ini, you can see something similar to the following:

station EDD	description = "GEOFON_Station"
name = EDD
network = GE
proc = edata_100

The “proc” attribute selects the stream processing scheme. If this attribute is missing, raw data is not accepted and you will get the error shown above. The stream processing schemes are deﬁned in the ﬁle streams.xml. The scheme “edata 100” is in fact based on two other schemes deﬁned in the following way:

<proc name="edata_100">
<using proc="generic_6x100"/>
<using proc="edata_aux"/>
</proc>

The scheme “generic 6x100” is deﬁned as follows:

<proc name="generic_6x100">
<tree>
<input name="Z" channel="Z" location="" rate="100"/>
<input name="N" channel="N" location="" rate="100"/>
<input name="E" channel="E" location="" rate="100"/>
<!--Uncomment this to enable 100Hz HH? Streams  -->
<!--                                            -->
<!--     <node stream="HH"/>                    -->
<!--Uncomment this to enable 50Hz SH? streams   -->
<!--                                            -->
<!--     <node filter="F96C" stream="SH"/>      -->
<node filter="FS2D5" stream="BH">
			<node filter="F96C">
<node filter="ULP" stream="LH">
<node filter="VLP" stream="VH"/>
</node>
</node>
</node>
</tree>
<tree>
<input name="Z1" channel="Z" location="" rate="100"/>
<input name="N1" channel="N" location="" rate="100"/>
<input name="E1" channel="E" location="" rate="100"/>
<!--Uncomment this to enable 100Hz HN? Streams  -->
<!--                                            -->
<!--     <node stream="HN"/>                    -->
<node filter="F96C" stream="SN"/>
</tree>
</proc>

This scheme tells the StreamProcessor how to create 12 streams (BHZ, BHN, BHE, LHZ, LHN, LHE, VHZ, VNH, VHE, SNZ, SNN, SNE) from 6 raw input channels (Z, N, E, Z1, N1, E1). Each raw data packet sent by a plug-in is labeled with channel ID in addition to station ID, so the StreamProcessor knows which packet belongs to which channel. Sometimes the channel ID’s are hardcoded within a plug-in, but often they can be deﬁned by a user. For example, in case of serial plug-in the following can be found in plugins.ini:

*  Keyword ’channel’ is used to map input channels to symbolic channel
*  names. Channel names are arbitrary 1..10-letter identifiers which
*  should match the input names of the stream processing scheme in
*  stream.xml, which is referenced from seedlink.ini
channel Z source_id=0
channel E source_id=1
channel N source_id=2
* "State of health" channels
channel        S0  source_id=16
channel        S1  source_id=17
channel        S2  source_id=18
channel        S3  source_id=19
channel        S4  source_id=20
channel        S5  source_id=21
channel        S6  source_id=22
channel        S7  source_id=23
channel        S8  source_id=24

If a plug-in uses a channel ID, which is not deﬁned in the stream processing scheme, a warning similar to the following is shown:

Jun 24 12:56:28 st55 seedlink: EDD channel X ignored

It is also important that the actual sample rate of raw channels is the same as deﬁned in the scheme, otherwise the time calculation is wrong and there will be apparent gaps in data:

Jun 26 02:26:18 st27 seedlink: AMZI : Z time gap -8.15 seconds (detected)

SeedLink’s I/O System

If a plug-in sends Mini-SEED data to SeedLink, the StreamProcessor will not be used and thus the SeedLink conﬁguration will be much simpler. However, still some problems may occur. First of all, the user must make sure that the Mini-SEED data that comes from the plug-in is correct big-endian Mini-SEED. Often there are errors in the data because the authors of the plug-ins have not taken the differences between computer architectures into account (e.g., the Mini-SEED data is little-endian, which is not valid and not supported by SeedLink).
Another source of problems is the corruption of SeedLink’s disk buffer (ﬁles in the directory pointed by the “ﬁlebase” parameter in seedlink.ini) due to disk errors (bad blocks, etc.), which can cause strange error messages and data gaps. In this case there are often disk errors shown in /var/log/messages. If the SeedLink’s disk buffer is corrupted, a simple ﬁx is to just delete its content.
Note that the SeedLink’s disk buffer is only used by the SeedLink server itself. Note that modifying the buffer while SeedLink is running is not allowed. When SeedLink is stopped, you can remove the buffer or use it for debugging purposes (the ﬁles in buffer are in Mini-SEED format).

SeedLink’s StreamMonitor

The module StreamMonitor is activated by setting the parameter “stream check” in seedlink.ini to “enabled”. In this case the time spans and gaps of streams are monitored (according to parameters “gap check pattern” and “gap treshold”) and the information can be requested from the server using slinktool. The stream information is also required for time window extraction.
When SeedLink exits, it saves the stream information of each station to a ﬁle called buffer.xml, which is located in a subdirectory within the SeedLink’s disk buffer. In this case SeedLink can start fast, because it can restore the stream information without scanning the disk buffer. However, if the parameters “stream check”, “gap check pattern” or “gap treshold” were meanwhile changed, the stored stream information may not be valid. Therefore the buffer.xml ﬁles must be removed before starting SeedLink. Scanning the disk buffer may take long time (depending on its size and harddisk speed) and during that time it is not possible to stop SeedLink (except with “kill -9”). If you believe that scanning the disk buffer is slower than it should be, check your operating system’s IDE DMA settings, however, fast DMA modes may also be less reliable.
] When using the StreamMonitor, make sure that “gap treshold” is not too low. In case of some digitizers, small gaps between records are normal due to timing errors. Also, “gap check pattern” should be set such that triggered streams are not included. Keeping the track of a large number of gaps may require more memory than available, causing a system crash.

Clients

If the SeedLink server seems to work correctly and data appears in the SeedLink’s disk buffer, but nothing is saved in the “archive” directory, most probably the “slarchive” client is not running properly.
The clients acquire data from the SeedLink server only via TCP/IP, even if they are running on the local machine. When a client starts, the sent SeedLink commands are listed in the SeedLink log ﬁle. Using slinktool, you can check which clients are connected and what is the status of connections (e.g., if there are many packets in the queue, the client may be hanging or there may be network errors that prevent it from getting data). If the StreamMonitor is enabled, slinktool can also show the time spans of streams in the buffer. You can also telnet to the SeedLink server and use the SeedLink commands directly:

$ telnet localhost 18000
Trying 139.17.3.25...
Connected to geofon.gfz-potsdam.de.
Escape character is ’ˆ]’.
HELLO
SeedLink v3.0 (2004.129)
GEOFON DC
BYE
Connection closed by foreign host.

Network connections can also be tested with slinktool (see chapter 5.3.14). The following command will request station APE from the GEOFON server and print verbose information about every packet received:

slinktool -v -ppp -S GE_APE geofon.gfz-potsdam.de:18000