In the other example How to filter syslog message by program, I described how  to filter syslog message created by a program.

However, sometime, message may not be that clear, for example, if a driver loaded into kernel, the its output by default goes to syslog and program name will be kernel too. Unless there is a special tag created by the driver, it's hard to seperated the message from syslog.

Here I use an example I did for mhvtl, a virtual tape library on Linux to show you how to seperate and discard message created by a program.

1. Identify the message created by mhvtl

Majority message look like below, To reduce number of different messages, surely not all message are listed below.

kernel: 0a 00 00 28 00 00 
kernel: mhvtl: CDB (160645782) 12 bytes
mhvtl: vtl_open: mhvtl33: opened
vtltape[21050]: processCommand(): 11700th contiguous WRITE_6 request
vtltape[21064]: processMessageQ(): Sender id: 30, msg : unload
vtllibrary[21084]: spc_inquiry():

As you can see, some message are useful, but some are not. They creates numberous message to /var/log/messages, which definitely not necessary.

2. Identify who generated those message

By using the ways described in How to filter syslog message by program, I identified two programs

vtltape and vtllibrary generated the following type of message below.

vtltape[21050]: processCommand(): 11700th contiguous WRITE_6 request
vtltape[21064]: processMessageQ(): Sender id: 30, msg : unload
vtllibrary[21084]: spc_inquiry():

Others are generated by kernel, no surprise, because mhvtl is loaded as a driver into kernel.

3. Filtering messages

Clearly, I need separate all message generated by mhvtl and put them to /var/log/mhvtl.log, And discard useless message.

Some message are useless, perhaps not really 'useless', but the volume of these message are huge, so have to discard them

vtltape[21050]: processCommand(): 11700th contiguous WRITE_6 request
kernel: 0a 00 00 28 00 00
kernel: mhvtl: CDB (160645782) 12 bytes

I created the following filters in /etc/rsyslog.conf to discard them, Filter them by message(2 of them generated by kernel)

### Discard useless kernel message generated by mhvtl
:msg, regex, ".*mhvtl: CDB .* bytes" ~
:msg, regex, "processCommand.*contiguous WRITE_6 request " ~
:msg, regex, " [[:xdigit:]][[:xdigit:]] [[:xdigit:]][[:xdigit:]] [[:xdigit:]][[:xdigit:]] [[:xdigit:]][[:xdigit:]] " ~
## End of kernel message disarding

These three types message are useful message and should be saved to /var/log/mhvtl.log

vtltape[21064]: processMessageQ(): Sender id: 30, msg : unload
vtllibrary[21084]: spc_inquiry():
mhvtl: vtl_open: mhvtl33: opened

Two of them can be selected by programname, the third is generated by kernel, but can be separated by message filter. Here is the lines in /etc/rsyslog.conf

#### separate mhvtl message from kernel
:msg, contains, "mhvtl"   /var/log/mhvtl.log
& ~
:programname, startswith , "vtl"      /var/log/mhvtl.log
& ~
#### End of separate mhvtl message from kernel

After done the above, I have only meaningful message left in /var/log/mhvtl.log, enough for debugging.

 

Note: My mhvtl logging configuration looks like below

# Set default verbosity [0|1|2|3]
VERBOSE=1

# Set kernel module debuging [0|1]
VTL_DEBUG=0