NXlog配置

NXlog文档：

http://nxlog.org/docs/nxlog-ce/nxlog-reference-manual.html

https://nxlog.org/documentation/nxlog-community-edition-reference-manual-v20928

Windows服务器上安装NXlog，然后配置为将Windows服务器上的事件日志发送到Linux syslog服务器（192.168.200.29:514）

<Extension _syslog>
    Module      xm_syslog #收集事件日志，所有的事件日志默认都被收集
</Extension>

<Input in> #配置input
    Module      im_msvistalog
# For windows 2003 and earlier use the following:
#   Module      im_mseventlog

</Input>

<Output out> #配置output
    Module      om_tcp #配置为使用tcp模式
    Host        192.168.200.29 #远端syslog server IP
    Port        514 #远端syslog server端口
    Exec        to_syslog_snare();
</Output>

<Route 1>
    Path        in => out #源输入对应输出
</Route>

配置完成后，如果nxlog出现如下错误：配置完成后，ERROR Couldn't read next event, corrupted eventlog?; The data is invalid.

则是因为windows8.1下windows事件日志种类过多导致，可以自定义需要保存的事件日志种类，设置如下：

<Extension _syslog>
    Module      xm_syslog
</Extension>

<Input in>
    Module      im_msvistalog
# For windows 2003 and earlier use the following:
#   Module      im_mseventlog

    ReadFromLast FALSE
    SavePos     FALSE
    Query       <QueryList>\
                    <Query Id="0">\
                        <Select Path="Security">*</Select>\ #配置为只发送Security Log
                    </Query>\
                </QueryList>

</Input>

<Output out>
    Module      om_tcp
    Host        192.168.200.29
    Port        514
    Exec        to_syslog_snare();
</Output>

<Route 1>
    Path        in => out
</Route>

配置为同时接收多个事件日志：

<Input in>
    Module      im_msvistalog
# For windows 2003 and earlier use the following:
#   Module      im_mseventlog

    ReadFromLast FALSE
    SavePos     FALSE
    Query       <QueryList>\
                    <Query Id="0">\
                        <Select Path="Application">*</Select>\
                        <Select Path="System">*</Select>\
                        <Select Path="Security">*</Select>\
                    </Query>\
                </QueryList>

</Input>

Linux收集到的日志如下：

每条事件日志对应一个log

如果Linux上某个目录下文件过多，删除方式如下：ls | xargs -n 10 rm -fr ls

## This is a sample configuration file. See the nxlog reference manual about the
## configuration options. It should be installed locally and is also available
## online at http://nxlog.org/docs/

## Please set the ROOT to the folder your nxlog was installed into,
## otherwise it will not start.

#define ROOT C:\Program Files\nxlog
define ROOT C:\Program Files (x86)\nxlog

Moduledir %ROOT%\modules
CacheDir %ROOT%\data
Pidfile %ROOT%\data\nxlog.pid
SpoolDir %ROOT%\data
LogFile %ROOT%\data\nxlog.log

<Extension _syslog>
    Module      xm_json #日志输出格式为json
</Extension>

<Input in>
    Module      im_msvistalog
# For windows 2003 and earlier use the following:
#   Module      im_mseventlog

    ReadFromLast FALSE
    SavePos     FALSE
    Query       <QueryList>\
                    <Query Id="0">\
                        <Select Path="Security">*</Select>\  #配置输入源为收集Security Log
                    </Query>\
                </QueryList>

</Input>

<Input systemin> #配置第二个输入源名为 systemin
    Module      im_msvistalog
# For windows 2003 and earlier use the following:
#   Module      im_mseventlog

    ReadFromLast FALSE
    SavePos     FALSE
    Query       <QueryList>\
                    <Query Id="0">\
                        <Select Path="System">*</Select>\ #配置收集System Log
                    </Query>\
                </QueryList>

</Input>

<Output out> #配置第一个输出方式，对应接收input in
    Module      om_file
    File 'E:\logtest\seclog.json' #输出到seclog.json文件
    Exec        to_json();
</Output>

<Output systemout> #配置第二个输出方式，对应接收input systemin
    Module      om_file
    File 'E:\logtest\syslog.json'
    Exec        to_json();
</Output>

<Route 1>
    Path        in => out #将第一个input in 对应到ouput out
</Route>

<Route 2>
    Path        systemin => systemout #将第二个input systemin 对应到 output systemout
</Route>

结果：

将两个input输出到同一个output：

<Route 1>
Path in, systemin => out
</Route>

按时间对output file进行rotate（未测试）：

define DIR C:\\logdir
<Output out>
Module om_file
File "%DIR%\\test.log"
<Schedule>
Every 1 sec
Exec out->rotate_to("%DIR%\\test."+ strftime(now(), "%Y%m%d%H%M%S"));
</Schedule>
</Output>

按大小对output file进行rotate：

<Extension _syslog>
    Module      xm_json
</Extension>

<Input secin>
    Module      im_msvistalog
# For windows 2003 and earlier use the following:
#   Module      im_mseventlog

    ReadFromLast FALSE
    SavePos     FALSE
    Query       <QueryList>\
                    <Query Id="0">\
                        <Select Path="Security">*</Select>\
                    </Query>\
                </QueryList>

</Input>

<Input systemin>
    Module      im_msvistalog
# For windows 2003 and earlier use the following:
#   Module      im_mseventlog

    ReadFromLast FALSE
    SavePos     FALSE
    Query       <QueryList>\
                    <Query Id="0">\
                        <Select Path="System">*</Select>\
                    </Query>\
                </QueryList>

</Input>

<Output secout>
    Module      om_file
    CreateDir   TRUE
    sync        FALSE
    File    "e:\logtest\sec_" + $Hostname + "_" + month(now()) + ".json"
    Exec if secout->file_size() > 20M \
        { \
             $newfile = "e:\logtest\sec_" + $Hostname + "_" + strftime(now(), "%Y%m%d%H%M%S") + ".json"; \
             secout->rotate_to($newfile); \
        };
    Exec        to_json();
</Output>

<Output systemout>
    Module      om_file
    File 'E:\logtest\syslog.json'
    Exec        to_json();
</Output>

<Route 1>
    Path        secin => secout
</Route>

<Route 2>
    Path        systemin => systemout
</Route>

output还可以编辑为如下，效果相同：

<Output secout>
    Module      om_file
    CreateDir   TRUE
    sync        FALSE
    File    "e:\logtest\sec_" + $Hostname + "_" + month(now()) + ".json" #此为当前正在写入的文件
    <Exec>
        if secout->file_size() > 20M
        {
             $newfile = "e:\logtest\sec_" + $Hostname + "_" + strftime(now(), "%Y%m%d%H%M%S") + ".json";
             secout->rotate_to($newfile);
        }
    </Exec>
    Exec        to_json();
</Output>

输出如下：

收集IIS Log：

IIS仍为源格式

<Input IIS01> #按照不同站点分为多个Input
 Module    im_file
 File      "d:\\iislog\W3SVC2\\\\u_ex*.log"
 SavePos   TRUE
</Input>

<Output IIS01>同其他</Output>

IIS log输出为json格式

<Extension w3c>
    Module xm_csv
    Fields $date, $time, $s-ip, $cs-method, $cs-uri-stem, $cs-uri-query, $s-port, $cs-username, $c-ip, $csUser-Agent, $cs-Referer, $sc-status, $sc-substatus, $sc-win32-status, $time-taken
    FieldTypes string, string, string, string, string, string, integer, string, string, string, string, integer, integer, integer, integer
    Delimiter ' '
    QuoteChar '"'
    EscapeControl FALSE
    UndefValue -
</Extension>

<Input iis01in>
    Module    im_file
    File    'E:\IISLog\W3SVC1\u_ex*'
    SavePos  TRUE

    Exec if $raw_event =~ /^#/ drop();                \
       else                            \
       {                            \
            w3c->parse_csv();                    \
            $EventTime = parsedate($date + " " + $time);    \
            $SourceName = "IIS";                \
            $raw_event = to_json();                \
       }
</Input>

<Output iis01out>
    Module      om_file
    CreateDir   TRUE
    sync        FALSE
    Exec $Hostname = 'server01';
    File    "e:\logtest\iis_" + $Hostname + "_" + month(now()) + ".json"
    <Exec>
        if iis01out->file_size() > 200M
        {
             $newfile = "e:\logtest\iis_" + $Hostname + "_" + strftime(now(), "%Y%m%d%H%M%S") + ".json";
             iis01out->rotate_to($newfile);
        }
    </Exec>
#    Exec        to_json();
</Output>

<Route iis01>
    Path        iis01in => iis01out
</Route>

Using the Query directive

The im_msvistalog has a Query directive which can be used to specify an XML Query that gets passed to the Windows EventLog API in order to read only the selected events. The Windows Event Viewer can help construct such XML queries. The following example will only collect only process creation event records from the Sysmon source.

Query <QueryList> \
        <Query Id="0">\
          <Select Path="Microsoft-Windows-Sysmon/Operational">*[System[(EventID='1')]]</Select>\
        </Query>\
      </QueryList>

The event records filtered with the Query directive do not reach NXLog so this might be slightly more efficient than the next native NXLog filtering method.

Filtering with NXLog's log processing language

The NXLog log processing language is available for use by all modules and may be easier to write than the XML query syntax provided by the Windows EventLog API that the im_msvistalog exposes. The following NXLog style filter statement achieves the same as the XML Query above.

Exec if not ($Channel == 'Microsoft-Windows-Sysmon' and $EventID == 1) drop();

The following filtering rule will remove event records that are HTTP network connections to a specific server:

Exec if $SourceName == 'Microsoft-Windows-Sysmon' and $DestinationPort == 80 and $DestinationIp == 10.0.0.1 drop();

nxlog-ce-2.9.1504出现如下报错，可以替换成版本“nxlog-ce-2.9.1347”解决
报错：ERROR if-else failed at line 61, character 312 in C:\Program Files (x86)\nxlog\conf\nxlog.conf. statement execution has been aborted; procedure 'parse_csv' failed at line 61, character 98 in C:\Program Files (x86)\nxlog\conf\nxlog.conf. statement execution has been aborted; Not enough fields in CSV input, expected 15, got 11 in input 'Exchange.asmx &CorrelationID=<empty>;&cafeReqId=bdbb53c3-5227-4601-8dbe-a5a7fb72c7b9; 80 xx\tuliqi 106.3.4.150 AppleExchangeWebServices/806.1+ExchangeSync/121 - 200 0 0 9'