In our previous article, we have covered with Splunk master setup. Once done with the basic software installation next step we need to focus on how to bring the logs into Splunk for indexing. And what else needs to be indexed, In Splunk administration perspective we are responsible to import the logs. Logs can be imported using forwarder by running few commands.
In our previous articles, we have seen how to install “Splunk Enterprise 7.0.2” and client package “Splunk Forwarder 7.0.2“.
To read more about Splunk find few topics as follows.
- Step by step guide to install with Splunk Enterprise 7.0.2
- Step by step guide to install with Splunk Forwarder 7.0.2
- Splunk forwarder installation using Ansible
- How to forward the logs from clients to Splunk Master using the forwarder
- How to run a simple query from Splunk Search & Reporting
Now let us see how to push the logs from client-server to “Splunk Enterprise“.
Before starting with client-side first we need to enable the receiver in Splunk Server. By default, there will be no receiver configured in the Splunk server.
Run below command to enable the receiver at port 9997. We can define any ports which are not in use.
While enabling it will prompt for Splunk username and password.
# ./splunk enable listen 9997
[root@splunk ~]# cd /opt/splunk/ [root@splunk splunk]# [root@splunk bin]# ./splunk enable listen 9997 Splunk username: admin Password: Listening for Splunk data on TCP port 9997. [root@splunk bin]#
To confirm whether it enabled we can check from the graphical interface under
Settings –> Forwarding and receiving –> Receive data
Moreover, do a telnet to Splunk server for port 9997 from any one of client machines.
# telnet splunk.oel7.local 9997
[root@ansiclient1 bin]# telnet splunk.oel7.local 9997 Trying 192.168.107.235... Connected to splunk.oel7.local. Escape character is '^]'.
we are good to go. Now back to the client side.
Start to forward all the required logs from clients to the receiver. By running below command we are authenticating with Splunk master to push the logs. While prompt for username/password provides it.
Use default password admin/changeme
# ./splunk add forward-server splunk.oel7.local:9997 -auth admin:changeme
[root@ansiclient1 bin]# ./splunk add forward-server splunk.oel7.local:9997 -auth admin:changeme Added forwarding to: splunk.oel7.local:9997. [root@ansiclient1 bin]#
Now, are have done with forwarding. But how the Splunk know which logs need to be pushed? For that, we required adding the specific logs. In my setup just I need to monitor system activity so I’m about to monitor only below logs.
Maybe in your case, this can be application logs, DB logs, Web application logs etc.
# ./splunk add monitor /var/log/messages # ./splunk add monitor /var/log/secure # ./splunk add monitor /var/log/yum.log
If we need to remove any one of logs from monitor replace add with delete.
# ./splunk delete monitor /var/log/yum.log
[root@ansiclient1 bin]# ./splunk add monitor /var/log/messages Added monitor of '/var/log/messages'. [root@ansiclient1 bin]# [root@ansiclient1 bin]# ./splunk add monitor /var/log/secure Added monitor of '/var/log/secure'. [root@ansiclient1 bin]# [root@ansiclient1 bin]# ./splunk add monitor /var/log/yum.log Added monitor of '/var/log/yum.log'. [root@ansiclient1 bin]#
It’s not compulsory to restart the service, In case if your changes not into effect do a restart for Splunk forwarder service.
# /etc/init.d/splunk restart
To list out which logs are getting pushed to “Splunk Enterprise Server” we can list using “Splunk list monitor” command.
By default Splunk will monitor its logs for any changes, At the end of the output, we are able to see the manually added monitor list.
# ./splunk list monitor
[root@ansiclient1 bin]# ./splunk list monitor Monitored Directories: $SPLUNK_HOME/var/log/splunk /opt/splunkforwarder/var/log/splunk/audit.log /opt/splunkforwarder/var/log/splunk/btool.log /opt/splunkforwarder/var/log/splunk/conf.log /opt/splunkforwarder/var/log/splunk/first_install.log /opt/splunkforwarder/var/log/splunk/license_usage.log /opt/splunkforwarder/var/log/splunk/mongod.log /opt/splunkforwarder/var/log/splunk/remote_searches.log /opt/splunkforwarder/var/log/splunk/scheduler.log /opt/splunkforwarder/var/log/splunk/searchhistory.log /opt/splunkforwarder/var/log/splunk/splunkd-utility.log /opt/splunkforwarder/var/log/splunk/splunkd_access.log /opt/splunkforwarder/var/log/splunk/splunkd_stderr.log /opt/splunkforwarder/var/log/splunk/splunkd_stdout.log /opt/splunkforwarder/var/log/splunk/splunkd_ui_access.log $SPLUNK_HOME/var/log/splunk/license_usage_summary.log /opt/splunkforwarder/var/log/splunk/license_usage_summary.log $SPLUNK_HOME/var/log/splunk/metrics.log /opt/splunkforwarder/var/log/splunk/metrics.log $SPLUNK_HOME/var/log/splunk/splunkd.log /opt/splunkforwarder/var/log/splunk/splunkd.log $SPLUNK_HOME/var/spool/splunk/...stash_new Monitored Files: $SPLUNK_HOME/etc/splunk.version /var/log/messages /var/log/secure /var/log/yum.log [root@ansiclient1 bin]#
To confirm whether we receiving logs in “Splunk Enterprise server” have a look into graphical interface by searching using hostname.
Click the Search & Reporting from left side top corner to do a search.
looking for a hostname by starting “host =” we can change the time range from the right side near to the search button.
It will auto-complete the hostname because it already indexed the required data’s which received from clients.
Below we are able to see the logs which are inbound until last 24 hours.
That’s it we have successfully forwarded logs from client servers to Splunk master using Splunk Forwarder.
Conclusion:
In this guide, we have seen how to forward the logs to Splunk master server from clients using the forwarder. In the upcoming guide let us see “How to run a simple query from Splunk Search & Reporting“.