Contents
Prerequisite.
We will install Suricata IDS and ElasticStack anad Kibana on the following servers
・1st server Suricata IDS & Filebeat : Debian12.8 IP address (192.168.11.83)
・2nd server ElasticStack & kibana : Ubunru24.04 IP address(192.168.11.200)
1st server Suricata Install
SURICATA IDS/IPS is an open source IDS that monitors communications on the network and detects suspicious traffic. Its basic mechanism is signature-based, so it can detect predefined unauthorized communications. Suricata is also characterized by its ability to provide protection as well as detection.
1.Suricata Install
①Install required packages
1 |
# apt install wget curl software-properties-common dirmngr apt-transport-https gnupg2 ca-certificates lsb-release debian-archive-keyring unzip -y |
➁Suricata install
1 2 |
# apt update # apt install suricata |
Check Version
1 2 |
# suricata -V This is Suricata version 6.0.10 RELEASE |
Stop the Suricata service as it needs to be configured first.
1 |
# systemctl stop suricata |
2.Configure Suricata
①Determine interface and IP address where Suricata will inspect network packets
1 2 3 |
# ip --brief add lo UNKNOWN 127.0.0.1/8 ::1/128 ens33 UP 192.168.11.83/24 |
Edit /etc/suricata/suricata.yaml file
1 2 3 4 5 6 7 8 9 10 11 12 13 14 |
# vi /etc/suricata/suricata.yaml # Line 18 : Change (own network) HOME_NET: "[192.168.11.0/24]" # Line 132 : Change community-id: false → community-id: true # Line 589 : Change af-packet: - interface: eth0 ↓ af-packet: - interface: ens33 ←Change to your own interface name |
➁Add rule set
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 |
# suricata-update -o /etc/suricata/rules 19/11/2024 -- 12:26:56 - <Info> -- Using data-directory /var/lib/suricata. 19/11/2024 -- 12:26:56 - <Info> -- Using Suricata configuration /etc/suricata/suricata.yaml 19/11/2024 -- 12:26:56 - <Info> -- Using /etc/suricata/rules for Suricata provided rules. 19/11/2024 -- 12:26:56 - <Info> -- Found Suricata version 6.0.10 at /usr/bin/suricata. 19/11/2024 -- 12:26:56 - <Info> -- Loading /etc/suricata/suricata.yaml 19/11/2024 -- 12:26:56 - <Info> -- Disabling rules for protocol http2 19/11/2024 -- 12:26:56 - <Info> -- Disabling rules for protocol modbus 19/11/2024 -- 12:26:56 - <Info> -- Disabling rules for protocol dnp3 19/11/2024 -- 12:26:56 - <Info> -- Disabling rules for protocol enip 19/11/2024 -- 12:26:56 - <Info> -- No sources configured, will use Emerging Threats Open 19/11/2024 -- 12:26:56 - <Info> -- Fetching https://rules.emergingthreats.net/open/suricata-6.0.10/emerging.rules.tar.gz. 100% - 4599733/4599733 19/11/2024 -- 12:26:59 - <Info> -- Done. 19/11/2024 -- 12:26:59 - <Info> -- Loading distribution rule file /etc/suricata/rules/app-layer-events.rules 19/11/2024 -- 12:26:59 - <Info> -- Loading distribution rule file /etc/suricata/rules/decoder-events.rules 19/11/2024 -- 12:26:59 - <Info> -- Loading distribution rule file /etc/suricata/rules/dhcp-events.rules 19/11/2024 -- 12:26:59 - <Info> -- Loading distribution rule file /etc/suricata/rules/dnp3-events.rules 19/11/2024 -- 12:26:59 - <Info> -- Loading distribution rule file /etc/suricata/rules/dns-events.rules 19/11/2024 -- 12:26:59 - <Info> -- Loading distribution rule file /etc/suricata/rules/files.rules 19/11/2024 -- 12:26:59 - <Info> -- Loading distribution rule file /etc/suricata/rules/http-events.rules 19/11/2024 -- 12:26:59 - <Info> -- Loading distribution rule file /etc/suricata/rules/ipsec-events.rules 19/11/2024 -- 12:26:59 - <Info> -- Loading distribution rule file /etc/suricata/rules/kerberos-events.rules 19/11/2024 -- 12:26:59 - <Info> -- Loading distribution rule file /etc/suricata/rules/modbus-events.rules 19/11/2024 -- 12:26:59 - <Info> -- Loading distribution rule file /etc/suricata/rules/nfs-events.rules 19/11/2024 -- 12:26:59 - <Info> -- Loading distribution rule file /etc/suricata/rules/ntp-events.rules 19/11/2024 -- 12:26:59 - <Info> -- Loading distribution rule file /etc/suricata/rules/smb-events.rules 19/11/2024 -- 12:26:59 - <Info> -- Loading distribution rule file /etc/suricata/rules/smtp-events.rules 19/11/2024 -- 12:26:59 - <Info> -- Loading distribution rule file /etc/suricata/rules/stream-events.rules 19/11/2024 -- 12:26:59 - <Info> -- Loading distribution rule file /etc/suricata/rules/tls-events.rules 19/11/2024 -- 12:26:59 - <Info> -- Ignoring file rules/emerging-deleted.rules 19/11/2024 -- 12:27:01 - <Info> -- Loaded 54361 rules. 19/11/2024 -- 12:27:01 - <Info> -- Disabled 14 rules. 19/11/2024 -- 12:27:01 - <Info> -- Enabled 0 rules. 19/11/2024 -- 12:27:01 - <Info> -- Modified 0 rules. 19/11/2024 -- 12:27:01 - <Info> -- Dropped 0 rules. 19/11/2024 -- 12:27:02 - <Info> -- Enabled 136 rules for flowbit dependencies. 19/11/2024 -- 12:27:02 - <Info> -- Backing up current rules. 19/11/2024 -- 12:27:02 - <Info> -- Writing rules to /etc/suricata/rules/suricata.rules: total: 54361; enabled: 40645; added: 54361; removed 0; modified: 0 19/11/2024 -- 12:27:02 - <Info> -- Writing /etc/suricata/rules/classification.config 19/11/2024 -- 12:27:02 - <Info> -- Testing with suricata -T. 19/11/2024 -- 12:27:28 - <Info> -- Done. |
Indicates that suricata-update has retrieved the free Emerging Threats ET Open Rules and saved it to Suricata's /etc/suricata/rules/suricata.rules file.
It also shows the number of rules processed, in this example 54361 were added, of which 40645 were activated.
➂Adding Rule Set Providers
List Default Providers
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 |
# suricata-update list-sources Name: et/open Vendor: Proofpoint Summary: Emerging Threats Open Ruleset License: MIT Name: et/pro Vendor: Proofpoint Summary: Emerging Threats Pro Ruleset License: Commercial Replaces: et/open Parameters: secret-code Subscription: https://www.proofpoint.com/us/threat-insight/et-pro-ruleset Name: oisf/trafficid Vendor: OISF Summary: Suricata Traffic ID ruleset License: MIT Name: scwx/enhanced Vendor: Secureworks Summary: Secureworks suricata-enhanced ruleset License: Commercial Parameters: secret-code Subscription: https://www.secureworks.com/contact/ (Please reference CTU Countermeasures) Name: scwx/malware Vendor: Secureworks Summary: Secureworks suricata-malware ruleset License: Commercial Parameters: secret-code Subscription: https://www.secureworks.com/contact/ (Please reference CTU Countermeasures) Name: scwx/security Vendor: Secureworks Summary: Secureworks suricata-security ruleset License: Commercial Parameters: secret-code Subscription: https://www.secureworks.com/contact/ (Please reference CTU Countermeasures) Name: abuse.ch/sslbl-blacklist Vendor: Abuse.ch Summary: Abuse.ch SSL Blacklist License: CC0-1.0 Replaces: sslbl/ssl-fp-blacklist Name: abuse.ch/sslbl-ja3 Vendor: Abuse.ch Summary: Abuse.ch Suricata JA3 Fingerprint Ruleset License: CC0-1.0 Replaces: sslbl/ja3-fingerprints Name: abuse.ch/sslbl-c2 Vendor: Abuse.ch Summary: Abuse.ch Suricata Botnet C2 IP Ruleset License: CC0-1.0 Name: abuse.ch/feodotracker Vendor: Abuse.ch Summary: Abuse.ch Feodo Tracker Botnet C2 IP ruleset License: CC0-1.0 Name: abuse.ch/urlhaus Vendor: abuse.ch Summary: Abuse.ch URLhaus Suricata Rules License: CC0-1.0 Name: etnetera/aggressive Vendor: Etnetera a.s. Summary: Etnetera aggressive IP blacklist License: MIT Name: tgreen/hunting Vendor: tgreen Summary: Threat hunting rules License: GPLv3 Name: malsilo/win-malware Vendor: malsilo Summary: Commodity malware rules License: MIT Name: stamus/lateral Vendor: Stamus Networks Summary: Lateral movement rules License: GPL-3.0-only Name: stamus/nrd-30-open Vendor: Stamus Networks Summary: Newly Registered Domains Open only - 30 day list, complete License: Commercial Parameters: secret-code Subscription: https://www.stamus-networks.com/stamus-labs/subscribe-to-threat-intel-feed Name: stamus/nrd-14-open Vendor: Stamus Networks Summary: Newly Registered Domains Open only - 14 day list, complete License: Commercial Parameters: secret-code Subscription: https://www.stamus-networks.com/stamus-labs/subscribe-to-threat-intel-feed Name: stamus/nrd-entropy-30-open Vendor: Stamus Networks Summary: Newly Registered Domains Open only - 30 day list, high entropy License: Commercial Parameters: secret-code Subscription: https://www.stamus-networks.com/stamus-labs/subscribe-to-threat-intel-feed Name: stamus/nrd-entropy-14-open Vendor: Stamus Networks Summary: Newly Registered Domains Open only - 14 day list, high entropy License: Commercial Parameters: secret-code Subscription: https://www.stamus-networks.com/stamus-labs/subscribe-to-threat-intel-feed Name: stamus/nrd-phishing-30-open Vendor: Stamus Networks Summary: Newly Registered Domains Open only - 30 day list, phishing License: Commercial Parameters: secret-code Subscription: https://www.stamus-networks.com/stamus-labs/subscribe-to-threat-intel-feed Name: stamus/nrd-phishing-14-open Vendor: Stamus Networks Summary: Newly Registered Domains Open only - 14 day list, phishing License: Commercial Parameters: secret-code Subscription: https://www.stamus-networks.com/stamus-labs/subscribe-to-threat-intel-feed Name: pawpatrules Vendor: pawpatrules Summary: PAW Patrules is a collection of rules for IDPS / NSM Suricata engine License: CC-BY-SA-4.0 Name: ptrules/open Vendor: Positive Technologies Summary: Positive Technologies Open Ruleset License: Custom Name: aleksibovellan/nmap Vendor: aleksibovellan Summary: Suricata IDS/IPS Detection Rules Against NMAP Scans License: MIT |
If you include the tgreen/hunting ruleset as an example
1 2 3 4 5 6 7 8 9 |
# suricata-update enable-source tgreen/hunting 19/11/2024 -- 12:31:11 - <Info> -- Using data-directory /var/lib/suricata. 19/11/2024 -- 12:31:11 - <Info> -- Using Suricata configuration /etc/suricata/suricata.yaml 19/11/2024 -- 12:31:11 - <Info> -- Using /etc/suricata/rules for Suricata provided rules. 19/11/2024 -- 12:31:11 - <Info> -- Found Suricata version 6.0.10 at /usr/bin/suricata. 19/11/2024 -- 12:31:11 - <Info> -- Creating directory /var/lib/suricata/update/sources 19/11/2024 -- 12:31:11 - <Info> -- Enabling default source et/open 19/11/2024 -- 12:31:11 - <Info> -- Source tgreen/hunting enabled |
Perform update
1 |
# suricata-update -o /etc/suricata/rules |
3.Suricata Configuration Testing
①Run Verification Tool
1 2 3 4 5 6 7 8 9 10 11 12 13 |
# suricata -T -c /etc/suricata/suricata.yaml -v 19/11/2024 -- 12:32:15 - <Info> - Running suricata under test mode 19/11/2024 -- 12:32:15 - <Notice> - This is Suricata version 6.0.10 RELEASE running in SYSTEM mode 19/11/2024 -- 12:32:15 - <Info> - CPUs/cores online: 2 19/11/2024 -- 12:32:15 - <Info> - fast output device (regular) initialized: fast.log 19/11/2024 -- 12:32:15 - <Info> - eve-log output device (regular) initialized: eve.json 19/11/2024 -- 12:32:15 - <Info> - stats output device (regular) initialized: stats.log 19/11/2024 -- 12:32:28 - <Info> - 1 rule files processed. 40896 rules successfully loaded, 0 rules failed 19/11/2024 -- 12:32:29 - <Info> - Threshold config parsed: 0 rule(s) found 19/11/2024 -- 12:32:29 - <Info> - 40899 signatures processed. 1191 are IP-only rules, 5120 are inspecting packet payload, 34385 inspect application layer, 108 are decoder event only 19/11/2024 -- 12:32:40 - <Notice> - Configuration provided was successfully loaded. Exiting. 19/11/2024 -- 12:32:40 - <Info> - cleaning up signature grouping structure... complete |
Suricata service restart
1 |
# systemctl start suricata |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 |
# systemctl status suricata ● suricata.service - Suricata IDS/IDP daemon Loaded: loaded (/lib/systemd/system/suricata.service; enabled; preset: enabled) Active: active (running) since Tue 2024-11-19 12:33:13 JST; 6s ago Docs: man:suricata(8) man:suricatasc(8) https://suricata-ids.org/docs/ Process: 3009 ExecStart=/usr/bin/suricata -D --af-packet -c /etc/suricata/suricata.yaml --pidfile /run/suricata.pid (code=exited, status=0/SUCCESS) Main PID: 3010 (Suricata-Main) Tasks: 1 (limit: 2264) Memory: 187.4M CPU: 6.568s CGroup: /system.slice/suricata.service └─3010 /usr/bin/suricata -D --af-packet -c /etc/suricata/suricata.yaml --pidfile /run/suricata.pid Nov 19 12:33:13 Lepard systemd[1]: Starting suricata.service - Suricata IDS/IDP daemon... Nov 19 12:33:13 Lepard suricata[3009]: 19/11/2024 -- 12:33:13 - <Notice> - This is Suricata version 6.0.10 RELEASE running in SYSTEM mode Nov 19 12:33:13 Lepard systemd[1]: Started suricata.service - Suricata IDS/IDP daemon. |
Check log files
1 2 3 4 5 6 7 8 9 10 11 12 |
# tail -f /var/log/suricata/suricata.log 19/11/2024 -- 12:33:13 - <Info> - fast output device (regular) initialized: fast.log 19/11/2024 -- 12:33:13 - <Info> - eve-log output device (regular) initialized: eve.json 19/11/2024 -- 12:33:13 - <Info> - stats output device (regular) initialized: stats.log 19/11/2024 -- 12:33:27 - <Info> - 1 rule files processed. 40896 rules successfully loaded, 0 rules failed 19/11/2024 -- 12:33:27 - <Info> - Threshold config parsed: 0 rule(s) found 19/11/2024 -- 12:33:28 - <Info> - 40899 signatures processed. 1191 are IP-only rules, 5120 are inspecting packet payload, 34385 inspect application layer, 108 are decoder event only 19/11/2024 -- 12:33:39 - <Info> - Going to use 2 thread(s) 19/11/2024 -- 12:33:39 - <Info> - Using unix socket file '/var/run/suricata-command.socket' 19/11/2024 -- 12:33:39 - <Notice> - all 2 packet processing threads, 4 management threads initialized, engine started. 19/11/2024 -- 12:33:39 - <Info> - All AFP capture threads are running. |
4.Testing the Suricata Rule
①Test ET Open rule number 2100498 using the following command
1 2 |
# curl http://testmynids.org/uid/index.html uid=0(root) gid=0(root) groups=0(root) |
②Check the log file using the specified rule number
1 2 3 |
# grep 2100498 /var/log/suricata/fast.log 11/19/2024-12:35:06.340641 [**] [1:2100498:7] GPL ATTACK_RESPONSE id check returned root [**] [Classification: Potentially Bad Traffic] [Priority: 2] {TCP} 18.65.159.75:80 -> 192.168.11.83:45552 |
③Check events in /var/log/suricata/eve.log
Install jq
1 |
# apt install jq |
Filter EVE Log events by searching for 2100498 signatures
Display alert objects with signature_id keys that match the values in 2100498
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 |
# jq 'select(.alert .signature_id==2100498)' /var/log/suricata/eve.json { "timestamp": "2024-11-19T12:35:06.340641+0900", "flow_id": 1085739939590100, "in_iface": "ens33", "event_type": "alert", "src_ip": "18.65.159.75", "src_port": 80, "dest_ip": "192.168.11.83", "dest_port": 45552, "proto": "TCP", "community_id": "1:xYwYl5+l5Grb9PM/cjkqvNDYva0=", "alert": { "action": "allowed", "gid": 1, "signature_id": 2100498, "rev": 7, "signature": "GPL ATTACK_RESPONSE id check returned root", "category": "Potentially Bad Traffic", "severity": 2, "metadata": { "created_at": [ "2010_09_23" ], "updated_at": [ "2019_07_26" ] } }, |
④Creating and Applying Custom Rules
Create the following custom signature to scan SSH traffic to non-SSH ports and include it in the file /etc/suricata/rules/local.rules (IPv4 only this time)
1 2 |
# vi /etc/suricata/rules/local.rules alert ssh any any -> 203.0.113.5 !22 (msg:"SSH TRAFFIC on non-SSH port"; flow:to_client, not_established; classtype: misc-attack; target: dest_ip; sid:1000000;) |
Editing suricata.yaml
1 2 3 4 5 6 |
# vi /etc/suricata/suricata.yaml Add local.rules at line 1923 rule-files: - suricata.rules - local.rules |
Verify SURICATA settings
1 2 3 4 5 6 7 8 9 10 11 12 13 |
# suricata -T -c /etc/suricata/suricata.yaml -v 19/11/2024 -- 12:39:04 - <Info> - Running suricata under test mode 19/11/2024 -- 12:39:04 - <Notice> - This is Suricata version 6.0.10 RELEASE running in SYSTEM mode 19/11/2024 -- 12:39:04 - <Info> - CPUs/cores online: 2 19/11/2024 -- 12:39:05 - <Info> - fast output device (regular) initialized: fast.log 19/11/2024 -- 12:39:05 - <Info> - eve-log output device (regular) initialized: eve.json 19/11/2024 -- 12:39:05 - <Info> - stats output device (regular) initialized: stats.log 19/11/2024 -- 12:40:29 - <Info> - 2 rule files processed. 40897 rules successfully loaded, 0 rules failed 19/11/2024 -- 12:40:29 - <Info> - Threshold config parsed: 0 rule(s) found 19/11/2024 -- 12:40:30 - <Info> - 40900 signatures processed. 1191 are IP-only rules, 5120 are inspecting packet payload, 34386 inspect application layer, 108 are decoder event only 19/11/2024 -- 12:40:40 - <Notice> - Configuration provided was successfully loaded. Exiting. 19/11/2024 -- 12:40:40 - <Info> - cleaning up signature grouping structure... complete |
Edit the custom signature you just created and convert it to use the drop action.
1 2 3 |
# vi /etc/suricata/rules/local.rules drop ssh any any -> 192.168.11.83 !22 (msg:"SSH TRAFFIC on non-SSH port"; flow:to_client, not_established; classtype: misc-attack; target: dest_ip; sid:1000000;) |
⑤Reconfigure SURICATA and reboot in IPS mode
1 2 3 4 |
# vi /etc/default/suricata Line 13行 LISTENMODE=nfqueue |
Reload systemd to detect new Suricata settings
1 |
# systemctl daemon-reload |
1 |
# systemctl restart suricata.service |
⑥Add the necessary rules for suricata to the UFW
Directly edit the firewall files in /etc/ufw/before.rules and /etc/ufw/before6.rules
1 2 3 4 5 6 7 8 9 10 |
# vi /etc/ufw/before.rules Add the following around line 19 ## Start Suricata NFQUEUE rules -I INPUT 1 -p tcp --dport 2244 -j NFQUEUE --queue-bypass -I OUTPUT 1 -p tcp --sport 2244 -j NFQUEUE --queue-bypass -I FORWARD -j NFQUEUE -I INPUT 2 -j NFQUEUE -I OUTPUT 2 -j NFQUEUE ## End Suricata NFQUEUE rules |
Similarly, change /etc/ufw/before6.rules
Restart UFW
1 |
# systemctl restart ufw.service |
Ensure that packets matching the aforementioned signature sid:2100498 are dropped
1 2 3 4 5 |
# vi /etc/suricata/rules/suricata.rules Find line matching sid:2100498 and change alert to drop drop ip any any -> any any (msg:"GPL ATTACK_RESPONSE id check returned root"; content:"uid=0|28|root|29|"; classtype:bad-unknown; sid:2100498; rev:7; metadata:created_at 2010_09_23, updated_at 2019_07_26;) |
Reload signatures
1 |
# kill -usr2 $(pidof suricata) |
Test this rule using curl
1 2 |
# curl --max-time 5 http://testmynids.org/uid/index.html curl: (28) Operation timed out after 5000 milliseconds with 0 out of 39 bytes received |
Checking eve.log using jq, "action" is set to "blocked
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 |
# jq 'select(.alert .signature_id==2100498)' /var/log/suricata/eve.json { "timestamp": "2024-11-19T12:56:44.447219+0900", "flow_id": 1561530059235987, "event_type": "alert", "src_ip": "18.65.159.75", "src_port": 80, "dest_ip": "192.168.11.83", "dest_port": 37592, "proto": "TCP", "community_id": "1:SQuT4qYsqwmMzchc9if4AnDNNQ0=", "alert": { "action": "blocked", "gid": 1, "signature_id": 2100498, "rev": 7, "signature": "GPL ATTACK_RESPONSE id check returned root", "category": "Potentially Bad Traffic", "severity": 2, "metadata": { "created_at": [ "2010_09_23" ], "updated_at": [ "2019_07_26" ] } }, |
Integration of ELK Stack and SURICATA
Install & configure Elastic Stack to visualize & search SURICATA logs
This section is basically done on a second Ubuntu24.04 server
1. Elasticsearch Install
1.1 Download and install a public signature key
1 |
# wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | gpg --dearmor -o /usr/share/keyrings/elasticsearch-keyring.gpg |
1.2 Save repository definition in /etc/apt/sources.list.d/elastic-8.x.list
1 |
# echo "deb [signed-by=/usr/share/keyrings/elasticsearch-keyring.gpg] https://artifacts.elastic.co/packages/8.x/apt stable main" | tee /etc/apt/sources.list.d/elastic-8.x.list |
1.3 Install Elasticsearch package
1 |
# apt update && apt install elasticsearch |
2. Elasticsearch Configuration
2.1 Edit Elasticsearch configuration file
1 2 3 4 5 6 7 8 9 10 11 12 |
# vi /etc/elasticsearch/elasticsearch.yml Line 57 : Add the ip address of the Elastic Stack server network.host: 192.168.11.200 Line 62 : Uncomment http.port: 9200 Temporarily disable xpack.security.enabled and xpack.security.enrollment.enabled Line 93 : change xpack.security.enabled: false Line 95 : change xpack.security.enrollment.enabled: false |
Save the file
2.2 9200/tcp Open hort
1 2 |
# ufw allow 9200/tcp # ufw reload |
2.3 Enable and check Elasticsearch service
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 |
# systemctl start elasticsearch.service Send curl request to Elasticsearch Host # curl -X GET "192.168.11.200:9200" { "name" : "Lion", "cluster_name" : "elasticsearch", "cluster_uuid" : "rCXdMlLVTRiTl3xmX33IwA", "version" : { "number" : "8.16.0", "build_flavor" : "default", "build_type" : "deb", "build_hash" : "12ff76a92922609df4aba61a368e7adf65589749", "build_date" : "2024-11-08T10:05:56.292914697Z", "build_snapshot" : false, "lucene_version" : "9.12.0", "minimum_wire_compatibility_version" : "7.17.0", "minimum_index_compatibility_version" : "7.0.0" }, "tagline" : "You Know, for Search" } |
Elasticsearch version 8.16.0 is running
2.4 Edit the Elasticsearch configuration file again
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 |
# vi /etc/elasticsearch/elasticsearch.yml xpack.security back into effect. Line 93 : change xpack.security.enabled: true Line 95 : change xpack.security.enrollment.enabled: true Added “verification_mode: none” at line 101 xpack.security.http.ssl: enabled: true keystore.path: certs/http.p12 verification_mode: none Added “verification_mode: none” at line 106 xpack.security.transport.ssl: enabled: true verification_mode: none keystore.path: certs/transport.p12 truststore.path: certs/transport.p12 |
Save the file and restart elasticsearch.service
1 |
# systemctl restart elasticsearch.service |
2.4 Create passwords for elastic and kibana_system
Be sure to copy the passwords for the elastic and kibana_system users as they will be used later.
The kibana_system user is used to configure Kibana
The elastic user is used to configure Filebeat and login to Kibana
Create password for [elastic] user
1 2 3 4 5 6 7 8 9 |
# cd /usr/share/elasticsearch/bin # ./elasticsearch-reset-password -u elastic This tool will reset the password of the [elastic] user to an autogenerated value. The password will be printed in the console. Please confirm that you would like to continue [y/N]y Password for the [elastic] user successfully reset. New value: 1y6En9+QRA+BduUbrSYJ |
Create password for [kibana_system]user
1 2 3 4 5 6 7 8 9 |
# cd /usr/share/elasticsearch/bin # ./elasticsearch-reset-password -u kibana_system This tool will reset the password of the [kibana_system] user to an autogenerated value. The password will be printed in the console. Please confirm that you would like to continue [y/N]y Password for the [kibana_system] user successfully reset. New value: PKZfJ9uAnMsPB5c8xdAz |
3. Configure Kibana
Kibana is installed after Elasticsearch is installed.
Install Kibana on the same machine as Elasticsearch
3.1 Kibana Install
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 |
# apt install kibana Reading package lists... Done Building dependency tree... Done Reading state information... Done The following NEW packages will be installed: kibana 0 upgraded, 1 newly installed, 0 to remove and 2 not upgraded. Need to get 345 MB of archives. After this operation, 1,038 MB of additional disk space will be used. Get:1 https://artifacts.elastic.co/packages/8.x/apt stable/main amd64 kibana amd64 8.16.0 [345 MB] Fetched 345 MB in 5s (63.2 MB/s) Selecting previously unselected package kibana. (Reading database ... 88571 files and directories currently installed.) Preparing to unpack .../kibana_8.16.0_amd64.deb ... Unpacking kibana (8.16.0) ... Setting up kibana (8.16.0) ... Creating kibana group... OK Creating kibana user... OK Kibana is currently running with legacy OpenSSL providers enabled! For details and instructions on how to disable see https://www.elastic.co/guide/en/kibana/8.16/production.html#openssl-legacy-provider Created Kibana keystore in /etc/kibana/kibana.keystore Scanning processes... |
3.2 Create xpack.security settings and encryption key
Make a copy of the encryption key
1 2 3 4 5 6 7 |
# cd /usr/share/kibana/bin/ # ./kibana-encryption-keys generate -q xpack.encryptedSavedObjects.encryptionKey: 02ed1d28734acc1a28875504ebe005d9 xpack.reporting.encryptionKey: e2dbb6668a6a67ffc58129f272d1bca6 xpack.security.encryptionKey: cc4e87c2d357b53a35d4e6f133a1e876 |
Add encryption key to kibana.yml configuration file
1 2 3 4 5 6 7 8 9 10 |
# vi /etc/kibana/kibana.yml Add encryption key to last line # Maximum number of documents loaded by each shard to generate autocomplete suggestions. # This value must be a whole number greater than zero. Defaults to 100_000 #unifiedSearch.autocomplete.valueSuggestions.terminateAfter: 100000 xpack.encryptedSavedObjects.encryptionKey: 02ed1d28734acc1a28875504ebe005d9 xpack.reporting.encryptionKey: e2dbb6668a6a67ffc58129f272d1bca6 xpack.security.encryptionKey: cc4e87c2d357b53a35d4e6f133a1e876 |
3.3 Other kibana.yml configuration file editing
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 |
# vi /etc/kibana/kibana.yml Line 6 : Uncomment server.port: 5601 Line 12 : Replaced with Elastic Stack IP address (192.168.11.200) server.host: "192.168.11.200" Line 44 : add elasticsearch.hosts: ["https://192.168.11.200:9200"] Line 49,50 : Uncomment, leave elasticsearch.username as default (kibana_system) and change elasticsearch.password to the kibana_system password you created elasticsearch.username: "kibana_system" elasticsearch.password: "PKZfJ9uAnMsPB5c8xdAz" Line 96 : add (Kibana skips hostname validation when making outbound SSL/TSL connections to Elasticsearch) #elasticsearch.ssl.verificationMode: full elasticsearch.ssl.verificationMode: 'none' |
file save
3.4 Kibana service launched
1 |
# systemctl start kibana.service |
Check the status of Kibana.service
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 |
# systemctl status kibana.service ● kibana.service - Kibana Loaded: loaded (/usr/lib/systemd/system/kibana.service; disabled; preset: enabled) Active: active (running) since Tue 2024-11-19 14:17:35 JST; 6s ago Docs: https://www.elastic.co Main PID: 15123 (node) Tasks: 11 (limit: 4556) Memory: 343.8M (peak: 343.8M) CPU: 6.422s CGroup: /system.slice/kibana.service mq15123 /usr/share/kibana/bin/../node/glibc-217/bin/node /usr/share/kibana/bin/../src/cli/dist Nov 19 14:17:35 Lion systemd[1]: Started kibana.service - Kibana. Nov 19 14:17:35 Lion kibana[15123]: Kibana is currently running with legacy OpenSSL providers enabled! For details and instructions on how to disable see htt> Nov 19 14:17:36 Lion kibana[15123]: {"log.level":"info","@timestamp":"2024-11-19T05:17:36.111Z","log.logger":"elastic-apm-node","ecs.version":"8.10.0","agent> Nov 19 14:17:36 Lion kibana[15123]: Native global console methods have been overridden in production environment. Nov 19 14:17:37 Lion kibana[15123]: [2024-11-19T14:17:37.353+09:00][INFO ][root] Kibana is starting Nov 19 14:17:37 Lion kibana[15123]: [2024-11-19T14:17:37.383+09:00][INFO ][node] Kibana process configured with roles: [background_tasks, ui] |
3.5 http,https port open
1 2 3 |
# ufw allow http # ufw allow https # ufw reload |
4. Confirmation of Kibana web page launch
Access your web browser and verify that Kibana is running properly.
Try accessing Kibana by typing "http://192.168.11.200:5601". However, keep port 5601 open.
1 2 |
# ufw allow 5601/tcp # ufw reload |
Log in with elastic as Username and password created for elastic.
Select "Add integrations" and press Enter
Create a new user account so that you do not have to use the elastic superuser account.
Click on the three-line symbol in the upper left corner and select Stack Management.
「security」「Users」
Click on the “Create user” button in the upper right corner
Enter new user information and assign the roles kibana_admin, kibana_system, monitoring_user, and editor in Privileges
Log out of the current profile and confirm that you can log in with the newly created user account.
Currently, we do not have Filebeat configured on the SURICATA host, so we do not have any data that can be displayed by Kibana.
Filebeat Installation and Configuration
This work will be done on the first Debian 12.8 server with Suricata installed
1. Filebeat Install
1.1 Download Public Signature Key
1 |
# wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | apt-key add - |
1.2 Save repository definition in /etc/apt/sources.list.d/elastic-8.x.list
1 |
# echo "deb https://artifacts.elastic.co/packages/8.x/apt stable main" | tee -a /etc/apt/sources.list.d/elastic-8.x.list |
1.3 Install Filebeat
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 |
# apt update && apt install filebeat Get:1 http://security.debian.org/debian-security bookworm-security InRelease [48.0 kB] Get:2 http://deb.debian.org/debian bookworm-backports InRelease [59.0 kB] Hit:3 http://ftp.jp.debian.org/debian bookworm InRelease Get:4 http://ftp.jp.debian.org/debian bookworm-updates InRelease [55.4 kB] Get:5 http://security.debian.org/debian-security bookworm-security/main Sources [125 kB] Get:6 http://security.debian.org/debian-security bookworm-security/main amd64 Packages [204 kB] Get:7 https://artifacts.elastic.co/packages/8.x/apt stable InRelease [10.4 kB] Get:8 http://deb.debian.org/debian bookworm-backports/main amd64 Packages.diff/Index [63.3 kB] Get:9 http://deb.debian.org/debian bookworm-backports/main Translation-en.diff/Index [63.3 kB] Get:10 http://ftp.jp.debian.org/debian bookworm/main amd64 DEP-11 Metadata [4,492 kB] Get:11 http://deb.debian.org/debian bookworm-backports/main amd64 Packages T-2024-11-19-0205.28-F-2024-11-14-0805.11.pdiff [17.6 kB] Get:11 http://deb.debian.org/debian bookworm-backports/main amd64 Packages T-2024-11-19-0205.28-F-2024-11-14-0805.11.pdiff [17.6 kB] Get:12 https://artifacts.elastic.co/packages/8.x/apt stable/main amd64 Packages [120 kB] Get:13 http://deb.debian.org/debian bookworm-backports/main Translation-en T-2024-11-18-1408.37-F-2024-11-14-0805.11.pdiff [7,803 B] Get:13 http://deb.debian.org/debian bookworm-backports/main Translation-en T-2024-11-18-1408.37-F-2024-11-14-0805.11.pdiff [7,803 B] Get:14 http://ftp.jp.debian.org/debian bookworm/non-free-firmware amd64 DEP-11 Metadata [15.5 kB] Fetched 5,281 kB in 3s (2,032 kB/s) Reading package lists... Done Building dependency tree... Done Reading state information... Done 7 packages can be upgraded. Run 'apt list --upgradable' to see them. W: https://artifacts.elastic.co/packages/8.x/apt/dists/stable/InRelease: Key is stored in legacy trusted.gpg keyring (/etc/apt/trusted.gpg), see the DEPRECATION section in apt-key(8) for details. Reading package lists... Done Building dependency tree... Done Reading state information... Done The following NEW packages will be installed: filebeat 0 upgraded, 1 newly installed, 0 to remove and 7 not upgraded. Need to get 55.9 MB of archives. After this operation, 206 MB of additional disk space will be used. Get:1 https://artifacts.elastic.co/packages/8.x/apt stable/main amd64 filebeat amd64 8.16.0 [55.9 MB] Fetched 55.9 MB in 3s (21.4 MB/s) Selecting previously unselected package filebeat. (Reading database ... 53274 files and directories currently installed.) Preparing to unpack .../filebeat_8.16.0_amd64.deb ... Unpacking filebeat (8.16.0) ... Setting up filebeat (8.16.0) ... |
1.4 Configure Filebeat to connect to Elasticsearch and Kibana
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 |
# vi /etc/filebeat/filebeat.yml Line 114 : Uncomment and enter the Elastic Stack IP address and Kibana port number (5601) host: "192.168.11.200:5601" Per Line 141 : Uncomment Enter the ip address of Elastic Stack and port number of elasticsearch hosts: ["https://192.168.11.200:9200"] Uncomment per line 148 protocol: "https" Uncomment lines 152 and 153, leave [username] as default, and enter the [elastic] user's password in [password]. username: "elastic" password: “Ma6sJuSBGPnP+vav9K_P" Added around line 154 ssl.verification_mode: 'none' |
Enable Filebeat's Suricata module, but import the Suricata module from the .yml file, as Elasticsearch 8.x has a bug in importing modules
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 |
# vi /etc/filebeat/filebeat.yml Comment out the entire Filebeat module section per line 57 and add the following line under reload.period # ============================== Filebeat modules ============================== #filebeat.config.modules: # Glob pattern for configuration loading #path: ${path.config}/modules.d/*.yml # Set to true to enable config reloading #reload.enabled: false # Period on which files under path should be checked for changes #reload.period: 10s filebeat.modules: - module: suricata eve: enabled: true var.paths: ["/var/log/suricata/eve.json"] |
Next, force the SURICATA module to run
1 2 |
# filebeat setup --pipelines --modules suricata -M "suricata.eve.enabled=true" -c /etc/filebeat/filebeat.yml Loaded Ingest pipelines |
1.5 Set up initial environment
Load SIEM dashboard and pipeline into Elasticsearch
Run the filebeat setup command
Make sure suricata-eve-* is listed