I’m using Discord to keep up with some technical discussion groups (e.g. TrueNAS and OBS) and I’ve even created a personal Discord server for my family. I’m also running Graylog for log aggregation of my home technology stack (switches, IP phones, Asterisk, Plex, Pihole, Router) and I was wondering if I could create automated alerts from Graylog to my Discord server? It is very timely as Graylog posted a blog entry just this past Friday on how to do this very thing. Now, before we go any further we need to be clear on what is required from a software perspective. You will need a working Graylog server version 4.0 or greater (I’m running version 4.0.5 on FreeBSD) and you will need to install and license the enterprise plug-in so that you can use the correlation engine. Don’t worry, the enterprise Graylog plugin is free for personal use under 5 GB of logs a day (I’m at like 400 Mb’s a day). You will also need a Discord server which are free to setup for personal use. Instructions for setting up Graylog can be found here and here.
Make sure you let your children know what you are up to.
Step 1: Inform Your Children
Children are super curious; they like to know when new network monitoring infrastructure is configured for your home network. I’ve found that a bit of subterfuge throws them off the scent.
Create a new #channel for your alerts
Step 2: In Discord, Setup a New Text Channel for Your Alerts (Optional)
This step is optional but useful if you want to separate alert content from your normal content in Discord. In your Text Channels menu for your Discord server (you must be the server owner or granted admin privs), you can click the plus icon to create a new text channel. I named mine graylog and defined a topic text. Click save. Please notice the Integrations menu item in the above graph. We will click on that next.
Add the Webhook which your Graylog server will call
Step 3: In Discord, Add a Webhook Integration
Within your text channel configuration you will see an Integrations menu item. Click Integrations and then click on Webhooks and New webhook.
You can add a unique icon to the automated messages
In the New Webhook dialog, you can name your webhook. I chose a descriptive name for this but yours can be more generic. Note the Channel the webhook will be in and you can copy the webhook URL (a url encoded string to your server). I got the Graylog icon from their Twitter feed @graylog2 account.
Graylog Event Definitions
Step 4: In Graylog, Add an Event Definition
In the Graylog Alerts menu, click Event Definitions and then the Create Event Definition button. I named my Event “SSHd Logon Open Too Long” to match to Graylog blog video which is linked at the start of this blog post. Click Next. I set the Condition Type to be “Event Correlation” and then set the Correlation rules to follow a sequence of events that are satisfied within 16 min. You can make it more or less but essentially I’m looking for SSH logins for more than 16 min.
The event rules you will correlate
I set Event #1 in the correlation to be the “SSHd Session Open” Event from my alerts. I added Event #2 and set it to “SSHd Session Closed” which SHOULD NOT OCCUR in the next 15 min. This means event #1 will fire and within 15 min a subsequent event #2 will not fire. The definition of those events will be covered later and is outside the scope of this article. click next.
Create a custom event field for the user_name
Step 5: Create Custom Event Field for User Name
Click Add Custom Field in the Fields item of the wizard. Set the name to user_name and click to make this a field key. Set the template to
template: ${source_user_name}
And make sure you do not click the value is required. Click Next.
Setup the Slack Notification (works with Discord as well)
Step 6: Setup the Notification to Discord
In the Notifications wizard item, click on the Add Notification button.
Add Notification dialog
Within Add Notification, you will set a descriptive title and description (I’ve again used the example they discussed in the Graylog blog video). You MUST select Notification type of Slack and you can change the highlight color. In the webhook URL field you will paste the URL copied from Step 3 above. Please append to the end of the Webhook URL “/slack” which will tell Discord this message has been formated for Slack and to handle that format type. You can specify the channel again and the custom message template. You can use your custom user_name filed in the message template. Click on the Execute Test Notification button to verify your test message is logged to Discord. Click Save and wait for your event notifications!
FreeBSD package management recently updated their Graylog package from 3.3.0 to 4.0.5 with enterprise plugins! This article is as-of April 2021 and you should upgrade your FreeBSD os or Jail version to 12.2 and upgrade your packages (see below).
Now Graylog software is written in Java and distributed as JAR (Java ARchive) files so you really don’t need to wait for FreeBSD packaging to upgrade your server. In fact, just a few weeks before I wrote this I had upgraded the server manually by downloading the latest Graylog Jars from graylog.org website and putting them in the appropriate server directory. This works but was very manual and could be prone to error. The FreeBSD package automates everything and makes it simple I will not discuss my manual upgrade process and suggest to you to just use the official package noted above (v4.0.5). I even applied the FreeBSD package over my manual upgrade and it handled everything gracefully (by backing up the jars I placed as .prev versions).
There are numerous enhancements to Graylog and the software pkg upgrade process was relatively straight forward. PLEASE NOTE that I had to upgrade Elasticsearch to either version 6 or 7 (I was on version 5). This is noted in the FreeBSD graylog-4.0.5_2 pkg release notes that you must upgrade Elasticsearch (it doesn’t tell you how).
If you attempt to start the graylog service on your FreeBSD instance without first upgrading Elasticsearch and you are running version 5 or below then the web interface will fail to start and the graylog server will repeatedly log an error connecting to Elasticsearch.
If you have data in your Elasticsearch version 5 instance and you use FreeBSD package manager to install elasticsearch6 then it will upgrade the binaries and remove the elesticsearch5 package. It left the data intact in my experiance and I had to do a bit of editing of the elasticsearch config yaml file to start the new elasticsearch6 instance. The edits were very minor. I essentially had my old version 5 yaml config file up and diffed it against the new version 6 example and updated accordingly. I didn’t take notes of my edits but again they were version small.
You may also need to update your graylog config xml file although your mileage might vary. The connection details to elasticsearch changes slightly from my version 5 to version 6 so I had to edit in graylog config.
I restarted the graylog service again and tailed the /var/log/graylog/server.log to verify that it successfully restarted. I was able to login again to the web interface and verified none of my data was lost and that my graylog server was healthy. I took an immediate archive of the data to compressed gzip tar file just in case. The reader will note I took a backup after the upgrade not before… I should have taken a backup before if I was being truly risk averse as a rollback option. tsk, tsk on me…
Graylog is an enterprise log aggregation and management framework similar to Splunk. Graylog, the company, is based in Houston, Texas (yay!) and boasts over 40k installations. What am I trying to accomplish by using Graylog at home? Well, my initial requirement is to collect logs from my network routers and incorporate that into my existing network monitoring using InfluxDB and Grafana. I’m a client of Comcast Xfinity and my network performance has really been poor in mid to late 2020. After multiple calls to Comcast and several tech visits, I’ve setup my own monitoring to “show” Comcast that I’m not getting the bandwidth they are billing me for. Log retention is low on my routers and cable modem (data rolls off quickly), a goal of this project is to retain at least 30-90 days of log messages from all network devices.
I’ll setup Graylog in a TrueNAS Core 12.0 Jail which will be running FreeBSD 12.2 release. The jail configuration isolates the software and configuration from other applications running on TrueNAS and it provides a stable environment with a large storage pool for log retention. In the Jails menu on TrueNAS Core 12.0, select the add button and the wizard will prompt you for input to create the Jail. Name your jail graylog and select the FreeBSD software release, then hit Next. I chose DHCP for my Networking configuration which will allow me easy SSH access to the “host”. Select Next and then select Submit to create the Jail. This tutorial will assume you have an understanding of basic Unix and SSH commands. After the wizard completes, you will want to click on the greater-than symbol in the graylog jail row in TrueNAS to show the jail details including the IP address. In this instance my graylog jail is 192.168.1.32 (take note!). In my network router I configured DHCP to give the same IP4 address to the MAC address for the graylog jail and to resolve the DNS hostname of graylog (so I don’t need to remember the IP4 address)
Test that the jail is up by pinging the “host”
rich@eragon:~$ ping graylog
PING graylog.pavlovs.ky (192.168.1.32) 56(84) bytes of data.
64 bytes from graylog.pavlovs.ky (192.168.1.32): icmp_seq=1 ttl=64 time=0.691 ms
64 bytes from graylog.pavlovs.ky (192.168.1.32): icmp_seq=2 ttl=64 time=0.619 ms
64 bytes from graylog.pavlovs.ky (192.168.1.32): icmp_seq=3 ttl=64 time=0.684 ms
64 bytes from graylog.pavlovs.ky (192.168.1.32): icmp_seq=4 ttl=64 time=0.752 ms
64 bytes from graylog.pavlovs.ky (192.168.1.32): icmp_seq=5 ttl=64 time=0.537 ms
^C
--- graylog.pavlovs.ky ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4006ms
rtt min/avg/max/mdev = 0.537/0.656/0.752/0.073 ms
Step 2: Setup SSH access and a local user
You will need to ssh into your TrueNAS box and use the iocage or the jexec command to remote into the jail on your first access. This is because you won’t have a local user account yet and the SSH daemon isn’t running. SSH into your TrueNAS (mine is named kidney) and type the JLS command to see the JID number of your jail. This will change with every restart so don’t write it down or memorize it. Use the jexec command to remote in.
kidney% jls
JID IP Address Hostname Path
12 graylog /mnt/PavPool/iocage/jails/graylog/root
kidney% sudo jexec 12 /bin/tcsh
We trust you have received the usual lecture from the local System
Administrator. It usually boils down to these three things:
#1) Respect the privacy of others.
#2) Think before you type.
#3) With great power comes great responsibility.
Password:
root@graylog:/ #
You will notice that you are now in a shell (a tcsh shell if you followed my above instructions) in the jail. Please note bash is not installed yet. You will be logged in as root. Update the packages and install bash first; you will need bash when you create your local user. You will also install the sudo package and configure it. sudo allows non-root users to run privileged commands.
root@graylog:/ # pkg update
Updating FreeBSD repository catalogue...
FreeBSD repository is up to date.
All repositories are up to date.
root@graylog:/ # pkg install bash sudo
Now you need to configure sudo by executing the visudo command. This command will edit the sudoers file to tell the system that your local account is allowed to run priveliged commands. Scroll down in the vi window until you see User Privilege
##
## User privilege specification
##
root ALL=(ALL) ALL
You should add a new row by putting cursor under the row with root like above and hitting the i key to insert. Type your user row as follows:
rich ALL=(ALL) ALL
Substitute username rich for whatever your username is. You may wonder that you haven’t created the user yet and that is ok! We will create the user next. Type SHIFT ZZ (that is hold the shift key and then type Z twice) to save the file.
Then, add your local user to enable easy logins as something other than root (not good to remote into things as root). In this example my local login name is rich with unix id of 1000 on all my boxes including TrueNAS <<< this is important. Type adduser to create the rich user with uid of 1000.
root@graylog:/ # adduser
Username: rich_
Full name: Rich P
Uid (Leave empty for default): 1000
Login group [rich_]:
Login group is rich_. Invite rich_ into other groups? []:
Login class [default]:
Shell (sh csh tcsh bash rbash git-shell nologin) [sh]: bash
Home directory [/home/rich_]:
Home directory permissions (Leave empty for default):
Use password-based authentication? [yes]:
Use an empty password? (yes/no) [no]:
Use a random password? (yes/no) [no]:
Enter password:
Next, edit the /etc/rc.conf file and add this line to the end of the file
sshd_enable="YES"
Next, start the sshd service to test that it works
root@graylog:/ # service sshd start
Now, try to ssh into your TrueNAS jail from your localhost
rich@eragon:~$ ssh graylog
Password for rich@graylog:
Last login: Fri Dec 4 18:11:47 2020 from eragon.pavlovs.ky
FreeBSD 12.2-RC3 7c4ec6ff02c(HEAD) TRUENAS
Welcome to FreeBSD!
To make it more secure you can add an SSO component or add ssh keys to each of your jails (i.e. no more typing your password). You should have convenient SSH access to your jail. Remember, with great power comes great responsibility. Also remember that you are no longer logging in as root so you’ll need to run sudo on privileged commands. Sudo should be all setup and we’ll test it in the next step.
Step 3: Install Graylog Packages
Type the following to install the graylog, elasticsearch and mongodb packages.
[rich@graylog ~]$ sudo pkg update
Updating FreeBSD repository catalogue...
FreeBSD repository is up to date.
All repositories are up to date.
[rich@graylog ~]$ sudo pkg install graylog elasticsearch6 mongodb44
Update 14-Dec: Updated post based on comments/feedback. I had made a mistake on the packages I used, which was corrected by a reader.
It may take a few mins to download and install those packages. MongoDB will be used for the graylog configuration only. All log data will be stored in the elasticsearch cluster. These packages are downloaded from FreeBSD fresh ports which is a very extensive software pkg repository. Please note as of this writing in Dec 2020, version 3.3.1 is the latest FreeBSD graylog package available but graylog itself is shipping version 4.0.2 on the graylog.org website.
Step 4: Configure Software
None of the above software will be running after it’s installation. Don’t start the services yet. First, edit /etc/rc.conf to tell TrueNAS to autostart the software when the jail starts. Add these lines to the end of the file.
Next, edit the /usr/local/etc/elasticsearch/elasticsearch.yml file to change a few parameters. Add the following to the bottom of the file and save it.
cluster.name: graylog
Start the mongoDB and the elasticsearch components by typing:
[rich@graylog ~]$ sudo service elasticsearch start
[rich@graylog ~]$ sudo service mongod start
Notice the output of the above service commands, if there are any errors starting the commands it should prompt you. You can check if both services are running by typing a netstat command.
This command shows which TCP ports are being listened to by server processes like mongo and elasticsearch. You will notice that 9200 and 9300 are elasticsearch. 22 is SSHD and 27017 is mongo. If your output from a netstat is similar then you are doing good so far. Now, we need to configure graylog and then start the service for the first time. The graylog config will be in the /usr/local/etc/graylog directory.
This will setup the initial config directory and change the owner of the files to graylog user and graylog group. That user and group should have been created when you pkg installed graylog. If you don’t have that user or group then go back to the package install step and check you did that correctly. Change directory back to /usr/local/etc/graylog and edit the graylog.conf file and ensure the variables are set as such.
The above config parameters are scattered throughout the config file. Your http_bind_address will be the jail ip4 address. Ensure that all those directories like the journal directory exist and are owned by graylog:graylog. Next, set the hashed admin password and seed by running these two commands on the command line.
Edit the graylog.conf file and follow the instructions to paste the password_seed and root_password_sha2 to set those values and then save the file. There are instructions inline in the conf file on the password seed and root password if you are confused. Make sure the file is only readable to the graylog user and group. Now you can finally start the Graylog service
Tail the /var/log/graylog/server.log file to ensure it starts correctly and no fatal errors. It takes a minute or two to startup.
Step 5: Test Jail Restart to Ensure Auto-Launch
If graylog started up correctly, go ahead and stop the jail in the TrueNAS Jails menu. Click the Edit Jail menu item under the graylog jail properties and in the Basic Properties sheet you will check the Auto-start checkbox and click save. Next you can click the start button on the graylog jail and wait a few min for everything to start up. You then should be able to access the graylog UI at http://<your ip address>:9000 (please note that you need to fill in your ip address and it will be different than mine.)
Further tutorials will include hardening + adding SSL and configuring syslog inputs.
