Installing Graylog 3 in a TrueNAS 12.0 Jail (FreeBSD)

Graylog is an enterprise log aggregation and management framework similar to Splunk. Graylog, the company, is based in Houston, Texas (yay!) and boasts over 40k installations. What am I trying to accomplish by using Graylog at home? Well, my initial requirement is to collect logs from my network routers and incorporate that into my existing network monitoring using InfluxDB and Grafana. I’m a client of Comcast Xfinity and my network performance has really been poor in mid to late 2020. After multiple calls to Comcast and several tech visits, I’ve setup my own monitoring to “show” Comcast that I’m not getting the bandwidth they are billing me for. Log retention is low on my routers and cable modem (data rolls off quickly), a goal of this project is to retain at least 30-90 days of log messages from all network devices.

While I didn’t find this other post until after I wrote my post; I will cross link to this other post on essentially the same thing 😉 https://sign13.wordpress.com/setting-up-graylog-in-a-freenas-jail/

Step 1: Create the Graylog Jail

I’ll setup Graylog in a TrueNAS Core 12.0 Jail which will be running FreeBSD 12.2 release. The jail configuration isolates the software and configuration from other applications running on TrueNAS and it provides a stable environment with a large storage pool for log retention. In the Jails menu on TrueNAS Core 12.0, select the add button and the wizard will prompt you for input to create the Jail. Name your jail graylog and select the FreeBSD software release, then hit Next. I chose DHCP for my Networking configuration which will allow me easy SSH access to the “host”. Select Next and then select Submit to create the Jail. This tutorial will assume you have an understanding of basic Unix and SSH commands. After the wizard completes, you will want to click on the greater-than symbol in the graylog jail row in TrueNAS to show the jail details including the IP address. In this instance my graylog jail is 192.168.1.32 (take note!). In my network router I configured DHCP to give the same IP4 address to the MAC address for the graylog jail and to resolve the DNS hostname of graylog (so I don’t need to remember the IP4 address)

Test that the jail is up by pinging the “host”

rich@eragon:~$ ping graylog
PING graylog.pavlovs.ky (192.168.1.32) 56(84) bytes of data.
64 bytes from graylog.pavlovs.ky (192.168.1.32): icmp_seq=1 ttl=64 time=0.691 ms
64 bytes from graylog.pavlovs.ky (192.168.1.32): icmp_seq=2 ttl=64 time=0.619 ms
64 bytes from graylog.pavlovs.ky (192.168.1.32): icmp_seq=3 ttl=64 time=0.684 ms
64 bytes from graylog.pavlovs.ky (192.168.1.32): icmp_seq=4 ttl=64 time=0.752 ms
64 bytes from graylog.pavlovs.ky (192.168.1.32): icmp_seq=5 ttl=64 time=0.537 ms
^C
--- graylog.pavlovs.ky ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4006ms
rtt min/avg/max/mdev = 0.537/0.656/0.752/0.073 ms

Step 2: Setup SSH access and a local user

You will need to ssh into your TrueNAS box and use the iocage or the jexec command to remote into the jail on your first access. This is because you won’t have a local user account yet and the SSH daemon isn’t running. SSH into your TrueNAS (mine is named kidney) and type the JLS command to see the JID number of your jail. This will change with every restart so don’t write it down or memorize it. Use the jexec command to remote in.

kidney% jls
   JID  IP Address      Hostname                      Path
    12                  graylog                       /mnt/PavPool/iocage/jails/graylog/root
kidney% sudo jexec 12 /bin/tcsh 

We trust you have received the usual lecture from the local System
Administrator. It usually boils down to these three things:

    #1) Respect the privacy of others.
    #2) Think before you type.
    #3) With great power comes great responsibility.

Password:
root@graylog:/ # 

You will notice that you are now in a shell (a tcsh shell if you followed my above instructions) in the jail. Please note bash is not installed yet. You will be logged in as root. Update the packages and install bash first; you will need bash when you create your local user. You will also install the sudo package and configure it. sudo allows non-root users to run privileged commands.

root@graylog:/ # pkg update
Updating FreeBSD repository catalogue...
FreeBSD repository is up to date.
All repositories are up to date.
root@graylog:/ # pkg install bash sudo

Now you need to configure sudo by executing the visudo command. This command will edit the sudoers file to tell the system that your local account is allowed to run priveliged commands. Scroll down in the vi window until you see User Privilege

##
## User privilege specification
##
root ALL=(ALL) ALL

You should add a new row by putting cursor under the row with root like above and hitting the i key to insert. Type your user row as follows:

rich ALL=(ALL) ALL

Substitute username rich for whatever your username is. You may wonder that you haven’t created the user yet and that is ok! We will create the user next. Type SHIFT ZZ (that is hold the shift key and then type Z twice) to save the file.

Then, add your local user to enable easy logins as something other than root (not good to remote into things as root). In this example my local login name is rich with unix id of 1000 on all my boxes including TrueNAS <<< this is important. Type adduser to create the rich user with uid of 1000.

root@graylog:/ # adduser
Username: rich_
Full name: Rich P
Uid (Leave empty for default): 1000
Login group [rich_]: 
Login group is rich_. Invite rich_ into other groups? []: 
Login class [default]: 
Shell (sh csh tcsh bash rbash git-shell nologin) [sh]: bash
Home directory [/home/rich_]: 
Home directory permissions (Leave empty for default): 
Use password-based authentication? [yes]: 
Use an empty password? (yes/no) [no]: 
Use a random password? (yes/no) [no]: 
Enter password: 

Next, edit the /etc/rc.conf file and add this line to the end of the file

sshd_enable="YES"

Next, start the sshd service to test that it works

root@graylog:/ # service sshd start

Now, try to ssh into your TrueNAS jail from your localhost

rich@eragon:~$ ssh graylog
Password for rich@graylog:
Last login: Fri Dec  4 18:11:47 2020 from eragon.pavlovs.ky
FreeBSD 12.2-RC3 7c4ec6ff02c(HEAD) TRUENAS 

Welcome to FreeBSD!

To make it more secure you can add an SSO component or add ssh keys to each of your jails (i.e. no more typing your password). You should have convenient SSH access to your jail. Remember, with great power comes great responsibility. Also remember that you are no longer logging in as root so you’ll need to run sudo on privileged commands. Sudo should be all setup and we’ll test it in the next step.

Step 3: Install Graylog Packages

Type the following to install the graylog, elasticsearch and mongodb packages.

[rich@graylog ~]$ sudo pkg update
Updating FreeBSD repository catalogue...
FreeBSD repository is up to date.
All repositories are up to date.

[rich@graylog ~]$ sudo pkg install graylog elasticsearch6 mongodb44

Update 14-Dec: Updated post based on comments/feedback. I had made a mistake on the packages I used, which was corrected by a reader.

It may take a few mins to download and install those packages. MongoDB will be used for the graylog configuration only. All log data will be stored in the elasticsearch cluster. These packages are downloaded from FreeBSD fresh ports which is a very extensive software pkg repository. Please note as of this writing in Dec 2020, version 3.3.1 is the latest FreeBSD graylog package available but graylog itself is shipping version 4.0.2 on the graylog.org website.

Step 4: Configure Software

None of the above software will be running after it’s installation. Don’t start the services yet. First, edit /etc/rc.conf to tell TrueNAS to autostart the software when the jail starts. Add these lines to the end of the file.

graylog_enable="YES"
elasticsearch_enable="YES"
mongod_enable="YES"

Next, edit the /usr/local/etc/elasticsearch/elasticsearch.yml file to change a few parameters. Add the following to the bottom of the file and save it.

cluster.name: graylog

Start the mongoDB and the elasticsearch components by typing:

[rich@graylog ~]$ sudo service elasticsearch start

[rich@graylog ~]$ sudo service mongod start

Notice the output of the above service commands, if there are any errors starting the commands it should prompt you. You can check if both services are running by typing a netstat command.

[rich@graylog ~]$ netstat -an | grep -i listen

tcp4       0      0 127.0.0.1.9200         *.*                    LISTEN     
tcp6       0      0 ::1.9200               *.*                    LISTEN     
tcp4       0      0 127.0.0.1.9300         *.*                    LISTEN     
tcp6       0      0 ::1.9300               *.*                    LISTEN     
tcp4       0      0 *.22                   *.*                    LISTEN     
tcp6       0      0 *.22                   *.*                    LISTEN     
tcp4       0      0 127.0.0.1.27017        *.*                    LISTEN  

This command shows which TCP ports are being listened to by server processes like mongo and elasticsearch. You will notice that 9200 and 9300 are elasticsearch. 22 is SSHD and 27017 is mongo. If your output from a netstat is similar then you are doing good so far. Now, we need to configure graylog and then start the service for the first time. The graylog config will be in the /usr/local/etc/graylog directory.

[rich@graylog ]$ cd /usr/local/etc/graylog

[rich@graylog ]$ sudo cp graylog.conf.example graylog.conf

[rich@graylog ]$ sudo cp log4j2.xml.example log4j2.xml   

[rich@graylog ]$ sudo mkdir server 

[rich@graylog ]$ sudo touch server/node-id 

[rich@graylog ]$ cd ..

[rich@graylog ]$ sudo chown -R graylog:graylog graylog

This will setup the initial config directory and change the owner of the files to graylog user and graylog group. That user and group should have been created when you pkg installed graylog. If you don’t have that user or group then go back to the package install step and check you did that correctly. Change directory back to /usr/local/etc/graylog and edit the graylog.conf file and ensure the variables are set as such.

is_master = true
node_id_file = /usr/local/etc/graylog/server/node-id
root_username = admin
root_timezone = UTC
bin_dir = /usr/local/share/graylog
data_dir = /usr/local/var/lib/graylog
plugin_dir = /usr/local/share/graylog/plugin

http_bind_address = 192.168.1.32:9000

message_journal_dir = /usr/local/var/lib/graylog/journal

The above config parameters are scattered throughout the config file. Your http_bind_address will be the jail ip4 address. Ensure that all those directories like the journal directory exist and are owned by graylog:graylog. Next, set the hashed admin password and seed by running these two commands on the command line.

pwgen -N 1 -s 96

echo -n yourpassword | shasum -a 256

Edit the graylog.conf file and follow the instructions to paste the password_seed and root_password_sha2 to set those values and then save the file. There are instructions inline in the conf file on the password seed and root password if you are confused. Make sure the file is only readable to the graylog user and group. Now you can finally start the Graylog service

[rich@graylog ]$ sudo service graylog start

[rich@graylog ]$ sudo tail -f /var/log/graylog/server.log

Tail the /var/log/graylog/server.log file to ensure it starts correctly and no fatal errors. It takes a minute or two to startup.

Step 5: Test Jail Restart to Ensure Auto-Launch

If graylog started up correctly, go ahead and stop the jail in the TrueNAS Jails menu. Click the Edit Jail menu item under the graylog jail properties and in the Basic Properties sheet you will check the Auto-start checkbox and click save. Next you can click the start button on the graylog jail and wait a few min for everything to start up. You then should be able to access the graylog UI at http://<your ip address>:9000 (please note that you need to fill in your ip address and it will be different than mine.)

Further tutorials will include hardening + adding SSL and configuring syslog inputs.