Adjust Info on Cerebro

Docker/docker-compose.yml

@@ -60,6 +60,7 @@ services:
   # Kibana : https://www.elastic.co/guide/en/kibana/6.8/index.html
   kibana:
     image: docker.elastic.co/kibana/kibana:7.11.1
+    entrypoint: ["echo", "Service Kibana disabled"]
     env_file:
       - kibana.env
     depends_on:
@@ -68,6 +69,7 @@ services:
       - 5601:5601
   cerebro:
     image: lmenezes/cerebro
+#    entrypoint: ["echo", "Service cerebro disabled"]
     ports:
       - 9001:9000
     links:

Docker/grafana/provisioning/datasources/automatic.yml

@@ -35,5 +35,5 @@ datasources:
   database: "pfsense_*"
   url: http://elasticsearch:9200
   jsonData:
-    esVersion: 60
+    esVersion: 70
-    timeField: "utc_timestamp"
+    timeField: "timestamp"

Docker/graylog.env

@@ -2,6 +2,6 @@
 GRAYLOG_PASSWORD_SECRET=somepasswordpepperzzz
 # Password: admin
 GRAYLOG_ROOT_PASSWORD_SHA2=8c6976e5b5410415bde908bd4dee15dfb167a9c873fc4bb8a81f6f2ab448a918
-GRAYLOG_HTTP_EXTERNAL_URI=http://localhost:9000/
+GRAYLOG_HTTP_EXTERNAL_URI=http://pfanalytics.home:9000/
 # TZ List - https://en.wikipedia.org/wiki/List_of_tz_database_time_zones
-GRAYLOG_TIMEZONE=Europe/Berlin
+GRAYLOG_TIMEZONE=Europe/Berlin

Docker/graylog/getGeo.sh

@@ -1,2 +1,2 @@
-curl --output /etc/graylog/server/mm.tar.gz "https://download.maxmind.com/app/geoip_download_by_token?edition_id=GeoLite2-City&date=20210216&suffix=tar.gz&token=v2.local.jM7J0O4PMocBknBIc2Hkh1gO4VKQ9sBPM72EOg5i9KVuJL_rOchpeHh7uA9k0cc752E1lj9pWMQsOofvbSqFWW7GcJdsWXXqDONgiyW7_Zxg6UVvREHEa7g9pd7tne5oZG-KZOZx-VjCM_g6CNb2ccblHVnEiAjD9jzSZdY8QcNNMu7qYBMfXvlMQKHlrJvTM0oJgg"
+curl -o /etc/graylog/server/mm.tar.gz 'https://download.maxmind.com/app/geoip_download?edition_id=GeoLite2-City&license_key=<YOURLICENSKEYGOESHERE>&suffix=tar.gz'
 tar zxvf /etc/graylog/server/mm.tar.gz -C /etc/graylog/server/ --strip-components=1

README.md

@@ -1,3 +1,5 @@
+# pfSense Analytics
+
 This Project aims to give you better insight of what's going on your pfSense Firewall. It's based on some heavylifting alrerady done by devopstales and opc40772. Since it still was a bit clumsy and outdated I wrapped some docker-compose glue around it, to make it a little bit easier to get up and running. It should work hasslefree with a current Linux that has docker and docker-compose. Thanks as well to MatthewJSalerno for some Streamlining of the Graylog provisioning Process.
 
 I have recently updated the whole stack to utilize Graylog 4 and Elasticsearch 7 and Grafana 7. I don't include any directions for Upgrading GL3/ES6 to GL4/ES7. 
@@ -10,6 +12,7 @@ This doc has been tested with the following Versions:
 | Grafana | 7.4.2 |
 | Graylog | 4.0.3 |
 | Cerebro | 0.9.3 | 
+| pfSense | 2.5.0 CE|
 
 
 If it's easier for you, you can find a video guide here:  https://youtu.be/uOfPzueH6MA (Still the Guide for GL3/ES6, will make a new one some day.)
@@ -82,7 +85,7 @@ A salt for encrypting your graylog passwords
 - GRAYLOG_PASSWORD_SECRET (Change that _now_)
 
 
-Edit Docker/graylog/getGeo.sh and insert _your_ tokenized Downloadlink of the Maxmind GeoIP Database. Create an account on https://www.maxmind.com/en/account/login and go to "My Account -> Download Files -> GeoLite2 City" and copy the Link "Download GZIP" to your getGeo.sh File. If you don't do that the geolookup feature for IP Addresses won't work.
+Edit `Docker/graylog/getGeo.sh` and insert _your_ license Key for the  Maxmind GeoIP Database. Create an account on https://www.maxmind.com/en/account/login and go to "My Account -> Manage License Keys -> Generate new License key" and copy the that Key to the placeholder in your getGeo.sh File. If you don't do that the geolookup feature for IP Addresses won't work.
 
 Finally, spin up the stack with:
 
@@ -114,12 +117,12 @@ your Graylog Instance on http://localhost:9000. Let's see if we can login with u
 
 Next we have to create the Index in Elasticsearch for the pfSense logs in System / Indices
 
-![Index](https://raw.githubusercontent.com/lephisto/pfsense-analytics/master/screenshots/SS_Indexcreation.png)
+![Index](https://raw.githubusercontent.com/lephisto/pfsense-analytics/master/screenshots/SS_Indexcreation4.png)
 
 
 Index shard 4 and Index replicas 0, the rotation of the Index time index and the retention can be deleted, closure of an index according to the maximum number of indices or doing nothing. In my case, I set it to rotate monthly and eliminate the indexes after 12 months. In short there are many ways to establish the rotation. This index is created immediately.
 
-![Indices](https://www.sysadminsdecuba.com/wp-content/uploads/2018/04/Graylog_-_Indices_and_Index_Sets_-_2018-04-04_20.30.42-1024x82.png)
+![Indices](https://raw.githubusercontent.com/lephisto/pfsense-analytics/master/screenshots/SS_Indexcreation_done.png)
 
 # 3. GeoIP Plugin activation
 
@@ -136,7 +139,7 @@ In Graylog go to System->Configurations and:
 
 This should look like:
 
-![Index](https://raw.githubusercontent.com/lephisto/pfsense-analytics/master/screenshots/SS_processorsequence.png)
+![Index](https://raw.githubusercontent.com/lephisto/pfsense-analytics/master/screenshots/SS_processorsequence4.png)
 
 
 2. In the Plugins section update enable the Geo-Location Processor
@@ -150,11 +153,11 @@ This content pack includes Input rsyslog type , extractors, lookup tables, Data
 
 We can take it from the Git directory or sideload it from github to the Workstation you do the deployment on:
 
-https://raw.githubusercontent.com/lephisto/pfsense-analytics/master/pfsense_content_pack/graylog3/pfanalytics.json
+https://raw.githubusercontent.com/lephisto/pfsense-analytics/master/pfsense_content_pack/graylog4/pfanalytics.json
 
 Once it's uploaded, press the Install button. If everthing went well it should look like:
 
-![dpi1](https://raw.githubusercontent.com/lephisto/pfsense-analytics/master/screenshots/contentpack.png)
+![dpi1](https://raw.githubusercontent.com/lephisto/pfsense-analytics/master/screenshots/SS_Contentpacks.png)
 
 Note the "pfintel" on the bottom of the list.
 
@@ -173,7 +176,7 @@ As previously explained, by default graylog for each index that is created gener
 
 Get the Index Template from the GIT repo you cloned or sideload it from:
 
-https://raw.githubusercontent.com/lephisto/pfsense-graylog/master/Elasticsearch_pfsense_custom_template/pfsense_custom_template_es6.json
+https://raw.githubusercontent.com/lephisto/pfsense-analytics/master/Elasticsearch_pfsense_custom_template/pfsense_custom_template_es7.json
 
 To import personalized template open cerebro and will go to more/index template
 
@@ -207,7 +210,7 @@ Once this procedure is done, we don't need Cerebro for daily work, so it could b
 
 We will now prepare Pfsense to send logs to graylog and for this in Status/System Logs/ Settings we will modify the options that will allow us to do so.
 
-We go to the Remote Logging Options section and in Remote log servers we specify the ip address and the port prefixed in the content pack in the pfsense input of graylog that in this case 5442.
+We go to the Remote Logging Options section and in Remote lo7g servers we specify the ip address and the port prefixed in the content pack in the pfsense input of graylog that in this case 5442.
 
 ![Pfsense](https://www.sysadminsdecuba.com/wp-content/uploads/2018/04/Pfsene-log-settings-1024x329.png)
 
@@ -257,4 +260,18 @@ Configure according your needs, I propose following Settings:
 | Datebase Top Talker Storage   | 365d  |   |
 
 
+# Disable Cerebro.
+
+Since Cerebro is mainly used for applying a custom Index Template, we don't need it in our daily routine and we can disable it. Edit your docker-compose.yml and remove the comment in the service block for Cerebro:
+
+```
+  cerebro:
+    image: lmenezes/cerebro
+    entrypoint: ["echo", "Service cerebro disabled"]
+```
+
+No need to restart the whole Stack, just stop Cerebro:
+
+`sudo docker-compose stop cerebro`
+
 That should do it. Check your DPI Dashboard and enjoy :)