Build and Run Beats on Raspberry Pi

Beats are the replacement for Logstash Forward from Elastic. They are data shippers for Elastic Search. Unfortunately, compiling them on the Raspberry Pi isn’t very straightforward.Elastic Beats

In this post I’ll show you how to build and run Beats on Raspberry Pi.

Install Go

Firstly, you’ll need to install go. Beats requires Go version 1.5.3 or higher (at the time of writing.) Unfortunately this process is a little roundabout as you must first install go 1.4.x in order to install 1.5.x. Convenient! </sarcasm>

If you’ve already followed my tutorial for building Logstash Forward on a Raspberry Pi you can skip the step for installing go 1.4. Otherwise grab the latest sources from the Go Downloads. I’ll be using 1.4.2 below, but you should update the wget link with the link of the version you plan on using. The below script will remove any existing go install, so be careful if you want another go version for other things.

which go && sudo rm -rf /usr/bin/go* /usr/local/go
cd ~
wget https://storage.googleapis.com/golang/go1.4.2.src.tar.gz
sudo tar -C /usr/local -xvf go1.4.*.tar.gz
cd /usr/local/go/src
sudo ./make.bash

That will take a while to build. Once it’s complete download and install Go 1.5.x (grabbing the latest version, as before):

cd ~
wget https://storage.googleapis.com/golang/go1.5.3.src.tar.gz
sudo tar -xvf go1.5.*.tar.gz
sudo mv go /usr/local/go1.5
cd /usr/local/go1.5/src
sudo GOROOT_BOOTSTRAP=/usr/local/go ./make.bash
sudo ln -s /usr/local/go1.5/bin/* /usr/bin

Compile Beats

Once you have go 1.5.x installed, you’re ready to get on the way to installing beats. First however, you’ll need to install a bunch of dependencies:

mkdir -p /usr/local/go1.5/src/github.com/elastic
sudo chown -R pi /usr/local/go1.5/src/github.com #change to your username if not "pi"
cd /usr/local/go1.5/src/github.com/elastic
git clone https://github.com/elastic/beats
export GOROOT=/usr/local/go1.5/

Next install of the go dependencies for the project:

export GOPATH=/usr/local/go1.5/src/github.com/elastic/beats

# Install easy go deps
go get golang.org/x/text/encoding/charmap
go get golang.org/x/text/encoding/simplifiedchinese
go get golang.org/x/text/encoding/unicode
go get golang.org/x/text/transform
go get gopkg.in/yaml.v2

go get github.com/garyburd/redigo/redis
go get github.com/nranchev/go-libGeoIP
go get github.com/satori/go.uuid

Now that we have all of the dependencies its time to actually go about compiling Beats. To do so, compile the individual beats you need. The process looks like this:

cd /usr/local/go1.5/src/github.com/elastic/beats

# Pick a beat to compile. Choose from "filebeat", "topbeat", or "packetbeat"
cd filebeat
make
sudo cp filebeat /usr/bin/ # can also use "sudo ln -s filebeat /usr/bin" if you plan on keeping the sources around

There you go! Compile each beat that you want to use and enjoy!

, , , , , ,

6 Responses to Build and Run Beats on Raspberry Pi

  1. Attila Pinter February 15, 2016 at 1:36 pm #

    Hi Michael,
    Thanks for the great tutorial, unfortunately for some reason I keep running into issues when I try to run the ‘go get’ commands. I usually get an error message like this:
    package errors: unrecognized import path “errors”
    package bytes: unrecognized import path “bytes”
    package io: unrecognized import path “io”
    package unicode/utf8: unrecognized import path “unicode/utf8”
    package strconv: unrecognized import path “strconv”

    I tried to Google the issue and see numerous issues like this and usually the issue is setting GOPATH and GOROOT although it is still not solving my problem :/

    Maybe you have any suggestions what I can try?

    Big thanks in advance!

    • Michael Blouin February 15, 2016 at 1:47 pm #

      Hey Attila,

      Sorry to hear you’re having trouble. This sounds alot like a path problem.

      Can you ensure that you set your GOPATH using the export command? (as in the code sample).


      export GOROOT=/usr/local/go1.5/src/
      export GOPATH=/usr/local/go1.5/src/github.com/elastic/beats

      Also can you please ensure that the path /usr/local/go1.5/src/github.com/elastic/beats exists, and is filled with the contents of the git repo you checked out previously? (https://github.com/elastic/beats)

      Are you doing any steps differently or in different paths?

      -Mike

      • Attila Pinter February 15, 2016 at 2:21 pm #

        Hi Michael,

        I can confirm that in my case I was following the instructions and got the above posted errors, but no worries I think in the meanwhile I solved the problem by doing it a bit differently. The only thing I changed is the GOROOT path: export GOROOT=/usr/local/go1.5/ and now it ran perfectly and just finished compiling filebeat. Not really sure what went wrong as this is the first time I was even installing golang. Maybe I do made a mistake somewhere :/

        Anyway great tutorial and helped me a lot to get started, appreciate your work!


        Attila

        • Michael Blouin February 15, 2016 at 2:23 pm #

          Thanks Attila!

          Looking back I think I may have enterred that path wrong. I’ll update the path set in the tutorial.

          -Mike

          • Attila Pinter February 17, 2016 at 4:42 pm #

            Not to worry mate, still works like a charm. The only thing what is missing or at least for me it was missing after compiling topbeat and filebeat is the god files what is being used as wrappers in the original init.d script from Elastic running parallel with the main programs. So what I did is removing the wrappers from the init.d for both, but still got an error running it for some reason which is strange because starting it from the shell runs fine (/usr/bin/filebeat -c /etc/filebeat/filebeat.yml same for topbeat, built them a proper place in /etc) so added a small modification to the scripts removing the same things so in case of filebeat it looks like this:

            #!/bin/sh

            BEGIN INIT INFO

            Provides: filebeat

            Required-Start: $local_fs $network $syslog

            Required-Stop: $local_fs $network $syslog

            Default-Start: 2 3 4 5

            Default-Stop: 0 1 6

            Short-Description: Sends log files to Logstash or directly to Elasticsearch.

            Description: filebeat is a shipper part of the Elastic Beats

            # family. Please see: https://www.elastic.co/products/beats

            END INIT INFO

            Do NOT “set -e”

            PATH should only include /usr/* if it runs after the mountnfs.sh script

            PATH=/sbin:/usr/sbin:/bin:/usr/bin
            DESC=”Sends log files to Logstash or directly to Elasticsearch.”
            NAME=”filebeat”
            DAEMON=/usr/bin/$NAME
            DAEMON_ARGS=”-c /etc/filebeat/filebeat.yml”
            TEST_ARGS=”-e -configtest”
            PIDFILE=/var/run/filebeat.pid
            #WRAPPER=
            #WRAPPER_ARGS=”-r / -n -p $PIDFILE”
            SCRIPTNAME=/etc/init.d/$NAME

            Exit if the package is not installed

            #[ -x “$DAEMON” ] || exit 0

            Read configuration variable file if it is present

            #[ -r /etc/default/$NAME ] && . /etc/default/$NAME

            Load the VERBOSE setting and other rcS variables

            . /lib/init/vars.sh

            Define LSB log_* functions.

            Depend on lsb-base (>= 3.2-14) to ensure that this file is present

            and status_of_proc is working.

            . /lib/lsb/init-functions

            #

            Function that calls runs the service in foreground

            to test its configuration.

            #
            do_test()
            {
            $DAEMON $DAEMON_ARGS $TEST_ARGS
            }

            #

            Function that starts the daemon/service

            #
            do_start()
            {
            # Return
            # 0 if daemon has been started
            # 1 if daemon was already running
            # 2 if daemon could not be started
            start-stop-daemon –start \
            –pidfile $PIDFILE \
            –exec $DAEMON — $DAEMON_ARGS \
            || return 2 &
            }

            #

            Function that stops the daemon/service

            #
            do_stop()
            {
            # Return
            # 0 if daemon has been stopped
            # 1 if daemon was already stopped
            # 2 if daemon could not be stopped
            # other if a failure occurred
            start-stop-daemon –stop –quiet –retry=TERM/5/KILL/5 –pidfile $PIDFILE
            RETVAL=”$?”
            [ “$RETVAL” = 2 ] && return 2
            # Wait for children to finish too if this is a daemon that forks
            # and if the daemon is only ever run from this initscript.
            # If the above conditions are not satisfied then add some other code
            # that waits for the process to drop all resources that could be
            # needed by services started subsequently. A last resort is to
            # sleep for some time.
            start-stop-daemon –stop –quiet –oknodo –retry=0/30/KILL/5 –exec $DAEMON
            [ “$?” = 2 ] && return 2
            # Many daemons don’t delete their pidfiles when they exit.
            rm -f $PIDFILE
            return “$RETVAL”
            }

            #

            Function that sends a SIGHUP to the daemon/service

            #
            do_reload() {
            #
            # If the daemon can reload its configuration without
            # restarting (for example, when it is sent a SIGHUP),
            # then implement that here.
            #
            start-stop-daemon –stop –signal 1 –quiet –pidfile $PIDFILE –exec $DAEMON
            return 0
            }

            case “$1” in
            start)
            [ “$VERBOSE” != no ] && log_daemon_msg “Starting $DESC” “$NAME”
            do_test
            case “$?” in
            0) ;;
            *)
            log_end_msg 1
            exit 1
            ;;
            esac
            do_start
            case “$?” in
            0|1) [ “$VERBOSE” != no ] && log_end_msg 0 ;;
            2) [ “$VERBOSE” != no ] && log_end_msg 1 ;;
            esac
            ;;
            stop)
            [ “$VERBOSE” != no ] && log_daemon_msg “Stopping $DESC” “$NAME”
            do_stop
            case “$?” in
            0|1) [ “$VERBOSE” != no ] && log_end_msg 0 ;;
            2) [ “$VERBOSE” != no ] && log_end_msg 1 ;;
            esac
            ;;
            status)
            status_of_proc “$NAME” && exit 0 || exit $?
            ;;
            #reload|force-reload)
            #
            # If do_reload() is not implemented then leave this commented out
            # and leave ‘force-reload’ as an alias for ‘restart’.
            #
            #log_daemon_msg “Reloading $DESC” “$NAME”
            #do_reload
            #log_end_msg $?
            #;;
            restart|force-reload)
            #
            # If the “reload” option is implemented then remove the
            # ‘force-reload’ alias
            #
            log_daemon_msg “Restarting $DESC” “$NAME”
            do_test
            case “$?” in
            0) ;;
            *)
            log_end_msg 1 # Old process is still running
            exit 1
            ;;
            esac

            do_stop
            case "$?" in
            0|1)
            do_start
            case "$?" in
            0) log_end_msg 0 ;;
            1) log_end_msg 1 ;; # Old process is still running
            *) log_end_msg 1 ;; # Failed to start
            esac
            ;;
            *)
            # Failed to stop
            log_end_msg 1
            ;;
            esac
            ;;

            *)
            #echo “Usage: $SCRIPTNAME {start|stop|restart|reload|force-reload}” >&2
            echo “Usage: $SCRIPTNAME {start|stop|status|restart|force-reload}” >&2
            exit 3
            ;;
            esac

            :

            Not posting topbeat as it is pretty much the same and would be an even longer post.
            sudo update-rc.d topbeat defaults 95 10
            sudo update-rc.d filebeat defaults 95 10

            Runs like magic. Might try and cross-compile as it is explained here: https://github.com/elastic/beats-packer
            Frankly I started, but had issues – again – with GOPATH on Arch for some strange reason. . . Will dig in to it later and if interested can share my findings here.
            Till that a short vid from my topbeat dash from my proto unit: https://www.youtube.com/watch?v=z8thfva1HDk&feature=youtu.be

            Big thanks again Mike for your tutorial, was a HUUUUUGE help and highly appreciate it!

            Br,
            A.

Trackbacks/Pingbacks

  1. Build and Run Logstash Forwarder on a Rasperry Pi - February 5, 2016

    […] Logstash Forwarder has been replaced by Beats. See my guide for building and running beats on the Raspberry Pi here. […]

Leave a Reply

Proudly made in Canada