Beats are the replacement for Logstash Forward from Elastic. They are data shippers for Elastic Search. Unfortunately, compiling them on the Raspberry Pi isn’t very straightforward.
In this post I’ll show you how to build and run Beats on Raspberry Pi.
Install Go
Firstly, you’ll need to install go. Beats requires Go version 1.5.3 or higher (at the time of writing.) Unfortunately this process is a little roundabout as you must first install go 1.4.x in order to install 1.5.x. Convenient! </sarcasm>
If you’ve already followed my tutorial for building Logstash Forward on a Raspberry Pi you can skip the step for installing go 1.4. Otherwise grab the latest sources from the Go Downloads. I’ll be using 1.4.2 below, but you should update the wget link with the link of the version you plan on using. The below script will remove any existing go install, so be careful if you want another go version for other things.
which go && sudo rm -rf /usr/bin/go* /usr/local/go cd ~ wget https://storage.googleapis.com/golang/go1.4.2.src.tar.gz sudo tar -C /usr/local -xvf go1.4.*.tar.gz cd /usr/local/go/src sudo ./make.bash
That will take a while to build. Once it’s complete download and install Go 1.5.x (grabbing the latest version, as before):
cd ~ wget https://storage.googleapis.com/golang/go1.5.3.src.tar.gz sudo tar -xvf go1.5.*.tar.gz sudo mv go /usr/local/go1.5 cd /usr/local/go1.5/src sudo GOROOT_BOOTSTRAP=/usr/local/go ./make.bash sudo ln -s /usr/local/go1.5/bin/* /usr/bin
Compile Beats
Once you have go 1.5.x installed, you’re ready to get on the way to installing beats. First however, you’ll need to install a bunch of dependencies:
mkdir -p /usr/local/go1.5/src/github.com/elastic sudo chown -R pi /usr/local/go1.5/src/github.com #change to your username if not "pi" cd /usr/local/go1.5/src/github.com/elastic git clone https://github.com/elastic/beats export GOROOT=/usr/local/go1.5/
Next install of the go dependencies for the project:
export GOPATH=/usr/local/go1.5/src/github.com/elastic/beats # Install easy go deps go get golang.org/x/text/encoding/charmap go get golang.org/x/text/encoding/simplifiedchinese go get golang.org/x/text/encoding/unicode go get golang.org/x/text/transform go get gopkg.in/yaml.v2 go get github.com/garyburd/redigo/redis go get github.com/nranchev/go-libGeoIP go get github.com/satori/go.uuid
Now that we have all of the dependencies its time to actually go about compiling Beats. To do so, compile the individual beats you need. The process looks like this:
cd /usr/local/go1.5/src/github.com/elastic/beats # Pick a beat to compile. Choose from "filebeat", "topbeat", or "packetbeat" cd filebeat make sudo cp filebeat /usr/bin/ # can also use "sudo ln -s filebeat /usr/bin" if you plan on keeping the sources around
There you go! Compile each beat that you want to use and enjoy!
Hi Michael,
Thanks for the great tutorial, unfortunately for some reason I keep running into issues when I try to run the ‘go get’ commands. I usually get an error message like this:
package errors: unrecognized import path “errors”
package bytes: unrecognized import path “bytes”
package io: unrecognized import path “io”
package unicode/utf8: unrecognized import path “unicode/utf8”
package strconv: unrecognized import path “strconv”
I tried to Google the issue and see numerous issues like this and usually the issue is setting GOPATH and GOROOT although it is still not solving my problem :/
Maybe you have any suggestions what I can try?
Big thanks in advance!
Hey Attila,
Sorry to hear you’re having trouble. This sounds alot like a path problem.
Can you ensure that you set your
GOPATHusing theexportcommand? (as in the code sample).export GOROOT=/usr/local/go1.5/src/
export GOPATH=/usr/local/go1.5/src/github.com/elastic/beats
Also can you please ensure that the path
/usr/local/go1.5/src/github.com/elastic/beatsexists, and is filled with the contents of the git repo you checked out previously? (https://github.com/elastic/beats)Are you doing any steps differently or in different paths?
-Mike
Hi Michael,
I can confirm that in my case I was following the instructions and got the above posted errors, but no worries I think in the meanwhile I solved the problem by doing it a bit differently. The only thing I changed is the GOROOT path: export GOROOT=/usr/local/go1.5/ and now it ran perfectly and just finished compiling filebeat. Not really sure what went wrong as this is the first time I was even installing golang. Maybe I do made a mistake somewhere :/
Anyway great tutorial and helped me a lot to get started, appreciate your work!
—
Attila
Thanks Attila!
Looking back I think I may have enterred that path wrong. I’ll update the path set in the tutorial.
-Mike
Not to worry mate, still works like a charm. The only thing what is missing or at least for me it was missing after compiling topbeat and filebeat is the god files what is being used as wrappers in the original init.d script from Elastic running parallel with the main programs. So what I did is removing the wrappers from the init.d for both, but still got an error running it for some reason which is strange because starting it from the shell runs fine (/usr/bin/filebeat -c /etc/filebeat/filebeat.yml same for topbeat, built them a proper place in /etc) so added a small modification to the scripts removing the same things so in case of filebeat it looks like this:
#!/bin/sh
BEGIN INIT INFO
Provides: filebeat
Required-Start: $local_fs $network $syslog
Required-Stop: $local_fs $network $syslog
Default-Start: 2 3 4 5
Default-Stop: 0 1 6
Short-Description: Sends log files to Logstash or directly to Elasticsearch.
Description: filebeat is a shipper part of the Elastic Beats
# family. Please see: https://www.elastic.co/products/beats
END INIT INFO
Do NOT “set -e”
PATH should only include /usr/* if it runs after the mountnfs.sh script
PATH=/sbin:/usr/sbin:/bin:/usr/bin
DESC=”Sends log files to Logstash or directly to Elasticsearch.”
NAME=”filebeat”
DAEMON=/usr/bin/$NAME
DAEMON_ARGS=”-c /etc/filebeat/filebeat.yml”
TEST_ARGS=”-e -configtest”
PIDFILE=/var/run/filebeat.pid
#WRAPPER=
#WRAPPER_ARGS=”-r / -n -p $PIDFILE”
SCRIPTNAME=/etc/init.d/$NAME
Exit if the package is not installed
#[ -x “$DAEMON” ] || exit 0
Read configuration variable file if it is present
#[ -r /etc/default/$NAME ] && . /etc/default/$NAME
Load the VERBOSE setting and other rcS variables
. /lib/init/vars.sh
Define LSB log_* functions.
Depend on lsb-base (>= 3.2-14) to ensure that this file is present
and status_of_proc is working.
. /lib/lsb/init-functions
#
Function that calls runs the service in foreground
to test its configuration.
#
do_test()
{
$DAEMON $DAEMON_ARGS $TEST_ARGS
}
#
Function that starts the daemon/service
#
do_start()
{
# Return
# 0 if daemon has been started
# 1 if daemon was already running
# 2 if daemon could not be started
start-stop-daemon –start \
–pidfile $PIDFILE \
–exec $DAEMON — $DAEMON_ARGS \
|| return 2 &
}
#
Function that stops the daemon/service
#
do_stop()
{
# Return
# 0 if daemon has been stopped
# 1 if daemon was already stopped
# 2 if daemon could not be stopped
# other if a failure occurred
start-stop-daemon –stop –quiet –retry=TERM/5/KILL/5 –pidfile $PIDFILE
RETVAL=”$?”
[ “$RETVAL” = 2 ] && return 2
# Wait for children to finish too if this is a daemon that forks
# and if the daemon is only ever run from this initscript.
# If the above conditions are not satisfied then add some other code
# that waits for the process to drop all resources that could be
# needed by services started subsequently. A last resort is to
# sleep for some time.
start-stop-daemon –stop –quiet –oknodo –retry=0/30/KILL/5 –exec $DAEMON
[ “$?” = 2 ] && return 2
# Many daemons don’t delete their pidfiles when they exit.
rm -f $PIDFILE
return “$RETVAL”
}
#
Function that sends a SIGHUP to the daemon/service
#
do_reload() {
#
# If the daemon can reload its configuration without
# restarting (for example, when it is sent a SIGHUP),
# then implement that here.
#
start-stop-daemon –stop –signal 1 –quiet –pidfile $PIDFILE –exec $DAEMON
return 0
}
case “$1” in
start)
[ “$VERBOSE” != no ] && log_daemon_msg “Starting $DESC” “$NAME”
do_test
case “$?” in
0) ;;
*)
log_end_msg 1
exit 1
;;
esac
do_start
case “$?” in
0|1) [ “$VERBOSE” != no ] && log_end_msg 0 ;;
2) [ “$VERBOSE” != no ] && log_end_msg 1 ;;
esac
;;
stop)
[ “$VERBOSE” != no ] && log_daemon_msg “Stopping $DESC” “$NAME”
do_stop
case “$?” in
0|1) [ “$VERBOSE” != no ] && log_end_msg 0 ;;
2) [ “$VERBOSE” != no ] && log_end_msg 1 ;;
esac
;;
status)
status_of_proc “$NAME” && exit 0 || exit $?
;;
#reload|force-reload)
#
# If do_reload() is not implemented then leave this commented out
# and leave ‘force-reload’ as an alias for ‘restart’.
#
#log_daemon_msg “Reloading $DESC” “$NAME”
#do_reload
#log_end_msg $?
#;;
restart|force-reload)
#
# If the “reload” option is implemented then remove the
# ‘force-reload’ alias
#
log_daemon_msg “Restarting $DESC” “$NAME”
do_test
case “$?” in
0) ;;
*)
log_end_msg 1 # Old process is still running
exit 1
;;
esac
do_stop
case "$?" in
0|1)
do_start
case "$?" in
0) log_end_msg 0 ;;
1) log_end_msg 1 ;; # Old process is still running
*) log_end_msg 1 ;; # Failed to start
esac
;;
*)
# Failed to stop
log_end_msg 1
;;
esac
;;
*)
#echo “Usage: $SCRIPTNAME {start|stop|restart|reload|force-reload}” >&2
echo “Usage: $SCRIPTNAME {start|stop|status|restart|force-reload}” >&2
exit 3
;;
esac
:
Not posting topbeat as it is pretty much the same and would be an even longer post.
sudo update-rc.d topbeat defaults 95 10
sudo update-rc.d filebeat defaults 95 10
Runs like magic. Might try and cross-compile as it is explained here: https://github.com/elastic/beats-packer
Frankly I started, but had issues – again – with GOPATH on Arch for some strange reason. . . Will dig in to it later and if interested can share my findings here.
Till that a short vid from my topbeat dash from my proto unit: https://www.youtube.com/watch?v=z8thfva1HDk&feature=youtu.be
Big thanks again Mike for your tutorial, was a HUUUUUGE help and highly appreciate it!
Br,
A.
make error
I am using
export GOROOT=/usr/local/go1.5/
export GOPATH=/usr/local/go1.5/src/github.com/elastic/beats
Command “/usr/local/go1.5/src/github.com/elastic/beats/libbeat/build/python-env/bin/python2 -u -c “import setuptools, tokenize;file=’/tmp/pip-build-KZfbdP/PyYAML/setup.py’;f=getattr(tokenize, ‘open’, open)(file);code=f.read().replace(‘\r\n’, ‘\n’);f.close();exec(compile(code, file, ‘exec’))” install –record /tmp/pip-vTMV95-record/install-record.txt –single-version-externally-managed –compile –install-headers /usr/local/go1.5/src/github.com/elastic/beats/libbeat/build/python-env/include/site/python2.7/PyYAML” failed with error code 1 in /tmp/pip-build-KZfbdP/PyYAML/
scripts/Makefile:199: recipe for target ‘python-env’ failed
make[1]: *** [python-env] Error 1
make[1]: Leaving directory ‘/usr/local/go1.5/src/github.com/elastic/beats/libbeat’
Makefile:21: recipe for target ‘testsuite’ failed
make: *** [testsuite] Error 1