graylog2 logo

Upgrading Graylog2 from 0.9.5p2 to 0.9.6 with data migration

graylog2 logoThe Graylog2 “party-gorilla” team around Lennart Koopmann just released the final version of graylog2 0.9.6. It includes major changes and new features, such as:

  • ElasticSearch as new message store
  • new analytics shell, a terminal like browser gui for running queries
  • faster long term graphs
  • internal message queing systems to absorb and resist load spikes
  • new stream rules and filters
  • a lot of small improvements and bugfixes

The following migration guide describes the manual upgrade process on debian in detail:

Download new 0.9.6 release, for example into /opt and unpack it
tar xzfv graylog2-server-0.9.6.tar.gz

Download elasticsearch, most recent version from here and unpack it, for example:
tar xzfv elasticsearch-0.18.6.tar.gz

configure basic elasticsearch values, see elasticsearch configuration for details, for example:
logs: /var/log/elasticsearch
data: /var/data/elasticsearch
name: graylog2

Download elasticsearch-servicewrapper (tanuki-wrapper) into your elasticserach/bin installation directory and unpack it there
mv master && unzip
mv elasticsearch-elasticsearch-servicewrapper-*/* . && rm -rf elasticsearch-elasticsearch-servicewrapper-*

Start elasticsearch instance bia servicewrapper in /bin/service
./elasticsearch start
Starting ElasticSearch...
Waiting for ElasticSearch.......
running: PID:14816

Check if your elasticsearch instance started successfully. The logfile (defaut here in /var/log/elasticsearch/graylog2.log) should show something like this:
[2011-12-23 22:21:03,711][INFO ][node ] [Celestial Madonna] {0.18.6}[14818]: initializing ...
[2011-12-23 22:21:03,716][INFO ][plugins ] [Celestial Madonna] loaded [], sites []
[2011-12-23 22:21:05,299][INFO ][node ] [Celestial Madonna] {0.18.6}[14818]: initialized
[2011-12-23 22:21:05,299][INFO ][node ] [Celestial Madonna] {0.18.6}[14818]: starting ...
[2011-12-23 22:21:05,352][INFO ][transport ] [Celestial Madonna] bound_address {inet[/]}, publish_address {inet[/]}
[2011-12-23 22:21:08,385][INFO ][cluster.service ] [Celestial Madonna] new_master [Celestial Madonna][WiNh0iYwQyeipER3PuXZSg][inet[/]], reason: zen-disco-join (elected_as_master)
[2011-12-23 22:21:08,408][INFO ][discovery ] [Celestial Madonna] graylog2/WiNh0iYwQyeipER3PuXZSg
[2011-12-23 22:21:08,415][INFO ][http ] [Celestial Madonna] bound_address {inet[/]}, publish_address {inet[/]}
[2011-12-23 22:21:08,416][INFO ][node ] [Celestial Madonna] {0.18.6}[14818]: started
[2011-12-23 22:21:08,419][INFO ][gateway ] [Celestial Madonna] recovered [0] indices into cluster_state

Get the MongoDB to ElasticSearch migrator script from joschi
git clone

you need at least to adapt to config/migrator.yml with your mongodb username, password and host, usage:
After you've downloaded the script you should edit the configuration file in config/migrator.yml. If the ElasticSearch and MongoDB servers are running on the same system (on localhost) and MongoDB doesn't need authentication you can keep the file as is.

Get required dependencies with Bundler:
bundle install

prepare a new graylog2.conf in /tmp/graylog2.conf with new settings (or just copy the new config and adapt your settings). The new default configuration settings are:
elasticsearch_url = http://localhost:9200/
elasticsearch_index_name = graylog2
force_syslog_rdns = false
mq_batch_size = 4000
mq_poll_freq = 1
mq_max_size = 0

be brave and do a “hole-in-one” migration from the migrator directory (the following line works depending from where your graylog2 servers resides):
ps aux | grep graylog2-server.jar | awk {'print $2'} | xargs kill; ruby migrator.rb;cp /tmp/graylog2.conf /etc/graylog2.conf/; cd ../graylog2-server-0.9.6/bin/;./graylog2ctl start

Get the new web interface from github
git clone

get required dependencies for the graylog2 web-interface
cd graylog2-web-interface; bundle install

Configure new web-interface (copy old configs, adapt new parameters, create new indexer.yml file with correct elasticsearch settings), be sure to check

Start the bundled server or adapt your apache/passenger settings

Login to your web-interface, check if latest events are dropping in.

Write a “thank you party-gorilla” tweet to the @graylog2 twitter account

Related Posts: