There is an easy way out there to improve performance on webmail clients like SquirrelMail and Rouncube.
Roundcube and similar clients opens a new IMAP connection on each request, e.g. opening the next e-Mail.
A caching proxy like imapproxy is reported to increase performance from 0.5 seconds up to 3 seconds per Action.
Installation on debian is very simple.
Just install the imapproxy package via aptitude install imapproxy.
In our setup the webmail client and imap server is running on different servers.
in /etc/imapproxy.conf we just change the configuration of our IMAP Server server_hostname mail.myhostname.com
the following important options could stay default but are worth mentioning:
listen_port 1143 - Port imapproxy is listening on
server_port 143 - Port of the imap server host)
After starting imapproxy we only have to change roundcube configuration in config/main.inc.php:
new analytics shell, a terminal like browser gui for running queries
faster long term graphs
internal message queing systems to absorb and resist load spikes
new stream rules and filters
a lot of small improvements and bugfixes
The following migration guide describes the manual upgrade process on debian in detail:
Download new 0.9.6 release, for example into /opt and unpack it https://github.com/downloads/Graylog2/graylog2-server/graylog2-server-0.9.6.tar.gz
tar xzfv graylog2-server-0.9.6.tar.gz
Download elasticsearch, most recent version from here and unpack it, for example: wget https://github.com/downloads/elasticsearch/elasticsearch/elasticsearch-0.18.6.tar.gz
tar xzfv elasticsearch-0.18.6.tar.gz
configure basic elasticsearch values, see elasticsearch configuration for details, for example: network.host: 192.168.164.100
path:
logs: /var/log/elasticsearch
data: /var/data/elasticsearch
cluster:
name: graylog2
Download elasticsearch-servicewrapper (tanuki-wrapper) into your elasticserach/bin installation directory and unpack it there wget https://github.com/elasticsearch/elasticsearch-servicewrapper/zipball/master
mv master elasticsearch-servicewrapper.zip && unzip elasticsearch-servicewrapper.zip
mv elasticsearch-elasticsearch-servicewrapper-*/* . && rm -rf elasticsearch-elasticsearch-servicewrapper-*
Start elasticsearch instance bia servicewrapper in /bin/service ./elasticsearch start
Starting ElasticSearch...
Waiting for ElasticSearch.......
running: PID:14816
Check if your elasticsearch instance started successfully. The logfile (defaut here in /var/log/elasticsearch/graylog2.log) should show something like this: [2011-12-23 22:21:03,711][INFO ][node ] [Celestial Madonna] {0.18.6}[14818]: initializing ...
[2011-12-23 22:21:03,716][INFO ][plugins ] [Celestial Madonna] loaded [], sites []
[2011-12-23 22:21:05,299][INFO ][node ] [Celestial Madonna] {0.18.6}[14818]: initialized
[2011-12-23 22:21:05,299][INFO ][node ] [Celestial Madonna] {0.18.6}[14818]: starting ...
[2011-12-23 22:21:05,352][INFO ][transport ] [Celestial Madonna] bound_address {inet[/192.168.164.100:9300]}, publish_address {inet[/192.168.164.100:9300]}
[2011-12-23 22:21:08,385][INFO ][cluster.service ] [Celestial Madonna] new_master [Celestial Madonna][WiNh0iYwQyeipER3PuXZSg][inet[/192.168.164.100:9300]], reason: zen-disco-join (elected_as_master)
[2011-12-23 22:21:08,408][INFO ][discovery ] [Celestial Madonna] graylog2/WiNh0iYwQyeipER3PuXZSg
[2011-12-23 22:21:08,415][INFO ][http ] [Celestial Madonna] bound_address {inet[/192.168.164.100:9200]}, publish_address {inet[/192.168.164.100:9200]}
[2011-12-23 22:21:08,416][INFO ][node ] [Celestial Madonna] {0.18.6}[14818]: started
[2011-12-23 22:21:08,419][INFO ][gateway ] [Celestial Madonna] recovered [0] indices into cluster_state
Get the MongoDB to ElasticSearch migrator script from joschi git clone https://github.com/joschi/graylog2-mongo-es-migrator.git
you need at least to adapt to config/migrator.yml with your mongodb username, password and host, usage: After you've downloaded the script you should edit the configuration file in config/migrator.yml. If the ElasticSearch and MongoDB servers are running on the same system (on localhost) and MongoDB doesn't need authentication you can keep the file as is.
Get required dependencies with Bundler: bundle install
prepare a new graylog2.conf in /tmp/graylog2.conf with new settings (or just copy the new config and adapt your settings). The new default configuration settings are: elasticsearch_url = http://localhost:9200/
elasticsearch_index_name = graylog2
force_syslog_rdns = false
mq_batch_size = 4000
mq_poll_freq = 1
mq_max_size = 0
be brave and do a “hole-in-one” migration from the migrator directory (the following line works depending from where your graylog2 servers resides): ps aux | grep graylog2-server.jar | awk {'print $2'} | xargs kill; ruby migrator.rb;cp /tmp/graylog2.conf /etc/graylog2.conf/; cd ../graylog2-server-0.9.6/bin/;./graylog2ctl start
Get the new web interface from github git clone https://github.com/Graylog2/graylog2-web-interface.git
get required dependencies for the graylog2 web-interface cd graylog2-web-interface; bundle install
Configure new web-interface (copy old configs, adapt new parameters, create new indexer.yml file with correct elasticsearch settings), be sure to check indexer.yml
mongoid.yml
general.yml
email.yml
Start the bundled server or adapt your apache/passenger settings
Login to your web-interface, check if latest events are dropping in.
Due to the lack of an official Apache Solr Package in Debian Squeeze 6.0 this post will briefly describe the installation of tomcat7 together with Apache (Lucene) Solr from Source.
Today the icinga team released the first stable version of their open source monitoring tool – Icinga 1.0.
The team behind icinga formed up earlier this year and forked nagios due to several reasons.
In this small howto i’d like to describe the installation of Icinga in cooperation with IDOUtils and NConf for easier configuration of multiple hosts and services on a fresh debian system:
# mysql -u root -p
mysql> CREATE DATABASE icinga;
GRANT USAGE ON \*.\* TO ‘icinga’@'localhost’ IDENTIFIED BY ‘MYSUPERHEROPASSWORD’ WITH MAX_QUERIES_PER_HOUR 0 MAX_CONNECTIONS_PER_HOUR 0 MAX_UPDATES_PER_HOUR 0;
GRANT ALL PRIVILEGES ON icinga.\* TO ‘icinga’@'localhost’;FLUSH PRIVILEGES ;
quit
Import corresponding database scheme for MySQL:
# cd /path/to/icinga-src/module/idoutils/db
# mysql -u root -p icinga < mysql.sql
Customize database credentials in /usr/local/icinga/etc/ido2db.cfg and take care of the following options:
Unpack the archive in one of your web directories, e.g. in /var/www/nconf
tar xzfv nconf-1.2.6-0.tgz
Make the new directory read/writeable to your webserver:
chown -R www-data:www-data nconf/
Create new file nconf.conf in /etc/apache2/conf.d:
#nconf.conf
Alias /nconf "/var/www/virtual/nconf"
# SSLRequireSSL
Options None
AllowOverride None
Order allow,deny
Allow from all
# Order deny,allow
# Deny from all
# Allow from 127.0.0.1
AuthName "Icinga Access"
AuthType Basic
AuthUserFile /usr/local/icinga/etc/htpasswd.users
Require valid-user
Reload apache
/etc/init.d/apache2 reload
Create database and user for NConf:
# mysql -u root -p
mysql> CREATE DATABASE nconf;
GRANT USAGE ON \*.\* TO ‘nconf’@'localhost’ IDENTIFIED BY ‘MYSUPERHEROPASSWORD’ WITH MAX_QUERIES_PER_HOUR 0 MAX_CONNECTIONS_PER_HOUR 0 MAX_UPDATES_PER_HOUR 0;
GRANT ALL PRIVILEGES ON nconf.\* TO ‘nconf’@'localhost’;
FLUSH PRIVILEGES ;
quit
Complete installation via NConf Web Gui, for example on
Put /usr/local/icinga/bin/icinga as NAGIOS_BIN
Configure your hosts and services in the NConf WebGui
make sure to delete the host and service notification state "[n]" editing your contacts, otherwise you won’t get any notification.
Edit the notify-host-by-email/notify-sercice-by-email command in misccommands section in order to point to the corresponding binary for the mail command, in my case /usr/bin/mail
After finishing host and services configuration, hit the Generate Nagios config Button.
Your nagios configuration is now saved in the /output directory of your NConf installation.
customize NConf deploy script in /var/www/virtualnconf/ADD-ONS/deploy_local.sh, for example:
if [ ! -e ${TEMP_DIR} ] ; then
mkdir -p ${TEMP_DIR}
fi
if [ ${OUTPUT_DIR}${CONF_ARCHIVE} -nt ${TEMP_DIR}${CONF_ARCHIVE} ] ; then
cp -p ${OUTPUT_DIR}${CONF_ARCHIVE} ${TEMP_DIR}${CONF_ARCHIVE}
tar -xf ${TEMP_DIR}${CONF_ARCHIVE} -C ${NAGIOS_DIR}
/etc/init.d/icinga reload
fi
exit
Configure cronjob to refresh your configuration regulary, if you want to
Dabei kann man beispielsweise seine Website mit einem einfachen (oder custom) HTTP Check überwachen, Netzwerk Checks über UDP, TCP oder einfachen Ping einrichten oder auch Dienste wie POP3, IMAP und SMTP prüfen lassen. In der kostenlosen Version kann man jedoch nur einen Host/IP überwachen. Allerdings kann man Meldungen dann an mehrere Kontakte verschicken, die wahlweise per EMail und/oder SMS über den Status benachrichtigt werden (20 SMS pro Monat sind inklusive).
Der Clou an der Sache ist, dass Pingdom Monitoring Hubs an verschiedenen Standorten auf der ganzen Welt betreibt, und so werden die Dienste beispielsweise aus Montreal (Kanada), London (UK), Dallas (Texas), Stockholm und Amsterdam überwacht!
Die Statistiken und Reports sind schön aufbereitet und es gibt sogar eine IPhone App!
Auf dem Google Data Center Efficiency Summit in Mountain View hat Google erstmals offiziell bestätigt, dass ihr RZ Design auf RZ Containern basiert. Ein Google Mitarbeiter (Jimmy Clidaras) erklärte mit Hilfe einer Präsentation die Besonderheiten und Möglichkeiten dieser seit 2005 im Einsatz befindlichen Architektur. Jeder der bis zu 12 Meter langen Container enthält dabei bis zu 1160 Server und verbraucht bis zu 250KW Strom. Der gezeigte Container Hangar enthält 45 Container, zum Teil sogar auf einem 2. Stock!
Über die Präsentation gibt es ein über 5 minütiges Video auf youtube, weitere Videos der Konferenz sollen noch folgen:
Auch schon offiziell ist die Bauweise der Google Webserver, welche mich doch stark an die RootServer bei 1&1 erinnern:
Vor kurzem schrieb ich über einen Festplatten Ausfall in einem Software Raid.
Dieser Ausfall wurde einem Nagios Überwachungssystem gemeldet, mit welchem wir unsere Server überwachen. Als RAID Systeme kommen “normale” Linux Software Raids mit md devices und ein Server mit einem 3ware Hardware Raid (3ware Inc 8000-series) zum Einsatz.
Für das 3Ware Raid verwenden wir das check_3ware_raid von nagiosexchange.org. Dafür muss wie im Artikel beschrieben noch sudoers um “nagios ALL=(ALL) NOPASSWD: /path/to/check_3ware_raid” erweitert werden. Der Check selbst wird dann beispielsweise im nrpe Client über ein Wrapper Skript ausgeführt, welches ein “sudo /path/to/check_3ware_raid” ausführt.
Zur Ausführung benötigt man noch das 3ware Command Line Interface (CLI). Für Debian bekommt man das einfach über debian-unoffical mirror, der noch zu den apt-Sourcen hinzugefügt werden muss:
# For getting 3dm2-package for 3ware
deb http://ftp.debian-unofficial.org/debian sid main contrib non-free restricted
befinden sich die benötigten Binaries auf eurem Server und die Funktion lässt sich mit dem Aufruf von
tw_cli /c0 show
Unit UnitType Status %Cmpl Stripe Size(GB) Cache AVerify IgnECC
——————————————————————————
u0 RAID-1 OK – – 698.637 ON – -
Port Status Unit Size Blocks Serial
—————————————————————
p0 OK u0 698.63 GB 1465149168 5QD2DXQ8
p1 OK u0 698.63 GB 1465149168 5QD0XX6A
testen.
Software Raids werden mit dem einfachen check_md_raid überwacht. Dazu wird der Check einfach als command oder im nrpe Client hinzugefügt und ausgeführt. Allerdings schreit er schon mit Critical, wenn nur ein Device Critical ist, und dies kann man auch nicht konfigurieren oder auf bestimmte md devices eingrenzen.