FOURPROC

Creating a Repository to Hold Example Code and Tools for LabKey Server

This weekend I created a new GitHub repository called samples. This repository was created to hold scripts, tools and/or sample code that LabKey wants to make available to our customers and other LabKey Server users.

Repository Information:

URL: https://github.com/LabKey/samples

What is in this repository as of Today?

Currently the repository contains two folders

ops:

  • Install and upgrade scripts to be used on Windows server (where LabKey is manually installed)
  • These scripts used to be maintained in a repository created by my personal account (https://github.com/bdconnolly/labkey-ops).

python:

  • sample python scripts for uploading files and downloading study archives to/from a LabKey Server

Over the next few months, I will be adding some additional python tools for import/export of Study Archives and for managing AWS instances.

.....

How LabKey uses Vagrant to make development easier

As you may or may not know, LabKey Server has the capability to offload certain types of processing to separate computers such as

  • Rserv: offload the execution of R reports/views to an Rserv running on a separate computer
  • Remote Pipeline: offload proteomics and sequence analysis computation to a separate computer
  • Galaxy: offload sequence analysis and genotyping to a separate Galaxy server.

Last month, one of LabKey’s developers, Kevin Krouse, wrote up instructions for using Vagrant to make development and testing of features using Rserv easier. I took his work and expanded it

  • to support Remote Pipeline servers
  • uses chef-solo to handle configuration during provisioning.
  • [update 9/11] Ben Bimber added support for his sequence analysis pipeline.

Instructions

Once these VMs have been installed, all you need to do is run vagrant up and the VM will be started on your dev machine.

.....

Create a DEBUG log file for your LabKey Server

There are times, when I am investigating a problem, it is helpful to turn on DEBUG logging for the LabKey Server. As you would expect, enabling DEBUG logging, produces LOTS of log entries, so it is not something that you want to keep enabled forever, but it can be helpful to enable while debugging.

There are a number of ways to enable DEBUG logging. The easiest will simply flood the labkey.loglog file with the DEBUG messages, which is not ideal. I like to write the DEBUG logs to a separate, new log file.

The LabKey Server uses log4j for logging. The configuration file is located at LABKEY_HOME/labkeywebapp/WEB-INF/classes/log4j.xml, where LABKEY_HOME is the installation directory for your LabKey Server.

With the changes documented below, the new logging behavior will be

  1. The log file, labkey-errors.log will continue to receive all LabKey-specific messages with priority of ERROR or greater
  2. The log file, labkey.log will continue to receive all LabKey-specific messages with priority of INFO or greater
  3. The new log file, labkey-debug.log will receive all LabKey-specific messages with priority of DEBUG or greater

Please be warned, the labkey-debug.log file can grow very fast. The configuration below sets the max log file size to 10MB. When the file size reaches 10MB, it will be automatically rollover to a new file. Four previous log files will be retained.

Enable DEBUG level logging

You can follow these instructions to enable DEBUG level logging for your LabKey Server

1) Stop your LabKey Server

2) Make a backup of the existing log4j configuration file

cd LABKEY_HOME/labkeywebapp/WEB-INF/classes/log4j.xml
cp log4j.xml log4j.xml.original

3) Edit log4j.xml file

4) Start your LabKey Server

Now if you go directory which holds the log files, CATALINA_HOME/logs you should see a file named labkey-debug.log.

Once you are done debugging your problem, you can disable the DEBUG logging by

1) Stop your LabKey Server

2) Restore the backup of the log4j configuration file

cd LABKEY_HOME/labkeywebapp/WEB-INF/classes/log4j.xml
cp log4j.xml.original log4j.xml

3) Start your LabKey Server

Using the DEBUG log file.

labkey-debug.log will contain lots and lots of DEBUG messages. The best way to use the new log file, is to use a tool like grep to only output the messages you are interested in. For example, if you are interested in tracking the progress of the deletion of a Project, you can use

tail -f labkey-debug.log | grep ContainerManager

or you see all messages about the deletion by running

less labkey-debug* | grep ContainerManager

.....

Using LabKey's sample backup script to backup your PostgreSQL database

Below you will find instructions for using LabKey’s sample database backup script to backup your PostgreSQL databases on Windows. This is a very simple script which backups up the databases to the specified directory. Please feel free to use it as a starting point for backing up the database for your LabKey Server.

What will this script do

This script can be used to perform a nightly backup (using pg_dump) of databases on a PostgreSQL server instance. The script will

  1. Perform a full backup of the postgres and labkey databases using pg_dump
  2. Dump archives will be placed in the %BACKUP_DIR% directory
  3. Dump archive file names will be in the format postgresql-backup-DB_YYYYMMDD.bak where
    • DB is the database name
    • YYYY is the year
    • MM is the month
    • DD is the day of the month
  4. Write status and error messages to STDOUT
  5. Write status and error messages to the logfile (%LOGFILE%)

Install the backup script

Install the sample backup script in a directory, such as c:\labkey\bin, on the server running your PostgreSQL database instance. In many cases, this will be the same server running your LabKey Server.

Customize the backup script for your server

You will need to customize the script for your server and environment.

  1. Change the name of the script to something more meaningful to you, such as labkey-database-backup.bat
  2. Open the script and edit the variables as needed. The important variables are
    • BACKUP_DIR: This is the directory where the backups will be stored
    • POSTGRES_HOME: The directory where your PostgreSQL database is installed
    • POSTGRES_USER: The user account to be used for the backup.
    • PGPASSFILE: Location of the pgpass file which will be used for authenticating to the PostgreSQL server

Create the pgpass file

To authenticate to the PostgreSQL database server, the script will use a pgpass file. This allows us to not have the database password in clear-text in the backup script.

  1. Create a pgpass file and place it in a secure location.
    • For example, you could place it in your HOME directory at C:\Users\ACCOUNTNAME\postgresql\pgpass.conf
    • where ACCOUNTNAME is your username.
  2. Add the login credentials to the pgpass file

    • The format of the pgpass is

        hostname:port:database:username:password
      
    • For example, if you plan on performing the backup using the postgres database user, the file may contain

        localhost:5432:postgres:postgres:THEPASSWORD
        localhost:5432:labkey:postgres:THEPASSWORD
      
  3. Secure the pgpass file

    • Using NTFS permissions, disable access to this file for everybody except for the ACCOUNTNAME user
  4. Edit the backup script and change the PGPASSFILE variable to be path to the file created in step 1.

Running the backup script.

To execute the backup script, you will need to open a command prompt and run

C:\labkey\bin\labkey-database-backup.bat

NOTE: This example assumes you have installed the backup script at C:\labkey\bin\labkey-database-backup.bat

The script will output status and error messages to the screen. In addition, it will write these messages, plus additional debug messages to a log file located at %BACKUP_DIR%\labkey-database-backup.log

Create a scheduled task [OPTIONAL]:

If you would like to perform a backup on a nightly basis, you can create scheduled task in windows.

The command to use is

C:\Windows\System32\cmd.exe /c "C:\labkey\bin\labkey-database-backup.bat"

The start directory to use will be the directory where the backup script is located (for example C:\labkey\bin)

NOTE: This example assumes you have installed the backup script at C:\labkey\bin\labkey-database-backup.bat

.....

Point release upgrades in PostgreSQL

With the recent PostgreSQL security update, I was forced to perform emergency maintenance on all my PostgreSQL servers to install the update. To remind myself of the process, I did a quick search and it turns out I could not find great instructions on installing “Point Releases” of PostgreSQL.

Here are my step by step instructions, for the next time I need to do this.

At LabKey, we run lots of different versions of PostgreSQL(for testing, etc) and I always install PostgreSQL from source. The instructions below will be for upgrading v9.0.

One of the great things about upgrading to a new point release, is that you are not required perform the usual methodology of

  1. Backup database
  2. Stop database server
  3. Install new version of database
  4. Restore database backup from Step #1

You can simply install the new version and start the server up and it should work. The methodology I used is

1. Backup the databases

[Optional] For safe keeping, backup the database using pg_dump

/usr/local/postgresql-9.0/bin/pg_dumpall -U postgres -W -f /backup/postgresql_backup_for_upgrade_to_9.0.13.out

2. Build the new version

sudo su - 
cd /usr/local/src
wget http://ftp.postgresql.org/pub/source/v9.0.13/postgresql-9.0.13.tar.gz
tar xzf postgresql-9.0.13.tar.gz
cd postgresql-9.0.13/
./configure --prefix=/usr/local/postgresql-9.0/
make

3. Shutdown PostgreSQL server

/etc/init.d/postgresql-9.0 stop

Backup the installation

For safe keeping, make a backup of the installed binaries, etc

If you have small databases or your PGDATA directory is not located in the PostgreSQL install directory , you can run

cd /usr/local
cp -Rp postgresql-9.0 postgresql-9.0_backup

If your databases are large, then you can run

cd /usr/local
tar --exclude="./postgresql-9.0/data" -czf postgresql_backup_for_upgrade_to_9.0.13.tar.gz postgresql-9.0 

Upgrade PostgreSQL to 9.0.13 and start the server

cd /usr/local/src/postgresql-9.0.13/
make install 
/etc/init.d/postgresql-9.0 start

That is it. All done.

.....