Migrate a Drupal site with CiviCRM from a custom LAMP environment to Virtualmin (1/2)

Okay LAMP is not so hype anymore, and you would prefer a LEMP stack or another approach like JAMStack, but if you have loads of legacy websites to maintain and have no resources for working on the possible side effects of a stack change (Apache modules, custom configuration of php.ini, ...), in some situations, it should be better to stick to LAMP.

This post goes a bit against the mainstream, with one purpose in mind: for a low budget, leave a custom LAMP server provisioning for a well maintained LAMP stack that comes with a smart GUI and helpers for common sysadmin tasks like:

• Several versions of PHP (5.x, 7.x) that can be defined per virtual host, so we can facilitate the transition of legacy websites to 7.x when 5.x will reach EOL
• A php.ini for each virtual host and each PHP version
• Backup system for files, configuration and databases
• Internal monitoring system for services
• Let's Encrypt ready

Install Virtualmin

A good way to run a first test is a cheap VPS where you can install a Linux system compatible with Virtualmin. You can get it from companies like Digital Ocean.

Especially for very low budget hosting with reasonable performances, you should also prefer a solution like a VPS that you can upgrade (ram/cpu wise) instead of container based solutions like Docker or Kubernetes.

If you can afford a bit more, have a look at container based services like Wodby.

We will use for this example:

• Ubuntu 16.04 LTS
• A Drupal 7.x site installed with CiviCRM 4.6.x LTS

I will try to stick to the GUI whenever possible, so it will give you an idea when a switch to the command line is needed.

Note that, once you have Virtualmin installed, you will have access to a terminal straight from the footer of the GUI.

Once you have your system ready, ssh into it and, in the /root directory, get Virtualmin.

wget http://software.virtualmin.com/gpl/scripts/install.sh

Basically, make it executable and launch the installer.

chmod +x install.sh
./install.sh

Here is a more detailed installation guide if needed.

Create the destination environment

Add a Virtual host

On your new Virtualmin enabled server, create a Virtualhost for your site.

In your browser, head to https://YOUR.VIRTUALMIN\_SERVER.IP.V4:10000

Then select the Virtualmin tab on the upper left and click on 'Create Virtual Server'.

Here is a screenshot of a minimal setup, with no mail hosting: just a website, ssl ready (enabling ports :80 and :443), with a database.

Add domain aliases for testing before the actual migration

We will add an alias for testing before changing the A records of our domain to the new server.

Select 'Create Virtual Server' then 'Alias of mysite.org', then define the temporary domain to be used. Leave the default options.

In your registrar admin interface create a new A record with the same name, pointing to your new server IP address.

Restart Apache

Go to the Webmin tab, then

Add a second database

By default, enabling the 'Create MySQL database' will create a database with the same name as the virtual host. We will use it for our Drupal site.

Here, we need another database for CiviCRM. Click on 'Edit databases' > 'Create a new database'.

If you want to finetune this setup on the mysql prompt, e.g. for one user per database:

CREATE DATABASE IF NOT EXISTS my_database;
CREATE USER 'my_database'@'localhost' IDENTIFIED BY 'my_password';
GRANT USAGE ON my_database.* TO 'my_user'@'localhost' IDENTIFIED BY 'my_password' WITH MAX_QUERIES_PER_HOUR 0 MAX_CONNECTIONS_PER_HOUR 0 MAX_UPDATES_PER_HOUR 0 MAX_USER_CONNECTIONS 0;
GRANT ALL PRIVILEGES ON my_database.* TO 'my_user'@'localhost';

Make a dump of the source environment databases

On your legacy server, create a temporary directory for the databases in your virtual host, so we can migrate them along with the files, on the next step.

cd /home/YOUR_VHOST_DIRECTORY/
mkdir -p private-files/tmp-sql
cd private-files/tmp-sql
mysqldump -u YOUR_USER_NAME -p YOUR_PASSWORD > drupal.sql
mysqldump -u YOUR_USER_NAME -p YOUR_PASSWORD > civicrm.sql

For a Drupal site, it should be preferable to use drush sql-dump or Backup Migrate but let's keep this post for general purpose.

Migrate the files and the databases

On your legacy server, use rsync, so you can re-roll the migration of the files later on, with changes only.

rsync -avxr --progress /path/to/source_directory/ root@YOUR.VIRTUALMIN_SERVER.IP.V4:/path/to/destination_directory/

This is kind of brute force, so a better option is to use git for the code and just rsync the sites/*/files directories (and if needed private files directories outside the docroot).

Example

You can have for the document root:

• Source (legacy server) /var/www/mysite/docroot/sites/default/files
• Destination (Virtualmin server) /home/mysite/public_html/sites/default/files

While adding the virtual host (see above), Virtualmin has created the default document root directory under /home/mysite/public_html.

On the Virtualmin server, the public_html directory should be empty if we do not have enabled stats like webalizer (see enabled features above).

Then, on the legacy server execute rsync

rsync -avxr --progress /var/www/mysite/docroot/sites/default/files/ root@YOUR.VIRTUALMIN_SERVER.IP.V4:/home/mysite/public_html/sites/default/files/

You could have private files, outside of the docroot, then run another rsync for this directory:

rsync -avxr --progress /var/www/mysite/private-files/ root@YOUR.VIRTUALMIN_SERVER.IP.V4:/home/mysite/private-files/

For any update of the files during the migration, re-run these two commands.

A brief reminder about security

In all cases, you should run a security audit before migrating the website to your brand new server.

For a Drupal 7 website that you didn't maintained, even if it looks to have the latest updates, make sure that you run a site audit with security review, hacked! and/or drupalgeddon. If you haven't heard yet of Drupageddon (also known as SA-CORE-2014-005 - Drupal core - SQL injection), have a look at this vulnerability that could have been exploited.

Import databases

On your Virtualmin server, import the two databases that have been migrated on your disk via rsync.

cd /path/to/source_directory/tmp_sql
mysql -u YOUR_USER_NAME -p YOUR_DRUPAL_DB_NAME < drupal.sql
mysql -u YOUR_USER_NAME -p YOUR_CIVICRM_DB_NAME < civicrm.sql

Dealing with PHP versions

Virtualmin comes by default with php 7.0. In some situations, you may want to co-install another version, like 5.6.

Before you start, have a look at the supported versions that mentions EOL (support / security).

Add the ppa for that.

sudo add-apt-repository ppa:ondrej/php

Then update/upgrade your system.

sudo apt update
sudo apt upgrade

Install PHP 5.6 and extensions

This makes sense for CiviCRM 4.6 LTS

sudo apt install pkg-php-tools php5.6 libapache2-mod-php5.6 php5.6-cgi php5.6-cli php5.6-common php5.6-curl php5.6-gd php5.6-imap php5.6-intl php5.6-mysql php5.6-pspell php5.6-sqlite3 php5.6-tidy php5.6-opcache php5.6-json php5.6-bz2 php5.6-mcrypt php5.6-readline php5.6-xmlrpc php5.6-enchant php5.6-xsl

Depending on your needs, you can install PHP 7.1 and 7.2 + extensions.

PHP 7.1

sudo apt install php7.1 libapache2-mod-php7.1 php7.1-cgi php7.1-cli php7.1-common php7.1-curl php7.1-gd php7.1-imap php7.1-intl php7.1-mysql php7.1-pspell php7.1-sqlite3 php7.1-tidy php7.1-opcache php7.1-json php7.1-bz2 php7.1-mcrypt php7.1-readline php7.1-xmlrpc php7.1-enchant php7.1-xsl

PHP 7.2

sudo apt install php7.2 libapache2-mod-php7.2 php7.2-cgi php7.2-cli php7.2-common php7.2-curl php7.2-gd php7.2-imap php7.2-intl php7.2-mysql php7.2-pspell php7.2-sqlite3 php7.2-tidy php7.2-opcache php7.2-json php7.2-bz2 php7.2-readline php7.2-xmlrpc php7.2-enchant php7.2-xsl

Some must have extensions for Drupal 8

If you plan to use Drupal 8, here is a list of packages that should be installed.

Check what is installed first, for a specific PHP version

# List all available extensions
/usr/bin/php7.1 -m

Virtualmin comes with the most popular extensions, but I tend to add these ones by default.

# Needed by the Drupal core
sudo apt install php7.1-gd
# Needed for Solarium (Solr)
sudo apt install php7.1-curl
# Needed for Drupal Commerce
sudo apt install php7.1-bcmath

Fix PHP versions discoverability

At the time of writing, we need to fix two issues between Virtualmin and several PHP versions to allow discoverability.

Fix 1.

For some reason the detected version of PHP are either not install correctly or Virtualmin detects them incorrectly. So in order for Virtualmin to work with PHP 5.6.x and PHP 7.0.x you will need to correct the location of where the php-cgi executables are. Instead of fixing them in the /home/fcgi-bin/*.fcgi files you will need to fix them on the file system. The reason for this is that Virtualmin will change the files back to what it detects and then you will start getting 500 errors. Type or paste the following to set the files correctly.

cd /usr/bin
sudo mv -f php5.6 php5.6.old
sudo mv -f php7.0 php7.0.old
sudo ln -s php-cgi5.6 php5.6
sudo ln -s php-cgi7.0 php7.0

This leads to errors like

PHP Fatal error: cli.php can only be run from command line. in /home/mysite/public_html/sites/all/modules/civicrm/bin/cli.class.php

If you plan to use the cli.php from CiviCRM (or any other command via a specific PHP version), just rename your original PHP version into /usr/bin/php5.6.cli (instead of php5.6.old) and adapt the call in your crontab, e.g.

# Execute the scheduled Jobs every 15 minutes
*/15 * * * * /usr/bin/php5.6.cli /home/mysite/public_html/sites/all/modules/civicrm/bin/cli.php -s site -u MY_DRUPAL_USER_NAME -p MY_DRUPAL_PASSWORD -e Job -a execute

Fix 2.

Comment out SetHandler application/x-httpd-php and SetHandler application/x-httpd-php-source. Do this for each php version in /etc/apache2/mods-available/php5.6.conf, /etc/apache2/mods-available/php7.0.conf (and any other version you may have).

See this post about these two fixes.

Restart Apache

sudo service apache2 restart

You can now select operational PHP versions per virtual host.

Configure disk space quotas

In your Virtualmin GUI, select 'Edit Virtual Server' then 'Quotas and Limits'

php.ini configuration

Select the php.ini that corresponds to the PHP version you set earlier.

A common need is to increase file max upload size In this example we will set the default value of 2M to 15M.

• upload_max_filesize = 15M
• post_max_size = 15M

Install a SSL certificate

Once you have your production domain pointing to the new server, select the Virtualmin tab then your virtual host.

Under 'Server Configuration' > 'Manage SSL Certificate', select the 'Let's Encrypt' tab and request a certificate.

You can then redirect http to https in your .htaccess file.

RewriteCond %{HTTPS} off
RewriteRule ^(.*)$ https://%{HTTP_HOST}%{REQUEST_URI} [L,R=301]

Note that wilcard certificates will be available on January 2018.

Troubleshooting

A quick look into /var/log/virtualmin/mysite.org_error_log will give you a hint about most issues.

Note that you can access it straight from the browser by selecting your virtual host then 'Logs and Reports' > 'Apache Error Log'.

.htaccess

Follow symbolic links

Option FollowSymLinks not allowed here

Change +FollowSymLinks into +SymLinksIfOwnerMatch

So it becomes

Options +SymLinksIfOwnerMatch

For a Drupal site, you should also change this in the .htaccess files that lives in the public (by default, sites/default/files) and private files directories (/home/mysite/private-files in this case).

Directory permissions

Server unable to read htaccess file, denying access to be safe

Chances are that the permissions of the directories (or one of the sub-directories) were not set correctly.

Set all the directories permissions to 755 (in this example the sites/default/files directory and its sub-directories).

find sites/default/files -type d -exec chmod 755 {} \;

Deprecated functions

mod_fcgid: stderr: PHP Deprecated: Methods with the same name as their class will not be constructors in a future version of PHP; Log has a deprecated constructor in /home/mysite/public_html/sites/all/modules/civicrm/packages/Log.php

Your code base is not ready yet for 7.x, switch to another PHP version (5.6 instead of 7.x).

Compatibility issue between CiviCRM 4.6.x and MySQL 5.7

In some cases (e.g. searching a contact) CiviCRM describes an error as

Sorry but we are not able to provide this at the moment. DB Error: unknown error

Enabling backtrace shows that that there is no support for sql_mode = 'ONLY_FULL_GROUP_BY';

See this issue, it is only addressed by CiviCRM 4.7.x https://issues.civicrm.org/jira/browse/CRM-18439

Here is the fix:

mysql > SET GLOBAL sql_mode=(SELECT REPLACE(@@sql_mode,'ONLY_FULL_GROUP_BY',''));

Note that this will not make the change persitent. On restart, the sql_mode variable will be set as previously:

mysql > SHOW GLOBAL VARIABLES;
| sql_mode | ONLY_FULL_GROUP_BY,STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION

To change it permanently, edit the /etc/mysql/mysql.conf/mysqld.cnf, then append this to the [mysqld] section.

sql_mode = STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION

The sql_mode value should be obtained by running the SET GLOBAL query according to your needs then pasting its result to the sql_mode key.

Then restart the mysql service:

sudo service mysql restart

Time out

Depending on your hardware or configuration, you could have errors like

mod_fcgid: read data timeout in 31 seconds
End of script output before headers: index.php

Select your virtual host then 'Services' > 'Configure Website' > 'Edit Directives'

Then try to increase the IPCCommTimeout default value (31).

If you have SSL enabled, do not forget to do the same in 'Configure Website for SSL'.

Memory

Another common error is

Fatal error: Allowed memory size of X bytes exhausted (tried to allocate Y bytes)...

In your php.ini configuration (see above), try to increase the PHP memory_limit default value of 128M to something that complies your available RAM.

I always tend to increase the Maximum packet size (max_allowed_packet) to 32 MB for a Drupal website.

The "" plugin does not exist

This error is really misleading, it can happen on a Drupal 8 website when GD is missing.

mod_fcgid: stderr: Uncaught PHP Exception Drupal\\Component\\Plugin\\Exception\\PluginNotFoundException: "The "" plugin does not exist."

Just install the GD for the PHP version that you plan to use.

sudo apt install php7.1-gd

Fix the locale warning while installing Debian packages

In this case, for belgian French.

locale-gen fr_BE.UTF-8
dpkg-reconfigure locales

What's next?

On the second part of this article, we will describe how to achieve the following tasks via the Virtualmin GUI:

• Configure backups of databases and files: manually and via cron, locally and on another storage (like AWS)
• Configure basic monitoring of the services
• Disable unused services on startup (like FTP)
• Enable new services on startup (like Solr)
• Create a virtual host configuration template
• Handle 301 redirects without altering your local Drupal .htaccess

Related article

• Install Solr 7 for Drupal 8 Search API on Ubuntu 16.04