Documentation

2017 software infrastructure

This migration combines numerous changes having as main goals:

  • general updates to the preinstalled software,
  • simplifying certain aspects of our software infrastructure.

For the oldest shared accounts, this migration also includes the migration to our own physical infrastructure.

This document describes the main incompatibilities introduced by this migration. We strive to be as thorough as possible, but it is impossible to be completely exhaustive. We strongly advise that you carry out a test to the migration in order to detect as many incompatibilities as possible.

This document does not describe the new features introduced by the 2017 software infrastructure, as this has been done in a blog post devoted to that topic.

Background information

  • You are absolutely required to use the new hostname format for accessing services that was introduced in 2015. The addresses to be used are in the alwaysdata administration, in each of the appropriate sections. The old formats, using the "alwaysdata.com" domain (for instance, mysql.alwaysdata.com and postgresql1.alwaysdata.com) or a point between the name of the service and the account (like mysql.account.alwaysdata.net) will altogether stop working.

  • Several files and directories in each account will be moved or deleted. Namely:

    • the four default files (php5.fcgi, php5.ini, php4.fcgi, php4.ini) in the ~/cgi-bin directory will be deleted, as well as the directory itself, if it becomes empty;
    • the various internal configuration files (~/.env.*, ~/admin/apache) will be moved to the ~/admin/config directory;
    • the ~/admin/ssl, ~/admin/session, and ~/admin/log/awstats directories will be deleted;
    • the logs will henceforth be kept in the ~/admin/logs directory, with a subdirectory per type (http, apache, etc.). The previous log directory (~/admin/log) will be moved into the ~/admin/logs/old directory.

Languages

PHP

  • The .php4 and .php5 file extensions will no longer be supported; by default only files with the .php extension will be interpreted by PHP. However, you can revert to the previous behavior by creating a ".htaccess" file at the root of your account. For example, if you want .php5 files to be interpreted by PHP, it should include:
AddHandler fcgid-script .php5
FCGIWrapper /usr/bin/php-cgi .php5
  • It will no longer be possible to run PHP 4 and PHP 5 simultaneously in the same account. PHP 4 will henceforth be regarded simply as a version of PHP as any other, having nothing special about it.

  • Versions 5.5.0, 5.5.1, and 5.5.6 will be replaced by version 5.5.19.

  • The only PHP binary available will be php. Do not use, for example, php-cgi.

  • Non-standard extensions (APC, Xapian, Xdebug, and Mapscript for PHP 5.3 and 5.2, and Fileinfo for 5.2) will be deleted. However, you may install them yourself in your own account.

  • Symfony is no longer preinstalled, you have to install it manually.

  • The calendar and intl extensions are no longer automatically activated, they has to be explicitely loaded in your php.ini if needed.

  • Some minor compilation options have been changed.

  • Internal Apache directives that configure PHP (using FastCGI) have been modified. It only matters if you explicitely referred to them, for instance if you have a Apache custom site and you had copy-pasted directives from the internal Apache configuration. We suggest you replace your Apache custom sites by Apache standard sites and put your own directives in the Additional directives for this virtual host (under Advanced Settings) or in a .htaccess file.

Python

  • The only Python binary available will be python. Do not use, for example, python2.6. In particular, consider replacing your shebangs (the first line in a script, for example #!/usr/bin/python), in case they don't already use python.

  • A few libraries previously preinstalled by default will be deleted, including Django. You should therefore manually install all the libraries you will need.

  • All WSGI applications will henceforth be served by uWSGI and no longer by mod_wsgi. In the vast majority of cases, this changes nothing for you. If you had compiled your own mod_wsgi with a custom Apache site, it will still work.

  • Version 2.5.5 will be replaced by 2.5.6, version 2.6.6 by 2.6.9 and version 3.1 by 3.3.6.

Ruby

  • The only Ruby binary available will be ruby. Do not use, for example, ruby1.8. In particular, consider replacing your shebangs (the first line in a script, for example #!/usr/bin/ruby), in case they don't already use ruby.

  • A few libraries previously preinstalled by default will be deleted, including Ruby on Rails. You should therefore manually install all the libraries you will need.

  • All Ruby Rack and Ruby on Rails applications will henceforth be served by uWSGI and no longer by Passenger. In the vast majority of cases, this changes nothing for you. If you have compiled your own Passenger with a custom Apache site, it will still work.

  • Versions 1.8.6 and 1.8.7 will be replaced by versions 1.8.7-p374 and version 1.9.2 by version 1.9.2-p320.

Databases

MySQL

We will replace MySQL by MariaDB 10.1, which is fully compatible with MySQL. MariaDB 10.1 corresponds to MySQL 5.6, or even 5.7. For more details on the compatibility, see https://mariadb.com/kb/en/mariadb/mariadb-vs-mysql-compatibility/

PostgreSQL

We will update PostgreSQL to version 9.6:
https://www.postgresql.org/docs/9.6/static/release.html

We will also update PostGIS to version 2.3:
http://postgis.net/docs/manual-2.3/PostGIS_Special_Functions_Index.html#NewFunctions

MongoDB

We will update MongoDB to version 3.2:
https://docs.mongodb.com/manual/release-notes/

Attention: the authentification mechanism changes. Users having all permissions have the readWrite role, users having read-only permissions have the read role.

CouchDB

We will update CouchDB to version 1.6:
http://docs.couchdb.org/en/1.6.0/whatsnew/

Miscellaneous

Apache

  • Apache will be updated to version 2.2.32:
    http://www.apache.org/dist/httpd/CHANGES_2.2

  • Files index.htm, index.cgi, index.pl, and index.xhtml will no longer be considered index files. Only index.php and index.html will, and index.php will henceforth take precedence over index.html.

  • If no index file is present, a Forbidden error page will be returned rather than listing files.
    They may be re-listed with the addition in a .htaccess of the Options +Indexes directive.

  • Modules mod_dav_svn, mod_wsgi, and Passenger will be deleted.

  • Files belonging to other modules will be moved, but the migration will automatically handle changing the paths of LoadModule directives in "custom Apache* sites which refer to them. Therefore, you need not worry about this.

Miscellaneous updates

A great number of applications and libraries will also updated. They are too numerous to list, but among the most important, which might cause incompatibilities are:

  • Erlang 19.1 (previously 15.b)
  • Haskell 7.10 (previously 6.8)
  • GDAL 1.10 (previously 1.5)
  • git 2.9 (previously 1.8)
  • ImageMagick 6.8 (previously 6.5)
  • Java JRE 7 et 8 (previously 6)
  • Lua 5.3 (previously 5.1)
  • MapServer 7.0 (previously 5.6)
  • Mercurial 3.9 (previously 1.3)
  • OCaml 4.01 (previously 3.10)
  • Perl 5.20 (previously 5.10)
  • PROJ 4.8 (previously 4.7)
  • Subversion 1.8 (previously 1.6)
  • Texlive 2014 (previously 2007)

VPS/dedicated environment

The following services, installed on demand, will also be upgraded:

  • RabbitMQ will be updated to version 3.5
  • Redis will be updated to version 3.2
  • Elasticsearch will be updated to version 5.5

Elimination of old libraries

If you've compiled your own applications, they might be linked to libraries or versions that will now be missing (because of an ABI bump). Your application, or some of its features, may stop working. You will need to recompile the applications in question.

Among the libraries for which this is most likely are:

  • OpenSSL. Version 0.9.8 will remain installed, alongside version 1.0.2, so your applications will continue to work. However, version 0.9.8 was old, and you are strongly advised to nevertheless recompile your applications to use a recent version of OpenSSL.
  • libreadline

Change to the IP address of the HTTP server

For some shared clients still hosted in our former physical infrastructure, their server's IP address will have to change. This will be noted, if necessary, in the migration details page, in the alwaysdata administration. If you use our DNS servers or a CNAME, you will need to do nothing. On the other hand, if you manually set a DNS record of type A or AAAA, you must change it after performing the migration, providing the new IP addresses.

The previous IP addresses will nevertheless continue to work for a few weeks longer (as a reverse proxy), but connecting through them will cause a slight increase in the latency of your site and will increase the probability of a malfunction. Therefore, you are strongly advised to change them as soon as possible.

Migration Test

It is strongly recommended that you perform a migration test prior to the actual process, to ensure that your applications will continue to work and to fix them if they don’t.

SSH Access

You will be able to connect to your account via SSH through a temporary server. Though the server is equipped with the 2017 version of the software architecture, it will have access to your actual files, not a copy. Any changes which are enacted will immediately be reflected on your account.

SSH access can be slow in comparison to your usual SSH server. This is a consequence of the test, but this slowness will not be present during the actual migration.

HTTP Access

You can test your sites using several methods:

  • by accessing their usual URL, then adding the suffix .migration.alwaysdata.net. For example, if your site is normally accessible via the address www.example.com, you will be able to access it through the test infrastructure by going to www.example.com.migration.alwaysdata.net.
    Warning: you will need to explicitly authorize the SSL certificate sent from *.migration.alwaysdata.net in your browser, as it will be invalid. This is only relevant during the migration test. Certificates will not be changed during the actual migration.
    Warning: some applications do a redirect to the usual URL, which prevents them from being tested using this method.

  • by using a browser extension that allows to override the Host header (and thus the requested site). For instance with Chrome, the Virtual Hosts extension. You'll have to connect by specifying the HTTP test server address (for instance, migration-test1.paris1.alwaysdata.com), but requesting your site address.

  • by modifying your hosts file to force using the HTTP test server IP address to connect to your sites. You can do that by directly editing the file, or by using a browser extension, for instance HostAdmin with Firefox.

Your applications will start up from a temporary server running the 2017 software infrastructure, as if the migration had taken place. Just like under SSH, the files from your account that this server has access to will be your actual files. This access will also run slowly, but you needn’t worry about this.

Warning: the internal configuration of your alwaysdata account on the temporary server is created when you perform the migration test, not afterward. For example, if you perform the test when PHP version 5.6.27 is defined in your environment and this version is then changed, changes will not be reflected on the temporary migration server. You would have to start the test again to register the changes. This is the same for changes made in the Web > Sites section.

Databases

When you perform the migration test, all your database structures and users will be copied onto a temporary server, which allows the new versions to run. You will then be able to access your copied data on this server while using your usual login details to run the tests. Unlike the files used by the SSH and HTTP servers, the files here are copies of your data. Each time you run the migration test, the previous copies of your data will be overwritten by the new data.

Migration tips

General strategy

It can be tempting to use the opportunity of migrating your account to do drastic changes to your web applications, their configuration or deployment. We suggest instead to do as few changes as needed to do the migration successfully (after you've tested it), and only then to do more invasive changes, once your account is on the new infrastructure.

The major gain to proceed this way is that once your account is on the new infrastructure, you can benefit from new options and much simplified deployment methods, in particular with Python and Ruby. For instance, if you had compiled your own interpreter or your own Apache modules (mod_wsgi, Passenger), it's quite likely that you'll be able to dispose of them once your account in on the new infrastructure.

You have references to the internal PHP configuration either in a .htaccess file or in a Apache custom site (maybe you've copied parts of the sites.conf configuration file). For instance, you have references to application/x-httpd-fastphp5.

Our internal PHP configuration has changed, so your configuration will stop working. If you use PHP and a Apache custom site, we suggest you convert it to the Apache standard type and put your specific Apache directives either in a .htaccess file or in the Additional directives for this virtual host field.

Error: "cannot open shared object file: No such file or directory"

This error indicates that a system library is missing. Even if the overwhelming majority of system libraries that have been upgraded on the new infrastructure remain compatible with the older versions (because the ABI remains the same), some do not (there's an ABI version change). If your applications depend on those libraries, they will not be able to find them anymore.

The following libraries are affected:

In general, those libraries are not used directly by applications, but by using a dependency (Python module, Ruby gem, PHP extension). Recompiling those modules on the new infrastructure will have them use the new version of the library, but then they won't work on the old infrastructure anymore.

Another solution is sometimes possible: indicate during the module compilation to disable some features to avoid using the mentioned library. For instance, you can compile the Pillow module without TIFF support using the following command:

pip install Pillow --global-option="build_ext" --global-option="--disable-tiff"

Migrating a Python application using FastCGI

You have a Python application (for instance, using Django) using FastCGI, with a .htaccess file and a .fcgi script. FastCGI will continue to work on the new infrastructure as before, but if you don't use a virtualenv, you'll have to manually install Python system libraries as they're no longer preinstalled on the new infrastructure.

  • in SSH, install all the libraries you need, for instance:

mkdir ~/python_libs; PYTHONPATH=~/python_libs easy_install --always-copy --install-dir ~/python_libs Django==1.6 flup==1.0.3.dev-20110405 psycopg==2.0.11

  • edit your .fcgi file to replace the shebang (first line), typically #!/usr/bin/python, with #!/usr/bin/eval PYTHONPATH=/home/foo/python_libs python, where foo should be replaced with your account name.

You use your own mod_wsgi module

You have compiled your own Python interpreter along with the mod_wsgi module, that you use with a Apache custom site. Make sure you've defined the WSGIPythonHome Apache directive, otherwise you may get errors on the new infrastructure, such as:

ImportError: No module named _sysconfigdata_nd

If you Python interpreter is in the /home/foo/python directory, you should use the following directive:

WSGIPythonHome /home/foo/python

You use the mod_wsgi system module

You have a Apache custom site that loads the system module /usr/lib/apache2/modules/mod_wsgi.so-2.6. As said earlier, this module disappears on the new infrastructure (in favour of WSGI). To make the migration easier, you can:

wget https://files.alwaysdata.com/migrations/software-2017/mod_wsgi.so-2.6

  • replace the /usr/lib/apache2/modules/mod_wsgi.so-2.6 path from directives in your Apache custom site with the path to the downloaded file on your account, for instance /home/foo/mod_wsgi.so-2.6.

Last updated: 10 Oct. 2017 at 16:19