Re-installing software Part IV : OwnCloud

Why install OwnCloud? That is a very good question. All the evidence online suggests that NextCloud, the recent fork of the OwnCloud, is seeing all of the development love right now. But I’m going to be honest and say that I have an almost of pathological love of things that come in two’s. I’m sure there’s a name for it, in some dictionary of mild phycological disorders – monophobia – fear of not having a spare.

Why have one when you can have two? Once you set aside the obvious and logical arguments ( such as: you won’t use it, it’s just using up resources, and it’s not as good as the other one) it’s hard to raise objections to the statement that ‘it’s always better to have two rather that one’. There are some exceptions: tax returns, brain tumours, rotten tomatoes. But you could argue that it’s always better to have zero rather than one of those. So the fact remains: two is always better.

Edit: This wisdom devoting so much time and energy to this idea was extremely questionable. It’s now December, and I started digging into this in October. I’ve been down so many rabbit holes, I feel like I might be sprouting ears, and all I have so far is an instance of something that I know I wont’ use. So please feel free to ignore me ignore me.

Rewind

The blog post you’re reading now is actually the result of several frustrating iterations. I’ve had retrace my steps a lot (admittedly made easier by the fact that I’m automating stuff as I go).

Iteration number one used a FreeBSD 11.1 jail template – the same one I’ve used for all the other jails I’ve described here. However, this is the first time I’ve had any dependency on the ports system. As you’ll see below, I tried to abstract away some knowledge about the packages I need to install by using the port for own cloud. This works well when all 3 bits of package infrastructure are in sync: the OS, the package repo and the ports repo. However, with my 11.1 template, I’ve got one element that’s out of whack: the ports tree.

So Iteration number one always ended in abject failure because:

1. The php version coming out of the latest ports repo is 7.1.
2. This gets a core installation working, but none of the pecl packages work (postgres drivers, redis, acpu packages) – so it gets you 80% there and the last 20% is impossible.
3. Installing any pecl package automatically removes the php71 packages, so I lost several installations to sudden php version changes.

With iteration number two, I decided that I needed to try to use a more appropriate ports tree. The the way to do this, apparently is to check out the ports tree from source. The ports tree SVN repo seems to be branched every quarter, so given that FreeBSD 11.1 went out of support in September, the 2018Q3 branch is probably the safest.

My goal with these setups is to to make them repeatable, ideally to be able to re-run them in a CI type system. So that makes me aware of external dependencies like the FreeBSD SVN servers. The infrastructure, as you would expect, is solid, geographically aware and highly available: but I don’t want to be ‘The guy that killed the Ports sever’. Also, cloning the repo takes forever and seems to be about 2Gigs on disk.

I settled on the read only git mirror of the ports tree SVN in Github ¹. That suited me fine because I could make a local clone, and then feed that clone into any FreeBSD 11.1 jails. And what turned out to be an efficient way to do that is to make another template with the right ports tree in it. This now gives me 2 jail templates:

root@freenas:~ # iocage list -t 
+-----+---------------------+-------+--------------+------+ | JID | NAME | STATE | RELEASE | IP4 | +=====+=====================+=======+==============+======+                                       
| -   | ansible_template    | down  | 11.1-RELEASE | DHCP |                                       
+-----+---------------------+-------+--------------+------+                                       
| -   | ansible_template_v2 | down  | 11.1-RELEASE | DHCP |                                       
+-----+---------------------+-------+--------------+------+

Step 1: Make a jail

So, pretending that we’re looking at this fresh, let’s get going. The first thing you usually need to do is make a safe place to put this new software: a jail. As with all great software, OwnCloud is going to jail. We will use our fancy new ansible ready, Ports tree included template and we will externalise the directory that holds the owncloud data (non database data).

iocage create -n owncloud01 dhcp=on allow_sysvipc=1 bpf=yes vnet=on -t ansible_template_v2 boot="on"
iocage fstab -a owncloud01 /mnt/vol01/apps/owncloud /var/www/owncloud/data nullfs rw 0 0

Step 2: Dependency management

The next thing we have to do is install all the pre-requisite packages. There is a trick to this, as OwnCloud (and NextCloud) are large software projects that have a LOT of php dependencies. Some of them are optional and enable plugins or functionality that is non-core, but some are mandatory for basic operation of the product. My preference would be to derive this list somehow, as I don’t want to maintain a list of OwnCloud dependencies that will become out of date at the whim of a developer.

Vermaden over his his blog describes how to use the ports system to list out the dependencies for any FreeBSD package using ‘make’ and the ports system. ²

Important: This couples us to the owncloud port. Is that a bad thing? I don’t know right now but I think it’s still better than keeping my own list of packages

So, if you execute something like this: make -C /usr/ports/www/owncloud run-depends-list`, you get a list of dependencies. With some ansible magic, you can capture this into a list, and then install those packages on the fly:

TASK [owncloud : debug] *****************************************
ok: [[email protected]] => { "packages_output.stdout_lines": [ "mysql56-client", "pecl-smbclient", "php56", "php56-bz2", "php56-ctype", "php56-curl", "php56-dom", "php56-fileinfo", "php56-filter", "php56-gd", "php56-hash", "php56-iconv", "pecl-intl", "php56-json", "php56-mbstring", "php56-pdo", "php56-posix", "php56-session", "php56-simplexml", "php56-xml", "php56-xmlreader", "php56-xmlwriter", "php56-xsl", "php56-wddx", "php56-zip", "php56-zlib", "php56-exif", "php56-ldap", "php56-openssl", "php56-pdo_mysql" ]
}

Step 3

The obligatory enablement of services:

- name: Set SYSRC config shell: "sysrc {{item.key}}={{item.value}}" with_dict: "{{ sysrc_config }}" notify: - restart nginx - restart php-fpm

Step 4 – Tune php and PHP-FPM

All of these steps are described on the owncloud 10 site ³. The summary steps for this installation:

1. Setup the php to nginx socket – /var/run/php-fpm.sock – in /usr/local/etc/php-fpm.conf
2. Setup the env[PATH] and other in the www section of /usr/local/etc/php-fpm.conf. Following the instructions in ³, we are told to use a different file (/usr/local/etc/php-fpm.d/www.conf), but this file is not included by the php-fpm.conf file by default in FreeBSD.
3. Copy php.ini-production to php.ini
4. Set the php opcache values as per ramsdenj ⁴
5. Set cgi.fix_pathinfo to 0 in the php.ini

Step 5 – Download and install owncloud

- name: download owncloud get_url: url=https://download.owncloud.org/community/owncloud-10.0.10.tar.bz2 dest=/root/owncloud-10.0.10.tar.bz2 sha256sum=a2efe484678c1659b9640ea247746a2174d77870d29c7d60abd565c20eb5aa84 register: owncloud_downloaded - name: install owncloud unarchive: src: /root/owncloud-10.0.10.tar.bz2 dest: /usr/local/www remote_src: yes when: owncloud_downloaded is changed notify: - restart nginx - restart php-fpm - name: set perms on owncloud dir file: path="" state=directory recurse="true" owner=www group=www # mode='u=rwX,go=rX' with_items: - "/usr/local/www" - "/usr/local/etc/nginx/cert"

There are a few small things we can do to in order to make occ easier to access:

- name: create the occ command template: src=occ.j2 dest=/usr/local/sbin/occ owner=root group=wheel mode='a+x'

Where this is what occ.j2 would look like:

#!/bin/sh
sudo -u www /usr/local/bin/php /usr/local/www/owncloud/occ "$@"

Step 7 – Configure Nginx

Much like the nextcloud config, the owncloud nginx config is pretty complicated. It is 168 lines long, and is mostly taken verbatim from the owncloud site ⁵.

My ansible role modifies this slightly for my own SSL certs.

Step 8 – Add in the cron script for ownclouds internal scheduling

- name: Cron for cron.php cron: name: "Cron for cron.php" minute: "*/15" job: "sudo -u www /usr/local/bin/php /usr/local/www/owncloud/cron.php"

A production ready owncloud installation needs caching set up as per these instructions. ⁶ They instructions advise you to use Redis as a distributed file lock and APCu as a memory cache. I landed up using that configuration, but mainly because I couldn’t get redis working as a memory cache.

Ideally, I’d like to install one redis instance and have it server multiple owncloud/nextcloud instances. This means using a non-default db index. See this example in your config/config.php

'memcache.local' => '\OC\Memcache\APCu', 'redis' => [ 'host' => 'redis01.home', 'port' => 6379, 'timeout' => 0, 'password' => '', 'dbindex' => 3 ], 'memcache.locking' => '\OC\Memcache\Redis',
# 'memcache.local' => '\\OC\\Memcache\\Redis',