Overriding the built-in Twig date filter

In one project that I'm working on, I'm using Twig and needed to format a date received from an API. The date string received is of the style "YYYYMMDD", however date produced an unexpected output.

Consider this:

{{ "20141216"|date('jS F Y') }}

creates the output:

20th May 1976

This surprised me. Then I thought about it some more and realised that the date filter is treating my date string as a unix timestamp. I investigated and discovered the problem in twig_date_converter:

    $asString = (string) $date;
    if (ctype_digit($asString) 
        || (!empty($asString) && '-' === $asString[0] && ctype_digit(substr($asString, 1)))) {
        $date = '@'.$date;
    }

    $date = new DateTime($date, $defaultTimezone);

This code tests to see if the dates string provided is a number (positive or negative) and then prepends an '@' symbol to the front. This has the effect of informing DateTime's constructor to treat $date as a unix timestamp.

Unfortunately, twig_date_converter is a function and so I couldn't override it, so I wrote my own extension that registers new date, date_modify filters and a new date function in order to solve my problem:

<?php
/*
 * Extension to provide updated date & date_modify filters along with an
 * updated date function which do not auto-convert strings of numbers to
 * a unix timestamp.
 *
 * Code within dateFilter(), modifyFilter() and dateFromString() extracted
 * from Twig/lib/Twig/Extension/Core.php which is (c) 2009 Fabien Potencier
 * and licensed as per https://github.com/twigphp/Twig/blob/master/LICENSE
 */
namespace My\Twig\Extension;

use Twig_Extension;
use Twig_SimpleFilter;
use Twig_SimpleFunction;
use DateTime;
use DateTimeInterface;
use DateTimeImmutable;

class DateExtension extends Twig_Extension
{
    public function getName()
    {
        return 'my_date';
    }

    public function getFilters()
    {
        return array(
            new Twig_SimpleFilter('date', [$this, 'dateFilter'],
                    ['needs_environment' => true]),
            new Twig_SimpleFilter('date_modify', [$this, 'modifyFilter'],
                    ['needs_environment' => true]),
        );
    }

    public function getFunctions()
    {
        return array(
            new Twig_SimpleFunction('date', [$this, 'dateFromString'],
                    ['needs_environment' => true]),
        );
    }

    public function dateFilter($env, $date, $format = null, $timezone = null)
    {
        if (null === $format) {
            $formats = $env->getExtension('core')->getDateFormat();
            $format = $date instanceof DateInterval ? $formats[1] : $formats[0];
        }

        if ($date instanceof DateInterval) {
            return $date->format($format);
        }

        return $this->dateFromString($env, $date, $timezone)->format($format);
    }

    public function modifyFilter($env, $date, $format = null, $timezone = null)
    {
        $date = $this->dateFromString($env, $date, false);
        $date->modify($modifier);

        return $date;
    }

    public function dateFromString($env, $date, $timezone)
    {
        // determine the timezone
        if (!$timezone) {
            $defaultTimezone = $env->getExtension('core')->getTimezone();
        } elseif (!$timezone instanceof DateTimeZone) {
            $defaultTimezone = new DateTimeZone($timezone);
        } else {
            $defaultTimezone = $timezone;
        }

        // immutable dates
        if ($date instanceof DateTimeImmutable) {
            return false !== $timezone ? $date->setTimezone($defaultTimezone) : $date;
        }

        if ($date instanceof DateTime || $date instanceof DateTimeInterface) {
            $date = clone $date;
            if (false !== $timezone) {
                $date->setTimezone($defaultTimezone);
            }

            return $date;
        }

        $date = new DateTime($date, $defaultTimezone);
        if (false !== $timezone) {
            $date->setTimezone($defaultTimezone);
        }

        return $date;
    }
}

This class simply registers new date, date_modify filters and a new date function to replace the ones in Twig core and then is a direct copy of the functions twig_date_format_filter, twig_date_modify_filter and twig_date_converter with the functionality above removed.

I also needed to register this extension with the Twig_Environment using: $env->addExtension(new \My\Twig\Extension\DateExtension()); and I'm done.

{{ "20141216"|date('jS F Y') }}

now correctly outputs:

16th December 2014

While, it's a shame I can't just override twig_date_converter, I'm glad that I can re-register the relevant Twig filters and function.

Converting databases between MySQL and SQL Server

I regularly deal projects that target SQL Server, but mostly develop against MySQL to avoid having to run a full Windows stack locally all the time. One of the nice things about PHP with DBAL and Migrations is that the database is pretty well abstracted from my code. Of course, this means that I don't target any of the specialist features, but for these projects, this hasn't been an issue.

To convert data from SQL Server to MySQL, I've found Intelligent Converters's MSSQL to MySQL tool to work very well for my needs. It allows me to grab the test database from the client and convert it to a MySQL dump file that I can then import.

Validating JSON with ZF2's Zend\Validator

Let's say that you have an admin form where the user can enter JSON and you'd like to validate that the JSON parses before allowing the user to submit. To do this, you can use the rather excellent jsonlint project by Jordi Boggiano. Obviously, add it via Compser :)

Usage is simple:

use Seld\JsonLint\JsonParser;
use Seld\JsonLint\ParsingException;

$parser = new JsonParser();

$result = $parser->lint($json);
if ($result instanceof ParsingException) {
    // $json is invalid JSON
}

We can wrap this up into a Zend Framework 2 validator quite easily:

<?php

namespace RKA\Validator;

use Zend\Validator\AbstractValidator;
use Seld\JsonLint\JsonParser;
use Seld\JsonLint\ParsingException;


class Json extends AbstractValidator
{
    const INVALID      = 'jsonInvalid';

    /**
     * Json parser
     *
     * @var \Seld\JsonLint\JsonParser
     */
    protected static $parser = null;

    /**
     * Validation failure message template definitions
     *
     * @var array
     */
    protected $messageTemplates = array(
        self::INVALID      => "Json is invalid: '%reason%'",
    );

    /**
     * Additional variables available for validation failure messages
     *
     * @var array
     */
    protected $messageVariables = array(
        'reason' => 'reason',
    );

    /**
     * @var string
     */
    protected $reason;

    /**
     * Returns true if and only if $value is valid JSON
     *
     * @param  string $value
     * @return bool
     */
    public function isValid($value)
    {
        $parser = new JsonParser();

        $result = $parser->lint($value);

        if ($result instanceof ParsingException) {
            $this->reason = $result->getMessage();
            $this->error(self::INVALID);
            return false;
        }

        return true;
    }
}

Simply add RKA\Validator\Json to the input filter of the element in question and we're done.

Autocomplete Phing targets on the command line

Shortly after writing my last post, it crossed my mind that it would be nice if I could autocomplete the targets in my Phing build file on the command line.

In order to do this, I read Buddy Lindsey's Quick and Dirty Write Your Own Bash Autocomplete and made it do what I needed!

Start by creating a new bash completion file in the bash_completion.d directory. This file needs executable permission. This directory can usually be found at /etc/bash_completion.d, but on OS X using Homebrew, it's at /usr/local/etc/bash_completion.d.

This is the file:

# Store this file in /etc/bash_completion.d/phing
 
_phing () {
    local cur prev
 
    COMPREPLY=()
    buildfile=build.xml
    _get_comp_words_by_ref cur prev
 
    [ ! -f $buildfile ] && return 0

    COMPREPLY=( $( compgen -W "$( phing -l | tr -s '\-' | sed s/^-/\|/ | tr -d '\|' \
        | sed s/\ \ .*\// \
        | sed s/Buildfile.*// | sed s/Default\ target:// | sed s/Subtargets:// \
        | sed s/Main\ targets:// \
        | tr -s ' ' \
        | sed 's/[^[:print:]]//g' | sed s/\\[.*// | tr '\n' ' ' | tr -s '\n' 2>/dev/null )" \
        -- "$cur" ) )
}
 
complete -F _phing phing

It's a bit messy because my knowledge of sed/tr isn't too hot. What we do is create a list of words for the -W option to compgen by running phing -l and then manipulating the output to remove everyting that isn't a target name using various sed and tr commands.

Once done, compgen creates the wordlist used by complete when you press tab after the word phing on the command line.

A few Phing tips

Following on from my last post, here's a few other Phing things that I've found helps me.

Hiding targets from Phing -l

I have a number of dependent targets in my build.xml that I don't want listed when I run phing -l. The easiest way to do this is to add the hidden="true" property to the targets I want to hide:

<?xml version="1.0"?>
<project name="buildfile" default="foo">

  <target name="foo" depends="bar,baz" />

  <target name="bar" hidden="true">
    <!-- tasks -->
  </target>

  <target name="baz" hidden="true">
    <!-- tasks -->
  </target>

</project>

The output of phing -l now shows just the foo target as I want:

$ phing -l
Buildfile: /Users/rob/tmp/build.xml
Default target:
-------------------------------------------------------------------------------
 foo

Subtargets:
-------------------------------------------------------------------------------
 foo

Main target vs subtarget

If you set a description on a target, then it becomes a Main target:

<?xml version="1.0"?>
<project name="buildfile" default="foo">

  <target name="foo" depends="bar,baz" description="Run foo to do stuff" />

  <!-- other hidden targets -->

</project>

The output is now:

$ phing -l
Buildfile: /Users/rob/tmp/build.xml
Default target:
-------------------------------------------------------------------------------
 foo

Main targets:
-------------------------------------------------------------------------------
 foo      Run foo to do stuff

It just looks better and provides additional information as well.

List available targets by default

If you run phing without a target, then the default target runs. If you've forgotten what that is, then this may not be the safest thing to happen. Ideally, the default target should do no harm and I've found it helpful to make the default target display a list of the available targets within the build file.

This is done by adding a hidden "list_targets" target and making it the default:

<?xml version="1.0"?>
<project name="buildfile" default="list_targets">

  <target name="list_targets" hidden="true">
    <exec command="phing -q -f ${phing.file} -l" passthru="true"/>
  </target>

  <target name="foo" depends="bar,baz" description="Run foo to do stuff" />

  <!-- other hidden targets -->

</project>

Essentially, we run phing -l against our current build file and set the passthru property so that the output is displayed. This gives us:

$ phing
Buildfile: /Users/rob/tmp/build.xml

buildfile > list:

Default target:
-------------------------------------------------------------------------------
 list

Main targets:
-------------------------------------------------------------------------------
 foo  Run foo to do stuff


BUILD FINISHED

Total time: 0.2176 seconds

Using Phing to SSH into a Vagrant box

Now that I've started using migrations, I've discovered a minor irritant.

I run this project on a Vagrant VM and have discovered that I keep forgetting to ssh into the vagrant box before running the migrations script. The obvious solution is to automate this and I decided to use Phing to do so.

Firstly, I needed to install the PHP ssh2 extension:

$ brew install libssh2
$ sudo pecl install pecl.php.net/ssh2-0.12

I'm on OS X, so installed libssh2 via homebrew and as the PECL ssh2 extension is marked as beta, the simplest way to install it is to reference the exact version.

Once this is done, we can now use Phing's SshTask in our build scripts.

This is a simple build.xml file to check it all works:

<?xml version="1.0"?>
<project name="sshtest" default="testssh">
  <target name="testssh">
    <ssh username="vagrant" password="vagrant" host="192.168.123.456"
        command="pwd" property="pwd" display="false" />
    <echo>The current working directory is ${pwd}</echo>
  </target>
</project>

The output looks like:

$ phing
Buildfile: /www/19ft/projectname/build.xml

sshtest > testssh:

     [echo] The current working directory is /home/vagrant


BUILD FINISHED

Total time: 0.3923 seconds

Now that it works, we can write a target that does something useful:

  <target name="migrate"  >
    <ssh username="vagrant" password="vagrant" host="192.168.123.456"
      command="cd /vagrant; php bin/migrations.php migrate"  />
  </target>

Now, I simply run `phing migrate` and the MySQL in my Vagrant box is migrated as I expect.

Obviously Phing isn't the only way to do this, but it worked for me.

My "new" blogging habit

I've just come across Andy Baio's Middling post and this bit really resonated with me:

Twitter and Waxy Links cannibalized all the smaller posts, and as my reach grew, I started reserving blogging for more "serious" stuff — mostly longer-form research and investigative writing.

From around September 2013, I found that I was in this situation. I was only posting here once or twice a month, mainly as I felt that I should only blog when I had something "worthy" to write about. I got of this rut nearly a year later as I started a new project that resulted in a number of fairly frequent "long" posts about Slim and ZF2 components, which rekindled my interest in writing here.

In March 2013, I gave myself permission to write more opinionated posts here but didn't actually do so until June 2014. Since then I've written a few more times on the things I care about and hope to continue to do so.

I've also started giving myself permission to write posts that are shorter tech-notes. These are more aide-mémoira that I want to remember that maybe someone else will find useful.

I would like to write shorter opinion pieces. Say 250ish words on items that catch my interest. I'm currently using Twitter for this, but sometimes 140 characters isn't enough and I can never find what I wrote again. This post is to give myself permission to do so.

Maybe I'll actually do it in less than a year from now!

Registering Doctrine Type Mappings for standalone migrations

Shortly after starting to use Doctrine Migrations as a standalone tool in my project, I came across this error message:

Unknown database type bit requested, Doctrine\DBAL\Platforms\MySqlPlatform may not support it.

This means that I have a column in my database of type bit which is used for booleans in SQL Server, but confuses the MySQL platform as it's not a default mapping.

To support this, you need to modify the database connection's Platform object to know about the new mapping. However, with the setup that I'm using, I didn't have access to the connection object that's automatically created in the Migrations AbstractCommand object.

After poking around in the code for a bit, I discovered that the solution is to create the connection object myself and then attach it as a new helper to the Console\Application object.

This is how to do to it within the migrations.php file.

Firstly, create the database connection object using the information in migrations-db.php. This code is lifted from AbstractCommand:

if (file_exists('migrations-db.php')) {
    $params = include 'migrations-db.php';
    if (!is_array($params)) {
        throw new \InvalidArgumentException('The connection file has to return an array.');
    }
    $connection = \Doctrine\DBAL\DriverManager::getConnection($params);
} else {
    throw new \InvalidArgumentException('Missing "migrations-db.php" file. Alternatively use --db-configuration.');
}

We can now set up the database type mapping for bit:

$platform = $connection->getDatabasePlatform();
$platform->registerDoctrineTypeMapping('bit', 'boolean');

Now, all we need to do is add the connection to the Console\Application's HelperSet so that the AbstractCommand can use it. We are already adding the DialogHelper, so we add a ConnectionHelper

use Doctrine\DBAL\Tools\Console\Helper\ConnectionHelper;

$helperSet = new Console\Helper\HelperSet();
$helperSet->set(new Console\Helper\DialogHelper(), 'dialog');
$helperSet->set(new ConnectionHelper($connection), 'connection');
$cli->setHelperSet($helperSet);

That's it. Our migrations system know nows that a database column of type bit is a boolean on MySQL as well as on SQL Server.

Routing specific traffic to the VPN on OS X

I have a client that requires me to use a VPN when connecting to their servers. I use OS X's built in L2TP VPN to connect, but don't want all my traffic going that way.

To do this, I unchecked the Advanced VPN setting "Send all traffic over VPN connection" in the Network preferences and then created the file /etc/ppp/ip-up like this:

sudo touch /etc/ppp/ip-up
sudo chmod 755 /etc/ppp/ip-up

The file itself is a bash script that runs various /sbin/route commands and looks similar to this:

/etc/ppp/ip-up:

#!/bin/sh
/sbin/route add -host 192.168.1.0/16 -interface ppp0

Now, whenever I connect to the VPN, only traffic for hosts on 192.168.1.x is sent to the client's VPN and we're both happy.

Using Doctrine Migrations as a standalone tool

My current project has reached the point where a good migrations system is required. As I'm targeting two different database engines (MySQL and MS SQL Server) and we're already using DBAL, it made sense to use Migrations from the Doctrine project.

To do this, I updated my composer.json with the following require statements:

"doctrine/migrations": "1.0.*@dev",
"symfony/console": "~2.5"

in order to install Migrations and also Symfony console component so that I can run it from the command line.

The CLI script

Migrations doesn't come with it's own CLI script, so you have to create your own. Fortunately, we can tweak the supplied phar-cli-stub.php. I placed it in bin/migrations.php:

/**
 * Command line script to run Migrations
 * Inspired by phar-cli-stup.php
 */

use Symfony\Component\Console;
use Doctrine\DBAL\Migrations\MigrationsVersion;
use Doctrine\DBAL\Migrations\Tools\Console\Command as MigrationsCommand;

require (__DIR__ . '/../vendor/autoload.php');

// Set current directory to application root so we can find config files
chdir(__DIR__ . '/..');

// Instantiate console application
$cli = new Console\Application('Doctrine Migrations', MigrationsVersion::VERSION);
$cli->setCatchExceptions(true);

$helperSet = new Console\Helper\HelperSet();
$helperSet->set(new Console\Helper\DialogHelper(), 'dialog');
$cli->setHelperSet($helperSet);

// Add Migrations commands
$commands = array();
$commands[] = new MigrationsCommand\ExecuteCommand();
$commands[] = new MigrationsCommand\GenerateCommand();
$commands[] = new MigrationsCommand\LatestCommand();
$commands[] = new MigrationsCommand\MigrateCommand();
$commands[] = new MigrationsCommand\StatusCommand();
$commands[] = new MigrationsCommand\VersionCommand();

// remove the "migrations:" prefix on each command name
foreach ($commands as $command) {
    $command->setName(str_replace('migrations:', '', $command->getName()));
}
$cli->addCommands($commands);

// Run!
$cli->run();

Note that, by default, the console commands are all prefixed with 'migrations:'. This makes sense as usually they are integrated with the rest of the Doctrine ORM command line application and so it avoids any clashes. In this case, this script is only dealing with migrations and I don't want to type more than I have to, so I remove the prefixes!

Configuration

Migrations requires two separate configuration files to work: migrations.yml and migrations-db.php. These need to be in the current directory, so I used chdir() to ensure that it is in a known location – the application root in this case.

migrations.yml is the config file that tells Migrations the names of things:

migrations.yml:

name: DBAL Migrations
migrations_namespace: Migrations
table_name: migration_versions
migrations_directory: migrations

In my case, I want to use my migrations files to be in a directory called migrations and use the namespace Migrations. The database table name that Migrations uses to keep track of its internal status is called migration_versions. I'm not too imaginative when it comes to naming things!

migrations-db.php must return an array that is used to configure Doctrine\DBAL\DriverManager::getConnection()

migrations-db.php:

<?php
return array(
        'dbname'   => 'albums',
        'user'     => 'rob',
        'password' => '123456',
        'host'     => 'localhost',
        'driver'   => 'pdo_mysql',
);

The documentation will help you find out what can go in here.

We're now ready to go!

On the command line, php bin/migrations.php status should work:

$ php bin/migrations.php 
Doctrine Migrations version 2.0.0-DEV

Usage:
  [options] command [arguments]

Options:
  --help           -h Display this help message.
  --quiet          -q Do not output any message.
  --verbose        -v|vv|vvv Increase the verbosity of messages
  --version        -V Display this application version.
  --ansi              Force ANSI output.
  --no-ansi           Disable ANSI output.
  --no-interaction -n Do not ask any interactive question.

Available commands:
  execute    Execute a single migration version up or down manually.
  generate   Generate a blank migration class.
  help       Displays help for a command
  latest     Outputs the latest version number
  list       Lists commands
  migrate    Execute a migration to a specified version or the latest available version.
  status     View the status of a set of migrations.
  version    Manually add and delete migration versions from the version table.

Creating a migration

run php bin/migrations.php generate and a new migration class will be created in the migrations directory. It has two methods: up() and down(). Both have an instance of Doctrine\DBAL\Schema\Schema passed and an you can do whatever you like in the function to manipulate your database to its next state. Again, the documentation is helpful!

For instance:

<?php

namespace Migrations;

use Doctrine\DBAL\Migrations\AbstractMigration;
use Doctrine\DBAL\Schema\Schema;

class Version20141027161210 extends AbstractMigration
{
    public function up(Schema $schema)
    {
        $myTable = $schema->createTable('artists');
        $myTable->addColumn('id', 'integer',
            ['unsigned' => true, 'autoincrement'=>true]);
        $myTable->addColumn('name', 'string', ['length' => 60]);
        $myTable->setPrimaryKey(['id']);
    }

    public function down(Schema $schema)
    {
        $schema->dropTable('artists');
    }
}

Running the migration

Simply run php bin/migrations.php migrate and Migrations will the up() method for all migration classes that haven't be run yet. In this case it will create a table called artists and update the migration_versions table with the version number of '20141027161210'.

To back out of a migration, simple run php bin/migrations.php migration {version number} and Migrations will run the appropriate down() (or up()) methods to get to that version number.

Finally you can run php bin/migrations.php status to find out the current state.

On the whole, it turns out that it's quite easy to use Migrations as a stand-alone tool outside of Doctrine ORM or Symfony!