Validating JSON with ZF2's Zend\Validator

Let's say that you have an admin form where the user can enter JSON and you'd like to validate that the JSON parses before allowing the user to submit. To do this, you can use the rather excellent jsonlint project by Jordi Boggiano. Obviously, add it via Compser :)

Usage is simple:

use Seld\JsonLint\JsonParser;
use Seld\JsonLint\ParsingException;

$parser = new JsonParser();

$result = $parser->lint($json);
if ($result instanceof ParsingException) {
    // $json is invalid JSON
}

We can wrap this up into a Zend Framework 2 validator quite easily:

<?php

namespace RKA\Validator;

use Zend\Validator\AbstractValidator;
use Seld\JsonLint\JsonParser;
use Seld\JsonLint\ParsingException;


class Json extends AbstractValidator
{
    const INVALID      = 'jsonInvalid';

    /**
     * Json parser
     *
     * @var \Seld\JsonLint\JsonParser
     */
    protected static $parser = null;

    /**
     * Validation failure message template definitions
     *
     * @var array
     */
    protected $messageTemplates = array(
        self::INVALID      => "Json is invalid: '%reason%'",
    );

    /**
     * Additional variables available for validation failure messages
     *
     * @var array
     */
    protected $messageVariables = array(
        'reason' => 'reason',
    );

    /**
     * @var string
     */
    protected $reason;

    /**
     * Returns true if and only if $value is valid JSON
     *
     * @param  string $value
     * @return bool
     */
    public function isValid($value)
    {
        $parser = new JsonParser();

        $result = $parser->lint($value);

        if ($result instanceof ParsingException) {
            $this->reason = $result->getMessage();
            $this->error(self::INVALID);
            return false;
        }

        return true;
    }
}

Simply add RKA\Validator\Json to the input filter of the element in question and we're done.

Autocomplete Phing targets on the command line

Shortly after writing my last post, it crossed my mind that it would be nice if I could autocomplete the targets in my Phing build file on the command line.

In order to do this, I read Buddy Lindsey's Quick and Dirty Write Your Own Bash Autocomplete and made it do what I needed!

Start by creating a new bash completion file in the bash_completion.d directory. This file needs executable permission. This directory can usually be found at /etc/bash_completion.d, but on OS X using Homebrew, it's at /usr/local/etc/bash_completion.d.

This is the file:

# Store this file in /etc/bash_completion.d/phing
 
_phing () {
    local cur prev
 
    COMPREPLY=()
    buildfile=build.xml
    _get_comp_words_by_ref cur prev
 
    [ ! -f $buildfile ] && return 0

    COMPREPLY=( $( compgen -W "$( phing -l | tr -s '\-' | sed s/^-/\|/ | tr -d '\|' \
        | sed s/\ \ .*\// \
        | sed s/Buildfile.*// | sed s/Default\ target:// | sed s/Subtargets:// \
        | sed s/Main\ targets:// \
        | tr -s ' ' \
        | sed 's/[^[:print:]]//g' | sed s/\\[.*// | tr '\n' ' ' | tr -s '\n' 2>/dev/null )" \
        -- "$cur" ) )
}
 
complete -F _phing phing

It's a bit messy because my knowledge of sed/tr isn't too hot. What we do is create a list of words for the -W option to compgen by running phing -l and then manipulating the output to remove everyting that isn't a target name using various sed and tr commands.

Once done, compgen creates the wordlist used by complete when you press tab after the word phing on the command line.

A few Phing tips

Following on from my last post, here's a few other Phing things that I've found helps me.

Hiding targets from Phing -l

I have a number of dependent targets in my build.xml that I don't want listed when I run phing -l. The easiest way to do this is to add the hidden="true" property to the targets I want to hide:

<?xml version="1.0"?>
<project name="buildfile" default="foo">

  <target name="foo" depends="bar,baz" />

  <target name="bar" hidden="true">
    <!-- tasks -->
  </target>

  <target name="baz" hidden="true">
    <!-- tasks -->
  </target>

</project>

The output of phing -l now shows just the foo target as I want:

$ phing -l
Buildfile: /Users/rob/tmp/build.xml
Default target:
-------------------------------------------------------------------------------
 foo

Subtargets:
-------------------------------------------------------------------------------
 foo

Main target vs subtarget

If you set a description on a target, then it becomes a Main target:

<?xml version="1.0"?>
<project name="buildfile" default="foo">

  <target name="foo" depends="bar,baz" description="Run foo to do stuff" />

  <!-- other hidden targets -->

</project>

The output is now:

$ phing -l
Buildfile: /Users/rob/tmp/build.xml
Default target:
-------------------------------------------------------------------------------
 foo

Main targets:
-------------------------------------------------------------------------------
 foo      Run foo to do stuff

It just looks better and provides additional information as well.

List available targets by default

If you run phing without a target, then the default target runs. If you've forgotten what that is, then this may not be the safest thing to happen. Ideally, the default target should do no harm and I've found it helpful to make the default target display a list of the available targets within the build file.

This is done by adding a hidden "list_targets" target and making it the default:

<?xml version="1.0"?>
<project name="buildfile" default="list_targets">

  <target name="list_targets" hidden="true">
    <exec command="phing -q -f ${phing.file} -l" passthru="true"/>
  </target>

  <target name="foo" depends="bar,baz" description="Run foo to do stuff" />

  <!-- other hidden targets -->

</project>

Essentially, we run phing -l against our current build file and set the passthru property so that the output is displayed. This gives us:

$ phing
Buildfile: /Users/rob/tmp/build.xml

buildfile > list:

Default target:
-------------------------------------------------------------------------------
 list

Main targets:
-------------------------------------------------------------------------------
 foo  Run foo to do stuff


BUILD FINISHED

Total time: 0.2176 seconds

Using Phing to SSH into a Vagrant box

Now that I've started using migrations, I've discovered a minor irritant.

I run this project on a Vagrant VM and have discovered that I keep forgetting to ssh into the vagrant box before running the migrations script. The obvious solution is to automate this and I decided to use Phing to do so.

Then add extension=ssh2.so to your /etc/php.ini file.

Firstly, I needed to install the PHP ssh2 extension:

$ brew install libssh2
$ sudo pecl install pecl.php.net/ssh2-beta

I'm on OS X, so installed libssh2 via homebrew and as the PECL ssh2 extension is marked as beta, so we have to specify that. We can accept the default for auto-detecting the libssh2 prefix.

Once this is done, we can now use Phing's SshTask in our build scripts.

This is a simple build.xml file to check it all works:

<?xml version="1.0"?>
<project name="sshtest" default="testssh">
  <target name="testssh">
    <ssh username="vagrant" password="vagrant" host="192.168.123.456"
        command="pwd" property="pwd" display="false" />
    <echo>The current working directory is ${pwd}</echo>
  </target>
</project>

The output looks like:

$ phing
Buildfile: /www/19ft/projectname/build.xml

sshtest > testssh:

     [echo] The current working directory is /home/vagrant


BUILD FINISHED

Total time: 0.3923 seconds

Now that it works, we can write a target that does something useful:

  <target name="migrate"  >
    <ssh username="vagrant" password="vagrant" host="192.168.123.456"
      command="cd /vagrant; php bin/migrations.php migrate"  />
  </target>

Now, I simply run `phing migrate` and the MySQL in my Vagrant box is migrated as I expect.

Obviously Phing isn't the only way to do this, but it worked for me.

My "new" blogging habit

I've just come across Andy Baio's Middling post and this bit really resonated with me:

Twitter and Waxy Links cannibalized all the smaller posts, and as my reach grew, I started reserving blogging for more "serious" stuff — mostly longer-form research and investigative writing.

From around September 2013, I found that I was in this situation. I was only posting here once or twice a month, mainly as I felt that I should only blog when I had something "worthy" to write about. I got of this rut nearly a year later as I started a new project that resulted in a number of fairly frequent "long" posts about Slim and ZF2 components, which rekindled my interest in writing here.

In March 2013, I gave myself permission to write more opinionated posts here but didn't actually do so until June 2014. Since then I've written a few more times on the things I care about and hope to continue to do so.

I've also started giving myself permission to write posts that are shorter tech-notes. These are more aide-mémoira that I want to remember that maybe someone else will find useful.

I would like to write shorter opinion pieces. Say 250ish words on items that catch my interest. I'm currently using Twitter for this, but sometimes 140 characters isn't enough and I can never find what I wrote again. This post is to give myself permission to do so.

Maybe I'll actually do it in less than a year from now!

Registering Doctrine Type Mappings for standalone migrations

Shortly after starting to use Doctrine Migrations as a standalone tool in my project, I came across this error message:

Unknown database type bit requested, Doctrine\DBAL\Platforms\MySqlPlatform may not support it.

This means that I have a column in my database of type bit which is used for booleans in SQL Server, but confuses the MySQL platform as it's not a default mapping.

To support this, you need to modify the database connection's Platform object to know about the new mapping. However, with the setup that I'm using, I didn't have access to the connection object that's automatically created in the Migrations AbstractCommand object.

After poking around in the code for a bit, I discovered that the solution is to create the connection object myself and then attach it as a new helper to the Console\Application object.

This is how to do to it within the migrations.php file.

Firstly, create the database connection object using the information in migrations-db.php. This code is lifted from AbstractCommand:

if (file_exists('migrations-db.php')) {
    $params = include 'migrations-db.php';
    if (!is_array($params)) {
        throw new \InvalidArgumentException('The connection file has to return an array.');
    }
    $connection = \Doctrine\DBAL\DriverManager::getConnection($params);
} else {
    throw new \InvalidArgumentException('Missing "migrations-db.php" file. Alternatively use --db-configuration.');
}

We can now set up the database type mapping for bit:

$platform = $connection->getDatabasePlatform();
$platform->registerDoctrineTypeMapping('bit', 'boolean');

Now, all we need to do is add the connection to the Console\Application's HelperSet so that the AbstractCommand can use it. We are already adding the DialogHelper, so we add a ConnectionHelper

use Doctrine\DBAL\Tools\Console\Helper\ConnectionHelper;

$helperSet = new Console\Helper\HelperSet();
$helperSet->set(new Console\Helper\DialogHelper(), 'dialog');
$helperSet->set(new ConnectionHelper($connection), 'connection');
$cli->setHelperSet($helperSet);

That's it. Our migrations system know nows that a database column of type bit is a boolean on MySQL as well as on SQL Server.

Routing specific traffic to the VPN on OS X

I have a client that requires me to use a VPN when connecting to their servers. I use OS X's built in L2TP VPN to connect, but don't want all my traffic going that way.

To do this, I unchecked the Advanced VPN setting "Send all traffic over VPN connection" in the Network preferences and then created the file /etc/ppp/ip-up like this:

sudo touch /etc/ppp/ip-up
sudo chmod 755 /etc/ppp/ip-up

The file itself is a bash script that runs various /sbin/route commands and looks similar to this:

/etc/ppp/ip-up:

#!/bin/sh
/sbin/route add -net 192.168.1.0/16 -interface ppp0

Now, whenever I connect to the VPN, only traffic for hosts on 192.168.1.x is sent to the client's VPN and we're both happy.

Using Doctrine Migrations as a standalone tool

My current project has reached the point where a good migrations system is required. As I'm targeting two different database engines (MySQL and MS SQL Server) and we're already using DBAL, it made sense to use Migrations from the Doctrine project.

To do this, I updated my composer.json with the following require statements:

"doctrine/migrations": "1.0.*@dev",
"symfony/console": "~2.5"

in order to install Migrations and also Symfony console component so that I can run it from the command line.

The CLI script

Migrations doesn't come with it's own CLI script, so you have to create your own. Fortunately, we can tweak the supplied phar-cli-stub.php. I placed it in bin/migrations.php:

/**
 * Command line script to run Migrations
 * Inspired by phar-cli-stup.php
 */

use Symfony\Component\Console;
use Doctrine\DBAL\Migrations\MigrationsVersion;
use Doctrine\DBAL\Migrations\Tools\Console\Command as MigrationsCommand;

require (__DIR__ . '/../vendor/autoload.php');

// Set current directory to application root so we can find config files
chdir(__DIR__ . '/..');

// Instantiate console application
$cli = new Console\Application('Doctrine Migrations', MigrationsVersion::VERSION);
$cli->setCatchExceptions(true);

$helperSet = new Console\Helper\HelperSet();
$helperSet->set(new Console\Helper\DialogHelper(), 'dialog');
$cli->setHelperSet($helperSet);

// Add Migrations commands
$commands = array();
$commands[] = new MigrationsCommand\ExecuteCommand();
$commands[] = new MigrationsCommand\GenerateCommand();
$commands[] = new MigrationsCommand\LatestCommand();
$commands[] = new MigrationsCommand\MigrateCommand();
$commands[] = new MigrationsCommand\StatusCommand();
$commands[] = new MigrationsCommand\VersionCommand();

// remove the "migrations:" prefix on each command name
foreach ($commands as $command) {
    $command->setName(str_replace('migrations:', '', $command->getName()));
}
$cli->addCommands($commands);

// Run!
$cli->run();

Note that, by default, the console commands are all prefixed with 'migrations:'. This makes sense as usually they are integrated with the rest of the Doctrine ORM command line application and so it avoids any clashes. In this case, this script is only dealing with migrations and I don't want to type more than I have to, so I remove the prefixes!

Configuration

Migrations requires two separate configuration files to work: migrations.yml and migrations-db.php. These need to be in the current directory, so I used chdir() to ensure that it is in a known location – the application root in this case.

migrations.yml is the config file that tells Migrations the names of things:

migrations.yml:

name: DBAL Migrations
migrations_namespace: Migrations
table_name: migration_versions
migrations_directory: migrations

In my case, I want to use my migrations files to be in a directory called migrations and use the namespace Migrations. The database table name that Migrations uses to keep track of its internal status is called migration_versions. I'm not too imaginative when it comes to naming things!

migrations-db.php must return an array that is used to configure Doctrine\DBAL\DriverManager::getConnection()

migrations-db.php:

<?php
return array(
        'dbname'   => 'albums',
        'user'     => 'rob',
        'password' => '123456',
        'host'     => 'localhost',
        'driver'   => 'pdo_mysql',
);

The documentation will help you find out what can go in here.

We're now ready to go!

On the command line, php bin/migrations.php status should work:

$ php bin/migrations.php 
Doctrine Migrations version 2.0.0-DEV

Usage:
  [options] command [arguments]

Options:
  --help           -h Display this help message.
  --quiet          -q Do not output any message.
  --verbose        -v|vv|vvv Increase the verbosity of messages
  --version        -V Display this application version.
  --ansi              Force ANSI output.
  --no-ansi           Disable ANSI output.
  --no-interaction -n Do not ask any interactive question.

Available commands:
  execute    Execute a single migration version up or down manually.
  generate   Generate a blank migration class.
  help       Displays help for a command
  latest     Outputs the latest version number
  list       Lists commands
  migrate    Execute a migration to a specified version or the latest available version.
  status     View the status of a set of migrations.
  version    Manually add and delete migration versions from the version table.

Creating a migration

run php bin/migrations.php generate and a new migration class will be created in the migrations directory. It has two methods: up() and down(). Both have an instance of Doctrine\DBAL\Schema\Schema passed and an you can do whatever you like in the function to manipulate your database to its next state. Again, the documentation is helpful!

For instance:

<?php

namespace Migrations;

use Doctrine\DBAL\Migrations\AbstractMigration;
use Doctrine\DBAL\Schema\Schema;

class Version20141027161210 extends AbstractMigration
{
    public function up(Schema $schema)
    {
        $myTable = $schema->createTable('artists');
        $myTable->addColumn('id', 'integer',
            ['unsigned' => true, 'autoincrement'=>true]);
        $myTable->addColumn('name', 'string', ['length' => 60]);
        $myTable->setPrimaryKey(['id']);
    }

    public function down(Schema $schema)
    {
        $schema->dropTable('artists');
    }
}

Running the migration

Simply run php bin/migrations.php migrate and Migrations will the up() method for all migration classes that haven't be run yet. In this case it will create a table called artists and update the migration_versions table with the version number of '20141027161210'.

To back out of a migration, simple run php bin/migrations.php migration {version number} and Migrations will run the appropriate down() (or up()) methods to get to that version number.

Finally you can run php bin/migrations.php status to find out the current state.

On the whole, it turns out that it's quite easy to use Migrations as a stand-alone tool outside of Doctrine ORM or Symfony!

Two incidents

An online friend of mine once said this about a sexism incident at a conference:

"If I had been there, ZERO chance I would've sat idly by and let someone treat you like that"

I like to believe that I have the same policy: If I am there when some casual sexism occurs, I will call it out.

It turns out that this is untrue.

One

I have witnessed a number of incidents of sexism at conferences that I've been to recently and I want to talk about one particular situation that occurred.

Around lunchtime at a recent conference, I was with a group of developer friends that included one woman. A man joined our group and made a joke to the woman along the lines of asking if she was here with her boyfriend. I understand that they are friends and that he was being ironic and humorous and certainly meant no personal insult. She responded with a cutting remark and that was that.

Except that wasn't that.

I did not speak out.

It was just a joke. It worked by demeaning women in my industry. I should have said something, especially as I was in the company of friends. The fact that I didn't speak up is unacceptable because jokes like this are actively harmful by contributing to the background tolerance of discrimination against women.

I am ashamed that I didn't speak up. I let my friends and myself down and and I apologise to everyone in our community for not doing my bit when I had the chance.

Two

On Saturday, I took my son to one of his extracurricular activity sessions. Afterwards, my son and I were talking with his coach about the rest of our weekend and how I'd be helping him with the electrical bits of building his new Tamiya Hornet remote controlled car. The coach commented that he would like to do the same sort of things with his future grandchildren and hoped that he wouldn't get a granddaughter.

Maybe it was because my son was there and I'm trying to help teach him about these things, or maybe it was because of the incidents that I'd witnessed recently at conferences, but this time, I spoke up.

I talked about my friend who took an Arduino and some coloured LEDs to create a fuzzy clock where the colour of LEDs indicates the time to the nearest hour. I'm fully intending to copy her project as it would be especially useful to me at night when I'm not wearing my glasses. We then talked a little about how a granddaughter could certainly be interested in building and running remote control cars!

Fin

I want to help reduce sexism in this world. More than one recent conference incident has reminded me that I have a long long way to go, but maybe there's some hope for me yet.

Even though I regularly fail, sometimes I succeed and I won't let my failures stop me from getting it right next time.

Why I care about codes of conduct

tl;dr

I want every conference to enthusiastically champion their code of conduct in order to publicly reassure attendees (women in particular) that any issues will be dealt with. I want all of us to actually notice jokes and conduct that make conferences uncomfortable for women and call out or report issues if we see them. I'm still learning how to do this, but the rest of this post goes into detail on why I think this is subject is important.

The detail

NYCC's Anti-Harassment Policy

Following up on my last post about codes of conducts, I was recently asked why I'm interested in them. While I responded directly, it crosses my mind that I should probably write down my thoughts on this and then I can practice good DRY practices and just link back here!

I am becoming increasingly concerned about the lack of women and non-white people I see at the conferences I go to. It was particularly noticeable at one conference I went to fairly recently and so I’ve been asking why this is. Many of people who inspire me are women, but I hardly see any women at conferences. So, I've started with the issue of the lack of women at conferences and in development in general.

Conferences are one of the best ways I know to become a better developer. Not only do you get exposed to new technologies and techniques via the talks, you get to meet other developers & make contacts that last beyond the event.

We know from US statistics that around 20% of all developers are women. Looking at the male to female ratios at most of the conferences that I go to, I don't even see that many women developers there. I'm assuming that they are choosing not to attend.

That's sad.

So I've started reading. It's hard to read without judging based on your own experiences. I’ve also talked to my wife who has gone to events in a different space and other women friends in a variety of industries and it has been eye-opening.

From my learning so far there seemed to be two main topics that kept coming up: atmosphere and safety.

One major concern I have is that a woman’s initial base assumption seems to be that she expects that something “uncomfortable” is to going happen to them over the course of an event. This runs the full range of things from physical things like being touched “by accident”, through having someone stand very close in her personal space and monopolise her to the point that she is “trapped” talking to this person to men staring at her breasts when talking to her. There's also the words she will hear – from casual comments about women belonging in the kitchen, to overhearing jokes about how bad women are at logical thinking all the way to being asked if she has a boyfriend and if not, being propositioned.

As a man, it’s taken me ages to understand why a casual comment to a woman about whether she has a boyfriend is so damaging. It seemed harmless to me. Just a joke. However, no one ever talks to me a conference with an underlying subtext that they are mainly interested in having sex with me rather than geeking out on tech and programming. No one ever questions my authority in my subject area. Even now, I have been places where I hear conversations between one of my female friends and another man and they tell me afterwards that the man clearly didn't think she was competent by his choice of words and demeanour and I didn’t even notice :(

Hence, I can completely understand why a woman would rather not go to a conference than have to deal with this underlying unspoken assumption that she isn't good enough to be a programmer. Don’t even get me started on the message that's sent when you see pictures of women in lingerie or short skirts in slide decks and sponsor advertising material. I’ve seen all these things at conferences before.

To my wife, her personal safety is very much an active thought in her mind when she meets any man. It's her default position. Apparently this is as much of a worry in a public social setting as on a lonely street at night; maybe more so. By safety, I don’t mean violent rape. I mean being hugged when you don’t want to be; being touched "by accident"; being cornered by someone. These things don't just happen at evening socials, they also happen in the coffee break. These things are happening in public places and and no one else in the room notices.

How are we supposed to stop things that we don’t even see from happening?

The logical thing to me is that if any woman experiences anything she is uncomfortable with (or a man sees something happening to a woman), they should feel able to report it in private and be 100% sure that they will be believed and looked after, no matter how minor the organisers may perceive this incident to be.

I don't think I can emphasise enough that so many incidents appear to be minor and "just the way things are" to men. I'm finding this really difficult to internalise and I'm, notionally at least, aware of the issue. "Just a joke", "That's just how Bob is", "That's not against the law", "It's just a pat on the bottom" are all ways to enable us to believe that there's nothing in our community that needs improving.

Coming back to the subject at hand, I think that a published code of conduct is a clear way to say to the attendees that the organisers know that these things can happen and that (a) they are publicly saying that they are unacceptable and that (b) they are signalling that they have a plan, already in place, for dealing with any incident that may happen.

I appreciate that other people don't agree with me and think that their conference doesn't need a code of conduct because they work towards improving diversity in other ways.

I disagree.

Personally, I see a code of conduct that is championed by the organisers as a great way to improve the inclusivity of an event which can only make for a better conference. I want to see them at every conference.

I was impressed to see that NYCC put their code of conduct on billboards throughout their conference! This is a commercial venture, so certainly they thought about the pros and cons of doing it.

However, the best way we can make conferences more welcoming to women is if we, the attendees, call out our fellow attendees on what they say and what they do that's unwelcoming. It takes courage to report a joke that isn't suitable for a professional conference, but if we all do it, it becomes the norm.

Further reading

If you are interested in some further reading, on this topic then I can recommend Codes of Conduct 101 + FAQ, Sexual harassment at technical conferences: A big no-no and Your conference needs an anti-harassment policy.

For some real-world experiences, try My first OSCON, This is why we can't have nice things, The Creepy Librarian Stalker Hypothesis & Please do not pat me on my head. There are many many more examples out there; try asking any woman who's been to a conference as I haven't found one yet that did not have a story to tell.

Finally, the discussions on this reddit thread and on BoingBoing about the very visible NYCC banners are interesting too.