Category Archives: Computing

Sharing host VPN with Vagrant

When moving a project into Vagrant, I realised that I needed the vagrant guest to share my OS X’s VPN.

I’m using a separate IP address within a private network like this:

config.vm.network :private_network, ip: "192.168.101.101"

So, after some Googling, I added this provider configuration setting:

  config.vm.provider :virtualbox do |vb|
      vb.customize ["modifyvm", :id, "--natdnshostresolver1", "on"]
  end

and now my VM is sending data over the VPN. Note that this will not work when using the Vagrant’s public_network setting though.

Changing the GitHub IRC hooks notification events

As joind.in uses GitHub to host its source code, we use the IRC hook to receive notifications to the IRC channel (#joind.in on freenode) when interesting things happen on the GitHub repositories.

We noticed recently that we were being notified about more types of things happening on some repositories compared to others, so I decided to investigate.

The code for this is here and a quick perusal of it shows that it will do something for these events:

  • push
  • commit_comment
  • pull_request
  • pull_request_review_comment
  • issues
  • issue_comment

However, when you go the administration page on the website, you cannot set which events to be notified on.

Fortunately, GitHub has an API and so you can quite easily manipulate the hooks using curl.

To read all the hooks attached to a project, you do this:

curl -u '{your github username}' -H "Accept: application/json" 
https://api.github.com/repos/{owner name}/{repository name}/hooks

so, for the joind.in API project, I do:

curl -u 'akrabat' -H "Accept: application/json" 

https://api.github.com/repos/joindin/joindin-api/hooks

This will return a list of hooks attached, including all the information about them. In this case, I’m interested in the IRC hook and there’s two interesting keys in the results:

    "url": "https://api.github.com/repos/joindin/joindin-api/hooks/932001",
    "events": [
      "push",
      "pull_request"
    ],

The url provides the information on the end point to use to change the data and the events tell me what’s set up. As you can see, only “push” and “pull_request” are set up for this repository.

To change this, we simply use CURL to POST a new set of events:

curl -u 'akrabat'  -H "Accept: application/json" -H "Content-type: application/json" -X PATCH 

https://api.github.com/repos/joindin/joindin-api/hooks/932001

-d '{"events":["push", "pull_request", "commit_comment", "pull_request_review_comment"]}'

Helpfully, the response includes a representation of the hook we have just modified, so we can see that the events list has changed. For more information, look at GitHub’s API documentation for hooks.

I therefore went through all the joind.in repositories and set them all to the same set of events for consistency. We’ll no doubt find out soon if that results in too many notifications to the channel!

rst2html does not support :start inline:

I’m currently writing some documentation in Restructured Text that I’m targeting at HTML and PDF using rst2html and rst2pdf.

For syntax highlighting, both rst2html and rst2pdf use Pygments, however rst2html doesn’t support any Pygments options.

So a typical PHP snippet in rst targeting rst2pdf, would be written as:

.. code-block: php
    :startinline: true

    $this = str_replace('foo', 'bar', $that);

The startinline Pygments option is to allow it to highlight the snippet, even though the opening <?php is missing.

If you run rst2html, you get this error:

(ERROR/3) Error in "code-block" directive: unknown option: "startinline".

aargh!

To remove the error, I’ve simply run my rst file through see first, so I have a create-html.sh script that does this:

#!/bin/bash

sed '/startinline/d' ${1}.rst > temp.rst
rst2html.py --syntax-highlight=short --stylesheet=syntax.css temp.rst > ${1}.html
rm temp.rst

The sed command removes any lines that contain the word startinline and obviously, this means that the HTML version doesn’t syntax highlight the snippet, but at least it doesn’t generate an error!

Some notes on git

This set of notes covers the main things that I think you need to know about working with git. It is not comprehensive and mainly serves as a reminder for myself.

Remotes

In a typical open source workflow using GitHub or BitBucket, you would fork the main repository into your own and then clone that copy to your local computer:

    git clone git@github.com:akrabat/joind.in.git

You then need to connect your local repository to the main repository. By convention, the main repository is known as upstream:

    cd joind.in
    git remote add upstream git://github.com/joindin/joind.in.git

To sync your local repository with upstream and update your copy back on BitBucket/GitHub:

    git checkout master
    git fetch upstream
    git merge --ff-only upstream/master
    git push origin

If the merge fails, then something bad has happened and you’ll need Google! One option is to force your master to match upstream using git reset --hard upstream/master and then git push --force origin.

Branching

The main branch in the repository is called master. Never ever code directly on master. Always create a branch and code on that and then merge back to master when complete.

(Note that the push command is also used throughout this document to sync your local repository with your remote on BitBucket/GitHub. If don’t want to publish, then don’t push.)

Create a branch:

    git checkout -b my-branch-name
    git push origin my-branch-name

List all branches:

    git branch -v

To list all remote branches too use the -a switch:

    git branch -v -a

Change from one branch to another:

    git checkout another-branch-name

Delete a branch:

    git branch -D my-branch-name
    git push origin :my-branch-name

Rebase master onto a branch:

If lots of people are working on the project, then master has probably changed a lot since you started. It’s going to make your merge back to master much easier if you update your branch so that all your changes appear to be after all the changes on master. This is known as rebasing:

    git fetch upstream
    git checkout my-branch-name
    git rebase -f upstream/master
    git push -f origin my-branch

Bring in a remote branch to your local repository:

If you want to work with a branch that’s on a remote repository, then you need to create your own tracking branch:

    git fetch upstream
    git checkout master
    git branch -t upstream/remote-branch-name
    git push origin remote-branch-name

The name of your local branch will match the remote branch name.

Sync remote branch with local one

    git fetch upstream
    git checkout remote-branch-name
    git merge upstream/remote-branch-name
    git push origin

Committing

To commit a change:

To commit a change, you first need to stage the files that you want to commit to the index:

    git add filename

You can then commit:

    git commit -m "my commit message"
    git push origin

Note that if you change a filename after adding to the index, then the change will not be committed unless your git add the file again. For information about a good commit message, read A Note About Git Commit Messages by Tim Pope.

Merging

Merge a branch into master:

    git checkout master
    git fetch upstream && git merge --ff-only upstream/master
    git merge --no-ff my-branch-name

Note that you don’t need to commit anything. You do need push though:

    git push

If there were conflicts, then you need to resolve them by editing the files appropriately (look for <<<<<<<). At this point, you do need commit your changes:

    git add .
    git commit
    git push

Back out a conflicted merge:

    git reset --hard HEAD

Working with your repository

To find out what’s happened:

    git reflog -10

This provides a list of the last 10 things that you’ve done on this repository across all branches.

    git log --oneline -10

This provides a list of the last 10 commits that’s happened on this branch

In both cases, the first column contains the commit hash reference that uniquely identifies each commit.

To find out current commit:

    git log -1

To undo all working changes

    git checkout .

To revert a commit:

Find the commit that you want to go back to via log or reflog:

    git reset --hard abcdef0

You can also use the ‘HEAD’ number from reflog:

    git reset --hard HEAD@{1}

To amend last commit message:

    git commit --amend -m "New commit message"

It’s best to do this before pushing. If you have pushed, then you need to force push using git push --force and expect to get lots of hassle from your co-workers.

Differences

To find out the differences between your edits and the last commit:

All files:

    git diff

One file:

    git diff -- my-filename

To find out the differences between current branch and master:

    git diff master..HEAD my-filename

To find out the differences between local master and origin’s master:

    git diff HEAD...origin/master

Some git aliases

Aliases provide a way to create new git commands and are usually used for creating shortcuts:

Type these from the command line:

    $ git config --global alias.st status
    $ git config --global alias.staged 'diff --staged'
    $ git config --global alias.unstage 'reset HEAD --'
    $ git config --global alias.last 'log -1 HEAD'

This gives you some new git commands:

  • git st => view current status
  • git staged => view the diff of what is currently staged
  • git unstage filename => Remove filename from the staging area
  • git last => view last commit

Git export from BitBucket

One of the consequences of moving to git is that I needed to get our deployment scripts working with it. The first requirement was an equivalent to svn export. This is simple enough, but, as usual, I wanted to save myself some effort and created a script that wrapped it up for me.

git-export.sh:

#!/bin/sh

#
# USAGE: export.sh REPO_NAME BRANCH_NAME [DIRECTORY_NAME] [BITBUCKET_ACCOUNT_NAME]
#
#

BITBUCKET_ACCOUNT_NAME=akrabat
GITROOT=git@bitbucket.org:${BITBUCKET_ACCOUNT_NAME}

if [ -z "$1" ]; then
    echo "Usage is git-export.sh {repo_name} [{branch_name = master}] [{dir_name = repo_name}]"
    exit 1
fi


REPO_NAME=$1
BRANCH_NAME=$2
THIS_DIR=$3

if [ -z "$BRANCH_NAME" ]; then
    BRANCH_NAME=master
fi
if [ -z "$THIS_DIR" ]; then
    THIS_DIR=$REPO_NAME
fi

if [ -d "$THIS_DIR" ]; then
    echo "Error: directory '$THIS_DIR' already exists."
    exit 1
fi

echo "Exporting $BRANCH_NAME branch of $REPO_NAME into ./$THIS_DIR..."
mkdir $THIS_DIR
git archive --format=tar --remote=$GITROOT/$REPO_NAME $BRANCH_NAME | tar -xf - -C $THIS_DIR

echo "Done"
echo ""

Not the most complicated script in the world, but I never need to think about it again and it’s easy to call from Phing and other deployment scripts and doesn’t require a local clone to exist.

Migrating to BitBucket from Subversion

I’ve recently started the process of moving all of my Subversion repositories to BitBucket. BitBucket is my preferred git hosting supplier as its pricing structure suits me much better; that is, I have lots of private repositories and GitHub is too expensive for my case!

As I needed to migrate over one hundred repositories from Subversion to BitBucket, I automated it via a simple shell script. Rather usefully, BitBucket has an API that let me create the repository using curl and then use git to do the rest.

I’ve put this script here in the hope that it may be useful to someone else. It’s certainly been useful to me!

#!/bin/sh

# Inspired by :
# * http://blogs.atlassian.com/2012/01/moving-confluence-from-subversion-to-git/
# * http://john.albin.net/git/git-svn-migrate
# * http://blog.woobling.org/2009/06/git-svn-abandon.html
# * http://stackoverflow.com/questions/10637378/how-do-i-convert-a-bare-git-repository-into-a-normal-one-in-place

# usage: svn2git.sh REPO
# NOTE that we assume that REPO lives underneath SVN_URL and is a standard subversion layout

# SET THESE CORRECTLY!
USERNAME=my_bitbucket_username
PASSWORD=my_bitbucket_password
AUTHORS_TEXT_FILE=~/authors.txt
SVN_URL=http://svn.example.com/svn


REPO=$1
if [ -z "$REPO" ]; then
    echo ""
    echo "USAGE: svn2git.sh {repo} [{delete_repo_first =1|0}]"
    echo ""
    exit;
fi
DELETE_REPO_FIRST=$2
if [ -z "$DELETE_REPO_FIRST" ]; then
    DELETE_REPO_FIRST=0
fi

BASE_DIR=/tmp
TEMP_REPO=${REPO}_tmp
BARE_REPO=${REPO}.git

echo ""
if [ $DELETE_REPO_FIRST -eq 1 ]; then
    echo "Delete repository on BitBucket first"
    curl -L --silent -X DELETE -u ${USERNAME}:${PASSWORD} https://api.bitbucket.org/1.0/repositories/${USERNAME}/${REPO}
fi
echo "Create repository on BitBucket"
r=`curl -L --silent -X POST -u ${USERNAME}:${PASSWORD} https://api.bitbucket.org/1.0/repositories/ -d name=${REPO} -d scm=git`
t=${r:0:11}
if [ "${r:0:11}" = "Bad Request" ]; then
    echo ${r}
    exit 1
fi

echo ""
echo "Clone from subversion into temporary repository"
cd ${BASE_DIR}
git svn clone --stdlayout --no-metadata 
 -A ${AUTHORS_TEXT_FILE} 
 ${SVN_URL}/${REPO}/ ${TEMP_REPO}

if [ "$?" -ne "0" ]; then
    echo "ERROR: Failed to convert $REPO from subversion to git"
    exit 1
fi

echo ""
echo "Create bare repository"
git init --bare ${BARE_REPO}
cd ${BASE_DIR}/${BARE_REPO}
git symbolic-ref HEAD refs/heads/trunk

echo ""
echo "Push temporary repository into bare one"
cd ${BASE_DIR}/${TEMP_REPO}
git remote add bare ${BASE_DIR}/${BARE_REPO}
git config remote.bare.push 'refs/remotes/*:refs/heads/*'
git push bare

echo ""
echo "Rename 'trunk' to 'master' as that's the git norm"
cd ${BASE_DIR}/${BARE_REPO}
git branch -m trunk master


echo ""
echo "Clean up branches and tags"
git for-each-ref --format='%(refname)' refs/heads/tags |
cut -d / -f 4 |
while read ref
do
  git tag "$ref" "refs/heads/tags/$ref";
  git branch -D "tags/$ref";
done

echo ""
echo "Convert to 'normal' repository from bare"
mkdir .git
mv * .git
git config --local --bool core.bare false


echo ""
echo "Push branches & tags to BitBucket"
git remote add upstream git@bitbucket.org:${USERNAME}/${REPO}
git push --all upstream
git push --tags upstream


echo ""
echo "Clean up"
cd ${BASE_DIR}
rm -rf ${BARE_REPO}
rm -rf ${TEMP_REPO}

echo ""
echo "All done"

Note that at the top there’s this section:

# SET THESE CORRECTLY!
USERNAME=my_bitbucket_username
PASSWORD=my_bitbucket_password
AUTHORS_TEXT_FILE=~/authors.txt
SVN_URL=http://svn.example.com/svn

You need to put your information here if you want to use this script :)

The authors.txt file is a simple text file that maps Subversion user names to git names and emails. It should look something like this:

rob = Rob Allen <rob@akrabat.com>
jsmith = John Smith <jsmith@example.com>

To use this script, I simply call:

svn2bitbucket.sh reponame

and it does its thing. Note that it’s not especially fast for a repository that has lots of history. You can also do svn2bitbucket.sh reponame 1 which will delete the repository on BitBucket first which allows you to re-run the transfer. Note that if you do this, then you will lose any commits you independently pushed to the git repository!