My journal of daily hacks

So long wordpress.com and thanks for all the fish

Janos — Mon, 09 Jan 2012 18:51:40 +0000

I migrated my blog and homepage in one place, a wordpress container:

http://www.janosgyerik.com/

There will be no new posts in this blog, please follow the new link above instead. In any case, I will not delete this blog in the foreseeable future.

Convert and mirror a Bazaar repository on Google Code and GitHub

Janos — Fri, 16 Dec 2011 22:36:03 +0000

An open source project should be as open as possible, for that reason I decided to make the source code of bashoneliners.com available in more than one VCS, for example in Subversion and Git in addition to Bazaar which remains the “master” branch.

Bazaar to Subversion

Thanks to the bzr-svn plugin which comes bundled with the latest versions of Bazaar this is really easy and straightforward.

Initial setup: import full history from Bazaar to Google Code

bzr co /path/to/bzr/repo /path/to/svn/pusher
cd /path/to/svn/pusher
bzr push https://PROJECT.googlecode.com/svn/trunk

Mirroring periodically

cd /path/to/svn/pusher
bzr up && bzr push

Occasionally, especially if you have a lot of changes to push at once, the connection can get interrupted, or even bzr could crash. No worries, just repeat the “bzr push”.

As usual with mirrors, if you ever commit to the Google Code repository with anything else other than the above script, the mirroring will stop working because your branches have diverged. If you ever do that by mistake it’s not the end of the world, you can reset the repository on Google Code and rebuild it again.

Bazaar to Git

There is a nice plugin for this too, but since I could not get it working easily on my Mac, I chose an alternative path that’s less pretty but very easy to get going, by using the Subversion mirror on Google Code as a stepping stone from Bazaar to Git.

Initial setup: import full history from Google Code to GitHub

git svn clone https://PROJECT.googlecode.com/svn/trunk /path/to/git/pusher
cd /path/to/git/pusher
git remote add origin git@github.com:repo-url
git push origin master

Mirroring periodically

cd /path/to/git/pusher
git svn rebase && git push origin master

As usual with mirrors, if you ever commit to the GitHub repository with anything else other than the above script, the mirroring will stop working because your branches have diverged. If you ever do that by mistake it’s not the end of the world, since with Git you can rewrite history

How to handle git svn rebase with conflicts

Janos — Wed, 30 Nov 2011 22:09:13 +0000

I’m still new to git and it was not very clear at first how to handle the case when you encounter conflicts during a git svn rebase. Some notes to help next time around:

At any time, you can do git rebase --abort to undo the rebase and be back to your untouched git branch.
There is NO:

git svn rebase --abort
git svn rebase --continue
git svn rebase --skip

There are only:

git rebase --abort
git rebase --continue
git rebase --info

Depending on the conflict you may have to do various things to resolve, such as:

Edit files and sort out the lines marked as conflicted.
Do git add path/to/file
Do git rm path/to/file

After resolving a conflict, do: git rebase --continue

At this step if git complains that nothing has changed, then you have to do: git rebase --skip

When resolving a conflict, if you know for a fact that the file in your master branch is the correct one, you can resolve the conflict with git checkout master path/to/file
During my rebase, somebody committed to subversion, which cause my next git rebase --continue to fail. I don’t know what is the best solution here, I did git rebase --abort and started over.

Converting Django models into Graphviz DOT files

Janos — Tue, 29 Nov 2011 19:23:38 +0000

This was really easy and the result is really good-lookin.

https://code.djangoproject.com/wiki/DjangoGraphviz

The article is almost accurate. Some really accurate steps:

cd /path/to/your/django/project
git clone https://github.com/django-extensions/django-extensions.git
cp django-extensions/django_extensions/management/modelviz.py . 
ln -s $PWD/django-extensions/django_extensions/templates/django_extensions templates/ 
python modelwiz.py name_of_django_app > app.dot
dot app.dot -Tpng -o app.png

(And of course you need http://www.graphviz.org/ first.)

A quite painful MySQL server migration with Japanese characters and utf-8

Janos — Thu, 17 Nov 2011 20:27:04 +0000

What was supposed to be a walk in the park, turned out to be quite painful and weirdo experience with mysql and Japanese text entries. The database in question is a mantis (bug tracker) database with a mix of English and Japanese entries. The database and all tables had utf8 character set, both at the source and destination sites. No matter what method I tried to dump and load, the Japanese characters became garbled.

I tried these for dumping:

mysqldump bugtracker > bugtracker.dump
mysqldump bugtracker -r bugtracker.dump
mysqldump bugtracker --default-character-set=utf8 -r bugtracker.dump

And these for loading:

mysql < bugtracker.dump
mysql --default-character-set=utf8 < bugtracker.dump
set names utf8; source bugtracker.dump

Any combination of the above dumping and loading methods resulted in garbled characters after loading.

At this point, I wanted to pinpoint whether the data gets broken during the dump or during the load. To confirm the correctness of the dump, I created an entry with both English and Japanese text in it, so that I can grep with en English pattern and see some Japanese text that should not be garbled. However, modern versions of mysql dump data as so-called extended inserts, which results in extremely long lines (spanning multiple screens), making the grep output unreadable. Luckily this behavior can be changed with a flag, so that entries are dumped as a single insert command per entry:

mysqldump bugtracker --skip-extended-insert --default-character-set=latin1 | grep mantis_bug_text.*English

This way I found the right way to dump, and to my surprise I had to use latin1 character set. But my troubles were not over yet. Although the dump was supposed to be correct, I still could not load it without breaking the Japanese. By chance, I spotted a suspicious looking snippet in the dump file:

SET NAMES latin1

Changing latin1 to utf8 and then loading it with any of the above methods finally did the trick.

In conclusion, not only I had to force latin1 character set when dumping, I also had to hand-edit the dump file to make it loadable. Sounds pretty crazy to me. I don’t understand the logic behind all this, but this is what worked.

My first Android app, a very simple tool to debug Bluetooth sensors, is now available on Android Market

Janos — Sun, 06 Nov 2011 17:41:52 +0000

BluetoothViewer is a simple Android application that can connect to any Bluetooth device and display incoming raw data, and can also send data from Android to the Bluetooth device. The purpose of the application is to confirm that a Bluetooth device is online and discoverable, accepting incoming connections, and to observe the raw data messages coming from the device.

We use this as a debugging tool for portable environmental sensors that can transmit data over Bluetooth. Naturally it can be useful to debug any kind of Bluetooth device.

The application is FREE and open-source, contributions are very much welcome. For more details see the project homepage:

http://blog.ecomobilecitizen.com/contact/bluetoothviewer/

By the way, publishing on Android Market was really a smooth ride. After publishing, the app’s page on the Market website was ready in a couple of minutes, and after a few hours I was able to find it using the Market application on my Android phone. Totally awesome!

Severed Fifth is totally awesome hard pounding metal, and that for free

Janos — Wed, 02 Nov 2011 21:34:57 +0000

Severed Fifth is a metal band (?), with so far two albums that are self-produced, self-released, written, performed and recorded by Jono Bacon, community manager of Ubuntu, author of the awesome (and free) book The Art of Community. I really didn’t expect much from a music made by just one guy, but man, this is totally awesome hard pounding metal! And the guy is releasing the albums completely for free (under Creative Commons Attribution ShareAlike license). That’s right, it’s just out there for everybody to enjoy, so if you like some good metal, listen to it at http://www.severedfifth.com/releases/ (my favourite is The Lake, when the heavy rhythmic pounding kicks in at about halfway)

Yes, this topic is right at home in a hacking journal

Decentralized development workflow with Bazaar and DVCS in general

Janos — Thu, 29 Sep 2011 17:33:04 +0000

Many of my friends are getting into Bazaar or other distributed version control these days.

Bazaar is in the same class as git and mercurial = distributed version control software or DVCS. This class is significantly more powerful compared to traditional version control software (VCS) such as Subversion. The feature set of DVCS is often a superset of VCS, so usually you can do everything with DVCS that you could do with VCS, and a whole lot more.

The most significant consequence of being “distributed” is that a “checkout” is never “just a checkout”, it’s a whole repository, with full history.

There are two checkout commands in Bazaar, “checkout” and “branch”, for two different workflows.

You use “bzr checkout” for a centralized workflow, like the one used with Subversion. That is, there is a central repository that all developers commit to, they checkout the project with “bzr checkout”, and when they commit with “bzr commit” it gets written to the central repository. To merge the changes by other developers, you do “bzr update”. And when you commit, if other developers had commits while you were working, then you have to do “bzr update” first before you can do “bzr commit”. In this workflow bazaar works exactly the same way as Subversion, with the added benefits of having the entire history on your computer, so you can do diff and log without hitting the network.

You use “bzr branch” for a de-centralized workflow, radically different from Subversion. The repository you branched from is called the “parent” branch and is remembered for reference, but your local branch is not bound to it in any way. Practically it means that all your commits will be local only, to your local branch, and not to the parent. And since your local branch is its own master, “bzr update” will do nothing, because you are in fact always up to date To merge changes done by other developers in the parent, use “bzr rebase”. This will basically rewind your local branch to the revision that you branched from, replay the changes made by others in the parent, and finally replay your own changes. The same way as with “svn update” (or “bzr update”), conflicts may happen during this operation. After resolving conflicts, you can continue with “bzr rebase-continue”.

Side note: the “rebase” command is not a standard command in Bazaar, it is in the “rewrite” plugin. If you don’t have the plugin you can accomplish basically the same with “bzr merge”, but “bzr rebase” is a tad bit cleaner.

When your changes are ready to be merged into the parent branch, you push your branch to a place where your team can access it, and the manager of the parent branch can merge from it and commit into the parent branch. In the end, there is no practical difference between the “checkout” and “branch” workflows, the end result is the same. As such, with a DVCS there is little reason to use the “checkout” workflow.

Being disconnected from a central branch gives you a lot of freedom, for example you can create experimental branches of your own without affecting your team members, you can hack and commit even when you don’t have internet, and of course have the entire project history with you at all times. There is also no central repository server that may crash, no need to setup one, no need to manage user accounts and authentication. Backup is almost unnecessary, if you have enough developers in your team, somebody will always have a copy of the repository so you could lose a few latest revisions only. Once you get comfortable with a DVCS, you will find that the tool doesn’t get in your way, you can happily slice and dice repositories. Branching and merging of branches is straightforward and perfectly natural with any DVCS, there is none of the black magic Subversion needs to have that working.

You can read more about this and other variations of the decentralized workflow in the Bazaar documentation. I use mainly Bazaar for its intuitive simplicity, Git with Eclipse projects due to a lack of a good Bazaar plugin. The various DVCS certainly have some differences, but the essence is the same, and whichever you use, it will make you a happier developer with less stress.

Email forwarding option missing on newly setup google hosted domains

Janos — Sun, 25 Sep 2011 16:42:48 +0000

My internet hosting company has a nice feature to make it really easy to use Gmail for all my hosted domains. I’ve been using this for several domains, and I typically create an info@my.domain.com account and have it forward all incoming emails to my personal email account. Usually you do this in Mail settings / Forwarding and POP/IMAP but recently this option disappeared on the the google hosted domains, there is only POP/IMAP Download instead. Google Help pages are useless, there is nothing written about why the option is was removed.

But guess what, the feature is actually still there! You just have to wait! It seems there is a delay, and the email forwarding option is not available soon after you setup the domain. I don’t know why, I haven’t found relevant documentation. I just experienced with several domains already that I just need to wait a couple of days, and eventually the option appears.

You can tell it’s a bug, because if you try to create a filter, you will find a link “Manage your forwarding addresses” which takes you to the POP/IMAP Download page, where in fact it is impossible to setup forwarding addresses.

In short, it’s a gotcha. If you are experiencing this, give it a few days, hopefully the option will appear eventually.

Mirror a Bazaar repository to Subversion on Google Code

Janos — Sat, 24 Sep 2011 17:26:59 +0000

In this example I use the “bashoneliners” project, replace the project name appropriately for your case.

1. Create a new empty Subversion repository on Google Code

2. Checkout from Google with bzr co https://bashoneliners.googlecode.com/svn/trunk

3. Pull from the original Bazaar repository:

bzr pull path_to_branch --overwrite

To keep the Subversion repository up to date on Google Code, simply run bzr pull without other arguments.

Remember that you must never commit on the mirror Subversion repository by other methods, that will mess things up. But that is normal for a “mirror” repository isn’t it.