I didn't really like any of the solutions from this post, so I came up with my own.
Idealy what needed is :nth-of-type selector which would make things way easier.
But unfortunately JQuery does not support it "due to their lack of real-world usefulness". Ehh..
So here's my solution which does the trick using :nth-child expression:
$("a.delete").click(function(event) {
event.preventDefault();
var current_cell = $(this).closest("td");
var nb_columns = current_cell.closest('table').find('tr:eq(1) td').length+1;
var column_to_delete = current_cell.prevAll("td").length+1;
$('table tr td:nth-child('+(nb_columns+'n-'+(nb_columns-column_to_delete))+')').remove();
});