Algorithm for equiprobable random square binary matrices with two non-adjacent non-zeros in each row and column
It would be great if someone could point me towards an algorithm that would allow me to :
- create a random square matrix, with entries 0 and 1, such that <
-
Here is a very fast approach of generating the matrix row by row, written in Java:
public static void main(String[] args) throws Exception { int n = 100; Random rnd = new Random(); byte[] mat = new byte[n*n]; byte[] colCount = new byte[n]; //generate row by row for (int x = 0; x < n; x++) { //generate a random first bit int b1 = rnd.nextInt(n); while ( (x > 0 && mat[(x-1)*n + b1] == 1) || //not adjacent to the one above (colCount[b1] == 2) //not in a column which has 2 ) b1 = rnd.nextInt(n); //generate a second bit, not equal to the first one int b2 = rnd.nextInt(n); while ( (b2 == b1) || //not the same as bit 1 (x > 0 && mat[(x-1)*n + b2] == 1) || //not adjacent to the one above (colCount[b2] == 2) || //not in a column which has 2 (b2 == b1 - 1) || //not adjacent to b1 (b2 == b1 + 1) ) b2 = rnd.nextInt(n); //fill the matrix values and increment column counts mat[x*n + b1] = 1; mat[x*n + b2] = 1; colCount[b1]++; colCount[b2]++; } String arr = Arrays.toString(mat).substring(1, n*n*3 - 1); System.out.println(arr.replaceAll("(.{" + n*3 + "})", "$1\n")); }It essentially generates each a random row at a time. If the row will violate any of the conditions, it is generated again (again randomly). I believe this will satisfy condition 4 as well.
Adding a quick note that it will spin forever for N-s where there is no solutions (like N=3).
讨论(0) -
This solution use recursion in order to set the cell of the matrix one by one. If the random walk finish with an impossible solution then we rollback one step in the tree and we continue the random walk.
The algorithm is efficient and i think that the generated data are highly equiprobable.
package rndsqmatrix; import java.util.ArrayList; import java.util.Collections; import java.util.List; import java.util.stream.IntStream; public class RndSqMatrix { /** * Generate a random matrix * @param size the size of the matrix * @return the matrix encoded in 1d array i=(x+y*size) */ public static int[] generate(final int size) { return generate(size, new int[size * size], new int[size], new int[size]); } /** * Build a matrix recursivly with a random walk * @param size the size of the matrix * @param matrix the matrix encoded in 1d array i=(x+y*size) * @param rowSum * @param colSum * @return */ private static int[] generate(final int size, final int[] matrix, final int[] rowSum, final int[] colSum) { // generate list of valid positions final List<Integer> positions = new ArrayList(); for (int y = 0; y < size; y++) { if (rowSum[y] < 2) { for (int x = 0; x < size; x++) { if (colSum[x] < 2) { final int p = x + y * size; if (matrix[p] == 0 && (x == 0 || matrix[p - 1] == 0) && (x == size - 1 || matrix[p + 1] == 0) && (y == 0 || matrix[p - size] == 0) && (y == size - 1 || matrix[p + size] == 0)) { positions.add(p); } } } } } // no valid positions ? if (positions.isEmpty()) { // if the matrix is incomplete => return null for (int i = 0; i < size; i++) { if (rowSum[i] != 2 || colSum[i] != 2) { return null; } } // the matrix is complete => return it return matrix; } // random walk Collections.shuffle(positions); for (int p : positions) { // set '1' and continue recursivly the exploration matrix[p] = 1; rowSum[p / size]++; colSum[p % size]++; final int[] solMatrix = generate(size, matrix, rowSum, colSum); if (solMatrix != null) { return solMatrix; } // rollback matrix[p] = 0; rowSum[p / size]--; colSum[p % size]--; } // we can't find a valid matrix from here => return null return null; } public static void printMatrix(final int size, final int[] matrix) { for (int y = 0; y < size; y++) { for (int x = 0; x < size; x++) { System.out.print(matrix[x + y * size]); System.out.print(" "); } System.out.println(); } } public static void printStatistics(final int size, final int count) { final int sumMatrix[] = new int[size * size]; for (int i = 0; i < count; i++) { final int[] matrix = generate(size); for (int j = 0; j < sumMatrix.length; j++) { sumMatrix[j] += matrix[j]; } } printMatrix(size, sumMatrix); } public static void checkAlgorithm() { final int size = 8; final int count = 2215764; final int divisor = 122; final int sumMatrix[] = new int[size * size]; for (int i = 0; i < count/divisor ; i++) { final int[] matrix = generate(size); for (int j = 0; j < sumMatrix.length; j++) { sumMatrix[j] += matrix[j]; } } int total = 0; for(int i=0; i < sumMatrix.length; i++) { total += sumMatrix[i]; } final double factor = (double)total / (count/divisor); System.out.println("Factor=" + factor + " (theory=16.0)"); } public static void benchmark(final int size, final int count, final boolean parallel) { final long begin = System.currentTimeMillis(); if (!parallel) { for (int i = 0; i < count; i++) { generate(size); } } else { IntStream.range(0, count).parallel().forEach(i -> generate(size)); } final long end = System.currentTimeMillis(); System.out.println("rate=" + (double) (end - begin) / count + "ms/matrix"); } public static void main(String[] args) { checkAlgorithm(); benchmark(8, 10000, true); //printStatistics(8, 2215764/36); printStatistics(8, 2215764); } }The output is:
Factor=16.0 (theory=16.0) rate=0.2835ms/matrix 552969 554643 552895 554632 555680 552753 554567 553389 554071 554847 553441 553315 553425 553883 554485 554061 554272 552633 555130 553699 553604 554298 553864 554028 554118 554299 553565 552986 553786 554473 553530 554771 554474 553604 554473 554231 553617 553556 553581 553992 554960 554572 552861 552732 553782 554039 553921 554661 553578 553253 555721 554235 554107 553676 553776 553182 553086 553677 553442 555698 553527 554850 553804 553444讨论(0) -
(Updated test results, example run-through and code snippets below.)
You can use dynamic programming to calculate the number of solutions resulting from every state (in a much more efficient way than a brute-force algorithm), and use those (pre-calculated) values to create equiprobable random solutions.
Consider the example of a 7x7 matrix; at the start, the state is:
0,0,0,0,0,0,0meaning that there are seven adjacent unused columns. After adding two ones to the first row, the state could be e.g.:
0,1,0,0,1,0,0with two columns that now have a one in them. After adding ones to the second row, the state could be e.g.:
0,1,1,0,1,0,1After three rows are filled, there is a possibility that a column will have its maximum of two ones; this effectively splits the matrix into two independent zones:
1,1,1,0,2,0,1 -> 1,1,1,0 + 0,1These zones are independent in the sense that the no-adjacent-ones rule has no effect when adding ones to different zones, and the order of the zones has no effect on the number of solutions.
In order to use these states as signatures for types of solutions, we have to transform them into a canonical notation. First, we have to take into account the fact that columns with only 1 one in them may be unusable in the next row, because they contain a one in the current row. So instead of a binary notation, we have to use a ternary notation, e.g.:
2,1,1,0 + 0,1where the 2 means that this column was used in the current row (and not that there are 2 ones in the column). At the next step, we should then convert the twos back into ones.
Additionally, we can also mirror the seperate groups to put them into their lexicographically smallest notation:
2,1,1,0 + 0,1 -> 0,1,1,2 + 0,1Lastly, we sort the seperate groups from small to large, and then lexicographically, so that a state in a larger matrix may be e.g.:
0,0 + 0,1 + 0,0,2 + 0,1,0 + 0,1,0,1Then, when calculating the number of solutions resulting from each state, we can use memoization using the canonical notation of each state as a key.
Creating a dictionary of the states and the number of solutions for each of them only needs to be done once, and a table for larger matrices can probably be used for smaller matrices too.
Practically, you'd generate a random number between 0 and the total number of solutions, and then for every row, you'd look at the different states you could create from the current state, look at the number of unique solutions each one would generate, and see which option leads to the solution that corresponds with your randomly generated number.
Note that every state and the corresponding key can only occur in a particular row, so you can store the keys in seperate dictionaries per row.
TEST RESULTS
A first test using unoptimized JavaScript gave very promising results. With dynamic programming, calculating the number of solutions for a 10x10 matrix now takes a second, where a brute-force algorithm took several hours (and this is the part of the algorithm that only needs to be done once). The size of the dictionary with the signatures and numbers of solutions grows with a diminishing factor approaching 2.5 for each step in size; the time to generate it grows with a factor of around 3.
These are the number of solutions, states, signatures (total size of the dictionaries), and maximum number of signatures per row (largest dictionary per row) that are created:
size unique solutions states signatures max/row 4x4 2 9 6 2 5x5 16 73 26 8 6x6 722 514 107 40 7x7 33,988 2,870 411 152 8x8 2,215,764 13,485 1,411 596 9x9 179,431,924 56,375 4,510 1,983 10x10 17,849,077,140 218,038 13,453 5,672 11x11 2,138,979,146,276 801,266 38,314 14,491 12x12 304,243,884,374,412 2,847,885 104,764 35,803 13x13 50,702,643,217,809,908 9,901,431 278,561 96,414 14x14 9,789,567,606,147,948,364 33,911,578 723,306 238,359 15x15 2,168,538,331,223,656,364,084 114,897,838 1,845,861 548,409 16x16 546,386,962,452,256,865,969,596 ... 4,952,501 1,444,487 17x17 155,420,047,516,794,379,573,558,433 12,837,870 3,754,040 18x18 48,614,566,676,379,251,956,711,945,475 31,452,747 8,992,972 19x19 17,139,174,923,928,277,182,879,888,254,495 74,818,773 20,929,008 20x20 6,688,262,914,418,168,812,086,412,204,858,650 175,678,000 50,094,203(Additional results obtained with C++, using a simple 128-bit integer implementation. To count the states, the code had to be run using each state as a seperate signature, which I was unable to do for the largest sizes. )
EXAMPLE
The dictionary for a 5x5 matrix looks like this:
row 0: 00000 -> 16 row 3: 101 -> 0 1112 -> 1 row 1: 20002 -> 2 1121 -> 1 00202 -> 4 1+01 -> 0 02002 -> 2 11+12 -> 2 02020 -> 2 1+121 -> 1 0+1+1 -> 0 row 2: 10212 -> 1 1+112 -> 1 12012 -> 1 12021 -> 2 row 4: 0 -> 0 12102 -> 1 11 -> 0 21012 -> 0 12 -> 0 02121 -> 3 1+1 -> 1 01212 -> 1 1+2 -> 0The total number of solutions is 16; if we randomly pick a number from 0 to 15, e.g. 13, we can find the corresponding (i.e. the 14th) solution like this:
state: 00000 options: 10100 10010 10001 01010 01001 00101 signature: 00202 02002 20002 02020 02002 00202 solutions: 4 2 2 2 2 4This tells us that the 14th solution is the 2nd solution of option 00101. The next step is:
state: 00101 options: 10010 01010 signature: 12102 02121 solutions: 1 3This tells us that the 2nd solution is the 1st solution of option 01010. The next step is:
state: 01111 options: 10100 10001 00101 signature: 11+12 1112 1+01 solutions: 2 1 0This tells us that the 1st solution is the 1st solution of option 10100. The next step is:
state: 11211 options: 01010 01001 signature: 1+1 1+1 solutions: 1 1This tells us that the 1st solutions is the 1st solution of option 01010. The last step is:
state: 12221 options: 10001And the 5x5 matrix corresponding to randomly chosen number 13 is:
0 0 1 0 1 0 1 0 1 0 1 0 1 0 0 0 1 0 1 0 1 0 0 0 1
And here's a quick'n'dirty code example; run the snippet to generate the signature and solution count dictionary, and generate a random 10x10 matrix (it takes a second to generate the dictionary; once that is done, it generates random solutions in half a millisecond):
function signature(state, prev) { var zones = [], zone = []; for (var i = 0; i < state.length; i++) { if (state[i] == 2) { if (zone.length) zones.push(mirror(zone)); zone = []; } else if (prev[i]) zone.push(3); else zone.push(state[i]); } if (zone.length) zones.push(mirror(zone)); zones.sort(function(a,b) {return a.length - b.length || a - b;}); return zones.length ? zones.join("2") : "2"; function mirror(zone) { var ltr = zone.join(''); zone.reverse(); var rtl = zone.join(''); return (ltr < rtl) ? ltr : rtl; } } function memoize(n) { var memo = [], empty = []; for (var i = 0; i <= n; i++) memo[i] = []; for (var i = 0; i < n; i++) empty[i] = 0; memo[0][signature(empty, empty)] = next_row(empty, empty, 1); return memo; function next_row(state, prev, row) { if (row > n) return 1; var solutions = 0; for (var i = 0; i < n - 2; i++) { if (state[i] == 2 || prev[i] == 1) continue; for (var j = i + 2; j < n; j++) { if (state[j] == 2 || prev[j] == 1) continue; var s = state.slice(), p = empty.slice(); ++s[i]; ++s[j]; ++p[i]; ++p[j]; var sig = signature(s, p); var sol = memo[row][sig]; if (sol == undefined) memo[row][sig] = sol = next_row(s, p, row + 1); solutions += sol; } } return solutions; } } function random_matrix(n, memo) { var matrix = [], empty = [], state = [], prev = []; for (var i = 0; i < n; i++) empty[i] = state[i] = prev[i] = 0; var total = memo[0][signature(empty, empty)]; var pick = Math.floor(Math.random() * total); document.write("solution " + pick.toLocaleString('en-US') + " from a total of " + total.toLocaleString('en-US') + "<br>"); for (var row = 1; row <= n; row++) { var options = find_options(state, prev); for (var i in options) { var state_copy = state.slice(); for (var j in state_copy) state_copy[j] += options[i][j]; var sig = signature(state_copy, options[i]); var solutions = memo[row][sig]; if (pick < solutions) { matrix.push(options[i].slice()); prev = options[i].slice(); state = state_copy.slice(); break; } else pick -= solutions; } } return matrix; function find_options(state, prev) { var options = []; for (var i = 0; i < n - 2; i++) { if (state[i] == 2 || prev[i] == 1) continue; for (var j = i + 2; j < n; j++) { if (state[j] == 2 || prev[j] == 1) continue; var option = empty.slice(); ++option[i]; ++option[j]; options.push(option); } } return options; } } var size = 10; var memo = memoize(size); var matrix = random_matrix(size, memo); for (var row in matrix) document.write(matrix[row] + "<br>");The code snippet below shows the dictionary of signatures and solution counts for a matrix of size 10x10. I've used a slightly different signature format from the explanation above: the zones are delimited by a '2' instead of a plus sign, and a column which has a one in the previous row is marked with a '3' instead of a '2'. This shows how the keys could be stored in a file as integers with 2×N bits (padded with 2's).
function signature(state, prev) { var zones = [], zone = []; for (var i = 0; i < state.length; i++) { if (state[i] == 2) { if (zone.length) zones.push(mirror(zone)); zone = []; } else if (prev[i]) zone.push(3); else zone.push(state[i]); } if (zone.length) zones.push(mirror(zone)); zones.sort(function(a,b) {return a.length - b.length || a - b;}); return zones.length ? zones.join("2") : "2"; function mirror(zone) { var ltr = zone.join(''); zone.reverse(); var rtl = zone.join(''); return (ltr < rtl) ? ltr : rtl; } } function memoize(n) { var memo = [], empty = []; for (var i = 0; i <= n; i++) memo[i] = []; for (var i = 0; i < n; i++) empty[i] = 0; memo[0][signature(empty, empty)] = next_row(empty, empty, 1); return memo; function next_row(state, prev, row) { if (row > n) return 1; var solutions = 0; for (var i = 0; i < n - 2; i++) { if (state[i] == 2 || prev[i] == 1) continue; for (var j = i + 2; j < n; j++) { if (state[j] == 2 || prev[j] == 1) continue; var s = state.slice(), p = empty.slice(); ++s[i]; ++s[j]; ++p[i]; ++p[j]; var sig = signature(s, p); var sol = memo[row][sig]; if (sol == undefined) memo[row][sig] = sol = next_row(s, p, row + 1); solutions += sol; } } return solutions; } } var memo = memoize(10); for (var i in memo) { document.write("row " + i + ":<br>"); for (var j in memo[i]) { document.write(""" + j + "": " + memo[i][j] + "<br>"); } }讨论(0) -
Just few thoughts. Number of matrices satisfying conditions for n <= 10:
3 0 4 2 5 16 6 722 7 33988 8 2215764 9 179431924 10 17849077140Unfortunatelly there is no sequence with these numbers in OEIS.
There is one similar (A001499), without condition for neighbouring one's. Number of nxn matrices in this case is 'of order' as A001499's number of (n-1)x(n-1) matrices. That is to be expected since number of ways to fill one row in this case, position 2 one's in n places with at least one zero between them is ((n-1) choose 2). Same as to position 2 one's in (n-1) places without the restriction.
I don't think there is an easy connection between these matrix of order n and A001499 matrix of order n-1, meaning that if we have A001499 matrix than we can construct some of these matrices.
With this, for n=20, number of matrices is >10^30. Quite a lot :-/
讨论(0) -
Intro
Here is some prototype-approach, trying to solve the more general task of uniform combinatorial sampling, which for our approach here means: we can use this approach for everything which we can formulate as SAT-problem.
It's not exploiting your problem directly and takes a heavy detour. This detour to the SAT-problem can help in regards to theory (more powerful general theoretical results) and efficiency (SAT-solvers).
That being said, it's not an approach if you want to sample within seconds or less (in my experiments), at least while being concerned about uniformity.
Theory
The approach, based on results from complexity-theory, follows this work:
GOMES, Carla P.; SABHARWAL, Ashish; SELMAN, Bart. Near-uniform sampling of combinatorial spaces using XOR constraints. In: Advances In Neural Information Processing Systems. 2007. S. 481-488.
The basic idea:
- formulate the problem as SAT-problem
- add randomly generated xors to the problem (acting on the decision-variables only! that's important in practice)
- this will reduce the number of solutions (some solutions will get impossible)
- do that in a loop (with tuned parameters) until only one solution is left!
- search for some solution is being done by SAT-solvers or #SAT-solvers (=model-counting)
- if there is more than one solution: no xors will be added but a complete restart will be done: add random-xors to the start-problem!
The guarantees:
- when tuning the parameters right, this approach achieves near-uniform sampling
- this tuning can be costly, as it's based on approximating the number of possible solutions
- empirically this can also be costly!
- Ante's answer, mentioning the number sequence A001499 actually gives a nice upper bound on the solution-space (as it's just ignoring adjacency-constraints!)
The drawbacks:
- inefficient for large problems (in general; not necessarily compared to the alternatives like MCMC and co.)
- need to change / reduce parameters to produce samples
- those reduced parameters lose the theoretical guarantees
- but empirically: good results are still possible!
Parameters:
In practice, the parameters are:
- N: number of xors added
- L: minimum number of variables part of one xor-constraint
- U: maximum number of variables part of one xor-constraint
N is important to reduce the number of possible solutions. Given N constant, the other variables of course also have some effect on that.
Theory says (if i interpret correctly), that we should use L = R = 0.5 * #dec-vars.
This is impossible in practice here, as xor-constraints hurt SAT-solvers a lot!
Here some more scientific slides about the impact of L and U.
They call xors of size 8-20 short-XORS, while we will need to use even shorter ones later!
Implementation
Final version
Here is a pretty hacky implementation in python, using the XorSample scripts from here.
The underlying SAT-solver in use is Cryptominisat.
The code basically boils down to:
- Transform the problem to conjunctive normal-form
- as DIMACS-CNF
- Implement the sampling-approach:
- Calls XorSample (pipe-based + file-based)
- Call SAT-solver (file-based)
- Add samples to some file for later analysis
Code: (i hope i did warn you already about the code-quality)
from itertools import count from time import time import subprocess import numpy as np import os import shelve import uuid import pickle from random import SystemRandom cryptogen = SystemRandom() """ Helper functions """ # K-ARY CONSTRAINT GENERATION # ########################### # SINZ, Carsten. Towards an optimal CNF encoding of boolean cardinality constraints. # CP, 2005, 3709. Jg., S. 827-831. def next_var_index(start): next_var = start while(True): yield next_var next_var += 1 class s_index(): def __init__(self, start_index): self.firstEnvVar = start_index def next(self,i,j,k): return self.firstEnvVar + i*k +j def gen_seq_circuit(k, input_indices, next_var_index_gen): cnf_string = '' s_index_gen = s_index(next_var_index_gen.next()) # write clauses of first partial sum (i.e. i=0) cnf_string += (str(-input_indices[0]) + ' ' + str(s_index_gen.next(0,0,k)) + ' 0\n') for i in range(1, k): cnf_string += (str(-s_index_gen.next(0, i, k)) + ' 0\n') # write clauses for general case (i.e. 0 < i < n-1) for i in range(1, len(input_indices)-1): cnf_string += (str(-input_indices[i]) + ' ' + str(s_index_gen.next(i, 0, k)) + ' 0\n') cnf_string += (str(-s_index_gen.next(i-1, 0, k)) + ' ' + str(s_index_gen.next(i, 0, k)) + ' 0\n') for u in range(1, k): cnf_string += (str(-input_indices[i]) + ' ' + str(-s_index_gen.next(i-1, u-1, k)) + ' ' + str(s_index_gen.next(i, u, k)) + ' 0\n') cnf_string += (str(-s_index_gen.next(i-1, u, k)) + ' ' + str(s_index_gen.next(i, u, k)) + ' 0\n') cnf_string += (str(-input_indices[i]) + ' ' + str(-s_index_gen.next(i-1, k-1, k)) + ' 0\n') # last clause for last variable cnf_string += (str(-input_indices[-1]) + ' ' + str(-s_index_gen.next(len(input_indices)-2, k-1, k)) + ' 0\n') return (cnf_string, (len(input_indices)-1)*k, 2*len(input_indices)*k + len(input_indices) - 3*k - 1) # K=2 clause GENERATION # ##################### def gen_at_most_2_constraints(vars, start_var): constraint_string = '' used_clauses = 0 used_vars = 0 index_gen = next_var_index(start_var) circuit = gen_seq_circuit(2, vars, index_gen) constraint_string += circuit[0] used_clauses += circuit[2] used_vars += circuit[1] start_var += circuit[1] return [constraint_string, used_clauses, used_vars, start_var] def gen_at_least_2_constraints(vars, start_var): k = len(vars) - 2 vars = [-var for var in vars] constraint_string = '' used_clauses = 0 used_vars = 0 index_gen = next_var_index(start_var) circuit = gen_seq_circuit(k, vars, index_gen) constraint_string += circuit[0] used_clauses += circuit[2] used_vars += circuit[1] start_var += circuit[1] return [constraint_string, used_clauses, used_vars, start_var] # Adjacency conflicts # ################### def get_all_adjacency_conflicts_4_neighborhood(N, X): conflicts = set() for x in range(N): for y in range(N): if x < (N-1): conflicts.add(((x,y),(x+1,y))) if y < (N-1): conflicts.add(((x,y),(x,y+1))) cnf = '' # slow string appends for (var_a, var_b) in conflicts: var_a_ = X[var_a] var_b_ = X[var_b] cnf += '-' + var_a_ + ' ' + '-' + var_b_ + ' 0 \n' return cnf, len(conflicts) # Build SAT-CNF ############# def build_cnf(N, verbose=False): var_counter = count(1) N_CLAUSES = 0 X = np.zeros((N, N), dtype=object) for a in range(N): for b in range(N): X[a,b] = str(next(var_counter)) # Adjacency constraints CNF, N_CLAUSES = get_all_adjacency_conflicts_4_neighborhood(N, X) # k=2 constraints NEXT_VAR = N*N+1 for row in range(N): constraint_string, used_clauses, used_vars, NEXT_VAR = gen_at_most_2_constraints(X[row, :].astype(int).tolist(), NEXT_VAR) N_CLAUSES += used_clauses CNF += constraint_string constraint_string, used_clauses, used_vars, NEXT_VAR = gen_at_least_2_constraints(X[row, :].astype(int).tolist(), NEXT_VAR) N_CLAUSES += used_clauses CNF += constraint_string for col in range(N): constraint_string, used_clauses, used_vars, NEXT_VAR = gen_at_most_2_constraints(X[:, col].astype(int).tolist(), NEXT_VAR) N_CLAUSES += used_clauses CNF += constraint_string constraint_string, used_clauses, used_vars, NEXT_VAR = gen_at_least_2_constraints(X[:, col].astype(int).tolist(), NEXT_VAR) N_CLAUSES += used_clauses CNF += constraint_string # build final cnf CNF = 'p cnf ' + str(NEXT_VAR-1) + ' ' + str(N_CLAUSES) + '\n' + CNF return X, CNF, NEXT_VAR-1 # External tools # ############## def get_random_xor_problem(CNF_IN_fp, N_DEC_VARS, N_ALL_VARS, s, min_l, max_l): # .cnf not part of arg! p = subprocess.Popen(['./gen-wff', CNF_IN_fp, str(N_DEC_VARS), str(N_ALL_VARS), str(s), str(min_l), str(max_l), 'xored'], stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE) result = p.communicate() os.remove(CNF_IN_fp + '-str-xored.xor') # file not needed return CNF_IN_fp + '-str-xored.cnf' def solve(CNF_IN_fp, N_DEC_VARS): seed = cryptogen.randint(0, 2147483647) # actually no reason to do it; but can't hurt either p = subprocess.Popen(["./cryptominisat5", '-t', '4', '-r', str(seed), CNF_IN_fp], stdin=subprocess.PIPE, stdout=subprocess.PIPE) result = p.communicate()[0] sat_line = result.find('s SATISFIABLE') if sat_line != -1: # solution found! vars = parse_solution(result)[:N_DEC_VARS] # forbid solution (DeMorgan) negated_vars = list(map(lambda x: x*(-1), vars)) with open(CNF_IN_fp, 'a') as f: f.write( (str(negated_vars)[1:-1] + ' 0\n').replace(',', '')) # assume solve is treating last constraint despite not changing header! # solve again seed = cryptogen.randint(0, 2147483647) p = subprocess.Popen(["./cryptominisat5", '-t', '4', '-r', str(seed), CNF_IN_fp], stdin=subprocess.PIPE, stdout=subprocess.PIPE) result = p.communicate()[0] sat_line = result.find('s SATISFIABLE') if sat_line != -1: os.remove(CNF_IN_fp) # not needed anymore return True, False, None else: return True, True, vars else: return False, False, None def parse_solution(output): # assumes there is one vars = [] for line in output.split("\n"): if line: if line[0] == 'v': line_vars = list(map(lambda x: int(x), line.split()[1:])) vars.extend(line_vars) return vars # Core-algorithm # ############## def xorsample(X, CNF_IN_fp, N_DEC_VARS, N_VARS, s, min_l, max_l): start_time = time() while True: # add s random XOR constraints to F xored_cnf_fp = get_random_xor_problem(CNF_IN_fp, N_DEC_VARS, N_VARS, s, min_l, max_l) state_lvl1, state_lvl2, var_sol = solve(xored_cnf_fp, N_DEC_VARS) print('------------') if state_lvl1 and state_lvl2: print('FOUND') d = shelve.open('N_15_70_4_6_TO_PLOT') d[str(uuid.uuid4())] = (pickle.dumps(var_sol), time() - start_time) d.close() return True else: if state_lvl1: print('sol not unique') else: print('no sol found') print('------------') """ Run """ N = 15 N_DEC_VARS = N*N X, CNF, N_VARS = build_cnf(N) with open('my_problem.cnf', 'w') as f: f.write(CNF) counter = 0 while True: print('sample: ', counter) xorsample(X, 'my_problem', N_DEC_VARS, N_VARS, 70, 4, 6) counter += 1Output will look like (removed some warnings):
------------ no sol found ------------ ------------ no sol found ------------ ------------ no sol found ------------ ------------ sol not unique ------------ ------------ FOUNDCore: CNF-formulation
We introduce one variable for every cell of the matrix. N=20 means 400 binary-variables.
Adjancency:
Precalculate all symmetry-reduced conflicts and add conflict-clauses.
Basic theory:
a -> !b <-> !a v !b (propositional logic)Row/Col-wise Cardinality:
This is tough to express in CNF and naive approaches need an exponential number of constraints.
We use some adder-circuit based encoding (SINZ, Carsten. Towards an optimal CNF encoding of boolean cardinality constraints) which introduces new auxiliary-variables.
Remark:
sum(var_set) <= k <-> sum(negated(var_set)) >= len(var_set) - kThese SAT-encodings can be put into exact model-counters (for small N; e.g. < 9). The number of solutions equals Ante's results, which is a strong indication for a correct transformation!
There are also interesting approximate model-counters (also heavily based on xor-constraints) like approxMC which shows one more thing we can do with the SAT-formulation. But in practice i have not been able to use these (approxMC = autoconf; no comment).
Other experiments
I did also build a version using pblib, to use more powerful cardinality-formulations for the SAT-CNF formulation. I did not try to use the C++-based API, but only the reduced pbencoder, which automatically selects some best encoding, which was way worse than my encoding used above (which is best is still a research-problem; often even redundant-constraints can help).
Empirical analysis
For the sake of obtaining some sample-size (given my patience), i only computed samples for N=15. In this case we used:
- N=70 xors
- L,U = 4,6
I also computed some samples for N=20 with (100,3,6), but this takes a few mins and we reduced the lower bound!
Visualization
Here some animation (strengthening my love-hate relationship with matplotlib):
Edit: And a (reduced) comparison to brute-force uniform-sampling with N=5 (NXOR,L,U = 4, 10, 30):
(I have not yet decided on the addition of the plotting-code. It's as ugly as the above one and people might look too much into my statistical shambles; normalizations and co.)
Theory
Statistical analysis is probably hard to do as the underlying problem is of such combinatoric nature. It's even not entirely obvious how that final cell-PDF should look like. In the case of N=odd, it's probably non-uniform and looks like a chess-board (i did brute-force check N=5 to observe this).
One thing we can be sure about (imho): symmetry!
Given a cell-PDF matrix, we should expect, that the matrix is symmetric (A = A.T). This is checked in the visualization and the euclidean-norm of differences over time is plotted.
We can do the same on some other observation: observed pairings.
For N=3, we can observe the following pairs:
- 0,1
- 0,2
- 1,2
Now we can do this per-row and per-column and should expect symmetry too!
Sadly, it's probably not easy to say something about the variance and therefore the needed samples to speak about confidence!
Observation
According to my simplified perception, current-samples and the cell-PDF look good, although convergence is not achieved yet (or we are far away from uniformity).
The more important aspect are probably the two norms, nicely decreasing towards 0. (yes; one could tune some algorithm for that by transposing with prob=0.5; but this is not done here as it would defeat it's purpose).
Potential next steps
- Tune parameters
- Check out the approach using #SAT-solvers / Model-counters instead of SAT-solvers
- Try different CNF-formulations, especially in regards to cardinality-encodings and xor-encodings
- XorSample is by default using tseitin-like encoding to get around exponentially grow
- for smaller xors (as used) it might be a good idea to use naive encoding (which propagates faster)
- XorSample supports that in theory; but the script's work differently in practice
- Cryptominisat is known for dedicated XOR-handling (as it was build for analyzing cryptography including many xors) and might gain something by naive encoding (as inferring xors from blown-up CNFs is much harder)
- for smaller xors (as used) it might be a good idea to use naive encoding (which propagates faster)
- XorSample is by default using tseitin-like encoding to get around exponentially grow
- More statistical-analysis
- Get rid of XorSample scripts (shell + perl...)
Summary
- The approach is very general
- This code produces feasible samples
- It should be not hard to prove, that every feasible solution can be sampled
- Others have proven theoretical guarantees for uniformity for some params
- does not hold for our params
- Others have empirically / theoretically analyzed smaller parameters (in use here)
讨论(0)
加载中...
- 热议问题