How do I properly manage a neo4j-community-3.1.0 lucene legacy index? (really weird issues)

Question

I am having weird issues with Neo4j's legacy indexing, and got stuck today. I need full text support, as I wish to run a performance comparison against Solr (which uses Lucene full text) to see how the different data model compares.

I have been following a few guides online, as well as various posts around here on stack.

I had success up until yesterday, where all of a sudden I had corrupted index files, as range queries were returning invalid and inconsistent results. So I am trying to set in stone exactly the steps I need to take.

I use the CSV bulk import tool to populate my database with about 4 million nodes with the label "record", and various nodes with labels like "data:SPD", "data:DIR", "data:TS", etc (using 2 labels, to represent that they are ocean data nodes, for different types of measurements).

The data model is simple. I have:

(r:record {meta:M, time:T, lat:L1, lon:L2})-[:measures]-(d:data {value:V})

M is a ID-like string which I use to keep track of my data internally for testing purposes. T is an epoch time integer. L1 / L2 are geo-spatial coordinate floats. My data nodes represent various kinds of collected data and not all records have the same data nodes. (Some have temperatures, wind speeds, wind directions, sea temperatures, etc). These values are all represented as floats. Each data node has a second label that says what kind of data it contains.

After I complete the import, I open up the shell and execute the following sequence:

index --create node_auto_index -t Node
index --set-config node_auto_index fulltext

I have the following configuration added to the default neo4j.conf file (this is there even before the CSV bulk import happens):

dbms.auto_index.nodes.enabled=true
dbms.auto_index.nodes.keys=meta,lat,lon,time

Before today, I would see that the fulltext command indeed worked by querying the shell:

index --get-config node_auto_index

returned something like:

{
    "provider": "lucene",
    "type": "fulltext"
}

I ran a series of tests on my data using the MATCH clause recently. I understand that this uses the more modern, schema indexing. My results were fine, and returned the expected data.

I read somewhere that since my data was imported prior to legacy index creation, I needed to manually index the relevant properties by doing something like this:

START n=node(*)
WITH n SKIP {curr_skip} LIMIT {fixed_lim}
WHERE EXISTS(n.meta)
SET n.time=n.time, n.lat=n.lat, n.lon=n.lon, n.meta=n.meta
RETURN n

Since I have 4 million records, my python handler does this as a series of batch operations by upping the {curr_skip} each time by {fixed_lim} and executing the query until I get 0 results.

Upon transitioning to my tests which involve the START clause yesterday, I found that using a lucene query like:

START r=node:node_auto_index(lon:[{} TO {}]) RETURN count(r)

(with a filled in range) was giving me bad results. Data that I expected to be returned was not. Furthermore, different ranges were yielding strange results such. Range (a,b) might yield 1000 results, but (a-e,b+e), a superset of the previous range would yield 0 results!! However, the exact same style of queries on time and lat seemed to be working perfectly! Even more so, I could do a multi facted query like:

START r=node:node_auto_index(time:[{} TO {}] AND lat:[{} TO {}]) RETURN count(r)

My best guess, was that somehow I corrupted the index files for lon.

The recommendations I have found online are to stop the database, go to /path/to/graph.db, and remove all of index*, and restart the database. Upon following these instructions today, I have discovered more weird behavior. I re-exceuted the same index creation / configuration statements from above, but after querying the configuration, find that the index type remains as an "type": "exact". Even stranger, is that the index files are not actually being created! There is no index directory being created under path/to/graph.db.

I am certain I have started the shell correctly by using:

neo4j-shell -path /path/to/graph.db/

If I try use index --create node_auto_index -t Node, I get an already exists notification, when it clearly does not.

---

For now, I think I am just going to start from scratch again and see if I can either reproduce these errors, or somehow bypass them.

Otherwise, if anyone with experience here has any idea of what might be going wrong, I would greatly appreciate some input!

---

UPDATE:

So I went ahead and started from scratch.

# ran my bulk import code
python3
>>> from mylib.module import load_data()
>>> load_data()
>>> # ... lots of printed stuff ...
IMPORT DONE in 3m 37s 950ms.
Imported:
  15394183 nodes
  15394171 relationships
  27651625 properties
Peak memory usage: 361.94 MB
>>> exit()

# switched out my new database
cd /path/to/neo4j-community-3.1.0
mv data/databases/graph.db data/databases/oldgraph.db
mv data/databases/newgraph.db data/databases/graph.db

# check neo4j is off
ps aux | grep neo

# neo4j shell commands
bin/neo4j-shell -path data/databases/graph.db/
... some warning about GraphAware Runtime disbaled.
... the welcome message

neo4j-sh (?)$ index --create node_auto_index -t Node
neo4j-sh (?)$ index --set-config node_auto_index fulltext
INDEX CONFIGURATION CHANGED, INDEX DATA MAY BE INVALID
neo4j-sh (?)$ index --get-config node_auto_index -t Node
{
    "provider": "lucene",
    "type": "exact"
}
neo4j-sh (?)$ exit # thought maybe I just had to restart

# try again
bin/neo4j-shell -path data/databases/graph.db/
neo4j-sh (?)$ index --get-config node_auto_index -t Node
{
    "provider": "lucene",
    "type": "exact"
}
neo4j-sh (?)$ index --set-config node_auto_index fulltext
INDEX CONFIGURATION CHANGED, INDEX DATA MAY BE INVALID
neo4j-sh (?)$ index --get-config node_auto_index -t Node
{
    "provider": "lucene",
    "type": "exact"
}
# hmmmmm
neo4j-sh (?)$ index --create node_auto_index -t Node
Class index 'node_auto_index' alredy exists
# sanity check
neo4j-sh (?)$ MATCH (r:record) RETURN count(r);
+----------+
| count(r) |
+----------+
| 4085814  |
+----------+
1 row
470 ms
neo4j-sh (?)$ exit

As you can see, even after recreating a fresh database, I am not able to activate a fulltext index now. I have no idea why it worked a few days prior and not now, as I am the only one who is working on this server! Perhaps I will have to even reinstall neo4j as a whole.

---

UPDATE / IDEA:

Ok I have a potential idea as to my problem, and I think it may be permissions related. I have a dashboard.py module which I have been using to orchestrate turning on/off solr and neo4j. The other day, I had some weird issues with not being able to execute the start/stop sequences from within my shell, so I messed with a lot of permissions.

Lets call me userA. I belong to groups groupA, groupB.

I remember running the following yesterday:

sudo chown -R $USER:groupB neo4j-community-3.1.0

I have noticed that all of the new database files my python scripts are producing belong to group groupA. Could this be the culprit?

I am having the weird error again where I can't recreate the index because it thinks it still exists after i deleted it. I am going to rerun the bulk import once again, and fix these permissions prior trying to set the full text index. Will update tonight.

EDIT:

This did not seem to have an effect :( I even tried chowning everything to root, both user and group to no avail. My lucene index will not change from exact to fulltext.

I am going to go ahead and do a full reinstall of everything now.

---

UPDATE:

Not even a full reinstall has worked.

I removed my entire neo4j-community-3.1.0 folder, and unpacked the tarball I had. I set ownership of the entire folder to my own (because it was nfsnobody previously):

chown -R $USER:mygroup neo4j-community-3.1.0 

I added the two lines to neo4j.conf:

dbms.auto_index.nodes.enabled=true
dbms.auto_index.nodes.keys=meta,lat,lon,time

I imported the data via bulk import tool, then did the same index creation / configuration commands as before. The index is still reporting that it is using an exact lucene index after it tells me the configuration changed.

I am at an utter loss here. Maybe I will just go ahead and try the START clauses tests I have anyways and see if they work.

---

UPDATE:

WOOOOW. I figured out my exact->fulltext issue!!!

The command:

index --set-config node_auto_index fulltext

needed to be:

index --set-config node_auto_index type fulltext

Incredible. What a doozy. The output message about the index being changed is really what threw me off, thinking that the command was being run correctly and that some other problem was at hand. Should I throw in a request on git for this? Is this command actually changing the index at all if I don't include type?

As for the invalid range queries, I am going to test this further soon. I believe that when I ran some code the first time around, I had a bug in my python handler that didn't loop over all the results, effectively missing out on some node's during manual indexing. Once I finish this process again, I will run my tests to check the results.