Composite key in Cassandra with Pig

问题

We have a CQL table that looks something like this:

CREATE table data (
  occurday  text,
  seqnumber int,
  occurtimems bigint,
  unique bigint,

  fields map<text, text>,

  primary key ((occurday, seqnumber), occurtimems, unique)
)

I can query this table from cqlsh like this:

select * from data where seqnumber = 10 AND occurday = '2013-10-01';

This query works and returns the expected data.

If I execute this query as part of a LOAD from within Pig, however, things don't work.

-- Need to URL encode the query
data = LOAD 'cql://ks/data?where_clause=seqnumber%3D10%20AND%20occurday%3D%272013-10-01%27' USING CqlStorage();

gives

InvalidRequestException(why:seqnumber cannot be restricted by more than one relation if it includes an Equal)
at org.apache.cassandra.thrift.Cassandra$prepare_cql3_query_result.read(Cassandra.java:39567)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:78)
at org.apache.cassandra.thrift.Cassandra$Client.recv_prepare_cql3_query(Cassandra.java:1625)
at org.apache.cassandra.thrift.Cassandra$Client.prepare_cql3_query(Cassandra.java:1611)
at org.apache.cassandra.hadoop.cql3.CqlPagingRecordReader$RowIterator.prepareQuery(CqlPagingRecordReader.java:591)
at org.apache.cassandra.hadoop.cql3.CqlPagingRecordReader$RowIterator.executeQuery(CqlPagingRecordReader.java:621)

Shouldn't these behave the same? Why is the version through Pig failing where the straight cqlsh command works?

回答1:

Hadoop is using CqlPagingRecordReader to try to load your data. This is leading to queries that are not identical to what you have entered. The paging record reader is trying to obtain small slices of Cassandra data at a time to avoid timeouts.

This means that your query is executed as

SELECT * FROM "data" WHERE token("occurday","seqnumber") > ? AND
token("occurday","seqnumber") <= ? AND occurday='A Great Day' 
AND seqnumber=1 LIMIT 1000 ALLOW FILTERING

And this is why you are seeing your repeated key error. I'll submit a bug to the Cassandra Project.

Jira: https://issues.apache.org/jira/browse/CASSANDRA-6151

来源：https://stackoverflow.com/questions/19189649/composite-key-in-cassandra-with-pig

标签

cassandra

apache-pig