pentaho

I'm partitioning a data frame with split() in order to use parLapply() to call a function on each partition in parallel. The data frame has 1.3 million rows and 20 cols. I'm splitting/partitioning by two columns, both character type. Looks like there are ~47K unique IDs and ~12K unique codes, but not every pairing of ID and code are matched. The resulting number of partitions is ~250K. Here is the split() line: system.time(pop_part <- split(pop, list(pop$ID, pop$code))) The partitions will then be fed into parLapply() as follows: cl <- makeCluster(detectCores()) system.time(par_pop <-

I am trying to add a new data type (Geometry) to Kettle. I have added a new Value type to org.pentaho.di.compatibility. I have added a ValueGeometry class and made the necessary modifications to ValueInterface and Value. The code compiles but the new data type does not show up in plugins like Select. What am I missing here? Also I'd appreciate if you could point me towards the source code for these plugins. Thanks. As of Kettle 5.0, it is possible to create a plugin to provide new Value types: http://jira.pentaho.com/browse/PDI-191 I have a plugin to add a key/value type (like java.util.Map):

I have a requirement such that whenever i run my Kettle job, the database connection parameters must be taken dynamically from an excel source on each run. Say i have an excel with column names : HostName, Username, Database, Password. i want to pass these connection parameters to my table input step dynamically whenever the job runs. This is what i was trying to do. You can achieve this by reading the DB connection parameters from a source (e.g. Excel or in my example a CSV file) storing the parameters in variables using the variables in your connection setting. Proceed as follows Create

Need to load data from a single file with a 100,000+ records into multiple tables on MySQL maintaining the relationships defined in the file/tables; meaning the relationships already match. The solution should work on the latest version of MySQL, and needs to use the InnoDB engine; MyISAM does not support foreign keys. I am a completely new to using Pentaho Data Integration (aka Kettle) and any pointers would be appreciated. I might add that it is a requirement that the foreign key constraints are NOT disabled. Since it's my understanding that if there is something wrong with the database's

Hi I am loading table A data from sql server to mysql using pentaho when loading data i need to get only last 7 days data from sql server A table to mysql In sql server createddate column data type is like datetime AND In mysql created_on column datatype is timestamp Here I used below query but i am getting only 5 days data Please help me in this issue select id, NewsHeadline as news_headline, NewsText as news_text, state, CreatedDate as created_on from News WHERE CreatedDate BETWEEN GETDATE()-7 AND GETDATE() order by createddate DESC Try something like: SELECT id, NewsHeadline as news