Dynamic Table Generation

Question

First let me describe my situation so that you might be able to help me better. There are two parts.

1: I have a program that runs and analyzes a bunch of files. I

Answer 1

Automatically creating tables could give you headaches if the number of tables becomes vast. Directory listings in ext3 for dirs with over 5000 items start to feel expensive, and certainly are rather time consuming when you get over 100,000 files. (The mysql CLI will attempt to cache all table names when it connects, and to skip that, you need to connect with the -A switch.)

You might consider using temporary tables for the creation of the report, and then possibly condensing the results down into a report string for retrieval later. Or, as JP mentions, a table structure that tags values and reports in the same table:

create table stddev (
    report_id int(11),
    field_name int(11), -- fk to field
    field_value double
);
create table reports (
    report_id int(11);
    report_name varchar(255);
);

And to get data for just one report, you would do a select specifying the report_id:

select * from stddev where report_id = 123;

I would create tables for your report names, field names, and you probably want to separate input values from derived/computed values saved for your reports.

Depending on how often you input new data, I wouldn't prematurely optimize from a few large tables to lots of small tables. Properly indexed, a large table can perform well.

However, what is the magnitude of data this application is processing? Were there reasons for using so many small tables to begin with?

I use PHP to process a lot of data. If your schema make sense, it makes the PHP code make more sense, too. If it were me, I wouldn't switch to a different programming language, I would stick with what I started until I came upon a real structural limitation of the language; for me switching to Rails would be a waste of time.