1. 技术文章
  2. How can I make my GTFS queries run faster?

How can I make my GTFS queries run faster?

由 倖福魔咒の 提交于 2019-12-04 13:35:09

The short answer is: Use table joins and indices.

Here's the longer answer:

You have the right idea here and your understanding of how the tables relate to one another is correct. However, by asking the DBMS to match field values from a list (using WHERE...IN) rather than joining tables together you are requiring it do a lot more work than it needs to.

What you really want to do is execute all this as a single query, using JOIN clauses to link the tables together. Try this, which additionally joins the calendars and calendar_dates tables to limit the results to only routes actually operating today:

SELECT DISTINCT r.id, r.route_long_name
  FROM (SELECT s.stop_id, (6371000 *
          acos(cos(radians(48.824699)) * cos(radians(s.stop_lat)) *
          cos(radians(2.3243) - radians(s.stop_lon)) +
          sin(radians(48.824699)) * sin(radians(s.stop_lat)))) AS distance
          FROM stops AS s) AS i_s
  INNER JOIN stop_times AS st ON st.stop_id = i_s.stop_id
  INNER JOIN (SELECT trip_id, route_id FROM trips AS t
                INNER JOIN (SELECT service_id FROM calendars
                              WHERE start_date <= '2014-09-09'
                                AND end_date >= '2014-09-09'
                                AND tuesday = 1
                              UNION
                                SELECT service_id FROM calendar_dates
                                  WHERE date = '2014-09-09'
                                    AND exception_type = 1
                              EXCEPT
                                SELECT service_id FROM calendar_dates
                                  WHERE date = '2014-09-09'
                                    AND exception_type = 2) AS c
                   ON c.service_id = t.service_id) AS t_r
    ON t_r.trip_id = st.trip_id
  INNER JOIN routes AS r ON r.route_id = t_r.route_id
  WHERE st.departure_time > '$now'
    AND i_s.distance < 200;

Here INNER JOIN is used to "add in" the columns of another table, including only those rows that match the condition in the ON clause. This should be much faster than generating a list of results with one query and then feeding it in to the next.

To get even better performance, though, you will want to create indices that prevent the DBMS having to scan linearly through tables. The rule of thumb is to have an index defined for each column used in either a JOIN or a WHERE clause. Here are the indices I defined, which you should find make the above query perform quite well:

CREATE INDEX calendar_dates_date_exception_type_service_id_index
  ON calendar_dates (date, exception_type, service_id);

CREATE INDEX trips_service_id_trip_id_route_id_index
  ON trips (service_id, trip_id, route_id);

CREATE INDEX stop_times_trip_id_departure_time_stop_id_index
  ON stop_times (trip_id, departure_time, stop_id);

CREATE INDEX routes_route_id_index ON routes (route_id);

CREATE INDEX stops_stop_id_index ON stops (stop_id);

The table schema I used was plain wrong, I should've built it myself or at least analyze it before using it.

Here's an updated schema:

CREATE TABLE `agency` (
    transit_system VARCHAR(50) NOT NULL,
    agency_id VARCHAR(100),
    agency_name VARCHAR(255) NOT NULL,
    agency_url VARCHAR(255) NOT NULL,
    agency_timezone VARCHAR(100) NOT NULL,
    agency_lang VARCHAR(100),
    agency_phone VARCHAR(100),
    agency_fare_url VARCHAR(100),
    PRIMARY KEY (agency_id)
);

CREATE TABLE `calendar_dates` (
    id INT(12) NOT NULL PRIMARY KEY AUTO_INCREMENT,
    transit_system VARCHAR(50) NOT NULL,
    service_id VARCHAR(255) NOT NULL,
    `date` VARCHAR(8) NOT NULL,
    exception_type TINYINT(2) NOT NULL,
    KEY `service_id` (service_id),
    KEY `exception_type` (exception_type)    
);

CREATE TABLE `calendar` (
    id INT(12) NOT NULL PRIMARY KEY AUTO_INCREMENT,
    transit_system VARCHAR(50) NOT NULL,
    service_id VARCHAR(255) NOT NULL,
    monday TINYINT(1) NOT NULL,
    tuesday TINYINT(1) NOT NULL,
    wednesday TINYINT(1) NOT NULL,
    thursday TINYINT(1) NOT NULL,
    friday TINYINT(1) NOT NULL,
    saturday TINYINT(1) NOT NULL,
    sunday TINYINT(1) NOT NULL,
    start_date VARCHAR(8) NOT NULL, 
    end_date VARCHAR(8) NOT NULL,
    KEY `service_id` (service_id)
);

CREATE TABLE `fare_attributes` (
    id INT(12) NOT NULL PRIMARY KEY AUTO_INCREMENT,
    transit_system VARCHAR(50) NOT NULL,
    fare_id VARCHAR(100),
    price VARCHAR(50) NOT NULL,
    currency_type VARCHAR(50) NOT NULL,
    payment_method TINYINT(1) NOT NULL,
    transfers TINYINT(1) NOT NULL,
    transfer_duration VARCHAR(10),
    exception_type TINYINT(2) NOT NULL,
    agency_id INT(100),
    KEY `fare_id` (fare_id)
);

CREATE TABLE `fare_rules` (
    id INT(12) NOT NULL PRIMARY KEY AUTO_INCREMENT,
    transit_system VARCHAR(50) NOT NULL,
    fare_id VARCHAR(100),
    route_id VARCHAR(100),
    origin_id VARCHAR(100),
    destination_id VARCHAR(100),
    contains_id VARCHAR(100),
    KEY `fare_id` (fare_id),
    KEY `route_id` (route_id)
);

CREATE TABLE `feed_info` (
    id INT(12) NOT NULL PRIMARY KEY AUTO_INCREMENT,
    transit_system VARCHAR(50) NOT NULL,
    feed_publisher_name VARCHAR(100),
    feed_publisher_url VARCHAR(255) NOT NULL,
    feed_lang VARCHAR(255) NOT NULL,
    feed_start_date VARCHAR(8),
    feed_end_date VARCHAR(8),
    feed_version VARCHAR(100)
);

CREATE TABLE `frequencies` (
    id INT(12) NOT NULL PRIMARY KEY AUTO_INCREMENT,
    transit_system VARCHAR(50) NOT NULL,
    trip_id VARCHAR(100) NOT NULL,
    start_time VARCHAR(8) NOT NULL,
    end_time VARCHAR(8) NOT NULL,
    headway_secs VARCHAR(100) NOT NULL,
    exact_times TINYINT(1),
    KEY `trip_id` (trip_id)
);

CREATE TABLE `routes` (
    transit_system VARCHAR(50) NOT NULL,
    route_id VARCHAR(100),
    agency_id VARCHAR(50),
    route_short_name VARCHAR(50) NOT NULL,
    route_long_name VARCHAR(255) NOT NULL,
    route_type VARCHAR(2) NOT NULL, 
    route_text_color VARCHAR(255),
    route_color VARCHAR(255),
    route_url VARCHAR(255),
    route_desc VARCHAR(255),
    PRIMARY KEY (route_id),
    KEY `agency_id` (agency_id),
    KEY `route_type` (route_type),
    CONSTRAINT `agency_id` FOREIGN KEY (`agency_id`) REFERENCES `agency` (`agency_id`)
);

CREATE TABLE `shapes` (
    id INT(12) NOT NULL PRIMARY KEY AUTO_INCREMENT,
    transit_system VARCHAR(50) NOT NULL,
    shape_id VARCHAR(100) NOT NULL,
    shape_pt_lat DECIMAL(8,6) NOT NULL,
    shape_pt_lon DECIMAL(8,6) NOT NULL,
    shape_pt_sequence TINYINT(3) NOT NULL,
    shape_dist_traveled VARCHAR(50),
    KEY `shape_id` (shape_id)
);

CREATE TABLE `stops` (
    transit_system VARCHAR(50) NOT NULL,
    stop_id VARCHAR(255),
    stop_code VARCHAR(50),
    stop_name VARCHAR(255) NOT NULL,
    stop_desc VARCHAR(255),
    stop_lat DECIMAL(10,6) NOT NULL,
    stop_lon DECIMAL(10,6) NOT NULL,
    zone_id VARCHAR(255),
    stop_url VARCHAR(255),
    location_type VARCHAR(2),
    parent_station VARCHAR(100),
    stop_timezone VARCHAR(50),
    wheelchair_boarding TINYINT(1),
    PRIMARY KEY (stop_id),
    KEY `zone_id` (zone_id),
    KEY `stop_lat` (stop_lat),
    KEY `stop_lon` (stop_lon),
    KEY `location_type` (location_type),
    KEY `parent_station` (parent_station)
);

CREATE TABLE `trips` (
    transit_system VARCHAR(50) NOT NULL,
    route_id VARCHAR(100) NOT NULL,
    service_id VARCHAR(100) NOT NULL,
    trip_id VARCHAR(255),
    trip_headsign VARCHAR(255),
    trip_short_name VARCHAR(255),
    direction_id TINYINT(1), #0 for one direction, 1 for another.
    block_id VARCHAR(11),
    shape_id VARCHAR(11),
    wheelchair_accessible TINYINT(1), #0 for no information, 1 for at least one rider accommodated on wheel chair, 2 for no riders accommodated.
    bikes_allowed TINYINT(1), #0 for no information, 1 for at least one bicycle accommodated, 2 for no bicycles accommodated
    PRIMARY KEY (trip_id),
    KEY `route_id` (route_id),
    KEY `service_id` (service_id),
    KEY `direction_id` (direction_id),
    KEY `block_id` (block_id),
    KEY `shape_id` (shape_id),
    CONSTRAINT `route_id` FOREIGN KEY (`route_id`) REFERENCES `routes` (`route_id`),
    CONSTRAINT `service_id` FOREIGN KEY (`service_id`) REFERENCES `calendar` (`service_id`)
);

CREATE TABLE `stop_times` (
    id INT(12) NOT NULL PRIMARY KEY AUTO_INCREMENT,
    transit_system VARCHAR(50) NOT NULL,
    trip_id VARCHAR(100) NOT NULL,
    arrival_time VARCHAR(8) NOT NULL,
    arrival_time_seconds INT(100),
    departure_time VARCHAR(8) NOT NULL,
    departure_time_seconds INT(100),
    stop_id VARCHAR(100) NOT NULL,
    stop_sequence VARCHAR(100) NOT NULL,
    stop_headsign VARCHAR(50),
    pickup_type VARCHAR(2),
    drop_off_type VARCHAR(2),
    shape_dist_traveled VARCHAR(50),
    KEY `trip_id` (trip_id),
    KEY `arrival_time_seconds` (arrival_time_seconds),
    KEY `departure_time_seconds` (departure_time_seconds),
    KEY `stop_id` (stop_id),
    KEY `stop_sequence` (stop_sequence),
    KEY `pickup_type` (pickup_type),
    KEY `drop_off_type` (drop_off_type),
    CONSTRAINT `trip_id` FOREIGN KEY (`trip_id`) REFERENCES `trips` (`trip_id`),
    CONSTRAINT `stop_id` FOREIGN KEY (`stop_id`) REFERENCES `stops` (`stop_id`)
);

CREATE TABLE `transfers` (
    id INT(12) NOT NULL PRIMARY KEY AUTO_INCREMENT,
    transit_system VARCHAR(50) NOT NULL,
    from_stop_id INT(100) NOT NULL,
    to_stop_id VARCHAR(8) NOT NULL,
    transfer_type TINYINT(1) NOT NULL,
    min_transfer_time VARCHAR(100)
);

I've put the xyz_id keys as PRIMARY KEY in their own table and as FOREIGN KEY in the others.
I still have some optimizations to make to this schema.

Now this query works in less than 1-5 seconds:

SELECT
    s.stop_id,
    (6371000*acos(cos(radians(48.1128135))*cos(radians(s.stop_lat))*cos(radians(-1.6470705)-radians(s.stop_lon))+sin(radians(48.1128135))*sin(radians(s.stop_lat)))) AS distance,
    t.route_id,
    st.*,
    t.*,
    r.*,
    c.*

FROM stop_times st

LEFT JOIN stops s USING (stop_id)
LEFT JOIN trips t USING (trip_id)
LEFT JOIN routes r USING (route_id)

LEFT JOIN calendar c ON c.service_id = t.service_id 

where
    c.start_date <= 20140915
    and c.end_date >= 20140915
    and c.sunday = 1

    and st.departure_time > '15:00:00'

HAVING
    distance < 200

ORDER BY st.departure_time ASC

I can only tell you that I tried the same thing with SQL, and it took forever, so I had to write a script, first in Perl (no gain), then in C++ (gain 35× faster).

标签
mysql
gtfs
易学教程内所有资源均来自网络或用户发布的内容，如有违反法律规定的内容欢迎反馈！
该文章没有解决你所遇到的问题?点击提问,说说你的问题,让更多的人一起探讨吧!