MySQL: Select query execution and result fetch time increases with number of connections

Question

My server application makes multiple connections to MySQL through separate threads. Each connection fires a SELECT query and fetches result which the application then caters back to its connected users.

I am using InnoDB. To my surprise I found it a very weird that if I increase number of connections to MySQL, query performance deteriorates and result fetch time also increases. Below is a table showing same.

This data is produced when I had 3333 records in MySQL table and the SELECT query based on random parameters given to it fetches around 450 records out of them. Each record has around 10 fields and all of them together contains 1.2 KB of data. (Thus, single SELECT query fetches 1.2 * 450 = 540 KB data in total)

╔═══════════╦═══════════════╦══════════════╗
║ Number of ║Query execution║ Result fetch ║
║connections║  time range   ║  time range  ║
║ to MySQL  ║ (in seconds)  ║ (in seconds) ║ 
╠═══════════╬═══════════════╬══════════════╣
║     1     ║ 0.02 to 0.06  ║ 0.03 to 0.18 ║
║     7     ║ 0.23 to 0.64  ║ 0.54 to 0.74 ║
║    17     ║ 0.32 to 1.71  ║ 0.53 to 1.18 ║
║    37     ║ 0.37 to 2.01  ║ 0.70 to 1.70 ║
║   117     ║ 1.13 to 3.29  ║ 2.48 to 3.25 ║
╚═══════════╩═══════════════╩══════════════╝

What I don't understand here is why does MySQL take more time when number of connections to it are increased? Especially when there are no updates being made to the table, MySQL should process SELECT request from each connection in separate thread. Thus concurrent processing of query. Hence, ideally there shouldn't be significant degrade in performance and fetch.

I won't mind to have single connection to DB but catch is that my server performance significantly degrades with it. Thousands of users (connected to my server) will have to wait for that single thread for their turn to come.

After going through some related questions on SO I tried increasing innodb_buffer_pool_size to 1 GB with no luck.

Here are my all InnoDB parameters:

innodb_adaptive_flushing    ON
innodb_adaptive_flushing_lwm    10
innodb_adaptive_hash_index  ON
innodb_adaptive_max_sleep_delay 150000
innodb_additional_mem_pool_size 2097152
innodb_api_bk_commit_interval   5
innodb_api_disable_rowlock  OFF
innodb_api_enable_binlog    OFF
innodb_api_enable_mdl   OFF
innodb_api_trx_level    0
innodb_autoextend_increment 64
innodb_autoinc_lock_mode    1
innodb_buffer_pool_dump_at_shutdown OFF
innodb_buffer_pool_dump_now OFF
innodb_buffer_pool_filename ib_buffer_pool
innodb_buffer_pool_instances    8
innodb_buffer_pool_load_abort   OFF
innodb_buffer_pool_load_at_startup  OFF
innodb_buffer_pool_load_now OFF
innodb_buffer_pool_size 1073741824
innodb_change_buffer_max_size   25
innodb_change_buffering all
innodb_checksum_algorithm   crc32
innodb_checksums    ON
innodb_cmp_per_index_enabled    OFF
innodb_commit_concurrency   0
innodb_compression_failure_threshold_pct    5
innodb_compression_level    6
innodb_compression_pad_pct_max  50
innodb_concurrency_tickets  5000
innodb_data_file_path   ibdata1:12M:autoextend
innodb_data_home_dir    
innodb_disable_sort_file_cache  OFF
innodb_doublewrite  ON
innodb_fast_shutdown    1
innodb_file_format  Antelope
innodb_file_format_check    ON
innodb_file_format_max  Antelope
innodb_file_per_table   ON
innodb_flush_log_at_timeout 1
innodb_flush_log_at_trx_commit  2
innodb_flush_method normal
innodb_flush_neighbors  1
innodb_flushing_avg_loops   30
innodb_force_load_corrupted OFF
innodb_force_recovery   0
innodb_ft_aux_table 
innodb_ft_cache_size    8000000
innodb_ft_enable_diag_print OFF
innodb_ft_enable_stopword   ON
innodb_ft_max_token_size    84
innodb_ft_min_token_size    3
innodb_ft_num_word_optimize 2000
innodb_ft_result_cache_limit    2000000000
innodb_ft_server_stopword_table 
innodb_ft_sort_pll_degree   2
innodb_ft_total_cache_size  640000000
innodb_ft_user_stopword_table   
innodb_io_capacity  200
innodb_io_capacity_max  2000
innodb_large_prefix OFF
innodb_lock_wait_timeout    50
innodb_locks_unsafe_for_binlog  OFF
innodb_log_buffer_size  268435456
innodb_log_compressed_pages ON
innodb_log_file_size    262144000
innodb_log_files_in_group   2
innodb_log_group_home_dir   .\
innodb_lru_scan_depth   1024
innodb_max_dirty_pages_pct  75
innodb_max_dirty_pages_pct_lwm  0
innodb_max_purge_lag    0
innodb_max_purge_lag_delay  0
innodb_mirrored_log_groups  1
innodb_monitor_disable  
innodb_monitor_enable   
innodb_monitor_reset    
innodb_monitor_reset_all    
innodb_old_blocks_pct   37
innodb_old_blocks_time  1000
innodb_online_alter_log_max_size    134217728
innodb_open_files   300
innodb_optimize_fulltext_only   OFF
innodb_page_size    16384
innodb_print_all_deadlocks  OFF
innodb_purge_batch_size 300
innodb_purge_threads    1
innodb_random_read_ahead    OFF
innodb_read_ahead_threshold 56
innodb_read_io_threads  64
innodb_read_only    OFF
innodb_replication_delay    0
innodb_rollback_on_timeout  OFF
innodb_rollback_segments    128
innodb_sort_buffer_size 1048576
innodb_spin_wait_delay  6
innodb_stats_auto_recalc    ON
innodb_stats_method nulls_equal
innodb_stats_on_metadata    OFF
innodb_stats_persistent ON
innodb_stats_persistent_sample_pages    20
innodb_stats_sample_pages   8
innodb_stats_transient_sample_pages 8
innodb_status_output    OFF
innodb_status_output_locks  OFF
innodb_strict_mode  OFF
innodb_support_xa   ON
innodb_sync_array_size  1
innodb_sync_spin_loops  30
innodb_table_locks  ON
innodb_thread_concurrency   8
innodb_thread_sleep_delay   0
innodb_undo_directory   .
innodb_undo_logs    128
innodb_undo_tablespaces 0
innodb_use_native_aio   OFF
innodb_use_sys_malloc   ON
innodb_version  5.6.28
innodb_write_io_threads 16

Can someone please throw light? This is bugging me for really long time.

(Note: I haven't mentioned actual query in this question because query is little complicated and this question is not about that query. But it's about performance degradation with increasing connections when query is same)

UPDATE 1

Here is SHOW CREATE TABLE output for my tables:

CREATE TABLE `profiles` (
  `SRNO` bigint(20) unsigned NOT NULL AUTO_INCREMENT,
  `HANDLE_FIRST` int(10) unsigned NOT NULL,
  `HANDLE_SECOND` bigint(20) unsigned NOT NULL,
  `USERID` binary(16) NOT NULL,
  `UNIQUESTRING` char(10) NOT NULL,
  `CLIENT_VERSION` smallint(5) unsigned NOT NULL,
  `ISCONNECTED` bit(1) NOT NULL,
  `ISPROFILEPRESENT` bit(1) NOT NULL,
  `USERNAME` varchar(32) CHARACTER SET utf8 COLLATE utf8_unicode_ci DEFAULT NULL,
  `GENDER` tinyint(1) DEFAULT NULL,
  `DND` bit(1) DEFAULT NULL,
  `STATUS` varchar(128) CHARACTER SET utf8 COLLATE utf8_unicode_ci DEFAULT NULL,
  `PROFILE_URL` varchar(128) DEFAULT NULL,
  PRIMARY KEY (`SRNO`),
  UNIQUE KEY `USERID` (`USERID`),
  KEY `USERID_INDEX` (`USERID`),
  KEY `UNIQUESTRING_INDEX` (`UNIQUESTRING`),
  KEY `ISCONNECTED_INDEX` (`ISCONNECTED`),
  KEY `ISPROFILEPRESENT_INDEX` (`ISPROFILEPRESENT`)
) ENGINE=InnoDB AUTO_INCREMENT=9250 DEFAULT CHARSET=utf8


CREATE TABLE `blockers` (
  `BLOCKER_PROFILE_SRNO` bigint(20) unsigned NOT NULL,
  `BLOCKED_PROFILE_SRNO` bigint(20) unsigned NOT NULL,
  UNIQUE KEY `BLOCKER_PROFILE_SRNO` (`BLOCKER_PROFILE_SRNO`,`BLOCKED_PROFILE_SRNO`),
  KEY `BLOCKER_PROFILE_SRNO_INDEX` (`BLOCKER_PROFILE_SRNO`),
  KEY `BLOCKED_PROFILE_SRNO_INDEX` (`BLOCKED_PROFILE_SRNO`),
  CONSTRAINT `fk_BlockedIndex` FOREIGN KEY (`BLOCKED_PROFILE_SRNO`) REFERENCES `profiles` (`SRNO`),
  CONSTRAINT `fk_BlockerIndex` FOREIGN KEY (`BLOCKER_PROFILE_SRNO`) REFERENCES `profiles` (`SRNO`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8

And here is the query I am running:

select  prfls.*
    from  profiles as prfls
    left outer join  blockers as blkr1 on blkr1.blocker_profile_srno = prfls.srno
      and  blkr1.blocked_profile_srno = 6443
    left outer join  blockers as blkr2 on blkr2.blocker_profile_srno = 6443
      and  blkr2.blocked_profile_srno = prfls.srno
    where  blkr1.blocker_profile_srno is null
      and  blkr2.blocker_profile_srno is null
      and  (      (prfls.uniquestring like 'phk5600dcc%')
              or  (prfls.uniquestring like 'phk5600dcf%')
           )
      and  prfls.isconnected=1
      and  prfls.isprofilepresent=1
    limit  450

This query is essentially a prepared statement where blocked_profile_srno, blocker_profile_srno and uniquestring parameters keep changing for each query. However blocked_profile_srno and blocker_profile_srno always remains equal (in above query their value is 6443). Table blockers is blank (I have it in place for future use but currently it has no data in it)

When 117 connections were simultaneously running queries, output of SHOW GLOBAL STATUS LIKE 'Threads_running'; was most of the time 1. However it sometimes went upto 27. At the same time, output of SHOW GLOBAL STATUS LIKE 'Max_used_connections'; was 130

UPDATE 2

I can gather from Rick James answer below that optimizing query reduces query execution time range. This time range still keeps increasing with number of connections but within acceptable range. This is why I've accepted the answer.

Answer 1

Probably each connection is doing a full table scan of profiles. Let's try to avoid that. When there are dozens of queries hitting the same table, there are locks that cause InnoDB to "stumble over itself". Either of these plans will both speed up the query and decrease the number of rows touched (hence decrease the locking). The use of the suggested "composite" index will speed up the query. But the OR gets in the way. I see two tricks to still have an index look at uniquestring, but avoid some or all of the OR.

(      (prfls.uniquestring like 'phk5600dcc%')
   or  (prfls.uniquestring like 'phk5600dcf%')
)

OR is hard to optimize.

Add this:

INDEX(isconnected, isprofilepresent, uniquestring)

Then...

Plan A:

prfls.uniquestring         like 'phk5600dc%' AND  -- note common prefix
(      (prfls.uniquestring like 'phk5600dcc%')
   or  (prfls.uniquestring like 'phk5600dcf%')
)

This assumes you can construct that common prefix.

Plan B (turn OR into UNION):

( SELECT ...
    WHERE prfls.uniquestring like 'phk5600dcc%' AND ...
    LIMIT 450 )
UNION ALL    -- ? You may want DISTINCT, if there could be dups
( SELECT ...
    WHERE prfls.uniquestring like 'phk5600dcf%' AND ...  -- the only diff
    LIMIT 450 )
LIMIT 450   -- yes, again

Plan A (if practical) takes advantage of what seems to be a common starting value. Plan B works regardless, but is probably a little slower, although still a lot faster than the original.

Other notes...

Indexes on flags (of which you have two) are almost never used. EXPLAIN SELECT ... will probably show that neither was used. Please provide the EXPLAIN for any SELECT that needs discussion.

A UNIQUE KEY is a KEY, so there is not need for the redundant index on USERID.

limit 450 -- Which 450 do you want? Without an ORDER BY, the query is allowed to give you any 450. (Of course, perhaps that is fine.) (And ORDER BY would probably slow down the query.)

My suggestions will not "solve" the problem, but they should increase the number of connections before the slow-down becomes noticeable.