EDIT
I cannot group by only with patient_id, I will get an error of sql_mode=only_full_group_by...
I
Solution 1: Change database to something using Window Functions, as apparently MySQL has big drawbacks on this field (for example). This could save your day
Solution 2: As probably the first solution won't be acceptable for you, you may try to use something like
select t1.patient_id, min(t3.date_of_visit) "date_of_visit"
from consultation t1
LEFT JOIN visit t3 ON t3.visit_id = t1.visit_id
group by t1.patient_id
You could join this as subquery or put it into some table (maybe even temporary). Join should be done by patient and date_of_visit then.
Unfortunatelly until you provide MCVE like Strawberry requests to, it will be hard to help more.