This query uses a user defined variable to calculate the distance from the target id, and a series of wrapper queries to get the results you want. Only one pass is made over the table, so it should perform well.
select * from (
select id, value from (
select *, (@x := ifnull(@x, 0) + if(id > 3, -1, 1)) row from (
select * from mytable order by id
) x
) y
order by row desc
limit 3
) z
order by id
See an SQLFiddle
If you don't care about the final row order you can omit the outer-most wrapper query.