which.min() returns the index of the (first) minimum or maximum of a numeric (or logical) vector. If there are multiple equal values as close to 1.43 as each other and you want to keep all of them, you can use filter():
a %>% group_by(id) %>% filter(abs(b - 1.43) == min(abs(b - 1.43)))
#Source: local data frame [2 x 2]
#Groups: id [2]
# id b
#
#1 A 1.5
#2 B 1.4
If you prefer sticking with the nth() function, and it is OK to have only one value for each group, you can wrap it within a summarize function so that it will be applied to each group, and also according to ?nth(), you need to pass the vector to the function as an argument as well:
a %>% group_by(id) %>% summarise(b = nth(b, which.min(abs(b-1.43))))
# A tibble: 2 × 2
# id b
#
#1 A 1.5
#2 B 1.4