If the important part (your veg.part) is already fenced in ,what is your problem? rabbits do not climb trees so they won't eat your olives.may be little by little you can even fence  a little bit more so you won't even se them anymore.
Live and let live.