Yes. I let my 2 year old wander around outside (in our 100% fenced in yard) sometimes. Not for long and not without watching from the house.
I guess it depends on your surroundings. I can't imagine someone coming into our yard. We have 6ft PVC fences that would be VERY hard to climb and latched gates leading to the front.
Is yours gated as well? Like the fence goes all around?
I totally let my 6 and 8 year old do it. We have no water/lakes or pools around. That would be a major factor for me....