Trespass
If you stray from the right of way onto private land, you are technically trespassing.
The landowner can insist that you return to the right of way or leave his/her property. If you refuse, the landowner may use a minimum degree of force to make you do so.
Trespassing itself is not an offence, but the landowner can sue you for damages. If you are found guilty, then you may have to pay damages and, more importantly, the landowner's legal fees.
Nobody wants to end up in court. Harmony can be maintained by showing respect for everyone's point of view. Landowners tend to get most annoyed by dogs running loose on their land upsetting wildlife and livestock, damage to crops, damage to walls and fences and discarded litter. Try to avoid all of these by following the Countryside Code.
Remember, you may have a right to be there, but it is not exclusive.