To explain, why downvotes will be locked too after some minutes: there can be "tactical downvoting". For example, one could downvote a competing answer, so the own answer gets higher, becoming better visible and perhaps earning an upvote, later removing the downvote to cover one’s track. There could even be a sock puppet, who downvotes a good answer which provokes an upvote of another reader, later the downvote is removed.
Sockpuppet actions and tactical voting can be detected and the effect can be removed, but only if there are traces.
Such kind of gaming might be a nuisance on bigger sites with a lot more users, it's much less probable here. However, our TeX site inherits such security features. So perhaps we suffer a bit because of years of experience on older sites, where they learned that voting needs a bit protection.
A voting system is not perfect, some rules help to keep it reasonable.
Back to a downvote by mistake:
If the vote came because of a misunderstanding, so perhaps the vote might be deserved somehow, because of the ambiguity of an answer. Is the answer edited, so improved, reversing the vote is possible.
If you voted by accident, just leave a comment telling that. The answerer receives the message and can edit, allowing you to undo the undesired action.
In any case, if helps a lot if the reason for the downvote is briefly explained by a comment. Besides that readers and the answerer can learn from the comment, it supports corrective action, so there's no need for a problem.