If you care about your health, you should be worried.
Maffetone P. The big book of endurance training and racing. Skyhorse Publishing, Inc.; 2010 Sep 22.
RHR is an important measure of overall health. It is a indicator of your aerobic function level.
In general, the lower RHR the better. As a healthy RHR could be considered lower than 60. The world standards are higher, because it is based on sample of people where are a lot of unhealthy people (Note the difference between "healthy" and "able to work").
A lot of people are fit (can finish marathon or deadlift 3xBW), but they are not healthy. Some people are also exceptions. If you care about health, do not listen exceptions or unhealthy but fit people.
Note: Make sure you measure your RHR correctly. Best is at the morning, directly in the bed before you get up. Repeat at least 10 days and make an average. Also It is negatively influenced by any stress in previous days (training, work stress, junk food...). It could be also influenced (in any direction) by some serious health issue.