For a given output route, the ort_nextdist field points to a linked list of routes for the same network and the same gateway but with possibly different distances. sort_dists(oroute) finds the best route to the network in the linked list that begins with oroute (sort_dists()'s only parameter), moves this route to the beginning of the linked list, and returns the route (which is, obviously, the beginning of the modified linked list). The route to the network with the smallest distance is the best route. If two routes to the network are tied for the smallest distance, the route with the greater preference is the best route.
The function name "sort_dists" is somewhat misleading since the function does not do a full sort of the routes. sort_dists() simply moves the best route to a network for a given gateway to the beginning of the linked list of routes. In this way, sort_dists() is similar to sort_gws(), which also doesn't do a full sort.