TODO：

• Complete Metrics Evaluation System Introduction；
• Add Metrics Evaluation tool code and Tutorial；

1. Metrics Evaluation System

To comprehensively evaluate the planning algorithm’s performance across various scenarios, our scoring system assesses four key categories of metrics: safety, efficiency, smoothness, and task completion. These metrics are aggregated to produce an overall score for the proposed algorithm, providing a holistic assessment of its capabilities.

Safety Metrics:

This metric evaluates the vehicle’s ability to avoid collisions with obstacles and road boundaries. Given the irregular characteristics of unstructured roads, the vehicle must maintain a sufficient safety margin from the boundaries. Specifically, the lateral distance between the ego vehicle and the road boundary must exceed a predefined threshold to ensure safety throughout the scenario.

Efficiency Metrics:

Efficiency is measured by how quickly the autonomous vehicle reaches the target area, provided it completes the scenario safely. Faster completion times indicate better performance.

Smoothness Metrics:

Unlike passenger vehicles, where algorithms focus on optimizing ride comfort based on passengers’ subjective experience, the primary goal for mining vehicles is driving smoothness. This metric is assessed by calculating the minimum and maximum longitudinal and lateral accelerations, as well as centripetal acceleration during turns to evaluate stability. Smoothness is crucial for ensuring stability and safety in challenging mining environments.

Task Completion:

Task completion is evaluated for both scenarios in which the vehicle reaches the target area and those in which it does not. If the target is not reached, the completion score is determined by the vehicle’s progress along the route, specifically by comparing the final driving distance to the total route distance. If the vehicle reaches the target, the quality of task completion is also assessed by evaluating the final yaw deviation, which measures the vehicle’s alignment with the road at the goal based on the yaw difference from the target orientation.

Note：”Dynamic Obstacle Avoidance Scenario Testing”and “Dynamic Obstacle Avoidance Scenario Testing” is different.

Other Metrics:

When testing the performance of the provided algorithm across a wide range of scenarios, key metrics like task completion rate and collision rate can also serve as important indicators, depending on the metrics prioritized.

Dynamic Obstacle Avoidance Scenario Testing

Table 1 Metric Design for Dynamic Obstacle Avoidance Scenario Testing.

Static Obstacle Avoidance Scenario Testing

The metrics design for the static scenario planning task is shown in Table 3 and differs slightly from the Task completion metrics used for the dynamic scenario planning task.

Table 3: Metric Design for Static Obstacle Avoidance Scenario Testing.

2. Metrics Evaluation Tool

TODO ……

Improve this page