Identify scenarios and referred objects in Argoverse 2 logs from natural-language prompts. Evaluation includes spatial tracking and temporal localization.

Preparing predictions

Upload a single JSON document, optionally gzip-compressed. Pickle uploads are not accepted. Use the repository's exporter and validator to prepare predictions.

Log IDs are the top-level keys. Each log maps prompts to lists of frames. This leaderboard requires tracking arrays at every required timestamp.

Predictions with tracks

For the spatiotemporal test track, each frame contains arrays of tracked objects. Use the log ID → prompt → frame list structure below:

{
  "<log_id>": {
    "<prompt>": [{
      "timestamp_ns": 315969904359876000,
      "track_id": [12],
      "score": [0.8],
      "label": [0],
      "name": ["REFERRED_OBJECT"],
      "translation_m": [[6759.5, 1596.4, 57.9]],
      "size": [[4.3, 1.7, 1.5]],
      "yaw": [-1.12],
      "dtypes": {
        "track_id": "int32", "label": "int32",
        "score": "float32", "translation_m": "float64",
        "size": "float32", "yaw": "float32"
      }
    }]
  }
}

All arrays in a frame must describe the same number of objects. Keep track IDs consistent across frames and unique within each frame. Translation and size have shape (N, 3), with coordinates and dimensions in meters; yaw is in radians. Optional velocity_m_per_s also has shape (N, 3). Other arrays have shape (N,). Dimensions must be nonnegative; zero-size legacy placeholders are preserved. Use empty arrays for frames with no detections, while keeping every required timestamp.

Labels 0, 1, and 2 correspond to REFERRED_OBJECT, RELATED_OBJECT, and OTHER_OBJECT. The name array must agree with the labels. Optional is_positive is a boolean for the whole frame. Optional dtypes preserve float32/float64 or int32/int64 arrays; defaults are float64 and int64. Do not include seq_id, description, or ego_translation_m: the evaluator obtains these from its trusted annotations.

Export your existing Python prediction dictionary, keyed by (log_id, prompt), with:

from submission_format.cli import write_predictions
write_predictions(predictions, "predictions.json.gz")

Upload the resulting .json or .json.gz file to the spatiotemporal test leaderboard. There is no schema_version or predictions wrapper. Pickles must be converted locally; they cannot be uploaded.

Evaluation

Tracking ranks by HOTA-Temporal and also reports HOTA-Track, Timestamp Balanced Accuracy, and Log Balanced Accuracy. Temporal phases rank by Timestamp Balanced Accuracy. Metrics are reported as percentages; referred objects are evaluated within 50 meters and the existing mapped-road filtering.

The public leaderboard evaluates the spatiotemporal test split. Anyone with a GitHub account may submit. Submissions are associated with your account; account names and aggregate scores appear publicly after successful evaluation. Prediction files remain private and are retained for 30 days. Test spatiotemporal method descriptions and links remain private.

Challenge information

Autonomous Vehicles (AVs) collect and pseudo-label terabytes of multi-modal data localized to HD maps during normal fleet tests. However, identifying interesting and safety critical scenarios from uncurated data streams is prohibitively time-consuming and error-prone. Although prior works have explored this problem in the context of structured queries and hand-crafted heuristics, we are hosting this challenge to solicit better end-to-end solutions to this important problem.

To this end, our benchmark includes 10,000 planning-centric natural language queries. Challenge participants can use all RGB frames, Lidar sweeps, HD Maps, and track annotations from the AV2 sensor dataset to find relevant actors in each log. Methods will be evaluated at three levels of spatial and temporal granularity. First, methods must determine if an action (defined by a natural language query) occurs in the log. Next, if the action occurs in the log, methods must temporally localize (e.g. find the start and end time) of the action. Lastly, methods must detect and track all objects relevant to the text description. Our primary evaluation metric is HOTA-temporal, a tracking metric that jointly considers detection and association accuracy for referred objects.

This is an ongoing benchmark. The public spatiotemporal test leaderboard ranks methods by HOTA-Temporal and also reports tracking and temporal localization metrics. New submissions do not reopen the past prize competition.

The sensor dataset will be used in this competition. An overview is shown below:

• 30 different object categories.
• 50 meter range detection.
• Lidar, synchronized camera imagery, and HD maps available.

Getting the data

• Please see the Argoverse User Guide for detailed instructions on how to download the sensor dataset.
• The scenario mining prompts and annotations are available to download from this link. See the User Guide for more details.

Preparing your submission

• Please see scenario mining submission tutorial for background on the benchmark. Its original EvalAI/pickle instructions have been replaced by this site’s JSON submission format.
• The challenge baseline repository contains some helpful code for formatting and visualizing submissions.

Citing

    @inproceedings{ davidson2026refav,
        author = {Davidson, Cainan and Ramanan, Deva and Peri, Neehar},
        title = {RefAV: Towards Planning-Centric Scenario Mining},
        booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
        year = {2026}
    }

    @INPROCEEDINGS{ wilson2021argoverse2,
        author = {Benjamin Wilson and William Qi and Tanmay Agarwal and John Lambert and Jagjeet Singh and Siddhesh Khandelwal and Bowen Pan and Ratnesh Kumar and Andrew Hartnett and Jhony Kaesemodel Pontes and Deva Ramanan and Peter Carr and James Hays},
        title = {Argoverse 2: Next Generation Datasets for Self-Driving Perception and Forecasting},
        booktitle = {Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks (NeurIPS Datasets and Benchmarks 2021)},
        year = {2021}
    }

Detailed evaluation

Tracking Evaluation

Dataset Splits

Submitted predictions are evaluated against the 150 logs in the AV2 Sensor dataset test set.

We randomly partitioned 1000 logs into the following splits:

• Train (700 logs)
• Validation (150 logs)
• Test (150 logs)

Evaluation Classes

These 3 classes are defined as follows:

The leaderboard HOTA-Temporal and HOTA-Track metrics are only calculated for the REFERRED_OBJECT class. Outputs from the AV2 evaluation script for RELATED_OBJECTs and OTHER_OBJECTs are intended to be secondary diagnostic metrics.

Additional Competition Details

• Language queries are object-centric -- all correspond to some set of objects.
• Most language queries are given from the third person persective (such as "ego vehicle turning left"). The language queries given from the first-person perspective (such as "the pedestrian on the right") describe objects from the point of view of the ego vehicle.
• Scenarios only involve objects within 50 meters from the ego vehicle and within 5 meters of a mapped road.
• Interacting objects within a scenario are at most 50 meters away from each other.
• All referred object tracks persist for at least 3 evaluation timestamps (1.5s).

Terms