What is this?

This repository is a demo leaderboard template. You can copy the leaderboard space and the two datasets (results and requests) to your org to get started with your own leaderboard!

The space does 3 things:

• stores users submissions, and sends them to the requests dataset
• reads the submissions depending on their status/date of creation, and launches evaluations through the main_backend.py file, using the Eleuther AI Harness. Results of running evaluations are then sent to results
• reads the results and displays them in a leaderboard.

You can also move the backend to its own space if you need, by grabbing main_backend and putting it in its own space, with a app.py which runs it every few minutes - it is probably the best solution.

Getting started

Defining environment variables

To get started on your own leaderboard, you will need to edit 2 files:

• src/envs.py to define your own environment variable (like the org name in which this has been copied)
• src/about.py with the tasks and number of few_shots you want for your tasks

Setting up fake results to initialize the leaderboard

Once this is done, you need to edit the "fake results" file to fit the format of your tasks: in the sub dictionary results, replace task_name1 and metric_name by the correct values you defined in Tasks above.

    "results": {
        "task_name1": {
            "metric_name": 0
        }
    }