Shot Categorizer 🎬

<br>

Contains fine-tuning and inference code for a shot categorizer model. We fine-tuned the model on
the microsoft/Florence-2-large and the
diffusers/ShotDEAD-v0 dataset.

• Fine-tuned model checkpoint: diffusers/shot-categorizer-v0
• Dataset: diffusers/ShotDEAD-v0
• HF Spaces for a quick try out: ariG23498/shot-categorizer-demo (thanks to @ariG23498!)

The model can be used to curate video datasets amongst other use cases.

Getting started

Install PyTorch and make sure you're on a CUDA-enabled GPU. The scripts were tested on 8xH100s.

Then install the dependencies pip install requirements.txt.

You can then launch training:

accelerate launch \
    --config_file=configs/accelerate-configs/ds.yaml train.py \
    --config=configs/training-configs/config.yaml

We provide a SLURM configuration in the run.slurm file.

Inference

python inference.py --image_path=<PATH_TO_AN_IMAGE>

Additionally, you can run the accuracy.py script to compute aggregate metrics on the indvidual sgot categories on your dataset.

Acknowledgement

Thanks to this blog post on fine-tuning Florence-2.