AutoScientist Challenge Part 2 — Data Visualization & Chart Interpretation

Small-model-first dashboard + native-language (Hindi/Punjabi) chart interpretation

Part A — Small Fine-Tuned Classifier vs. Large Hosted Model

Legal-aid intake routing task, 60-row live benchmark

DistilBERT (66M, self-hosted)
Llama-3.3-70B-Instruct-Turbo (hosted API)

Accuracy

Small
DistilBERT: 74.2% accuracy
74.2%
Large
Llama-3.3-70B: 70.0% accuracy
70.0%

Macro-F1

Small
DistilBERT: 0.600 macro-F1
0.600
Large
Llama-3.3-70B: 0.391 macro-F1
0.391

Latency (lower is better)

Small
DistilBERT: 57ms avg
57ms
Large
Llama-3.3-70B: 1083ms avg
1083ms

Cost per 1,000 queries (lower is better)

Small
DistilBERT: $0.00 (self-hosted)
$0.00
Large
Llama-3.3-70B: $0.46 per 1k
$0.46
DistilBERT (small, self-hosted) — practitioner recommendation
यदि आपके लिए प्रति 1,000 क्वेरी $0.00 की शून्य लागत और 57ms की कम लेटेंसी महत्वपूर्ण है और 74.2% सटीकता पर�याप्त है, तो बड़े होस्टेड API मॉडल के बजाय इस स्व-होस्टेड डिस्टिलबर्ट का चयन करें, अन्यथा उच्च सटीकता के लिए बड़े मॉडल पर विचार करें।
If zero cost per 1,000 queries and low 57ms latency matter to you and 74.2% accuracy is sufficient, choose this self-hosted DistilBERT instead of the large hosted API model — otherwise consider the larger model for higher accuracy.
Llama-3.3-70B-Instruct-Turbo (large, hosted API) — practitioner recommendation
70.0% सटीकता और $0.46 प्रति 1,000 क्वेरी की लागत के बावजूद, 1083ms की उच्च लेटेंसी के कारण केसवर्कर को तेज़ प्रतिक्रिया के लिए छोटे स्व-होस्टेड वर्गीकारक का चयन करना चाहिए।
Despite 70.0% accuracy and $0.46 per 1,000 query cost, the high latency of 1083ms means the caseworker should choose the smaller self-hosted classifier for a faster response.

Live benchmark: 60-row sample from the legal-aid intake routing test set (Part 1), measured 2026-07-28. Large-model calls via Together AI (meta-llama/Llama-3.3-70B-Instruct-Turbo, $1.04/1M tokens). Vernacular recommendations generated by AutoScientist (universal_prompt, Hindi/Devanagari output) from the raw benchmark numbers. Directly operationalizes Mozilla's "State of Open Source AI" (Jul 2026) finding that closed models captured ~96% of revenue on ~80% of usage share under metered pricing — this benchmark shows exactly where that gap comes from on a real routing task.

Part B — Native-Language Chart Interpretation (VLM)

LoRA-fine-tuned google/gemma-3-4b-it-VLM that answers chart questions directly in Hindi, Punjabi, or English — trained via AutoScientist's real training API (client.autoscientist.create()), the first live use of that capability in this project. 1,260 rows (420 EN + 420 HI + 420 PA), 100 synthetic charts. Result: win rate 0.56 (target 0.60).

Differentiator vs. the one other live entrant in this category found at submission time (doraking/chartwise-autoscientist-data, 751 downloads, English-only): same chart-QA task shape, genuinely native-language rather than a translation layer bolted on.

Model weights (HF) Dataset (HF) Model (Kaggle) Dataset (Kaggle)