Speeding Up JSON Extraction with Tiny LLMs: A Breakthrough!

This project showcases impressive performance gains using small, open source Large Language Models (LLMs) for a practical text extraction task. The incredibly low latency and high throughput demonstrate the potential of these models for real-world applications. The innovative post-processing technique for proper nouns is a clever solution that further enhances accuracy.

• •Achieved impressive speed and efficiency extracting data into JSON format using a 3B parameter LLM.
• •Achieved less than 500ms latency and 30 RPM throughput on an L4 GPU.
• •Data quality and post-processing steps (like Levenshtein distance) significantly improved accuracy, especially with proper nouns.