experience

Artifical intelligence Research Intern

Malaviya National Institute of Technology (MNIT) • May 2026 – July 2026

company: Malaviya National Institute of Technology (MNIT)

Duration: May 2026 – July 2026
Role: AI Research Intern
Research Domain: Information Retrieval, Retrieval-Augmented Generation (RAG), Large Language Models (LLMs), Hybrid Search, Explainable AI

Overview

Worked as an AI Research Intern at Malaviya National Institute of Technology (MNIT), Jaipur, contributing to the development of a large-scale Retrieval-Augmented Generation (RAG) framework for long-document question answering and evidence retrieval.

The internship focused on building a production-scale retrieval pipeline capable of efficiently searching hundreds of thousands of document chunks while maintaining low latency, reduced memory footprint, and high retrieval accuracy.

The project involved designing and optimizing every stage of the retrieval pipeline, including document preprocessing, dense retrieval, sparse retrieval, reranking, evidence fusion, metadata management, indexing, benchmarking, and memory optimization.

Research Problem

Large Language Models possess limited context windows and cannot directly reason over massive document collections.

Traditional retrieval systems also struggle because:

  • Sparse lexical search (BM25) misses semantic matches.
  • Dense retrieval may overlook exact keyword matches.
  • Long documents exceed transformer context lengths.
  • Multiple retrieval stages repeatedly reload metadata into memory.
  • Large-scale indexing introduces high RAM overhead and I/O latency.

The objective was to engineer a scalable hybrid retrieval pipeline capable of:

  • Retrieving highly relevant evidence
  • Reducing GPU and CPU memory usage
  • Supporting hundreds of thousands of chunks
  • Improving retrieval accuracy
  • Optimizing inference latency

Key Contributions

  • Developed a scalable document preprocessing pipeline that performs sentence-aware chunking with configurable sliding-window overlap and token-aware segmentation. The pipeline generates all downstream retrieval artifacts—including manifest.csv, retrieval_map.csv, docs.jsonl, and chunk_text.parquet—in a single preprocessing pass, eliminating redundant preprocessing steps.
  • Engineered a hybrid retrieval framework by integrating sparse lexical retrieval using Lucene BM25 with dense semantic retrieval using Qwen3-Embedding-4B embeddings indexed through HNSW Approximate Nearest Neighbor Search, enabling both keyword-level precision and semantic search capabilities.
  • Implemented a multi-query retrieval strategy where each user query is expanded into multiple semantically diverse reformulations before retrieval. The expanded queries are independently searched across dense and sparse indices, significantly improving recall for ambiguous and underspecified information needs.
  • Designed a Multi-Query Reciprocal Rank Fusion (RRF) module to aggregate retrieval results from multiple expanded queries. The fusion strategy improves retrieval robustness, increases evidence diversity, reduces dependence on a single query formulation, and consistently enhances Recall@K.
  • Integrated a dual-stage reranking pipeline combining a Qwen Cross-Encoder reranker for semantic relevance estimation and pairwise document scoring with ColBERT’s late-interaction architecture for token-level similarity matching. Final rankings are produced through weighted ensemble fusion to maximize retrieval precision.
  • Redesigned metadata management across the retrieval pipeline by replacing repeated loading of the complete manifest.csv with stage-specific metadata artifacts. Introduced a lightweight retrieval_map.csv containing only chunk_id and doc_id, reducing unnecessary RAM consumption, redundant CSV parsing, disk I/O, and overall retrieval latency.
  • Optimized the BM25 indexing workflow by generating docs.jsonl directly during preprocessing rather than reconstructing it immediately before indexing. This eliminated repeated manifest loading, reduced preprocessing duplication, and streamlined the sparse indexing pipeline.
  • Implemented a compressed chunk_text.parquet lookup store containing chunk identifiers and corresponding text for reranking and generation stages. Leveraging Parquet’s columnar storage format reduced memory footprint, improved read efficiency, and accelerated downstream evidence retrieval.
  • Built large-scale retrieval indices using Lucene BM25 for sparse retrieval and SentenceTransformers with Qwen3-Embedding-4B embeddings for dense retrieval over HNSW indices, supporting datasets containing tens of thousands of documents and hundreds of thousands of chunks.

Tech Stack Used

Category Technologies
Programming Language Python
Deep Learning & AI PyTorch, Hugging Face Transformers, SentenceTransformers
Large Language Models (LLMs) Qwen3-Embedding-4B, Qwen Reranker
Retrieval & Information Retrieval Retrieval-Augmented Generation (RAG), Hybrid Search, Dense Retrieval, Sparse Retrieval, BM25, Lucene, HNSW Indexing, ColBERT, Reciprocal Rank Fusion (RRF), Multi-Query Retrieval, Cross-Encoder Reranking
Data Processing Pandas, NumPy, PyArrow, Apache Parquet
Performance Engineering Memory Optimization, Metadata Engineering, Pipeline Optimization, Performance Benchmarking, GPU Computing
Infrastructure & Development CUDA, NVIDIA A10 GPU, Linux, Conda, Bash, VS Code Remote
Core Expertise Production AI Systems, Information Retrieval, Large Language Models (LLMs), Retrieval Pipeline Engineering, Scalable Indexing, Hybrid Retrieval Systems

Artifical intelligence and machine learning Intern

Coregen • Feb 2026 - May -2026

Duration: May 2026 – July 2026
Role: AI Research Intern
Research Domain: Information Retrieval, Retrieval-Augmented Generation (RAG), Large Language Models (LLMs), Hybrid Search, Explainable AI

Overview

Worked as an AI/ML Intern at CoreGen, contributing to the research and development of Generative AI and Computer Vision applications. Developed Retrieval-Augmented Generation (RAG) systems, semantic search pipelines, and deep learning solutions while building end-to-end AI applications using modern LLM frameworks, vector databases, and full-stack technologies.

Tech Stack

Category Technologies
Programming Languages Python, SQL, JavaScript, TypeScript
AI & Deep Learning PyTorch, TensorFlow, Scikit-learn, OpenCV
Generative AI LangChain, LlamaIndex, Hugging Face Transformers, Retrieval-Augmented Generation (RAG), Prompt Engineering
Vector Databases FAISS, ChromaDB
Web Development Flask, REST APIs, MERN Stack
Tools & Platforms Git, GitHub, Docker, Linux, Jupyter Notebook

Projects

LLM-powered Knowledge Assistant

Developed a domain-specific Retrieval-Augmented Generation (RAG) system capable of answering natural language queries using external knowledge sources.

Key Contributions

  • Developed Retrieval-Augmented Generation (RAG) pipelines for knowledge-grounded question answering.
  • Implemented semantic search using vector embeddings with FAISS and ChromaDB.
  • Designed document ingestion, chunking, embedding generation, and retrieval workflows.
  • Integrated Large Language Models with retrieval pipelines for accurate response generation.

Satellite Image Super-Resolution

Designed deep learning models for enhancing low-resolution satellite imagery to improve spatial resolution and visual quality.

Key Contributions

  • Developed super-resolution models using deep learning techniques.
  • Applied Computer Vision algorithms for image reconstruction.
  • Improved image quality for downstream geospatial analysis tasks.

Full-Stack AI Dashboard

Built a production-ready dashboard for visualizing AI predictions and monitoring inference results.

Key Contributions

  • Developed a MERN-based dashboard for AI model visualization.
  • Integrated backend inference APIs with interactive frontend analytics.
  • Designed responsive interfaces for displaying prediction outputs and model insights.

Key Contributions

  • Developed scalable Retrieval-Augmented Generation (RAG) pipelines for enterprise AI applications.
  • Built semantic search systems using vector embeddings and FAISS / ChromaDB.
  • Designed modular LLM workflows using LangChain and LlamaIndex.
  • Implemented REST APIs for AI inference and backend integration.
  • Worked on Computer Vision pipelines for satellite image enhancement.
  • Developed interactive dashboards for AI model monitoring and visualization.
  • Optimized data preprocessing and model inference workflows for improved efficiency.
  • Collaborated using Git-based software development practices within an agile development team.

Skills Demonstrated

  • Retrieval-Augmented Generation (RAG)
  • Large Language Models (LLMs)
  • Semantic Search
  • Vector Databases
  • Prompt Engineering
  • LangChain
  • LlamaIndex
  • Hugging Face Transformers
  • Computer Vision
  • Deep Learning
  • Satellite Image Processing
  • REST API Development
  • MERN Stack Development
  • Docker
  • Linux Development
  • Git & GitHub
  • AI System Deployment