BACKEND ENGINEER | AI & ML DEVELOPER

Avneesh Rai

I Design Backend Systems That Don’t Break At Scale.

What I Dominate

High-Performance Backend Architecture

AI Agents & Automation

Lead Generation Infrastructure

Database Systems Design

AWS Cloud & DevOps Automation

High-Performance Backend Architecture

AI Agents & Automation

Lead Generation Infrastructure

Database Systems Design

AWS Cloud & DevOps Automation

High-Performance Backend Architecture

AI Agents & Automation

Lead Generation Infrastructure

Database Systems Design

AWS Cloud & DevOps Automation

BACKEND ENGINEER | AI & ML DEVELOPER

Avneesh Rai

I Design Backend Systems That Don’t Break At Scale.

TypeScript

React

Node.js

Python

Docker

TypeScript

React

Node.js

Python

Docker

TypeScript

React

Node.js

Python

Docker

TypeScript

React

Node.js

Python

Docker

System Design

SQL & NoSQL

REST & GraphQL

Git & CI/CD

Cloud (AWS/GCP)

System Design

SQL & NoSQL

REST & GraphQL

Git & CI/CD

Cloud (AWS/GCP)

System Design

SQL & NoSQL

REST & GraphQL

Git & CI/CD

Cloud (AWS/GCP)

System Design

SQL & NoSQL

REST & GraphQL

Git & CI/CD

Cloud (AWS/GCP)

Testing & TDD

Data Structures

Algorithms

Security

AI / ML APIs

Testing & TDD

Data Structures

Algorithms

Security

AI / ML APIs

Testing & TDD

Data Structures

Algorithms

Security

AI / ML APIs

Testing & TDD

Data Structures

Algorithms

Security

AI / ML APIs

My Story

Full Time

OrbitAim

Backend Team Lead

Present

Current

Mar-2025

First Role

Intern

Epack Prefab

Python Developer

College

Indian Institute Of Information Technology, Lucknow

B.Tech, Computer Science

2021 – 2025

Graduate

2018 – 2020

Higher Secondary

School

Central Hindu School

Science

Recent Builds

Projects

4 total · swipe to browse

1 / 4

Live

01 / 04

AI / Data Engineering

News Scraper & Lead Intelligence Engine

Built an async Python pipeline that aggregates business news from RSS feeds and Vertex AI Search, scrapes articles via Tor, classifies leads with Bedrock LLMs, enriches company data, performs semantic deduplication using embeddings, and stores structured results in PostgreSQL.

Python 3.11

asyncio

Pydantic v2

+14

View details→

Live

AI / Data Engineering

News Scraper & Lead Intelligence Engine

01 / 04

Overview

A fully automated, asynchronous news-intelligence pipeline built in Python. It ingests articles from 20 RSS feeds (The Hindu, Zee News, Hindustan Times, India TV, Times of India, ET Now, Indian Express, PR Newswire) and Google Vertex AI Discovery Search. Each article is full-text scraped via a Tor hidden-service API, then passed through a two-stage AWS Bedrock (Nova Lite) LLM pipeline: first to classify whether it is a business lead, then to extract structured entities (companies, contacts, signals, project details). Confirmed leads are enriched with LinkedIn data via Unipile. Article summaries and categories are generated by Bedrock, keywords by YAKE. Every article summary is embedded with Alibaba-NLP/gte-large-en-v1.5 (1024-dim) and stored in a pgvector store in PostgreSQL to detect semantically similar prior coverage. Results are fanned out across a normalised 14-table schema.

Key features

Dual-source Ingestion

Collects articles from 20 RSS feeds across 16 Indian news outlets and from Google Vertex AI Discovery Search using configurable search terms stored in PostgreSQL. Both sources write to the same flat JSON staging file with deduplication.

Tor-routed Full-text Scraping

Routes each article URL through a remote onion-service scraper API. Distinguishes TTL-expiry (0x06), host-down (0x01), and unreachable (0x04) SOCKS errors, applies targeted retry/circuit-restart logic per error class, and records scraper diagnostics in the DB for monitoring.

Two-stage LLM Lead Pipeline

Stage 1 (classification): Bedrock Nova Lite determines whether the article is a business lead, returning confidence (HIGH/MEDIUM/LOW) and structured reasons. Stage 2 fires only for confirmed leads — extracting companies, contacts, business signals, project details, funding sources, and stated goals.

Unipile Company Enrichment

For each extracted company, Unipile is queried to resolve LinkedIn profile URL, logo, website, employee count range, and headquarters city/country. Enriched data is merged into the company record before DB insertion.

Semantic Deduplication with pgvector

Each confirmed-lead article summary is embedded with Alibaba-NLP/gte-large-en-v1.5 (1024-dim). The embedding is compared against all existing vectors in PostgreSQL via pgvector cosine similarity; near-duplicate articles are stored as 'similar' rather than 'unique'.

Normalised 14-table PostgreSQL Schema

Articles fan out across primary, main, full-news, vectors, category, keywords, and search-term tables — plus leads-summary, lead-companies, lead-contacts, lead-signals, lead-projects, processed, and failed tables — enabling granular querying and per-stage failure tracking.

Performance

News Sources Integrated

RSS Feeds Monitored

1024

Vector Embedding Dimensions

2-stage

LLM Pipeline Architecture

Normalised DB Tables

100+

Articles Processed per Engine Run

Built with

Python 3.11

asyncio

Pydantic v2

asyncpg

PostgreSQL

pgvector

Amazon Bedrock (Nova Lite)

Sentence Transformers — GTE-Large-en-v1.5

Tor / SOCKS5h

YAKE

feedparser

Google Vertex AI Discovery Search

Unipile API

AWS Secrets Manager

python-dotenv

filelock

pytz

Project details

6 months

Duration

Production

Status

Solo

Team

2024

Year

View project View code

Replaces manual business-development research across 16 Indian news outlets with a fully automated pipeline — extracting structured lead intelligence from hundreds of articles per run without human intervention.

Impact

Now Playing

My Playlist

Curated with love

3 tracks

Sunflower

Post Malone

2:38

Shinunoga E-Wa

Fujii Kaze

3:17

You Haunt Me

Citizen

3:42

About Me

Avneesh Rai

Subject Profile

I'm a Python Backend Developer & AI/ML Enthusiast. Curiosity is the thread that runs through everything I do. I enjoy building things, understanding how they work, and exploring unfamiliar territory—whether that's a new technology, a new idea, or a new experience.

I'm always looking for opportunities to grow, create, and challenge myself. My goal is simple: build meaningful things, keep learning relentlessly, and make each year more interesting than the last.

→ Curious by default. Building by choice.

Based inGhazipur, IN

RoleBackend Developer

FocusAI & ML Dev

Exp.1+ Years

AvailableOpen to offers