Data Engineer — ML Training Data Pipeline

We use cookies to improve your experience, personalize content, and analyze traffic. Accept all, decline non-essential, or manage your preferences. See our Cookie Policy.

openSettings()">Manage cookies

check(event) ; career-cookie-consent => acceptAll()" mousedown="lyte-button => check(event)" final-style="" final-class="lyte-button primary cookie-accept-btn lytePrimaryBtn" lyte-rendered=""> {{getI18n("zr.cw.cookie.accept.all")}} Accept all

check(event) ; career-cookie-consent => declineAll()" mousedown="lyte-button => check(event)" final-style="" final-class="lyte-button primary cookie-decline-btn lytePrimaryBtn" lyte-rendered=""> {{getI18n("zr.cw.cookie.decline.all")}} Decline non-essential

DATAECONOMY | Full time

Data Engineer — ML Training Data Pipeline

Hyderabad, India | Posted on 09/08/2026

check(event) ; career-website-detail => apply(record.id,meta)" mousedown="lyte-button => check(event)" final-style="background-color:#6875E2;border-color:#6875E2;color:white;" final-class="lyte-button lyteBackgroundColorBtn lyteSuccess" lyte-rendered=""> I'm interested

check(event)" mousedown="lyte-button => check(event)" final-style="background-color: #fff; color:#FFFFFF;border-color:#FFFFFF;" final-class="lyte-button lyteColorBtn" lyte-rendered=""> {{getI18n("zr.cw.sh.job")}} Share job via email

Share this job with your network

• openurl(event)">
• openurl(event)">
• openurl(event)">
• openurl(event)">
• openurl(event)">
• copyLink(event)">

• loadlistpage(event)"> Job listing
• Job details

Job Information

• Date Opened 09/08/2026
• Job Type Full time
• Industry IT Services
• City Hyderabad
• State/Province Telangana
• Country India
• Zip/Postal Code 500081

Job Description

Job Title:Data Engineer - ML Training Data Pipeline

Notice period: 0-30 Days

Experience : 5+ Years

Location: Hyderabad OR Pune

We are looking for Data Engineer - ML Training Data Pipeline who can Build and maintain the data pipeline that transforms raw production traces into high-quality training datasets for LLM fine-tuning-ingestion, deduplication, format conversion, quality filtering, and train/test splitting at scale on AWS.

What We Expect:

• Build

end-to-end data pipelines: raw trace ingestion → dedup → format conversion → quality gating → training-ready datasets

• Process

large-scale JSONL data on AWS S3 (tens of thousands of traces per batch)

• Convert

between chat-completion formats (e.g., OpenAI → Llama 3.1 tool-calling format)

• Implement

smart deduplication and sampling to balance training distribution

• Design

identity-aware train/test splits that measure true generalization

• Build

data validation gates to detect schema drift and format anomalies

• Create

a continuous pipeline that auto-processes new production traces for retraining

Requirements

• Experience: 6+

years data engineering focused on ML data pipelines

• Python: Strong — pandas,

pyarrow, JSONL processing at scale

• ML

Data Libraries: HuggingFace Datasets, Arrow-based storage

• Data

Formats: Multi-turn conversation/chat data structures and tokenizer-specific formatting

• Deduplication: Content

hashing, identity-based grouping strategies

• AWS: S3,

EC2, batch processing workflows

Preferred (Not Required): LLM training data prep (chat templates, tool-calling schemas); Axolotl or similar dataset formats; data versioning (DVC, LakeFS); browser-automation trace data or Playwright.

Benefits

• Comprehensive Medical Coverage:

Health insurance of INR 5.0 Lakhs for you and your family (up to 6 members), ensuring complete peace of mind.

• Robust Protection Plans:

Group Personal Accident Insurance and Group Term Life Insurance to safeguard you and your loved ones.

• Retirement Benefits:

PF and Gratuity provided as per standard government regulations.

• Flexible Work Options:

Enjoy hybrid work arrangements & flexible working hours.

• Generous Leave Policy:

21 days of annual leave, in addition to 10 company-declared holidays.

• Employee Well-being Spaces:

Access to a dedicated break-out area with round-the-clock refreshments for relaxation and rejuvenation.

check(event) ; career-website-detail-template-2 => apply(record.id,meta)" mousedown="lyte-button => check(event)" final-style="background-color:#6875E2;border-color:#6875E2;color:white;" final-class="lyte-button lyteBackgroundColorBtn lyteSuccess" lyte-rendered=""> I'm interested

onBlurDropdownCallback(this)">

filterAction(this)"> {{criteriaIndex}}

{{field}} {{captialize(condition)}}

( ) open()">{{cxPropHelpText}}

',0),'on','')}} {{if(showViewAll,'viewAllMode','')}} {{if(expHandlers(stackCollapsed,'&&',expHandlers(getLength(messages),'>',3)),'is-collapsed','')}} {{if(expHandlers(getLength(messages),'>',3),'has-stack-controls','')}} {{if(stackExpanding,'is-expanding','')}}" tabindex="{{if(expHandlers(stackCollapsed,'&&',expHandlers(getLength(messages),'>',3)),'0','-1')}}" focusin="rec-msg-push-notify => expandNotificationStack(event)" focusout="rec-msg-push-notify => collapseNotificationStack(event)" click="rec-msg-push-notify => expandNotificationStack(event)">

',cxPropOpenSearch),'?:',expHandlers('#dd-sch-','+',id),'')}}">

{{cxPropField.consent_details.description}}

{{record.Posting_Title}}

keydown(event,this)" focus="lyte-number => focus(event,this)">

{{record.Posting_Title}}

• openurl(event)">
• openurl(event)">
• openurl(event)">
• openurl(event)">
• openurl(event)">
• copyLink(event)">

• '),'disN','')}}" click="career-website-job-listing-layout1 => clickJob(this)">

{{record.Posting_Title}}

enableNav()">

enableNav()">

{{candidate.Email}}

keyup(event,this)" keypress="lyte-search => keypress(event,this)">

{{topMessage}}

Job Details

{{topMessage}}

showPrevJob()"> previous showNextJob()">next

{{title}}

{{score}}%

Job Description

{{unescape(sanitizeHTML(descriptionHTML))}}

hideJobDetail()">

{{message}}

Step {{curStepInMandatorySecPrompt}}/{{totalNumOfStepsInMandatorySecPrompt}}

submit()">

For privacy and security purposes, please go through the following points and provide consent.

sendPrivacyConsentMessage('Yes, I accept')">Accept sendPrivacyConsentMessage('No, I decline')">Decline

change(event,this)"> click(event)" keydown="lyte-fileupload => keydown(event)">

rowMouseDown(event)">

• loadlistpage(event)"> {{getI18n("zr.careers.publicpage.meta.joblisting")}}

• loadlistpage(event)"> {{getI18n("zr.careers.publicpage.meta.joblisting")}}

{{getI18n("zr.cw.autofill.head")}}

{{cxPropBodyMessage}}

{{getI18n("zr.careers.autopopulate.success")}}

{{cxPropUploadedFileName}} clearResume()">{{getI18n("crm.email.relay.label.clear")}}

{{getI18n("zr.cw.apply.sign",cxPropCompanyInfo.name)}}

easy_apply()">

seek_apply()"> {{if(cxPropDetails.applyWithSeekButton,cxPropDetails.applyWithSeekButton.buttonLabel,getI18n("zr.quickapply.apply.toggle","Seek"))}} {{getI18n('zr.eeo.questionnaire.portal.maintitle')}}

{{botName}}

{{getI18n("zr.zia.sb.consent.msg1")}}