Medior Data Engineer – Extraction & Ingestion (Python)

What you will be doing

You are co-responsible for the front door of our central data platform

• Lead design & delivery of Python ingestion jobs (scrapers, API connectors) with robust error handling, idempotency and documented backfill strategies.

• Define the first steps of ELT: validation, normalization and schema mapping and agree on data contracts with downstream consumers.

• Own observability & reliability: define Service Level Indicators and Objectives, add logs/metrics/alerts, drive and track error budgets.

• Set standards & mentor: create connector templates/runbooks, lead code reviews, pair with juniors, and coach on clean code and debugging.

• Design for ethics & resilience: handle pagination, throttling and anti-bot challenges respectfully; follow legal guidelines; secure secrets/PII via Azure Key Vault and least privilege access.

• Embed AI enrichment (Azure OpenAI & Cognitive Services) to classify, summarize and deduplicate content; set acceptance criteria for model outputs.

Projects you’ll tackle in the first year

• Lead migration processes of legacy scrapers to Azure Functions with retry logic, exponential back-off and structured logs, plan phases, risks and rollbacks.

• Connector standardization: ship reusable templates, runbooks and SLIs/SLOs for ingestion, and guide the team to adopt them.

• Document-to-insight pipeline: evolve extraction/vectorization/summarization at scale; improve throughput and cost per document.

• AI-powered news crawler: raise precision/recall and define evaluation metrics for summarization/deduplication.

Ongoing work you will contribute to

• Build & maintain Python ingestion jobs and first-step ELT pipelines with clear ownership boundaries.

• Deploy and observe services through Azure DevOps and Docker; proactively optimize cost/performance in Azure.

• Keep our data platform reliable, scalable and secure; lead post-incident improvements.

• Define and follow data-engineering best practices for design, testing and documentation; guide their adoption.

• Build and maintain cloud infrastructure in Azure to support scalable and secure data ingestion processes.

Tech stack

Daily tools

• Python (including Pandas, Selenium, and internal SDKs)

• API integration (REST, GraphQL)

• Azure DevOps (Git version control, CI/CD pipelines)

• Docker & Azure Container Registry

• Azure Function Apps & Logic Apps

You’ll also touch

• Azure AI Services & AI Foundry

• SQL (SQL Server Management Studio / Azure Data Studio)

• Azure Key Vault

• Azure Storage Services (Blob, Table, Queue)

• Azure Monitor / Application Insights

Haven’t used every tool yet? If you’ve mastered similar tech and love learning, we’ll back you.

Who you'll be working with:

Data Ingestion Architect • Python Data Engineers • Insights Analysts • Product Manager (Amsterdam‑based, collaborative and hands‑on)

Your Talent

• 4–6 years of professional software engineering experience, with strong Python skills.

• Proven end-to-end ownership of ingestion projects (from design to production) or major refactors in production.

• Led design reviews, wrote technical docs/runbooks.

• Git power user branching, rebasing, PR reviews and CI checks are second nature.

• Strong on clean code, debugging, architectural trade-offs and performance tuning in Azure.

• Nice to have: web scraping at scale, Selenium, SQL optimization, event-driven/streaming (e.g., Kafka/Event Hubs), Pandas/NumPy, experience in mentoring junior team members

What we offer:

• €55,000–€65,000 annually, including 8% holiday allowance.

• Performance bonus up to 10% of gross salary.

• Hybrid work: 2–5 days/week in our Amsterdam HQ.

• Personal growth plan and budget for both hard & soft skills.

• A supportive team of experienced engineers (pair‑programming, architecture sparring).

• Regular company events, dinners, BBQs, boat trips, padel tournaments, because fun fuels focus.

‍A-INSIGHTS

We enable the world’s biggest food companies like McCain, Royal FrieslandCampina, Ahold Delhaize, Puratos, Savencia and more to save time, deepen insights and seamlessly share knowledge. With a strong product, a clear mission and rapid growth ahead, there’s never been a better time to join us.

Our Amsterdam team is a vibrant mix of 15 + nationalities who blend curiosity with craft (and occasionally turn each other into works of art).

Ready to be our next secret ingredient?

Application process

• Online Skill‑Based Assessment: If we see a match, we'll invite you to take an online assessment.

• Screening Call: If the assessment is successful, you will receive a call from us with some follow‑up questions.

• In‑Person Interview: Intro chat in our Amsterdam office to assess team and culture fit.

• Case Preparation: You’ll work on a case to prepare for the next interview.

• In‑Depth Interview: We review your case and discuss your technical skills. Plenty of room for your questions.

• Offer: If everything aligns, we’ll make you an offer you can’t refuse.

  We aim to finalize the hiring process in 3–6 weeks.

‍We’re serious about inclusion

Research shows some groups only apply when they meet every requirement. If you’re excited by this role but your experience doesn’t line up perfectly, please apply anyway—we’d love to hear your story.

Ready to be our next secret ingredient?

Have Questions? For any inquiries or additional information, please contact Nofar Biran. 

• Email: nofar.biran@a-insighst.eu

• WhatsApp: +31 (0)611532011