As a Senior Data Engineer, you will play a pivotal role in the extraction, transformation, and delivery of data from various sources. You will also be responsible for implementing MLOps practices, serving APIs, performing web scraping, and automating data processes. Your deep expertise in machine learning and proficiency in technology stacks such as GCP, Linux, BigQuery, and Airflow will be essential in delivering high-quality data solutions.
- Data products integrations – DMP (Lotame), 3rd party software (Comscore, Chartbeat, etc.),& Google’s products (GA, GTM)
- Implementation & development of Data Engineering pipelines from data ingestion, ETL/ELT, middleware & back-end developments i.e. API as a service and building microservices for data pipelines.
- Cloud Engineering domain expertise – setting up and provisioning cloud services i.e. VMs, DBs, cloud security, etc.
- Domain knowledge and experiences in Big Data Analytics, Media & Entertainment
- Lead the extraction of data from media social platforms (e.g., Facebook, Twitter, Instagram) and APIs using advanced tools and techniques.
- Develop, deploy, and maintain scalable and efficient data pipelines to ingest, transform, and load data from various sources into the data warehouse.
- Collaborate closely with data scientists and analysts to understand data requirements and design optimal solutions.
- Implement MLOps practices, including model deployment, monitoring, and version control, to ensure smooth integration of machine learning models into production environments.
- Serve APIs to enable real-time access to data for internal and external stakeholders.
- Lead web scraping efforts to gather relevant data from websites and integrate it into the data ecosystem.
- Automate data processes, ensuring efficiency, reliability, and accuracy in data handling and processing.
- Monitor and optimize data pipelines and workflows for performance, reliability, and scalability.
- Collaborate with cross-functional teams, including data scientists, analysts, and software engineers, to implement end-to-end data solutions.
- Stay up-to-date with emerging technologies, industry trends, and best practices in data engineering, machine learning, and cloud platforms.
Skills/ Qualifications/ Experience:
- Bachelor’s or Master’s degree in Computer Science, Engineering, or a related field.
- Minimum 7 years of professional experience as a Senior Data Engineer or a similar role.
- Proficiency in data extraction from media social platforms (e.g., Facebook Graph API, Twitter API, Instagram API) and other external data sources.
- Strong experience in developing and operating data pipelines using tools like Apache Airflow or similar workflow management systems.
- Solid understanding of MLOps principles and experience deploying machine learning models into production environments.
- Knowledge of web scraping techniques and tools (e.g., Beautiful Soup, Scrapy).
- Strong programming skills in languages such as Python, and SQL, and familiarity with Linux environments.
- Experience working with cloud platforms, especially Google Cloud Platform (GCP), including services like BigQuery, Dataflow, Pub/Sub, and Cloud Storage.
- Ability to design, optimize, and tune data pipelines for performance, scalability, and reliability.
- Excellent problem-solving skills and attention to detail.
- Strong communication and collaboration skills to lead and work effectively in a cross-functional team environment.
- Experience with cloud platforms, such as GCP, AWS, or Azure.
- Knowledge of machine learning concepts and experience working with ML frameworks (e.g., TensorFlow, PyTorch).
- Familiarity with containerization technologies (e.g., Docker, Kubernetes).
- RDBMS – MySQL, PostGRES, Oracle
- NoSQL – MongoDB, Elasticsearch
- Cloud Engineering – AWS, GCP, Azure, Kubernetes & Containerize (Docker)
- Distributed Computing Domain Knowledge – Spark, Hadoop (optional), Hive, PIG
- Software Engineering – Python, JAVA, JS
- Domain knowledge in Media, Digital Ads & Digital Marketing
- Work with a young and vibrant team in a fast-paced startup environment, embracing new and exciting challenges daily.
- Opportunity to work on exciting projects, leveraging your skills in media social data extraction, data pipeline development, MLops, web scraping and automation.
- Stay ahead of the curve by upskilling your tech knowledge and adapting to the latest technologies in the field.
- Experience upgrading career development within the company, unlocking new and exciting opportunities.
- Enjoy an attractive remuneration package that rewards your dedication and commitment.
- Benefit from flexible working arrangements, including a hybrid mode setup
- Engage in monthly team events/activities that foster camaraderie and create lasting memories.
- Join a team that values work-life balance and supports you in finding the perfect equilibrium
If this opportunity captures your interest, we welcome you to share your CV/Resume along with your introduction, notice period (if applicable), and your expected salary range to firstname.lastname@example.org.