Python + PySpark - Junior Developer Job at Compunnel, Inc., New York, NY

SlVHUXVINm41WFQ3R0pjVjhFY2NyY1ppTFE9PQ==
  • Compunnel, Inc.
  • New York, NY

Job Description

The Python + PySpark + AWS Consultant will be responsible for designing, developing, and optimizing data engineering solutions using Python and PySpark. The role requires hands-on experience in building scalable ETL pipelines, working with cloud platforms such as AWS or Azure, and ensuring data integrity and performance optimization. The candidate should have strong SQL skills and a deep understanding of data engineering principles. Key Responsibilities

  • Develop and maintain data processing workflows using Python and PySpark
  • Design and implement ETL pipelines for structured and unstructured data
  • Optimize Spark-based data processing for efficiency and scalability
  • Deploy and manage data solutions on AWS or Azure
  • Write and optimize SQL queries for data transformation and analysis
  • Troubleshoot and resolve performance issues in data pipelines
  • Work closely with cross-functional teams to ensure data reliability and integrity
Required Qualifications
  • 5+ years of experience in data engineering
  • Strong proficiency in Python and object-oriented programming
  • Hands-on experience with PySpark for large-scale data processing
  • Proficiency in SQL for data manipulation and query performance tuning
  • Experience with AWS or Azure for cloud-based data solutions
  • Knowledge of ETL processes and data pipeline automation
  • Experience with Hadoop is acceptable
Preferred Qualifications
  • Experience in optimizing Spark jobs for performance and cost efficiency
  • Familiarity with DevOps practices for data engineering
  • Understanding of data governance, security, and compliance best practices
#J-18808-Ljbffr Compunnel, Inc.

Job Tags

Similar Jobs

Universal General Contractors

Project Manager Job at Universal General Contractors

 ...The Project Manager is responsible for overseeing and managing all aspects of our construction projects from conception to completion. This...  ...Communication Problem-Solving Time Management Contract Negotiation Salary will commensurate with experience.The... 

Headway

LCSW (Virtual) Job at Headway

 ...Remote Licensed Clinical Social Worker (LCSW) Wage: Between $95-$122 an hour Are you a Licensed Clinical Social Worker looking...  ...more through insurance! Headway supports thousands of mental health clinicians across the United States in our mission to rewire the... 

Holland Special Delivery

Dedicated Route - 3rd Shift - CDL Truck Driver Job at Holland Special Delivery

 ...holidays and equipment. We do everything that we say! We're looking for drivers that deliver products to our customers in a safe and timely manner. *Advantages of driving Local at Holland Special Delivery:** Home daily!* Dedicated Route * *3rd Shift Premium**... 

Medworks

Registered Needed For Great Team! High Paying 13 Week Assignment Job at Medworks

 ...a local group in the Towson area. This 13 week assignment offers extension potential, a supportive team and guaranteed 40 hours. We pay our Registered Nurses hourly, ensuring they are paid for drive time in addition to mileage. The ideal RN will have 2+ years of home health... 

CHRISTUS Health

CLS, Clinical Laboratory Scientist II - Atlanta Lab - PRN Job at CHRISTUS Health

 ...accredited laboratory within the last 5 years. Licenses, Registrations, or Certifications Must possess Certification for MT, MLS, CLS, or MLT from ASCP, AMT, AAB, or HEW/HHS If certification with an expiration date is held, the certification must be maintained and...