As a global leader in artificial intelligence technology, Quantcast is using machine learning to inspire human learning to help brands grow in the AI-era. Founded in 2006, Quantcast is the the world’s largest AI-driven direct audience insights and measurement platform. Measuring over 100 million web and mobile destinations, Quantcast has created the “consumer behavior graph” becoming the real-time pulse of the Internet helping marketers, publishers, agencies and consultancies understand, optimize and predict consumer interactions to drive business outcomes. Headquartered in San Francisco. Quantcast employs approximately 800 employees in 22 offices across 10 countries.
Quantcast
CA, USStaff Software Engineer (Lead), Scale and Transport
At Quantcast, data is king, and the
Scale & Transport team
knows how to process it. The unique features of our custom map-reduce implementation include a revolutionary petabyte sort mechanism, an efficient and flexible petabyte join and lookup, based on innovative data indexing and partitioning mechanisms, dynamic task size adjustment in running jobs, and a self-tuning system that learns and improves job performance without developer intervention. In the last 5 years it has allowed Quantcast to scale from processing 400 to 40,000 terabytes of data per day. For a workload consisting primarily of ETL, analytics, and data mining, our big data processing platform is on average 4 to 5 times more efficient than the most popular open source map-reduce system. Now Quantcast is looking to marry the ergonomics of Spark with the adaptability and performance of our home-grown stack. Are you the Staff Engineer who can bring us into the next generation of distributed computing? You would integrate and improve new open source technologies and invent your own. Responsibilities:
- Co-own the vision for peta-scale computation at Quantcast, from strategy to implementation
- Work with engineers using Quantcast’s distributed computing infrastructure to anticipate and meet their needs, with an eye to business value
- Design and develop high performance distributed computing systems
- Diagnose and solve big data problems with system engineering solutions
- Perform design, architecture, and code reviews
- Mentor and grow junior team members
- Innovate in the big data space: lead, not follow
Requirements:
- BS in Computer Science or equivalent experience
- 8 years of industry experience in software development
- Experience working & optimizing the performance of large-scale distributed systems
- Proficient in C++ and/or Java
- Strong software development skills (version control, code review, testing, release, monitoring, debugging)
- Proficiency and high level of insight into Linux OS and running software systems on Linux