Location: Australia
Workplace: remote
Overview
Are you passionate about languages, writing, and quality evaluation? We are launching an exciting language evaluation project and are looking for Chinese-speaking experts to join our team.
What you will do:
Identify realistic search-related failures where an AI model provides inadequate responses. Create search queries, test them, document what went wrong, and write instructions and evaluation criteria for correct answers.
Common issues: local relevance, source quality, outdated information, language mismatches, incomplete answers, and region-specific inapplicability. Testing multiple queries before finding failures is expected.
Project Details:
- Start Date: ASAP
- Duration: Ongoing Project.
- Commitment: 20 hours per week.
- Job Type: Freelance contract.
- Location: Remote, Australia
- Rate: $40 per hour.
Requirements:
- Research skills — ability to find and verify information from multiple sources; comfort navigating different websites, databases, and documentation
- Critical thinking — Can spot contradictions between sources, identify outdated information, and recognize when something doesn't apply to a specific locale/context
- Attention to detail — Can catch nuances in language, local variations (prices, regulations, requirements), and edge cases
- Problem-solving mindset — Comfortable iterating and testing when things don't work; doesn't get frustrated by trial- and-error
- Domain knowledge — Ideally someone with background in research, quality assurance, compliance, or similar work that requires precision and source verification
- Writing clarity — Can articulate instructions and rubrics clearly so others understand the logic behind their reasoning
Locale knowledge — Deep familiarity with their specific locale (local regulations, business practices, cultural nuances) is valuable for spotting region-specific failures
Location: Remote; Slovakia (Remote); Spain (Remote); Czechia (Remote); Switzerland (Remote); Portugal (Remote); USA (Remote); Austria (Remote); Iceland (Remote); Hungary (Remote); Ireland (Remote); UK (Remote); Romania (Remote); Poland (Remote)
Workplace: Remote
WHO WE ARE
Hi, we're DuckDuckGo, the online protection company and remote-first team of 400+ on a mission to raise the standard of trust online. Founded in 2008 and profitable since 2014, our annual revenue is in the hundreds of millions USD and millions use our browser on Mac https://spreadprivacy.com/introducing-duckduckgo-for-mac/, Windows https://spreadprivacy.com/windows-browser-open-beta/, iOS https://spreadprivacy.com/duckduckgo-privacy-browser-ios14/, and Android https://spreadprivacy.com/introducing-app-tracking-protection/, our search engine https://duckduckgo.com/, and the DuckDuckGo subscription https://duckduckgo.com/pro. We also offer private, useful, and optional AI, including Duck.ai https://duck.ai/, which lets you chat privately with ChatGPT, Claude, and other AIs, all in one place.
Our culture https://duckduckgo.com/how-we-work of trust, inclusivity, and empowered project management underpins everything we do, where each team member takes full ownership of their projects, from scoping and execution to postmortem. If you're seeking end-to-end ownership of your work, you've come to the right place!
YOUR TEAM AND ROLE
Working on the Data Science Functional Team, you'll drive product and business impact through statistical modeling, experiment design, Bayesian marketing analysis, and classifier implementation across our search engine, instant answers, AI, and revenue optimization, and work on related projects. Recent projects include:
- Defining core growth metrics for our products and building the infrastructure to monitor trends and anomalies
- Measuring the impact of major marketing campaigns for Duck.ai http://duck.ai, translating full-funnel data into actionable decisions
- Building monitoring pipelines to detect unexpected shifts in search traffic patterns faster and more reliably
- Expanding evaluation datasets for page context understanding across complex scenarios like multi-tab browsing and content reattachment
- Improving the trigger systems that determine when AI-assisted answers surface in DuckDuckGo Search, balancing quality and our privacy-first approach
Our data meets rigorous privacy standards and is necessarily sparse, so you'll need to think creatively to surface actionable insights in an evidence-first culture where quantitative rigor is the baseline, not the exception.
ABOUT YOU
- You have 7+ years of experience in Data Science and a track record of leading high-complexity projects from scoping to production with minimal direction.
- You can do 2 or more of the following:
- Deploy and iterate on ML in production environments
- Develop and deploy NLP solutions
- Apply knowledge of LLMs including fine-tuning, LLM-as-judge, and AI-assisted development
- Conduct marketing campaign analysis using Bayesian methods
- You write production-grade code in Python or a comparable high-level language.
- You have advanced SQL and query optimization skills, familiarity with columnar databases like BigQuery, Clickhouse, Redshift, or Druid; experience with a BI tool like Tableau or PowerBI is a plus.
- You can design, implement, and analyze experiments across product surfaces, and are comfortable deriving insights from sparse or privacy-constrained data.
- You collaborate effectively with product teams and contribute to overall team throughput.
COMPENSATION
This role is paid on a regional pay scale, in local currency, and stock options:
- United States: $190,500 USD annually
- United Kingdom: £134,600 GBP annually
- Canada: C$253,500 CAD annually
- Europe/Rest of World: €157,350 EUR annually
In Spain, pay is expected to transition to the EUR scale on January 1, 2027, and in other Europe/Rest of World countries on February 1, 2027.
Compensation is transparent across the organization, and all team members within the same professional level and global region receive the same compensation.
Eligibility for company-sponsored health benefits is limited to team members based in the United States. This program does not extend to team members located in other countries, such as Canada or the UK.
Our Team Member Support Guide https://duckduckgo.com/assets/hiring/team_support_guide.pdf explains how we prioritize your wellbeing including paid parental leave, office setup, and co-working allowances.
HIRING PROCESS
Hiring works best when it's a two-way street. Learn how we help you get to know DuckDuckGo, envision your future role here, and find out more about how we hire https://duckduckgo.com/how-we-hire.
DIVERSITY, EQUITY, AND INCLUSION
DuckDuckGo provides equal work opportunities to all team members and applicants, and it prohibits discrimination and harassment of any type on the basis of race, color, ethnicity, caste, religion, age, sex (including pregnancy), national origin, disability status, genetics, protected veteran status, sexual orientation, gender identity or expression, or any other characteristic protected by our policies or federal, state, or local laws.
We want to ensure that our hiring process is accessible. If you need reasonable accommodation for any part of the application process because of a medical condition or disability, please send an email to careers@duckduckgo.com to let us know the nature of your request.
PLEASE NOTE THAT:
- You’ll be required to attend meetings on camera via video conferencing
- Expect to travel at least two times a year: once for our all-hands meetup and again for a team retreat (each around 4-5 days). While extenuating circumstances may impact attendance, everyone is strongly encouraged to attend.
- While we offer a flexible work arrangement with no core hours, expect an average full-time commitment of 40 hours per week.
- A successful candidate must pass a background check as a condition of joining the team.
- By applying for this role, you confirm that all information submitted is accurate and complete. You further acknowledge that providing false or fraudulent information during the application process is cause for denial of an offer, revocation of any existing offer, or other adverse action, up to and including termination after the start of your commencement of work.
DISCLOSURE STATEMENT: USE OF AI IN HIRING PROCESS
As part of our commitment to enhancing our recruitment process, we utilize artificial intelligence (AI) technology to assist in reviewing and summarizing job applications and test projects, including those tools integrated into our recruitment vendor platforms. We use AI to flag potentially fraudulent applications, analyze and summarize applicants’ experience, interviews, and project performance, and help streamline our selection process.
KEY PRINCIPLES:
- Data Privacy: All information provided in your application will be handled in accordance with our Recruiting Privacy Policy https://duckduckgo.com/static-assets/files/pages/careers/DuckDuckGo-Recruiting-Privacy-Policy-effective-February-5-2025.pdf. We ensure that your personal information is protected and used solely for recruitment purposes.
- Human Oversight and Accountability: The AI technology is designed to support our hiring team by providing insights and summaries of applications and evaluations of test projects against scoring rubrics. All final evaluations and hiring decisions, however, will be made by our hiring team, who will consider the AI's input alongside other factors.
- Transparency: We believe in transparency regarding our hiring practices. If you have any questions about how AI is used in our recruitment process, please feel free to reach out to us.
By submitting your application, you acknowledge and consent to the use of AI technology in our review process. If you would like to request an alternative selection process, please contact us as at careers@duckduckgo.com. Thank you for your interest in joining DuckDuckGo!
#LI-DNI
• ClickHouse — основная экспертиза: проектирование MergeTree, партиционирование, TTL, оптимизация запросов. • ETL/ELT и оркестрация: Airflow, dbt, data quality, тесты, мониторинг свежести данных, data contracts.
- Разрабатывать и развивать пайплайны данных и платформенный слой, от которого зависят AI-продукты компании; - Анализировать использование LLM: стоимость, экономику токенов и кеширования, финансовую отчетность по расходам на AI;
– Поддержка заинтересованных сторон: проверка данных по мере необходимости, расследование расхождений между панелями мониторинга, обеспечение доступности платформы бизнес-аналитики. – Разработка и поддержка ETL/ELT-конвейеров для аналитических отчетов (когортичный доход, LTV, удержание, KPI по креативам, аналитика после покупки).
Lead the design and development of computer vision systems for biometrics (face attributes, detection, quality, and recognition). Rigorous fairness analysis and benchmarking of biometric models across various datasets and operating conditions.
-Define the long-term technical roadmap for Growth and Engagement ML systems, ensuring scalability, reliability, and measurable business impact. -Architect and deploy production-grade ML pipelines and real-time decisioning systems that power personalization, notification dispatch, and onboarding flows.
Location: Remote
Workplace: Remote
We are building AI to simulate the world through merging art and science.
We believe that world models are at the frontier of progress in artificial intelligence. Language models alone won’t solve the world’s hardest problems – robotics, disease, scientific discovery. Real progress requires models that experience the world and learn from their mistakes, the same way that humans do. And this kind of trial and error can be massively accelerated when done in simulation, rather than in the real world.
World models offer the most clear path to general-purpose simulation, changing how stories are told, how scientific progress is made and how the next frontiers of humanity are reached.
Our team consists of creative, open minded, caring and ambitious people who are determined to change the world. We aspire to continuously build impossible things and our ability to do so relies on building an incredible team. If you are driven to do the same, we'd love to hear from you.
*Titles posted to our careers page are updated regularly to best advertise the role and expected seniority, but internally we have a flat titling system with two types of titles: one for individual contributors and one for people managers.
ABOUT THE ROLE
We're looking for an ML infrastructure engineer to own model evaluation at Runway, end to end. Every decision we make about a model – which checkpoint to keep training, what to ship to millions of users, which datamix and architecture shows the most promise – rests on evals. Today that work is spread across the organization. You'll turn it into one platform.
Your job will be to design and build the systems that generate samples at scale, score them with automated metrics and human annotations, track results across checkpoints and releases, and put the answers in front of researchers in minutes rather than days. You'll define what "better" means operationally – how we measure it, how confident we are, and how a result becomes a ship/no-ship decision.
You'll be embedded within research teams as a member of ML Platform, and you'll set the technical direction for evals across the company. This is an extremely high-leverage role: the quality of our models is bounded by how well we can measure them.
A PEEK AT OUR TECHNICAL STACK
Our inference and training code is written in Python (PyTorch) and is cloud-native. We leverage multiple clouds and use a combination of Kubernetes native and home-grown tooling for efficient job orchestration.
We’re big users of agentic development and operations. We have a robust internal platform and provide broad access to the latest models and harnesses.
You’ll have access to best-in-class models, agents, GPUs, storage and cloud services you need to drive a world class evaluation pipeline that helps us deliver frontier models.
WHAT YOU'LL DO
- Own the evaluation platform end to end: the tooling and systems required to generate, annotate, review and adapt at frontier scale
- Define the CLIs, APIs, GUIs and storage layers required to make this process seamless, fast, sophisticated and collaborative
- Work directly with research teams on video, image, audio, agents, and robotics to understand what they need to measure and build it, then generalize the result into the platform
- Help set the standards for how we evaluate models at Runway: reproducibility, metric definitions, reporting formats, and when a result is trustworthy enough to act on
- Support broad adoption of the platform and its integration throughout Runway’s research efforts across training, production model serving and more
- Contribute broadly as a member of the ML Platform team to tools and systems that help Runway train and serve frontier models
WHAT YOU'LL NEED
- 5+ years of experience building ML infrastructure or data platforms in production environments, with at least some of that time spent on evaluation, experimentation, or benchmarking systems
- Strong Python and PyTorch, and hands-on experience running large batch GPU workloads on Kubernetes
- Experience designing data pipelines and storage for large volumes of media or model outputs, with attention to versioning and reproducibility
- Familiarity with experimental statistics: paired comparisons, confidence intervals, multiple-comparison pitfalls, inter-rater agreement
- Comfort building internal tools end to end, from the command line to the browser
- Ability to lead a broad technical area: gather requirements from many teams, set direction, make tradeoffs, and drive a roadmap without waiting to be told
- Familiarity with the full model development lifecycle: data, training, evaluation, serving
- Self-starter who can work embedded with research teams and move fast
- Strong systems thinking and pragmatic approach to production reliability
- Humility and open mindedness; at Runway we love to learn from one another
NICE TO HAVE
- Track record of building evaluation suites for generative models
- Hands-on work with LLM- or VLM-as-judge pipelines
- Experience with online experimentation platforms and connecting offline metrics to product outcomes
- Prior work evaluating agents or robotics policies
Runway strives to recruit and retain exceptional talent from diverse backgrounds while ensuring pay equity for our team. Our salary ranges are based on competitive market rates for our size, stage and industry, and salary is just one part of the overall compensation package we provide.
There are many factors that go into salary determinations, including relevant experience, skill level and qualifications assessed during the interview process, and maintaining internal equity with peers on the team. The range shared below is a general expectation for the function as posted, but we are also open to considering candidates who may be more or less experienced than outlined in the job description. In this case, we will communicate any updates in the expected salary range.
Lastly, the provided range is the expected salary for candidates in the U.S. Outside of those regions, there may be a change in the range, which again, will be communicated to candidates.
WORKING AT RUNWAY
Great things come from great teams. https://www.youtube.com/watch?v=kwmj4ato2kw&ab_channel=Runway We’d love to hear from you.
We’re committed to creating a space where our employees can bring their full selves to work and have equal opportunity to succeed. So regardless of race, gender identity or expression, sexual orientation, religion, origin, ability, age, veteran status, if joining this mission speaks to you, we encourage you to apply.
More about Runway
- Universal World Simulator https://runwayml.com/world-simulator.html
- GWM-1 https://runwayml.com/research/introducing-runway-gwm-1
- Gen-4.5 https://runwayml.com/research/introducing-runway-gen-4.5
- General World Models https://runwayml.com/research/introducing-general-world-models
- Robotics SDK https://runwayml.com/research/introducing-runway-gwm-1#robotics-section
- Conversational Real-time Agents https://runwayml.com/research/introducing-runway-gwm-1#avatars-section
- Runway Studios https://runwayml.com/studios
We're excited to be recognized as a best place to work:
Crain's https://www.crainsnewyork.com/awards/best-places-work-2023 | InHerSight https://www.inhersight.com/companies/best/city/new-york-city-ny | BuiltIn NYC https://builtin.com/awards/new-york-city/2024/best-places-to-work | INC https://www.inc.com/best-workplaces/2024
• Design and develop human-like, action-level chatbots that not only converse naturally but also initiate and execute user workflows inside the product. • Participate in the architecture and development of an intelligent, context-aware translation system that preserves meaning, tone, and style.
Location: Philippines; India
Workplace: remote
About the Role:
Review pre-segmented audio recordings, verify word-level segment accuracy, and correct timestamp or transcription errors to support training data for speech-to-speech AI models
Location: India/ Philippines (Remote)
Engagement Type: Freelance (Pilot)
Rate: USD 4.55/hour
Estimated Hours: 30 hours/week
Key Responsibilities
Listen to short audio clips and review automatically generated word-level segments (start timestamp, end timestamp, transcription)
Verify segment boundaries align with actual speech onset and offset
Review start/end timestamps when boundaries are inaccurate
Identify and flag tasks for discard per defined criteria (unintelligible audio, non-target language, sensitive content, no voice activity, etc.)
Maintain consistency and accuracy across a detailed, evolving style guide
Requirements
English fluency, including strong command of spoken/colloquial forms (contractions, filler words, informal speech)
Prior experience with audio transcription, data annotation, or speech/NLP data labeling preferred
Comfortable using annotation/labeling tools
Ability to reference and apply a detailed style guide consistently
Ability to work independently and adapt to periodic guideline updates
#L1-CC1
Задачи:
— улучшать рекомендации
— искать гипотезы
— прототипировать на GPU
— выкатывать в A/B-тесты
Требования:
— опыт 3+ года
— знание Python, SQL, Spark, Docker, Git, ClickHouse, Redis, Deep Learning
Условия:
— удалённая работа или офис
— фултайм
УдалённоseniorГеография найма не уточнена
УдалённоseniorRemote; Slovakia (Remote); Spain (Remote); Czechia (Remote); Switzerland (Remote); Portugal (Remote); USA (Remote); Austria (Remote); Iceland (Remote); Hungary (Remote); Ireland (Remote); UK (Remote); Romania (Remote); Poland (Remote)
УдалённоСША; Lastly, the provided range is the expected salary for candidates in the U.S
Данные и ML вакансии - удалённо - страница 9 · Job Hunters
