Product Analyst - Intelligence

Location: Indonesia; India Workplace: fully We're Osome - an international fintech startup making the lives of entrepreneurs easier. We help thousands of businesses kick admin, accounting and bookkeeping out of their day-to-day, so they can spend more time on what's important to them. We've developed a unique solution that combines SaaS with a human-in-the-loop approach to provide full-fledged services in real-time. We're experiencing tremendous growth in both clients and team members. We have over 500 people in our global offices. We're looking for more bright minds who'd love to change the world by solving complex problems. About the role: As Product Analyst – Intelligence, you’ll help Osome understand how well our AI-powered features and agents perform, where they fall short, and how they improve customer and operational outcomes. You’ll partner with Product, Engineering, and Operations to turn product usage, AI outputs, and human feedback into clear recommendations. Your work will span product analytics, AI evaluation, and experimentation, using our data stack: dbt, BigQuery, Spectacles, and Looker. What you'll do: Define and track metrics for AI-powered features, including adoption, task completion, output accuracy, human intervention, response time, and cost per completed task. Analyse AI outputs, product usage, and user feedback to identify failure patterns, explain changes in performance, and recommend improvements to the Intelligence team. Build and maintain dashboards in Looker to monitor AI quality, customer experience, and operational impact, including time saved, rework, and escalations. Partner with Product and Engineering to define tracking events, validate instrumentation, and maintain reliable datasets that connect AI activity to product and business outcomes. Work with domain experts to build and maintain evaluation datasets, scoring criteria, and human-review processes; check that automated evaluation scores reflect real-world quality. Design and analyse experiments to compare changes to prompts, models, and workflows, measuring quality improvements alongside cost, speed, and customer outcomes. Communicate findings clearly and help the team define launch criteria, monitor performance after release, and prioritise opportunities for reliable automation. Who you are: A degree in a quantitative discipline, or equivalent practical experience in analytics. Proven experience in product analytics or a similar analytical role, with examples of insights that influenced product decisions. Advanced SQL skills and experience working with complex datasets, data quality checks, and BI dashboards. Strong grounding in statistics and experimentation, including defining success metrics, interpreting results, and explaining uncertainty. Ability to turn ambiguous product questions into measurable analyses and communicate practical recommendations to technical and non-technical stakeholders. Familiarity with AI/LLM products and their evaluation challenges, including variable outputs, incorrect answers, and the need for human review. Nice to have: Python skills for data analysis, working with APIs, or automating evaluation workflows. Hands-on experience analysing AI-powered features, building evaluation datasets, or using LLM evaluation and observability tools. Experience with dbt, BigQuery, Looker, Amplitude, or similar tools. Experience in fintech, accounting, or operational automation, and handling sensitive data responsibly.