Applied AI Engineer

Location: Warsaw, PL We bring startups and disruptive tech companies to the local market, offering engineers the opportunity to work on innovative, high-impact products. Join us and become a member of their R&D team while we take care of employment, HR support, a benefits package, and an enjoyable work environment. Handle (formerly 44pixels) is a London-based, VC-backed startup founded in 2024. It started as a venture studio building AI-first mobile apps (Cue — AI meeting companion, Wordcast — audio-reading app, Clara — AI receptionist) and has since focused on one product: Handle. ‍ Handle is an AI agent for owners of local service businesses (groomers, roofers, bookkeepers, caterers, etc.). The owner chats with it like with ChatGPT, and it handles routine work: replying to customer messages, chasing invoices, answering reviews. It connects to Gmail, Outlook, Google Calendar, WhatsApp, Messenger, QuickBooks, Xero and other tools. The owner decides what Handle can access and what needs approval. We’re looking for a hands-on Applied AI Engineer to own the quality of Handle’s AI agent. You’ll expand and run the agent’s evals and benchmarks, drive the hill-climb loop on its skills, and help make the agent reliable, useful, and competitive. ‍ Handle’s engineers have already laid the groundwork for evals and own the harness, tracing, and deployment they run on. You’ll take it from there, focusing on measuring and improving agent quality. Build and own the eval suite: scenarios, rubrics, graders, benchmarks, and quality gates. Form research hypotheses, run controlled experiments, and separate model effects from prompt, tool, skill, and agent-structure effects. Version skills and changes, rerun the full evaluation suite, diagnose regressions, and follow improvements through to verified results. Evaluate conversation quality, style, user understanding, and task reliability. Test meaningful new models and assess portability, failure modes, and when a simpler single-model strategy is better. Recommend agent or skill restructuring when repeated failures show that local fixes are not enough. Proven experience in building and running evals for an LLM product in production, ability to write the code. Deep interest in agent behavior, model capabilities, and evaluation. Enough technical fluency to run experiments, inspect failures, and collaborate closely with engineers. Ability to move between theory and concrete product problems. Proactive taste: you notice patterns, propose better approaches, and follow them through. Hands-on post-training or fine-tuning of open-weight models. At On The Spot, we’re creating a workplace people would want to stay in: supportive, well-organized, and fair in terms of conditions and benefits. ‍ We’ll share more about our benefit package during the interviews. In the meantime, you can always check our Glassdoor reviews .