{"id":2446,"date":"2026-07-08T08:53:06","date_gmt":"2026-07-08T08:53:06","guid":{"rendered":"https:\/\/tradefinder.help\/?p=2446"},"modified":"2026-07-08T08:53:06","modified_gmt":"2026-07-08T08:53:06","slug":"mindrift-senior-software-engineer-ai-agent-evaluation","status":"publish","type":"post","link":"https:\/\/tradefinder.help\/?p=2446","title":{"rendered":"Mindrift: Senior Software Engineer &#8211; AI Agent Evaluation"},"content":{"rendered":"<p><img decoding=\"async\" src=\"https:\/\/we-work-remotely.imgix.net\/logos\/0171\/5754\/logo.gif?ixlib=rails-4.0.0&amp;w=50&amp;h=50&amp;dpr=2&amp;fit=fill&amp;auto=compress\" \/><\/p>\n<p>\n  <strong>Headquarters:<\/strong> Canada<br \/>\n    <br \/><strong>URL:<\/strong> <a href=\"https:\/\/mindrift.ai\/\" target=\"_blank\" rel=\"noopener\">https:\/\/mindrift.ai\/<\/a>\n<\/p>\n<p><em>Please submit your CV in English and indicate your level of English proficiency.<\/em><\/p>\n\n<p>Mindrift connects specialists with project-based AI opportunities for leading tech companies, focused on testing, evaluating, and improving AI systems.\u00a0<strong>Participation is project-based, not permanent employment.<\/strong><\/p>\n\n<p><strong>What this opportunity involves<\/strong>\u00a0<\/p>\n<p>We&#8217;re building a dataset to evaluate AI coding agents &#8211; how well a model handles real-world developer tasks.<\/p>\n\n<p>You&#8217;ll create challenging tasks and evaluation criteria within realistic simulated environments:<\/p>\n<ul>\n<li>Build realistic developer environments &#8211; a virtual company with codebase, infrastructure, and context (tickets, docs, conversations) that forms a believable development history<\/li>\n<li>Design tasks from intermediate states of these environments &#8211; craft the prompt, define what &#8220;solved&#8221; means, and ensure the task is solvable by an AI agent<\/li>\n<li>Write tests that verify agent solutions &#8211; accept all valid approaches and reject incorrect ones, neither too strict nor too lenient<\/li>\n<li>Iterate on tasks and tests based on QA feedback &#8211; review agent solutions, analyze failures, and refine until the evaluation is fair and robust<\/li>\n<\/ul>\n<p><strong>What this is NOT<\/strong><\/p>\n<ul>\n<li>Not data labeling<\/li>\n<li>Not prompt engineering<\/li>\n<li>Not writing code from scratch &#8211; the agent writes most of the code; you guide and evaluate<\/li>\n<\/ul>\n<p><strong>What we look for<\/strong><\/p>\n<ul>\n<li>5+ years in software development<\/li>\n<li>Core stack: Python (FastAPI), JavaScript\/TypeScript (React), Docker, Postgres, Kafka, Redis<\/li>\n<li>Experience writing tests (functional, integration)<\/li>\n<li>English proficiency &#8211; B2+<\/li>\n<\/ul>\n<p><strong>Why this is hard\u00a0<\/strong><\/p>\n<p>Frontier models are already good at coding. Creating a task that genuinely challenges the best models is non-trivial. You need to deeply understand where models fail and what scenarios reveal the difference between a good and a bad solution. Tasks have many valid solutions &#8211; writing tests that accept all correct solutions and reject incorrect ones is harder than it sounds.<\/p>\n\n<p><strong>How it works<\/strong><\/p>\n<p>Apply \u2192 Pass qualification(s) \u2192 Join a project \u2192 Complete tasks \u2192 Get paid<\/p>\n\n<p><strong>Effort estimate<\/strong><\/p>\n<p>Tasks for this project are estimated to take 20 hours to complete, depending on complexity. This is an estimate and not a schedule requirement; you choose when and how to work. Tasks must be submitted by the deadline and meet the listed acceptance criteria to be accepted.<\/p>\n\n<p><strong>Compensation<\/strong><\/p>\n<p>Up to <strong>$50\/hr equivalent<\/strong>, depending on level and pace. Tasks are estimated at ~20 hours each; you set your own schedule.<\/p>\n<p><strong>To apply:<\/strong> <a href=\"https:\/\/weworkremotely.com\/remote-jobs\/mindrift-senior-software-engineer-ai-agent-evaluation\" target=\"_blank\" rel=\"noopener\">https:\/\/weworkremotely.com\/remote-jobs\/mindrift-senior-software-engineer-ai-agent-evaluation<\/a><\/p>","protected":false},"excerpt":{"rendered":"<p>Headquarters: Canada URL: https:\/\/mindrift.ai\/ Please submit your CV in English and indicate your level of English proficiency. Mindrift connects specialists with project-based AI opportunities for leading tech companies, focused on testing, evaluating, and improving AI systems.\u00a0Participation is project-based, not permanent employment. What this opportunity involves\u00a0 We&#8217;re building a dataset to evaluate AI coding agents &#8211; [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-2446","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"featured_image_url":[],"post_author":"sonifashionhouse@gmail.com","assigned_categories":"Uncategorized","jetpack_featured_media_url":"","_links":{"self":[{"href":"https:\/\/tradefinder.help\/index.php?rest_route=\/wp\/v2\/posts\/2446","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/tradefinder.help\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/tradefinder.help\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/tradefinder.help\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/tradefinder.help\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=2446"}],"version-history":[{"count":0,"href":"https:\/\/tradefinder.help\/index.php?rest_route=\/wp\/v2\/posts\/2446\/revisions"}],"wp:attachment":[{"href":"https:\/\/tradefinder.help\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=2446"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/tradefinder.help\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=2446"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/tradefinder.help\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=2446"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}