Evaluation Scenario Writer

5 days ago

Pretoria, Gauteng, South Africa Mindrift Full time R20 000 - R25 000 per year

This opportunity is only for candidates currently residing in the specified country. Your location may affect eligibility and rates. Please submit your resume in English and indicate your level of English.

At Mindrift, innovation meets opportunity. We believe in using the power of collective human intelligence to ethically shape the future of AI. 

What we do

The Mindrift platform, launched and powered by Toloka, connects domain experts with cutting-edge AI projects from innovative tech clients. Our mission is to unlock the potential of GenAI by tapping into real-world expertise from across the globe.

About the Role

We're looking for someone who can design realistic and structured evaluation scenarios for LLM-based agents. You'll create test cases that simulate human-performed tasks and define gold-standard behavior to compare agent actions against. You'll work to ensure each scenario is clearly defined, well-scored, and easy to execute and reuse. You'll need a sharp analytical mindset, attention to detail, and an interest in how AI agents make decisions.

Although every project is unique, you might typically:

  • Designing structured test scenarios based on real-world tasks.
  • Defining the golden path and acceptable agent behavior.
  • Annotating task steps, expected outputs, and edge cases.
  • Working with devs to test your scenarios and improve clarity.
  • Reviewing agent outputs and adapting tests accordingly

How to get started

Simply apply to this post, qualify, and get the chance to contribute to projects aligned with your skills, on your own schedule. From creating training prompts to refining model responses, you'll help shape the future of AI while ensuring technology benefits everyone.

  • Bachelor's and/or Master's Degreein Computer Science, Software Engineering, Data Science / Data Analytics, Artificial Intelligence / Machine Learning, Computational Linguistics / Natural Language Processing (NLP), Information Systems or other related fields.
  • Background in QA, software testing, data analysis, or NLP annotation.
  • Good understanding of test design principles (e.g., reproducibility, coverage, edge cases).
  • Strong written communication skills in English.
  • Comfortable with structured formats like JSON/YAML for scenario description.
  • Can define expected agent behaviors (gold paths) and scoring logic.
  • Basic experience with Python and JS.
  • Curious and open to working with AI-generated content, agent logs, and prompt-based behavior.
  • You are ready to learn new methods, able to switch between tasks and topics quickly and sometimes work with challenging, complex guidelines.
  • Our freelance role is fully remote so, you just need a laptop, internet connection, time available and enthusiasm to take on a challenge.

Nice to Have

  • Experience in writing manual or automated test cases.
  • Familiarity with LLM capabilities and typical failure modes.
  • Understanding of scoring metrics (precision, recall, coverage, reward functions).

Contribute on your own schedule, from anywhere in the world. This opportunity allows you to:

  • Get paid for your expertise, with rates that can go up to $22/hour depending on your skills, experience, and project needs.
  • Take part in a flexible, remote, freelance project that fits around your primary professional or academic commitments.
  • Participate in an advanced AI project and gain valuable experience to enhance your portfolio.
  • Influence how future AI models understand and communicate in your field of expertise.

  • YouTube Channel Writer

    1 week ago

    Pretoria, Gauteng, South Africa UFD Full time R400 000 - R600 000 per year

    About the jobJoin Our YouTube Channel TeamAre you a tech enthusiast with a knack for storytelling? We're looking for a creative and talented Tech Content Writer to join our dynamic YouTube team If you love geeking out over the latest in PC hardware, creating captivating scripts, tinker with the latest gear, and coming up with original ideas that make...

  • Intermediate Software Developer

    3 days ago

    Pretoria, Gauteng, South Africa Interdot Solutions Full time R120 000 - R240 000 per year

    Duties and Responsibilities:Support existing applications and develop new functionality and components.Rewrite legacy system developed in MS Access into web applicationRefactor and improve the code to keep the systems maintainable and adaptable to new requirements.Work effectively in a team environment and follow development processes and...

  • Software Developer

    1 week ago

    Pretoria, Gauteng, South Africa Betway Africa Full time R800 000 - R1 200 000 per year

    Kick-start your career in the online gaming world and experience the very latest in technology and innovation.Software DevelopmentThe Software Developer develops software solutions by studying information needs; conferring with users; studying systems flow, data usage and work processes; investigating problem areas and following the software development...

  • Business Analyst

    3 days ago

    Pretoria, Gauteng, South Africa Boardroom Appointments Full time R120 000 - R250 000 per year

    Our client in the Financial sector is looking for a Business AnalystMinimum Requirements:QualificationsIT qualification, Certificate in BA endorsed by IIBA, Safe Certification.5 - 8 years of BA experience, managing multiple IT Projects and exposure to complex projects, business exposure, and industry exposure. Banking Experience:Collections &...

  • 360 Recruitment Specialist

    5 days ago

    Pretoria, Gauteng, South Africa First Point Group Full time R400 000 - R800 000 per year

    First Point Groupis a global specialist in the technology recruitment sector, with over 15 years of experience supporting the telecoms and tech industry across 6 continents. As a market leader, we have built lasting relationships with leading companies and innovative startups, helping professionals find roles where they can excel and drive industry change....

  • Senior Institutional Researcher

    7 days ago

    Pretoria, Gauteng, South Africa University of Pretoria Full time R1 000 - R2 500 per year

    UP Professional and SupportDEPARTMENT OF INSTITUTIONAL PLANNING, MONITORING, AND EVALUATIONSENIOR INSTITUTIONAL RESEARCHERPEROMNES POST LEVEL 6In pursuit of the ideals of excellence and diversity, the University of Pretoria wishes to invite applications for the following vacancy.The University of Pretoria's commitment to quality makes us one of the top...

  • Youtube Channel Writer

    2 weeks ago

    Pretoria, South Africa Ufd Full time

    About the job Join Our YouTube Channel Team Are you a tech enthusiast with a knack for storytelling? We’re looking for a creative and talented Tech Content Writer to join our dynamic YouTube team. If you love geeking out over the latest in PC hardware, creating captivating scripts, tinkering with the latest gear, and coming up with original ideas that make...

  • Software Developer

    3 weeks ago

    Pretoria, South Africa Betway Africa Full time

    Kick-start your career in the online gaming world and experience the very latest in technology and innovation. Software Development The Software Developer develops software solutions by studying information needs, conferring with users, studying systems flow, data usage and work processes, investigating problem areas and following the software development...

  • Dutch Language Specialist

    1 week ago

    Pretoria, South Africa Invisible Agency Full time

    Are you a Dutch language expert eager to shape the future of AI? Large‑scale language models are evolving from clever chatbots into powerful engines of linguistic discovery. With high‑quality training data, tomorrow's AI can democratize world‑class education, keep pace with cutting‑edge research, and streamline communication for Dutch speakers...

  • Educational Psychologist

    1 day ago

    Pretoria, South Africa Dalitso Holdings Full time

    To align and standardise all specialist practices, processes, services, solutions and support consistent with the governance mechanisms and operating frameworks benchmarked and ensure the availability of real time accurate data and information for effective decision-making and information sharing. **Responsibilities**: - Establish and maintain...

  • Software Developer

    3 weeks ago

    Pretoria, South Africa Betway Africa Full time

    Kick-start your career in the online gaming world and experience the very latest in technology and innovation. Software Development The Software Developer develops software solutions by studying information needs, conferring with users, studying systems flow, data usage and work processes, investigating problem areas and following the software development...

  • Test Specialist

    1 week ago

    Pretoria, South Africa iSanqa Resourcing Full time

    About the job Test Specialist (Contract) - Gauteng/Hybrid - ISB7402579 Our client requires the services of aSoftware Engineer (Advanced) -Midrand/Menlyn/Rosslyn/Home Office rotation Amazing brand with cutting edge technology Excellent teams in Global team collaboration High work-life balance with Flexible hours Agile working environment **POSITION**:...

  • Senior Analyst

    3 weeks ago

    Pretoria, South Africa Financial Intelligence Centre Full time

    Job Purpose The purpose of this job is to apply technical expertise and judgment in the interpretation and presentation of a variety of complex financial and factual data, to assist with identifying subjects and associated links with the proceeds of crime, in order to aid criminal investigations, the recovery of proceeds of crime, criminal prosecutions and...

  • Senior Analyst

    4 days ago

    Pretoria, South Africa Financial Intelligence Centre Full time

    Job Purpose The purpose of this job is to apply technical expertise and judgment in the interpretation and presentation of a variety of complex financial and factual data, to assist with identifying subjects and associated links with the proceeds of crime, in order to aid criminal investigations, the recovery of proceeds of crime, criminal prosecutions and...

  • Senior Analyst Market Surveillance and Data Analysis Department

    1 day ago

    Pretoria, South Africa Financial Sector Conduct Authority Full time

    **Purpose of the Job**: To analyse data systems, create automated systems to retrieve information from database and compile reports. The department will also use data to solve problems, to turn data into information and information into insight that will lead to informed business decisions, and to create detailed specifications that specify data sources,...

  • Senior Analyst

    3 weeks ago

    Pretoria, South Africa Financial Intelligence Centre Full time

    Job Purpose The purpose of this job is to apply technical expertise and judgment in the interpretation and presentation of a variety of complex financial and factual data, to assist with identifying subjects and associated links with the proceeds of crime, in order to aid criminal investigations, the recovery of proceeds of crime, criminal prosecutions and...