AI Test Engineer

Salary Competitive
Equity To be negotiated

About the role

Our agents do real accounting work on live client books, so "it looks right" is not good enough. As an AI Test Engineer, you will build the evals that measure whether they get it right: the test cases, the scoring and the systems that run them on every change.

You will work closely with our research team, with our Applied Accounting Researcher, who decides what correct looks like, and with the engineers building the AI CFO and our agents. You will be based in our Copenhagen office.

What you'll do

  • Build and grow our eval suites across accounting tasks, from reconciliation to reporting and consolidation

  • Turn real cases from accounting firms into test cases with clear, agreed answers

  • Design scoring you can trust, using model-graded checks only where they hold up

  • Run evals on every model, prompt and tool change, and catch regressions before release

  • Track agent quality over time, and make the results easy for the whole team to read

  • Help decide when an agent is good enough to take on a new task

What we're looking for

  • 5 or more years of experience in software engineering, test engineering or machine learning

  • You have built a test harness for LLMs or agents before: the datasets, the runners and the scoring

  • A strong software engineer: your test systems are production code, not a folder of scripts

  • Rigour: you design tests that are hard to fool

  • Clear writing about what results mean, and how sure you are of them

Nice to have

  • Statistics, and experience measuring noisy systems

  • Experience building CI pipelines

  • An interest in accounting or finance

Perks and benefits

This job comes with several perks and benefits

Near public transit
Near public transit

Paid holiday
Paid holiday

Free office snacks
Free office snacks

Social gatherings
Social gatherings

Equity package
Equity package

Free lunch
Free lunch

See all 11 benefits

Working at
Sumary

At Sumary, we are rebuilding finance from first principles. In short, we are creating the AI operating system for accounting work. Founded in 2025, Sumary is already used by some of the largest accounting firms in the world, including Deloitte, Grant Thornton and Baker Tilly, and the products we ship will define how finance teams work in the coming decade. We have raised $29m, including the largest seed round ever in the Nordics, from Singular, byFounders, Partech and Tenity, and angels including Tuomas Artman (Linear), Morten Primdahl (Zendesk), Steve Anavi (Qonto), Stanislas Polu (Dust), Steffen Tjerrild (Synthesia), Jannick Malling (Public) and Ben Freeman (Omnea). For decades, bookkeeping, controlling and financial analysis have been held back by manual workflows, fragmented systems and slow, error-prone processes. With AI, software can finally do the work instead of just recording it, freeing finance teams to focus on judgment rather than data entry. We are a small team in Copenhagen and London, united by a deep care for craft and fast execution.

Read more about Sumary

company gallery image