Use Devin for one integration, one migration, and a review

Answer.AI tested Devin on three coding tasks, but the migration stalled and the security pass mixed valid findings with mistakes.

Done with DevinCursor· Intermediate· Jan 2025

What they did
Answer.AI handed Devin around 20 real assignments through Slack and let it work in a containerized setup with browser access, an editor, and a shell. The jobs included making synthetic data for Braintrust, moving a Python project to nbdev, and checking a repository under 700 lines for security flaws. Team members reviewed the output and frequently jumped in to complete the work.
How it went
The Braintrust integration was dropped and later redone in Cursor. The nbdev move never got past the initial setup, and the vulnerability check surfaced some real problems along with false alarms and invented issues. The overall tally was 14 failures, 3 successes, and 3 inconclusive results.
Tips
Keep tasks narrow and clearly bounded, especially for reviews and setup-heavy changes. If the agent cannot verify a claim, have it label that instead of pushing ahead with guesses.

Try it yourself with Devin

Work on [repo name] in three separate passes: first, add an integration for [observability tool] that sends [specific data] into [destination]; second, migrate [module or project] to [target tool or version]; third, review the code for security issues in a small repo of about [size]. For each pass, show the evidence you used, note anything you could not confirm, and prepare a separate draft pull request without merging it. Before you make any irreversible change, such as sending data, deleting files, or submitting a pull request, stop and ask me to approve it first.

Fill in the [brackets]. Agents can make mistakes, so review anything before it is sent, paid or deleted.

Read the original on answer.ai ↗

Get 3 agent uses like this every morning

With the day's top AI news. Free, and one click to unsubscribe.