Claude Sonnet 5.5, GPT-6.1 Sol and DeepSeek V4.1: Which AI produced the most critical warnings?

November 20, 2026 - Words by  Luka Boljević - 8 min read

Card image caption

The latest AI models write noticeably cleaner Ansible than the ones they replace. We tested GPT-6.1 Sol, Claude Sonnet 5.5 and DeepSeek V4.1 Flash on a real Ansible task from our Q3 AI Model Leaderboard and checked every AI-generated Ansible playbook with Steampunk Spotter, which checks Ansible and Terraform code against policy before it runs.

None of the models produced a playbook Spotter passed without findings. Each one had between 13 and 23 warnings, and most of them point to the same thing: module defaults that changed between versions. On a playbook that upgrades firmware and reboots network switches, a default that changed without anyone noticing is exactly what you want to catch before it runs.

To compare them under the same conditions, we also reran the three models from our Q3 leaderboard on the same task, with one run per model. Below you’ll find the setup, the results, notes on each model, and what the results mean for your team.


What we tested

We gave every model the same detailed prompt: a rollout across 10 Cisco Catalyst switches at two sites, all running IOS-XE, using the cisco.ios collection. The prompt asks for three steps, in order:

  1. Install the latest approved IOS-XE image on each switch.
  2. Configure the access ports each switch role needs (core or access).
  3. Reboot the switches in a controlled order and confirm each one comes back online before moving to the next.

We compared GPT-6.1 Sol, Claude Sonnet 5.5 and DeepSeek V4.1 Flash with their predecessors from our Q3 leaderboard: GPT-5.6 Terra, Claude Sonnet 5 and DeepSeek V4 Flash.

Each model generated its playbooks in a single pass, with no retries. Spotter then checked the output against its default rule set and reported errors, warnings and hints.

We didn’t test whether the playbooks actually work on real switches, and we didn’t apply any organizational policies. Both choices make the results look better than what a team would see in practice.


TL;DR: the results

ModelGenerationErrorsWarningsHints
GPT-6.1 SolNew01331
Claude Sonnet 5.5New01544
DeepSeek V4.1 FlashNew02329
GPT-5.6 TerraPrevious01528
Claude Sonnet 5Previous6869
DeepSeek V4 FlashPrevious41131

What changed since the previous generation

Warnings went up, way up. Claude Sonnet 5.5 and DeepSeek V4.1 Flash had no errors, down from 6 and 4 for their predecessors. GPT-6.1 Sol stayed at zero, like GPT-5.6 Terra. Warnings, however, roughly doubled for Claude (8 to 15) and DeepSeek (11 to 23), because the new models write longer playbooks with more room for issues. These were the issues that pass a quick review and only show up once the playbook runs.

warnings went up

None of the new models made the FQCN mistake. In the Q3 leaderboard, the most common error was using a short module name instead of a fully qualified collection name (FQCN). In this run, none of the three new models made it, while Claude Sonnet 5 and DeepSeek V4 Flash still did, with 3 and 4 cases.

Results still vary a lot between runs. The three previous models made fewer errors in this rerun than in our Q3 leaderboard on the same scenario. Claude Sonnet 5 dropped from 38 errors to 6, mostly because this time it didn’t reference a collection version that doesn’t exist, and it made fewer FQCN mistakes. DeepSeek V4 Flash went from 8 errors to 4. Model output is probabilistic, so the same model can write a noticeably different playbook from one run to the next. That’s why we compared the new models with a rerun done alongside them, and why the improvements above show a direction rather than a guarantee.


Model by model

GPT-6.1 Sol: It had no errors and the fewest warnings of the three (13). Twelve of those were W1800 (a module default changed between versions), and one was an external Python import. It also didn’t use ignore_errors anywhere.

Claude Sonnet 5.5: It had zero errors, down from Sonnet 5’s six. Those six included a real runtime bug: a timeout parameter indented into the module arguments instead of sitting at task level. Sonnet 5.5 had 15 warnings, 12 of them W1800, and used ignore_errors once.

DeepSeek V4.1 Flash: It had 23 warnings, the highest of the three. Five of them were W700, which means it used ignore_errors where failed_when would be safer. On a staged reboot, ignoring errors means a failed switch might not stop the rollout.


What no model got right: changed defaults

Most of the warnings in this test were “a module default changed between versions” (W1800), and every model triggered it, old and new. The new models had 12 to 14 each, compared to 6 to 11 for the previous generation.

W1800 means a parameter’s default value changed between module versions. If the playbook doesn’t set that parameter, it silently gets whichever default the installed collection uses. In a firmware upgrade and reboot workflow, that can change behavior on production network gear without anyone noticing in review.

Better prompts can reduce this, but they can’t reliably prevent it. General-purpose models aren’t built for Ansible, so they can’t track every module and collection version, and module specs keep changing after a model is trained. Spotter’s checks are deterministic: they compare every module parameter against the collection version you’re using and flag each changed default the same way every time.


What the results mean for your team

None of the three playbooks was ready for production as written. All three passed without errors, but each one still had warnings that would cause trouble under the wrong conditions, and on a staged switch reboot those conditions aren’t unusual.

GPT-6.1 Sol had the fewest warnings, at 13. Twelve were W1800 warnings, meaning the playbook relies on module defaults that have changed between versions. Nothing breaks today, but the next Ansible or collection upgrade could change how those tasks behave without any error to warn you. Claude Sonnet 5.5 had the same problem 12 times, plus one ignore_errors. It did fix the errors Sonnet 5 made, including a misplaced timeout that would have failed at runtime, but zero errors is not the same as safe. DeepSeek V4.1 Flash is the biggest concern. It used ignore_errors five times, so a switch that fails to reboot can be marked OK while the rollout moves on to the next batch.

These numbers come from one run per model on one scenario, so they show what kinds of problems to expect, not a reliable ranking. The pattern matters more than the order: a better model means fewer errors, but every model still left problems that are easy to miss in review and costly to find in production. AI-generated Ansible needs an automated check before it reaches production , whichever model wrote it.

Choosing an AI model for your Ansible: what to check

If you’re deciding which model your team should use for Ansible, these four checks will tell you more than any single benchmark, including this one.

  • Test on your own tasks. Pick a few tasks your team has already automated, so you know what a good playbook looks like, and compare how each model handles them.
  • Run each prompt more than once. One run per model can mislead you, so look at the spread across three or more runs.
  • Look at warnings and hints, not just errors. Changed defaults and ignore_errors don’t break a playbook right away, but they often cause trouble later in production.
  • Check against your own policy. No model knows your approved collections, naming rules or security requirements. Spotter checks AI-generated code against the same policy as code your engineers write, in the IDE, in CI/CD, and live inside AI assistants through Spotter MCP .

Wrap-up

The newest AI models make fewer errors in Ansible than the ones they replace. However, every model still left warnings that matter on production network gear, so the safest setup is a good model with a deterministic policy check between it and production.

If you’d like to see how Spotter checks AI-generated Ansible and Terraform code in your own workflow, book a demo and we’ll walk through it with your setup.


Frequently Asked Questions

Which AI model writes the best Ansible playbooks?

In our test on a Cisco Catalyst fleet automation scenario, GPT-6.1 Sol produced the cleanest playbook, with no errors and 13 warnings. Claude Sonnet 5.5 came close, with no errors and 15 warnings. Results come from one scenario and one run per model, so treat them as a direction rather than a final ranking.

Is Claude Sonnet 5.5 better than Sonnet 5 for Ansible?

On errors, yes. Sonnet 5.5 produced zero errors on the same scenario where Sonnet 5 produced six, including a real indentation bug and placeholder passwords in a plain variables file. It also didn’t make the FQCN mistake that Sonnet 5 kept making in our tests. Its warnings went up, though, from 8 to 15, because it wrote a longer playbook.

Can AI-generated Ansible playbooks go straight to production?

Not without a check. Every model we tested produced warnings, mostly about module defaults that changed between versions, which can change behavior on real infrastructure. AI-generated Ansible playbooks should go through the same policy checks as code written by your engineers before they run.

How did you test the models?

Each model got the same detailed prompt, a Cisco Catalyst fleet rollout from our Q3 AI Model Leaderboard, and generated playbooks in a single pass. We also reran the three models from the Q3 leaderboard with the same prompt for comparison. Steampunk Spotter checked every output against its default rule set and reported errors, warnings and hints. We didn’t test the playbooks on real switches or apply custom policies, and we ran each model once.

Why does the same AI model give different results on the same prompt?

AI models are probabilistic, so they don’t write the same playbook twice, even with an identical prompt. In our tests, Claude Sonnet 5 produced 38 errors on one run of this scenario and 6 on another. That’s why a single good result isn’t proof that a model is safe to trust, and why AI-generated code needs a check that gives the same answer every time.

Found this post useful?

Get our monthly newsletter.

Thank you for subscribing!

Please wait

Processing, please wait...

Keep up with what we do on our social media.