Proprietary models outperform open-source models by a wide margin across all plotting libraries. GPT-4o achieves the highest execution pass rates and the strongest judge-based scores, followed by its lightweight variant GPT-4o-mini. These results indicate more reliable execution and better semantic alignment with task instructions, especially in complex visualization settings. In contrast, open-source models like LLaMA and Qwen2.5-Instruct underperform consistently across all metrics.

Performance varies across plotting libraries. While most models perform reliably on matplotlib and seaborn, results on plotly are significantly lower, especially for open-source models. Execution pass rates often drop below 35%, and task/visual scores degrade accordingly. Generated plots frequently fail to reflect the intended semantics or completeness, revealing the challenge posed by plotly's verbose syntax and less frequent API exposure in training corpora.

VisCoder significantly outperforms its Qwen2.5-Coder-Instruct baselines across all libraries. At the 3B scale, it improves execution success and semantic alignment, especially on plotly and seaborn. At 7B, VisCoder even outperforms GPT-4o-mini on these two libraries, although slightly trailing on matplotlib. These gains highlight the impact of domain-specific instruction tuning for visualization code generation.

GPT-4o demonstrates strong self-debugging capabilities, reaching near-perfect execution success with multiple attempts. VisCoder also benefits substantially under this evaluation protocol. VisCoder-7B surpasses 90% execution rate on matplotlib and seaborn, with large gains in task and visual scores across correction rounds. These results show VisCoder’s ability to generalize debugging behaviors learned from training, even without plot-specific correction examples.

Execution pass rates increase steadily over self-debug rounds for most models and libraries, indicating the overall effectiveness of the protocol. The first attempt typically yields the largest improvement, with smaller gains in subsequent rounds. This pattern suggests that a simple retry mechanism informed by execution feedback can recover a substantial portion of initial failures.

Compared to their Qwen2.5-Coder baselines, VisCoder models show smaller per-round gains but consistently achieve higher final performance. This indicates that VisCoder tends to generate stronger initial outputs and applies more stable corrections. VisCoder-7B is particularly strong on seaborn, approaching GPT-4o by the final round.

Even the strongest model GPT-4o does not reach perfect execution after self-debug. Its performance on seaborn plateaus after three rounds, leaving non-trivial failure cases. In contrast, VisCoder-3B stands out among smaller models — outperforming GPT-4o-mini on seaborn and performing competitively elsewhere. Smaller models generally plateau earlier with fewer gains.

VisCoder-7B demonstrates strong self-correction ability on shallow, structural errors. AttributeErrors in Seaborn are reduced from 15 to 2, and TypeErrors in Plotly from 3 to 1. These errors usually stem from invalid method calls or argument mismatches and are easily identified by diagnostic outputs. VisCoder learns to correct them consistently through retry-based feedback.

Semantic failures such as KeyError and ValueError are harder to resolve. On Plotly, ValueErrors drop only slightly (29 → 23), while KeyErrors remain unchanged. These errors require dynamic reasoning about data structures, but VisCoder’s retry attempts often rely on the same faulty assumptions. Symbolic corrections alone are insufficient for resolving such semantically grounded failures.