Learn extra at:
All 11 fashions additionally struggled with agentic benchmarks designed to evaluate actual world problem-solving talents round normal information, security, and coding. Claude 3.5 Sonnet and o1 ranked the very best on this space, notably when it got here to extra structured duties with express aims. Nonetheless, all fashions had a tough time with software program engineering and different duties requiring open-ended reasoning and planning.
Multimodality is turning into more and more necessary for AI programs, because it permits fashions to course of totally different inputs. To measure this, Vector developed the Multimodal Huge Multitask Understanding (MMMU) benchmark, which evaluates a mannequin’s skill to cause about pictures and textual content throughout each multiple-choice and open-ended codecs. Questions cowl math, finance, music and historical past and are designated as “straightforward,” “medium,” and “onerous.”
In its analysis, Vector discovered that o1 exhibited “superior” multimodal understanding throughout totally different codecs and issue ranges. Claude 3.5 Sonnet additionally did effectively, however not at o1’s stage. Once more, right here, researchers discovered that almost all fashions dropped in efficiency when given tougher, open-ended duties.
