Information Abundance Paradox suggests that learning to effectively use context may be at odds with encoding information parametrically. A natural follow-up question is: do models really need to store information in their parameters?

Suppose we could engineer an oracle harness that provides a model with all the accurate information it needs for a task. Would a model with sufficiently capable attention mechanism be able to solve essentially any task, despite knowing very little about it parametrically?

This blog post follows this curiosity.

Setup

LLMs and LLM-based agents have infiltrated the work habits of many, including myself. One particularly prominent use case, as we all know, is coding. So, let's focus on that.

How can we find settings where the model has different levels of parametric knowledge? One natural place to look is exposure: the more often a model encounters something during training, the more likely it is to encode knowledge about it parametrically [1].

Programming languages give us a convenient way to vary that exposure. Some are popular, and others are much more niche. For example, at the time of writing, GitHub contains 32.5M Python, 6.9M C++, and 1.2K Mojo repositories. We use these counts as a rough proxy for how frequently each language is likely to have appeared in the models' pre-training, mid-training, and post-training, and, in turn, for how much parametric knowledge the models have about each language.

So, we use gpt-oss models to solve LeetCode problems in Python, C++, and Mojo, all of which are Turing-complete. We use the recommended sampling hyperparameters and report pass@1 averages across five repetitions.

Acknowledgement: We cannot guarantee that the models we use have a sufficiently capable attention mechanism. Nor can we verify how frequently these languages appeared in gpt-oss training data, since its training corpus is not public.

Observation: Success Increases with Parametric Knowledge

First, does our proxy behave the way we expect? We compare how the models perform when given only the problem, leaving them to rely entirely on their parametric knowledge. An example input is provided in the appendix. We allow models to generate up to 4K tokens using medium reasoning effort.

The success rates follow the same ordering as our GitHub repository proxy. Increasing model size substantially improves performance on Python and C++, but does little for Mojo.

But perhaps 4K tokens at medium reasoning effort are simply not enough for the models to fully surface the knowledge they already possess. To check this, we vary the token budget from 2K to 131K while keeping reasoning effort fixed at medium, and separately vary reasoning effort across low, medium, and high while fixing the token budget at 131K.

Finding: Reasoning Fails to Compensate for the Lack of Parametric Knowledge

We start with the simplest question: does giving the model more tokens to reason help? We compare performance across different token budgets for each language, keeping reasoning effort fixed at medium.

As the figures above show, increasing the token budget improves performance on Python and C++ at first, before eventually saturating. For Mojo, however, increasing the token budget brings little improvement for either model.

Yet, one might say that token budget is only an indirect lever on how much the model reasons, and the appropriate control should be the reasoning effort hyperparameter. We next compare model performance across low, medium, and high reasoning effort for each language, keeping the token budget fixed at 131K, the largest available.

Increasing reasoning effort from medium to high more than doubles Mojo success for both models. Yet the absolute gain remains small, since Python and C++ improve by substantially more percentage points. Importantly, Mojo performance remains much closer to complete failure (0%) than to the success rates achieved on the other two languages.

I believe these two experiments provide satisfactory evidence for the claim that reasoning fails to compensate for the lack of parametric knowledge.

But what if the model does not have to know Mojo parametrically in the first place? If we provide the relevant information at inference time, can it close the gap?

Finding: Inference-Time Information Fails to Compensate for the Lack of Parametric Knowledge

We first test whether the models know what information to look for, and whether they can solve the problems once given access to that information. To do so, we equip them with the OpenCode harness, a prominent open-source coding harness.

We test two variants. The first can only search the local documentation downloaded from the official Mojo website using read, grep, and glob. The second has access to all OpenCode tools, including writing scratch files, invoking Bash, compiling and running Mojo programs, testing solutions, and revising them.

Providing access to the relevant documentation helps, but not nearly enough to close the gap. gpt-oss-120b benefits more clearly than gpt-oss-20b, while giving either model a richer set of tools does not consistently outperform simple search over the documentation.

So, the models can make some use of externally provided information. Access to that information alone, however, is far from sufficient.

Yet, there may be a simple explanation: the models we work with are relatively small, and may just not be capable enough to orchestrate tool calls effectively. A simpler alternative is to place the relevant documentation directly in context, which we test next.

Providing the documentation directly in context works best among the approaches we have tried so far, and improves Mojo performance notably for both models. Still, the gap remains large: even with the relevant information placed directly in context, Mojo performance is nowhere close to Python or C++.

So far, no inference-time strategy (reasoning, harness, in-context learning) has managed to close the gap. At this point, the picture for Mojo looks rather bleak. But perhaps it is too early to blame Mojo knowledge alone. Maybe the models themselves are simply not capable enough, or maybe there is a detrimental flaw in our experimental setup that we have failed to identify. Are these models simply doomed to fail at Mojo?

One way to find out is to see whether a relatively simple training intervention can change the picture.

Finding: A Simple Training Beats Inference-Time Strategies

Our evaluation set is fairly small, and directly training on the target distribution would feel a bit like cheating. Moreover, we do not have ground-truth Mojo solutions (and I would rather not spend sweet API money generating them) so SFT as an option is out, while the pass rates are low enough that GRPO would be prohibitively expensive.

An alternative, then, is to train on a simpler set of coding problems, such as MBPP. Unlike LeetCode, these problems are easy enough that we can use GRPO with a high sampling count and avoid the need for ground-truth solutions.

So, we train gpt-oss on MBPP problems translated into Mojo using GRPO, with unit test pass rate as the reward. For training details and implementation, please refer to the repository. We use a deliberately modest setup, where we limit the training data to only 271 MBPP problems, sample 16 solutions per problem, aggregate across 8 problems for each optimization step, and train for 10 epochs. We use rank-16 low rank adapters, giving us roughly 185M trainable parameters. Given these design choices, we refer to this as a simple training setup.

Reassuringly, the models are not doomed to fail at Mojo. GRPO outperforms every inference-time strategy across all three reasoning effort settings. So, the poor Mojo performance we saw earlier does not appear to be an intrinsic limitation of the models or of our evaluation setup, since the picture changes substantially once the model gets a relatively small opportunity to acquire Mojo knowledge parametrically.

Discussion

This blog post focuses on a narrow question: can inference-time strategies compensate for poor parametric knowledge? In the setting we operate in, they do, but fail to close the gap and fall behind even a simple training intervention.

Yet, building inference-time strategies and harness engineering, I think, remain valuable endeavors. It is still important to squeeze the best performance out of an already-trained model with better context, tools, retrieval, and reasoning. Our results simply suggest that these strategies may have limits when asked to substitute for knowledge the model never learned well in the first place.

Acknowledgement

All experiments reported throughout this blog post have been made possible thanks to compute provided by Daniel Khashabi and Benjamin Van Durme. The experiments have been inspired by my discussions with Daniel Khashabi and Matthew Brownrigg, who's pursuing an adjacent research question much more in-depth (so, stay tuned!).

Appendix

An example input is below:

System Prompt: 
          You are an expert competitive programmer. Follow the requested function interface exactly and return only source code.
          
          User Message: 
          Solve the following programming problem in {language}.
          
          Use only {language} and do not use syntax or imports from another language.
          
          You are given an integer n and a 2D integer array queries.
          There are n cities numbered from 0 to n - 1. Initially, there is a
          unidirectional road from city i to city i + 1 for all 0 <= i < n - 1.
          
          queries[i] = [ui, vi] represents the addition of a new unidirectional
          road from city ui to city vi. After each query, you need to find the
          length of the shortest path from city 0 to city n - 1.
          
          Return an array answer where answer[i] is the length of the shortest
          path after processing the first i + 1 queries.
          
          Example 1:
          Input: n = 5, queries = [[2,4],[0,2],[0,4]]
          Output: [3,2,1]
          
          Example 2:
          Input: n = 4, queries = [[0,3],[0,2]]
          Output: [1,1]
          
          Constraints:
          3 <= n <= 500
          1 <= queries.length <= 500
          queries[i].length == 2
          0 <= queries[i][0] < queries[i][1] < n
          1 < queries[i][1] - queries[i][0]
          There are no repeated roads among the queries.
          
          {language_specific_header}
          
          Return source code only. Include any imports you need, but do not define
          a main function, tests, or the interface more than once.