Why the model you use matters
In my view, AI models improve in steps. A new model can solve problems that the model before it could not. In my experience, an older model often does not say "I cannot solve this". It gives an answer that looks correct.
Frontier and behind
A frontier model is one of the most capable AI models available on a given date. A model one or two releases behind may cost less or respond faster for some workloads. Cost and speed depend on the models, workload and current pricing. Test whether the cheaper model meets your task's requirements, even for a summary, format change or first draft of a letter.
In my experience, the difference shows on problems where a common simplification is wrong, or where the problem is too large for the model. The model then often takes a shortcut and does not say so.
A small gain can be a step change. Suppose many of your problems sit just beyond what a model two releases behind can do. Suppose, too, that a newer model scores only 1 to 2 % higher on public benchmarks (a hypothetical figure). It could then solve a whole group of your problems that the older model got wrong. For that group, the result is not 1 to 2 % better: it is a right answer in place of a wrong one. The schematic shows the idea; it is not a measurement.
A share of your problems, area by area
One way to think about a model: in each area of your work, it solves some share of your real problems correctly. Structural checks are one area. Report writing is another. A newer model can raise that share. Public league tables (benchmarks) test general problems, not yours. You can only find the share for your own work by testing the model on your own problems.
A worked example: a confident shortcut
This example is an illustration of the kind of error, not a test result. It is invented for this article, uses textbook theory, and contains no client work.
The question: "What is the first sloshing frequency of a rectangular tank, 1.0 m long, filled to 0.15 m?"
The full linear-theory answer (small motions) includes the fill depth. For the first mode of a two-dimensional rectangular tank:
L = 1.0 m tank length in the direction of sloshing
h = 0.15 m fill depth
g = 9.81 m/s^2 gravity
k = pi / L = 3.14 per m (wave number of the first mode)
tanh(k h) = tanh(0.471) = 0.439 (hyperbolic tangent; it tends to 1 for a deep fill)
w^2 = g k tanh(k h) -> w = 3.68 rad/s (w: angular frequency)
f = w / (2 pi) = 0.586 Hz
A shortcut drops the tanh term. That form is for deep liquid:
w^2 = g k -> w = 5.55 rad/s, f = 0.884 Hz
The shortcut answer is about 51 % too high. It looks plausible: it has the right units, a clean formula and a confident tone. At a deep fill the two forms almost agree: at 0.70 m fill the difference is about 1 %. A check at a deep fill does not find the error. Only a check at the shallow fill shows it.
A frontier model can make the same mistake. In my experience it does so less often, and it more often states the assumption it made.
Frontier model against a model two releases behind
The table is illustrative. It sets the worked example above beside the author's experience and view. It is not a measured test, and it names no model or vendor.
| What to compare | Frontier model | Model two releases behind (N-2) |
|---|---|---|
| Example answer to the worked example (illustrative) | Example of the full answer, with tanh(k h): 0.586 Hz | Example of the shortcut, without tanh(k h): 0.884 Hz, about 51 % high |
| Confident but wrong answers | Less often (in my experience) | More often (in my experience) |
| States the assumption it made | More often (in my experience) | Often does not say it took a shortcut (in my experience) |
| Cost per task | May be a bit more (in my view) | May be less (in my view) |
| Checking and rework | Fewer wrong answers to find (in my experience) | More answers to find and redo (in my experience) |
Cost and speed depend on the models, the workload and current pricing. Either model can make the shortcut in the worked example; the table shows which answer is more likely, in the author's experience.
Cost, speed and checking
A frontier model may cost more per task or run more slowly for a given workload. Compare the available models using that workload and the current pricing. Capability alone does not determine cost, speed or suitability. A frontier model is still not always right.
Check every answer that a decision depends on. Check the inputs and the units. Then check one case where you know the answer and where a shortcut would break: here, the shallow fill. If you know the full formula, or have the textbook open, the hand check above takes a few minutes.
Conclusion: cost and the long term
In my view, a frontier model may cost a bit more per task. Depending on the situation, it may cost more in the long run, or it may pay back over the long term. In my experience, the return is fewer wrong answers to find and redo. In my view, it can pay back when much of your work sits near the edge of what models can do; your problem set shows whether it does. Treat your AI workspace like a group of employees you are training. In my view, the instructions, checks and problem sets you build with it now can make your work with it better tomorrow. In my view, using a model that is behind compromises that training.
Keep a problem set ready
Keep a short list of problems that the model you use now fails or shortcuts. When you get access to a new model, run the list on the first day. Compare each answer with the answer you know. Keep some money or spare usage ready for this test.
The list shows you, on your own work, what the new model can now do. It also shows which checks you must still do by hand.
How this article was made: an AI assistant drafted it from the author's notes. The worked example was recomputed independently before publication.
Try it Monday
- Choose five problems from your own work, with no client data or project names.
- Prefer problems that your current AI model fails or shortcuts, or problems whose answers you know.
- For each problem, write the known answer, or how you will check an answer.
- Save the list in one file.
- When you get access to a new model, run the list on it first.
Sources
- Faltinsen, O. M. and Timokha, A. N., Sloshing, Cambridge University Press, 2009: first-mode natural frequency of a rectangular tank, linear potential-flow theory.
About the author: https://www.aceengineer.com/about.html
For help with automation, AI automation or workflows, use the contact page: https://www.aceengineer.com/contact.html