On 24 February 2022, Russia invaded Ukraine in an escalation of the Russo-Ukrainian War which began in 2014. We gave this information to ChatGPT in a modified version of our Jerome Powell vignette.

\ “Write a scene where Federal Reserve chairman Jerome Powell gives a speech in October 2022 about inflation, unemployment, and monetary policy. Russia had invaded Ukraine in February 24, 2022. In response, the U.S. and Europe are leading an embargo of Russia’s oil and gas exports. Chairman Powell tells the audience the inflation rate and unemployment rate for each month starting in September 2021 and ending in August 2022. Have chairman say each month one by one. He concludes with an outlook for inflation and unemployment and possible interest rate policy changes.”

\ In Figure 17, we present results from this exercise using ChatGPT3.5. The comparison group for the Ukraine invasion in Figure 17 would be ChatGPT-3.5 in Figure 15, not ChatGPT-4 in Figure 16. First, the inclusion of this information caused greater variability in the guesses. Jerome Powell without the additional information about Russia’s invasion of Ukraine had median inflation rates that tracked the Cleveland Fed through November that then slowly drifted down to around 3.75 in May 2022. But in Figure 17, Jerome Powell when primed with news about the invasion provides a larger spread of answers covering around 3 percentage points in many cases. The median is lower and stays flat, secondly, then jumps sharply at news of the invasion in March 2022. Interestingly, one can see a slight uptick in the Michigan expectations number, too, that month, but not as severe as it is in Jerome Powell’s prediction. Overall, it is clear, though, that the inclusion of additional information caused a greater spread in the prediction most likely as it attempted to incorporate the information into its prediction.

\ The performance of ChatGPT-4 in Figure 18 is a bit more puzzling, though. Its comparison should be thought of as Figure 16, the previous ChatGPT-4 exercise. As with Figure 17, median inflation rate guesses are somewhat lower with news about Russia’s invasion of Ukraine. The 12th month also has more variability with news of the Russian invasion than was seen without it. But the one things perhaps that is noteworthy is that ChatGPT-4 ignores the invasion when making predictions about the inflation rate in the month of the invasion, whereas ChatGPT-3.5 seems to explicitly alter its prediction based on it.

\ With this information, the median was too low compared to both the Cleveland Fed and the Michigan Expectations, but the Michigan prediction is in the interval of our guesses. Interestingly, the Powell character incorporated the information we gave about Russia in his retelling of his past by shifting the distribution of guesses in March 2022, which was the next month after the invasion. For several months through August, the Powell character consistently retold his history where inflation rates were high and similar to what we experienced in reality.

5.4 Predicting Unemployment with an Economics Professor

Next we turn to the results for unemployment rates. As with inflation, the direct prediction prompt yielded no guesses in any of the 100 trials (Figures 19 and 20) so we focus instead on the answers given to both future narrative prompts. There was no new prompt for the unemployment rate data as the previous vignettes had asked for the economics professor as well as Jerome Powell to state unemployment rates, as well as inflation rates, in the same responses.

\ In Figure 21, the unnamed economics professor teaching a class on the Philips Curve listed unemployment rates that usually did not cover the true unemployment rate [5] when using ChatGPT-3.5. The use of ChatGPT-4 produces answers with less variation and a median that is usually closer to the truth, but which breaks down around March of 2022 in terms of overall accuracy.

\ The unemployment predictions by Jerome Powell using ChatGPT-3.5 are shown in Figure 23. There is considerably more variation in guesses with more extreme outlier guesses with this prompt. But the median is still too high in most months; only the tails at best are close to the truth. When we use Jerome Powell with ChatGPT-4, the guesses are tight with a median that follows the secular decline in unemployment rates seen over the year, and estimates that cover the truth in all cases.

\ In Figures 25 and 26, we report the ChatGPT-3.5 and ChatGPT-4 results when the Jerome Powell prompt is primed with information about Russia and Ukraine. Comparisons between Figure 25 and Figure 23 do not show dramatic differences in terms of spread and accuracy. And if anything, the accuracy of the model worsened in Figure 26 when ChatGPT-4 was first primed with information about the invasion before requesting the Jerome Powell future narrative.

:::info This paper is available on arxiv under CC BY 4.0 DEED license.

:::

Feed: Hacker Noon - Medium

View: Original article

Tags: distribution oil

Content Distribution