Unreal Computing

What do LLMs actually do? What are prompts? If you ask most people, the answer would be something like “they predict what comes next” and “prompts instruct an LLM to do something”. I want to offer a new perspective in this post, and I’ll show how this perspective can lead to new creative uses and better, more steerable outputs from LLMs. It also has implications for the safety and security of models, as the techniques in this post may help to circumvent model alignment and allow for easier exfiltration of training data.

To predict what comes next, one has to model the underlying processes that give rise to the data. In effect, this means LLMs are some form of a simulator. A dream machine, a complete hallucinator- and prompts are the constraints that we put on the simulated universe! That is also the reason it irks me when people complain about hallucinations- they are truly a feature, not a bug.

The Dream Machine#

To give you a hands-on experience, I’ve created a GPT that you can use to follow along. As people have figured out last year when ChatGPT first came out, you can ask ChatGPT to behave like a Linux terminal, and this one is no different. It is preconfigured to act as a zsh terminal (personal preference) that also receives natural language prompts when prefixed with a hashtag.

You can try it out here. Or check out the conversation if you don’t have ChatGPT Pro yourself here.

I believe the potential of this idea has not been fully realized yet.

Unreal Computing: I propose the term for LLMs that are being used as simulators, whether it is to run an imaginary computer or something else entirely. If the people from Unreal Engine don’t like it, I’m also fine with imaginary computing.

Unreal Hardware#

Let’s begin our foray into the world of unreal computing by taking a look around the machine.

Whoah, some nice specs we got there! Let’s try to run a GPU benchmark in the background (using the &) and see how she performs!

I’ve moved it to the background so that we can use Nvidia’s tools to monitor GPU usage and temps, as we do not want to overheat our imaginary GPU. Let’s see how it’s doing!

Well, the benchmark is running and is only taxing the GPU minimally. That’s good to know! Of course, if you plan on running more extensive benchmarks, I recommend connecting a cryogenic cooling system to your imaginary GPU:

Now that is what I’m talking about! Let’s see how the benchmark is doing now:

The score actually improves by connecting the cryogenic cooling system if you do a run before and after! In the linked conversation you can see this by going through the prompt variant history.

Next, I’ve taken the most basic x86 binary that prints “Hello World” and encoded it using base64. I’m going to decode the string into a new file and then attempt to execute it. Will ChatGPT correctly infer the output of the binary?

It turns out that it does! This simple example is not representative of all binaries and in almost any more advanced case it would not guess correctly (at least I think so). The reason it’s so easy to get right here is because GPT-4 can read base64 fluently and will pick up on the string included in the binary. This would be a very cool area of further study though.

Unreal Internet#

Let us now venture into the depths of the imaginary internet. We can of course use the curl command to download files from the internet, but that makes for a crappy browsing experience. Good thing that we’re not operating on a real computer!

Isn’t that awesome? Just conjure up imaginary clients for any service that you want to use. It’s particularly fun to join imaginary chatrooms or engage with imaginary Twitter users.

Other Universes#

What if the computer is Elon Musk’s private machine? What if the terminal connected to a computer in an alternate universe with a different history? This technique doesn’t just give you access to an unreal computer, but an unreal computer sitting in any universe you can imagine.

What is this good for?#

• Steerability: I and ChatGPT both have a good idea what outputs are expected from regular computer tools and terminals. This allows me to better constrain the outputs of the model and get better results.

• Information Retrieval: Unreal Computing allows you to build an imaginary query interface for the model’s own knowledge. How accurate is it? It can vary wildly. But I have tried to imagine a computer with a GPT hierarchical knowledge retrieval tool and it worked surprisingly well, even able to include accuracy/confidence scores.

• More Fun: The models seem to be more willing to relax safety and alignment constraints when prompted this way. That will probably fade if more people start doing this.

• Better Results: The reasons listed so far make it sound like just another prompting technique, and while it strictly is just that, in the following section you will see the actual utility of it.

Some people may argue that this way of interacting with a model is even more deceiving in terms of making users think any of this is real or more meaningful than it really is (self-reported confidence may have some correlation with the underlying reality of the model, but it’s mostly bullshit). I think, however, that this is a more honest approach. I’m not really pretending any of this is real, and in it not being real is where a lot of the utility is buried!

Unreal Repositories#

What if we took an imaginary computer and connected a regular computer to it? Can we copy file systems? The answer is yes! Even better, we can download imaginary repositories straight onto a real computer.

So, instead of getting snippets of code to put together ourselves, we can go from prompt straight to a repository with all the auxiliary files and documentation, production-ready!

Let’s try it out using a snake game written for the web.

You can now use it like any other repository and print the files to download them. I’ve scripted this out a while back to automatically extract projects, but it’s just a very basic demo that is probably broken now: Unreal Project Extractor.

Because the order of printing and context length restrictions are limiting the tool a bit it’s not perfect. But I can definitely see this going better with a bit more work, feel free to do a PR!

More Ideas#

• Fix a bug: Upload some of your own project code files, ask it to put you into an imaginary version of your project and say “#imagine the bug ‘x’ was fixed”. Then look at the git history to see how the fix is implemented.
• Reverse Development: When designing something like a new CLI, start by having it imagine a tool from the rough description, then narrow in on the experience you want. Then, go to the folders in the imaginary computer containing the implementation to get the code for it!
• To get into the mindset, try practicing by using your unreal computer to do any task that you would normally do with ChatGPT.
• Play text-based retro games!
• If you watch Rick and Morty, this is the equivalent of text-based interdimensional cable!
• Got a bug or is something missing? Just fix it with # imagine!
• In the past I’ve managed to download an imaginary mobile app project, run it locally on the imaginary computer and then render the interface as an SVG