On the spot generated code: LLMs and Python

Disclaimer: applying this in a real world scenario generally is a terrible idea, don’t create a startup based in this idea (unless you invite me to join)

I have been using LLMSs to genterate code for a while Somewhat like two years now. So far I have been just asking them to generate some code, copy it and the paste it to run with extremely diverse results. Next, all the big AI LLM-based companies created CLI interfaces that gets access to your filesystem and command line, and directly did the breaking changes to your codebase. This is the old way to create code with LLMs, there is a brand new unsafe way to create code, I present to you, the “on the spot generated code”, which is as expensive and unpredictable as it sounds.

Previously, you served as a middle point between the LLM and the execution. In Python we can make use of the prototype-based type system, which allows us to use methods thad do not exist. Furthermore, it allows to execute python code from a string, by using the exec function. This means that you can:

1. Define a object that doesn’t have any methods
2. Call methods that are not defined.
3. Intercet the call to those methods by using python magic functions.
4. Generate the code with a LLM in the spot
5. Run that python code with exec

Let’s explain it from the ground up.

Create code specifically with the LLM

In this case I have used llm library to connect to models. Specifically, I am using the OpenAI API. However, the same can be done with other models.

We need to call a LLM and receive code, we cannot receive any other kind of output since it would break the execution. To do this a schema can be used. Specifically, this would result in just code being returned:

    schema = {
        "type": "object",
        "properties": {"code": {"type": "string", "description": "Python code to execute"}},
        "required": ["code"],
        "additionalProperties": False,
    }

The exec method

Python magic methods

Python classes have a set of special methods that are defined with double underscores. These methods are called magic methods or dunder methods. They are used to define the behavior of objects in various common situations. For example, the __init__ method is called when an object is created. The programmer should define a behaviour for that method, by initializing the object with the needed variables.

Specially, for this use case, we are interested in three of these methods:

1. __getattr__: allows to access attributes
2. __setattr__: allows to set the value of attributes
3. __call__: allows to call methods

The key point of these methods is that they are called before doing the action. For example, if you have the following code

class Value:
    def __init__(self, value):        
        self.value = value

    def __getattr__(self, name):                  
        print(f"Hello there!")
       return self.__dict__[name]

x = Value(10)
print(x.value)

The output will be:

Hello there!
10

This means that we can replace the print function with a call to the LLM, and return whatever value it generates or store it.