class langtorch.Activation

As introduced on the page langtorch.tt, a design principle of LangTorch is to maintain a correspondence between neural network components from torch and LangTorch objects that we can better understand using these existing taxonomies. Because of this, we call an "activation" the LLM call done at the end of a TextModule "layer" (from activation function). The activation is typically an LLM call performed in parallel for each entry of the processed TextTensor. Each entry of the input tensor corresponds to a separate LLM call, which can be a single user message or a complete message history (see. Representing chat histories with Text).

Note

Note that a Text whose content has a key than is neither system, user nor assistant, the input will be interpreted as one long user message.

Initialization

The Activation class supports any OpenAI model and its parameters by default. You can initialize an activation using the Activation class directly or through its alias OpenAI:

from langtorch import Activation, OpenAI

# Use either:
llm = Activation("gpt-4", system_message="You are an expert assistant.", temperature=1.2, max_tokens=20)
# or:
llm = OpenAI("gpt-4", system_message="You are an expert assistant.", temperature=1.2, max_tokens=20)

When initializing an Activation, you can provide a system_message that guides the model's behavior. This system message can be overridden for individual entries in the input TextTensor by including a key-value pair of the form ("system", "your system message") in the corresponding Text object.

Here's an example of using Activation with a TextTensor:

from langtorch import TextTensor

user_messages = TextTensor([
    "What is the capital of France?",
    "Can you explain the theory of relativity?",
    {"system": "You are a math tutor.", "user": "What is the Pythagorean theorem?"}
])

llm_responses = llm(user_messages)

In this example, the first two entries in user_messages will use the default system message, while the third entry overrides the system message with "You are a math tutor."

Activations are instances of TextModule themselves, but are more commonly used in conjunction with another TextModule to perform a two-step operation of formatting prompts and obtaining completions for them. Apart from performing parallel calls, to speed up inference and decrease costs Activation caches results of deterministic calls (T = 0) and skips redundant parallel calls. If you want to cache and reuse results even for activations with non-zero temperature, set cache=True. The last non-standard init argument is keep_history, which False by default makes the module output only the completion, and the input + completion appended if True.

The rest of the Activation arguments are parameters of the API call:

Parameter	Description
model	The ID of the language model to use for processing.
system_message	Predefined system message to guide the model's response.
temperature	Controls the randomness of the generated output. Lower values make the output more deterministic.
n	Number of completions to generate for each input entry. The output TextTensor will have an additional first dimension for the different completions.
max_tokens	The maximum number of tokens to generate in the response.
tools	List of tools described by JSON schema or Python functions. The schema is generated based on the function's docstring and argument type hints.
tool_choice	none means the model will not call a function and instead generates a message. auto means the model can pick between generating a message or calling a function. Specifying a particular function via {"type": "function", "function": {"name": "my_function"}} forces the model to call that function.
frequency_penalty	Number between -2.0 and 2.0. Positive values penalize new tokens based on their existing frequency, decreasing the likelihood of repeating the same line verbatim.
presence_penalty	Number between -2.0 and 2.0. Positive values penalize new tokens based on whether they appear in the text so far, increasing the model's likelihood to talk about new topics.

For a full list of parameters, refer to the OpenAI API documentation.

Using

When creating an Activation with tools, if the model decides to use that tool, the output JSON representing the tool input will be an entry of the form {"tool_call": JSON}. The example below shows creating an Activation with a tool from the tool's JSON schema and how the reponse encodes the tool input:

llm = Activation(model="gpt-3.5-turbo", T=0.9, tools=[{"type": "function",  
                  "function": {  
                    "name": "get_current_weather",  
                    "description": "Get the current weather in a given location",  
                    "parameters": {  
                      "type": "object",  
                      "properties": {  
                        "location": {  
                          "type": "string",  
                          "description": "The city and state, e.g. San Francisco, CA"  
                        },  
                        "unit": {  
                          "type": "string",  
                          "enum": ["celsius", "fahrenheit"]  
                        }  
                      },  
                      "required": ["location"]  
                    }}}], tool_choice="auto")

response = llm(TextTensor("What is the current weather in San Francisco?")).item()
print(response.items())  
# Outputs: [('assistant', ('tool_call', '{"id": "call_6bII1dQHLYWiar5W1eSRDCQu", "type": "function", "function": {"name": "get_current_weather", "arguments": "{\\"location\\":\\"San Francisco\\"}"}}'))]

# Example code to handle both tool calls and message answers
for (k, v) in response.items():  
    if isinstance(v, tuple) and v[0] == "tool_call":  
        tool_call = json.loads(v[1])  
        name = tool_call["function"]["name"]  
        arguments = json.loads(tool_call["function"]["arguments"])  
        print(f"Tool Call: {name}, Arguments: {arguments}")
    else: # Chat Message Case
        print(f"Assistant Response: {v}")