Human-in-the-loop

There are certain tools that we don’t trust a model to execute on its own. One thing we can do in such situations is require human approval before the tool is invoked.

Setup

We’ll need to install the following packages:

npm install langchain @langchain/core @langchain/openai readline zod

We’ll use readline to handle accepting input from the user.

LangSmith

Many of the applications you build with LangChain will contain multiple steps with multiple invocations of LLM calls. As these applications get more and more complex, it becomes crucial to be able to inspect what exactly is going on inside your chain or agent. The best way to do this is with LangSmith.

Note that LangSmith is not needed, but it is helpful. If you do want to use LangSmith, after you sign up at the link above, make sure to set your environment variables to start logging traces:

export LANGCHAIN_TRACING_V2=true
export LANGCHAIN_API_KEY=YOUR_KEY

Chain

Suppose we have the following (dummy) tools and tool-calling chain:

import { ChatOpenAI } from "@langchain/openai";
import {
  Runnable,
  RunnableLambda,
  RunnablePassthrough,
} from "@langchain/core/runnables";
import { StructuredTool } from "@langchain/core/tools";
import { JsonOutputToolsParser } from "langchain/output_parsers";
import { z } from "zod";

class CountEmails extends StructuredTool {
  schema = z.object({
    lastNDays: z.number(),
  });

  name = "count_emails";

  description = "Count the number of emails sent in the last N days.";

  async _call(input: z.infer<typeof this.schema>): Promise<string> {
    return (input.lastNDays * 2).toString();
  }
}

class SendEmail extends StructuredTool {
  schema = z.object({
    message: z.string(),
    recipient: z.string(),
  });

  name = "send_email";

  description = "Send an email.";

  async _call(input: z.infer<typeof this.schema>): Promise<string> {
    return `Successfully sent email to ${input.recipient}`;
  }
}

const tools = [new CountEmails(), new SendEmail()];

const model = new ChatOpenAI({
  model: "gpt-3.5-turbo",
  temperature: 0,
}).bind({
  tools,
});

/**
 * Function for dynamically constructing the end of the chain based on the model-selected tool.
 */
const callTool = (toolInvocation: Record<string, any>): Runnable => {
  const toolMap: Record<string, StructuredTool> = tools.reduce((acc, tool) => {
    acc[tool.name] = tool;
    return acc;
  }, {});
  const tool = toolMap[toolInvocation.type];
  return RunnablePassthrough.assign({
    output: (input, config) => tool.invoke(input.args, config),
  });
};

// .map() allows us to apply a function to a list of inputs.
const callToolList = new RunnableLambda({ func: callTool }).map();
const chain = model.pipe(new JsonOutputToolsParser()).pipe(callToolList);

await chain.invoke("How many emails did I get in the last 5 days?");

[ { type: "count_emails", args: { lastNDays: 5 }, output: "10" } ]

Adding human approval

We can add a simple human approval step to our toolChain function:

import * as readline from "readline";
import { JsonOutputToolsParser } from "langchain/output_parsers";
import { callToolList, model } from "./helpers.js";

// Use readline to ask the user for approval
function askQuestion(question: string): Promise<string> {
  const rl = readline.createInterface({
    input: process.stdin,
    output: process.stdout,
  });

  return new Promise((resolve) => {
    rl.question(question, (answer) => {
      rl.close();
      resolve(answer);
    });
  });
}

async function humanApproval(toolInvocations: any[]): Promise<any[]> {
  const toolStrs = toolInvocations
    .map((toolCall) => JSON.stringify(toolCall, null, 2))
    .join("\n\n");
  const msg = `Do you approve of the following tool invocations\n\n${toolStrs}

Anything except 'Y'/'Yes' (case-insensitive) will be treated as a no.\n`;

  // Ask the user for approval
  const resp = await askQuestion(msg);
  if (!["yes", "y"].includes(resp.toLowerCase())) {
    throw new Error(`Tool invocations not approved:\n\n${toolStrs}`);
  }
  return toolInvocations;
}

const chain = model
  .pipe(new JsonOutputToolsParser())
  .pipe(humanApproval)
  .pipe(callToolList);

const response = await chain.invoke(
  "how many emails did i get in the last 5 days?"
);
console.log(response);
/**
Do you approve of the following tool invocations

{
  "type": "count_emails",
  "args": {
    "lastNDays": 5
  }
}

Anything except 'Y'/'Yes' (case-insensitive) will be treated as a no.
y
[ { type: 'count_emails', args: { lastNDays: 5 }, output: '10' } ]
 */
const response2 = await chain.invoke(
  "Send sally@gmail.com an email saying 'What's up homie'"
);
console.log(response2);
/**
Do you approve of the following tool invocations

{
  "type": "send_email",
  "args": {
    "message": "What's up homie",
    "recipient": "sally@gmail.com"
  }
}

Anything except 'Y'/'Yes' (case-insensitive) will be treated as a no.
y
[
  {
    type: 'send_email',
    args: { message: "What's up homie", recipient: 'sally@gmail.com' },
    output: 'Successfully sent email to sally@gmail.com'
  }
]
 */

API Reference:

• JsonOutputToolsParser from langchain/output_parsers

Examine the LangSmith traces from the code above here and here.