Pairwise Embedding Distance

One way to measure the similarity (or dissimilarity) between two predictions on a shared or similar input is to embed the predictions and compute a vector distance between the two embeddings.

You can load the pairwise_embedding_distance evaluator to do this.

Note: This returns a distance score, meaning that the lower the number, the more similar the outputs are, according to their embedded representation.

tip

See this section for general instructions on installing integration packages.

npm

npm install @langchain/openai

Yarn

yarn add @langchain/openai

pnpm

pnpm add @langchain/openai

import { OpenAIEmbeddings } from "@langchain/openai";
import { loadEvaluator } from "langchain/evaluation";

const embedding = new OpenAIEmbeddings();

const chain = await loadEvaluator("pairwise_embedding_distance", { embedding });

const res = await chain.evaluateStringPairs({
  prediction: "Seattle is hot in June",
  predictionB: "Seattle is cool in June.",
});

console.log({ res });

/*
  { res: { score: 0.03633645503883243 } }
*/

const res1 = await chain.evaluateStringPairs({
  prediction: "Seattle is warm in June",
  predictionB: "Seattle is cool in June.",
});

console.log({ res1 });

/*
  { res1: { score: 0.03657957473761331 } }
*/

API Reference:

• OpenAIEmbeddings from @langchain/openai
• loadEvaluator from langchain/evaluation