[BUG]: Method 'GetTokens' in type 'LLamaSharp.KernelMemory.LLamaSharpTextEmbeddingGenerator' from assembly 'LLamaSharp.KernelMemory, Version=0.14.0.0, Culture=neutral, PublicKeyToken=null' does not have an implementation.

[srulyg]

Description

This is my code:

using LLama.Common;
using LLama;
using LLama.Native;
using System;
using LLamaSharp.KernelMemory;
using Microsoft.KernelMemory.Configuration;
using Microsoft.KernelMemory;
using System.Diagnostics;
using LLama.Abstractions;
using Microsoft.KernelMemory.AI;
using Microsoft.KernelMemory.AI.LlamaSharp;

namespace consoleapp_llamasharp
{
internal class Program
{
public static async Task Main(string[] args)
{
Program p = new Program();

        string modelPath = @"C:\Users\Llama-2-7b-chat-hf-finetune-q5_k_m-v1.0\Llama-2-7b-chat-hf-finetune-q5_k_m-v1.0.gguf";
        LLama.Common.InferenceParams infParams = new() { AntiPrompts = ["\n\n"], MaxTokens = 100, TokensKeep = 100 };
        LLamaSharpConfig lsConfig = new(modelPath) { DefaultInferenceParams = infParams };
        SearchClientConfig searchClientConfig = new() { MaxMatchesCount = 1, AnswerTokens = 100 };
        TextPartitioningOptions parseOptions = new() { MaxTokensPerParagraph = 300, MaxTokensPerLine = 100, OverlappingTokens = 30 };

        try
        {
            IKernelMemory memory = new KernelMemoryBuilder()
            .WithLLamaSharpDefaults(lsConfig)
            .WithSearchClientConfig(searchClientConfig)
            .With(parseOptions)
            .Build();



            // Ingest documents (format is automatically detected from the filename)
            string documentFolder = @"C:\Users\\te.pdf";
            string[] documentPaths = Directory.GetFiles(documentFolder, "*.txt");
            for (int i = 0; i < documentPaths.Length; i++)
            {
                await memory.ImportDocumentAsync(documentPaths[i], steps: Constants.PipelineWithoutSummary);
            }

            // Allow the user to ask questions forever
            while (true)
            {
                Console.Write("\nQuestion: ");
                string question = Console.ReadLine() ?? string.Empty;
                MemoryAnswer answer = await memory.AskAsync(question);
                Console.WriteLine($"Answer: {answer.Result}");
            }

        }
        catch (Exception e)
        {

            Console.WriteLine(e.Message);
        }

    }
}

}

Just try running this, and you'll get this error:

Method 'GetTokens' in type 'LLamaSharp.KernelMemory.LLamaSharpTextEmbeddingGenerator' from assembly 'LLamaSharp.KernelMemory, Version=0.14.0.0, Culture=neutral, PublicKeyToken=null' does not have an implementation.

Reproduction Steps

Just run the code provided above.

Environment & Configuration

• Operating system:
• .NET runtime version:
• LLamaSharp version:
• CUDA version (if you are using cuda backend):
• CPU & GPU device:
  Latest version of LLamaSharp (14)

Known Workarounds

No response

[imabdul-dev]

Facing the same issue

[srulyg]

Facing the same issue

Do you have any workaround?

[SpaceAntelope]

It's because of changes in the latest version of the Microsoft.KernelMemory packages. One can either use older versions or work around the problem by implementing the GetTokens method themselves:

edit: this was wrong, as far as I can tell from looking in the Microsoft.KernelMemory repository GetTokens is supposed to return text tokens, not embeddings. Here's a version that returns the proper tokens:

// Based on the code in the CountTokens method:
public IReadOnlyList<string> GetTokens(string text)
{
    // !!WRONG!! return _embedder.Context.Tokenize(text, special: true).Select(x => x.ToString()).ToList();
   var embeddings = _context.Tokenize(text, special: true);
   var decoder = new StreamingTokenDecoder(_context);
   return embeddings
      .Select(x => { decoder.Add(x); return decoder.Read(); })
      .ToList()
      .AsReadOnly();
}

The classes that need updating are LLamaSharpTextEmbeddingGenerator and LLamaSharpTextGenerator which you can copy/paste to your project from the repository. Add GetTokens and change

builder.WithLLamaSharpTextEmbeddingGeneration(new LLamaSharpTextEmbeddingGenerator(config, weights));
builder.WithLLamaSharpTextGeneration(new LlamaSharpTextGenerator(weights, context, executor, config?.DefaultInferenceParams));

to

builder.WithCustomEmbeddingGenerator(new TextEmbeddingGenerator(config, weights));
builder.WithCustomTextGenerator(new TextGenerator(weights, context, executor, config?.DefaultInferenceParams));

where TextEmbeddingGenerator is whatever you named your LLamaSharpTextEmbeddingGenerator with GetTokens added, same with TextGenerator.

The code that registers the above is found in method WithLLamaSharpDefaults in BuilderExtensions.cs in project LLamaSharp.KernelMemory, as are LLamaSharpTextEmbeddingGenerator and LlamaSharpTextGenerator.

[srulyg]

It's because of changes in the latest version of the Microsoft.KernelMemory packages. One can either use older versions or work around the problem by implementing the GetTokens method themselves:

// Based on the code in the CountTokens method:
public IReadOnlyList<string> GetTokens(string text)
{
    return _embedder.Context.Tokenize(text, special: true).Select(x => x.ToString()).ToList();
}

The classes that need updating are LLamaSharpTextEmbeddingGenerator and LLamaSharpTextGenerator which you can copy/paste to your project from the repository. Add GetTokens and change

builder.WithLLamaSharpTextEmbeddingGeneration(new LLamaSharpTextEmbeddingGenerator(config, weights));
builder.WithLLamaSharpTextGeneration(new LlamaSharpTextGenerator(weights, context, executor, config?.DefaultInferenceParams));

to

builder.WithCustomEmbeddingGenerator(new TextEmbeddingGenerator(config, weights));
builder.WithCustomTextGenerator(new TextGenerator(weights, context, executor, config?.DefaultInferenceParams));

where TextEmbeddingGenerator is whatever you named your LLamaSharpTextEmbeddingGenerator with GetTokens added, same with TextGenerator.

The code that registers the above is found in method WithLLamaSharpDefaults in BuilderExtensions.cs in project LLamaSharp.KernelMemory, as are LLamaSharpTextEmbeddingGenerator and LlamaSharpTextGenerator.

Is there any workaround for my project that is using the NuGet package?

[martindevans]

A PR to bring us up to date to the latest KernelMemory version would be appreciated :)

[SpaceAntelope]

Is there any workaround for my project that is using the NuGet package?

@srulyg using the same version of the Microsoft.KernelMemory.Abstractions as LLamaSharp.KernelMemory as of 0.14 should eliminate the error, but if you need something that only works in later versions of Microsoft.KernelMemory you would probably be out of luck.

dotnet add package Microsoft.KernelMemory.Abstractions  --version 0.66.240709.1

[srulyg]

Thank you @SpaceAntelope ! This finally worked for me.

[SpaceAntelope]

A PR to bring us up to date to the latest KernelMemory version would be appreciated :)

I gave it a shot here #862