结合SK和ChatGLM3B+whisper+Avalonia实现语音切换城市

先创建一个Avalonia的MVVM项目模板,项目名称`GisApp`

项目创建完成以后添加以下nuget依赖

<PackageReference Include="Mapsui.Avalonia" Version="4.1.1" />

<PackageReference Include="Microsoft.Extensions.DependencyInjection" Version="8.0.0" />

<PackageReference Include="Microsoft.Extensions.Http" Version="8.0.0" />

<PackageReference Include="Microsoft.SemanticKernel" Version="1.0.0-beta8" />

<PackageReference Include="NAudio" Version="2.2.1" />

<PackageReference Include="Whisper.net" Version="1.5.0" />

<PackageReference Include="Whisper.net.Runtime" Version="1.5.0" />

Mapsui.Avalonia是Avalonia的一个Gis地图组件
Microsoft.Extensions.DependencyInjection用于构建一个DI容器
Microsoft.Extensions.Http用于注册一个HttpClient工厂
Microsoft.SemanticKernel则是SK用于构建AI插件
NAudio是一个用于录制语音的工具包
Whisper.net是一个.NET的Whisper封装Whisper用的是OpenAI开源的语音识别模型
Whisper.net.Runtime属于Whisper

修改`App.cs`

打开App.cs，修改成以下代码

public partial class App : Application

{

    public override void Initialize()

    {

        AvaloniaXamlLoader.Load(this);

    }

    public override void OnFrameworkInitializationCompleted()

    {

        if (ApplicationLifetime is IClassicDesktopStyleApplicationLifetime desktop)

        {

            var services = new ServiceCollection();

            services.AddSingleton<MainWindow>((services) => new MainWindow(services.GetRequiredService<IKernel>(), services.GetRequiredService<WhisperProcessor>())

            {

                DataContext = new MainWindowViewModel(),

            });

            services.AddHttpClient();

            var openAIHttpClientHandler = new OpenAIHttpClientHandler();

            var httpClient = new HttpClient(openAIHttpClientHandler);

            services.AddTransient<IKernel>((serviceProvider) =>

            {

                return new KernelBuilder()

                    .WithOpenAIChatCompletionService("gpt-3.5-turbo-16k", "fastgpt-zE0ub2ZxvPMwtd6XYgDX8jyn5ubiC",

                        httpClient: httpClient)

                    .Build();

            });

            services.AddSingleton(() =>

            {

                var ggmlType = GgmlType.Base;

                // 定义使用模型

                var modelFileName = "ggml-base.bin";

                return WhisperFactory.FromPath(modelFileName).CreateBuilder()

                    .WithLanguage("auto") // auto则是自动识别语言

                    .Build();

            });

            var serviceProvider = services.BuildServiceProvider();

            desktop.MainWindow = serviceProvider.GetRequiredService<MainWindow>();

        }

        base.OnFrameworkInitializationCompleted();

    }

}

OpenAIHttpClientHandler.cs，这个文件是用于修改SK的访问地址，默认的SK只支持OpenAI官方的地址并且不能进行修改！

public class OpenAIHttpClientHandler : HttpClientHandler

{

    protected override Task<HttpResponseMessage> SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)

    {

        if (request.RequestUri.LocalPath == "/v1/chat/completions")

        {

            var uriBuilder = new UriBuilder("http://您的ChatGLM3B地址/api/v1/chat/completions");

            request.RequestUri = uriBuilder.Uri;

        }

        return base.SendAsync(request, cancellationToken);

    }

}

修改ViewModels/MainWindowViewModel.cs

public class MainWindowViewModel : ViewModelBase

{

    private string subtitle = string.Empty;

    public string Subtitle

    {

        get => subtitle;

        set => this.RaiseAndSetIfChanged(ref subtitle, value);

    }

    private Bitmap butBackground;

    public Bitmap ButBackground

    {

        get => butBackground;

        set => this.RaiseAndSetIfChanged(ref butBackground, value);

    }

}

ButBackground是显示麦克风图标的写到模型是为了切换图标
Subtitle用于显示识别的文字

添加SK插件

创建文件/plugins/MapPlugin/AcquireLatitudeLongitude/config.json：这个是插件的相关配置信息

{

  "schema": 1,

  "type": "completion",

  "description": "获取坐标",

  "completion": {

    "max_tokens": 1000,

    "temperature": 0.3,

    "top_p": 0.0,

    "presence_penalty": 0.0,

    "frequency_penalty": 0.0

  },

  "input": {

    "parameters": [

      {

        "name": "input",

        "description": "获取坐标",

        "defaultValue": ""

      }

    ]

  }

}

创建文件/plugins/MapPlugin/AcquireLatitudeLongitude/skprompt.txt：下面是插件的prompt，通过以下内容可以提取用户城市然后得到城市的经纬度

请返回{{$input}}的经纬度然后返回以下格式，不要回复只需要下面这个格式：

{

    "latitude":"",

    "longitude":""

}

修改Views/MainWindow.axaml代码，将[素材](# 素材)添加到Assets中，

<Window xmlns="https://github.com/avaloniaui"

        xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"

        xmlns:vm="using:GisApp.ViewModels"

        xmlns:d="http://schemas.microsoft.com/expression/blend/2008"

        xmlns:mc="http://schemas.openxmlformats.org/markup-compatibility/2006"

        mc:Ignorable="d" d:DesignWidth="800" d:DesignHeight="450"

        x:Class="GisApp.Views.MainWindow"

        x:DataType="vm:MainWindowViewModel"

        Icon="/Assets/avalonia-logo.ico"

        Width="800"

        Height="800"

        Title="GisApp">

    <Design.DataContext>

        <vm:MainWindowViewModel />

    </Design.DataContext>

    <Grid>

        <Grid Name="MapStackPanel">

        </Grid>

        <StackPanel HorizontalAlignment="Right" VerticalAlignment="Bottom" Background="Transparent" Margin="25">

            <TextBlock Foreground="Black" Text="{Binding Subtitle}" Width="80" TextWrapping="WrapWithOverflow" Padding="8">

            </TextBlock>

            <Button Width="60" Click="Button_OnClick" Background="Transparent" VerticalAlignment="Center" HorizontalAlignment="Center">

                <Image Name="ButBackground" Source="{Binding ButBackground}" Height="40" Width="40"></Image>

            </Button>

        </StackPanel>

    </Grid>

</Window>

修改Views/MainWindow.axaml.cs代码

public partial class MainWindow : Window

{

    private bool openVoice = false;

    private WaveInEvent waveIn;

    private readonly IKernel _kernel;

    private readonly WhisperProcessor _processor;

    private readonly Channel<string> _channel = Channel.CreateUnbounded<string>();

    private MapControl mapControl;

    public MainWindow(IKernel kernel, WhisperProcessor processor)

    {

        _kernel = kernel;

        _processor = processor;

        InitializeComponent();

        mapControl = new MapControl();

        // 默认定位到深圳

        mapControl.Map = new Map()

        {

            CRS = "EPSG:3857",

            Home = n =>

            {

                var centerOfLondonOntario = new MPoint(114.06667, 22.61667);

                var sphericalMercatorCoordinate = SphericalMercator

                    .FromLonLat(centerOfLondonOntario.X, centerOfLondonOntario.Y).ToMPoint();

                n.ZoomToLevel(15);

                n.CenterOnAndZoomTo(sphericalMercatorCoordinate, n.Resolutions[15]);

            }

        };

        mapControl.Map?.Layers.Add(Mapsui.Tiling.OpenStreetMap.CreateTileLayer());

        MapStackPanel.Children.Add(mapControl);

        DataContextChanged += (sender, args) =>

        {

            using var voice = AssetLoader.Open(new Uri("avares://GisApp/Assets/voice.png"));

            ViewModel.ButBackground = new Avalonia.Media.Imaging.Bitmap(voice);

        };

        Task.Factory.StartNew(ReadMessage);

    }

    private MainWindowViewModel ViewModel => (MainWindowViewModel)DataContext;

    private void Button_OnClick(object? sender, RoutedEventArgs e)

    {

        if (openVoice)

        {

            using var voice = AssetLoader.Open(new Uri("avares://GisApp/Assets/voice.png"));

            ViewModel.ButBackground = new Avalonia.Media.Imaging.Bitmap(voice);

            waveIn.StopRecording();

        }

        else

        {

            using var voice = AssetLoader.Open(new Uri("avares://GisApp/Assets/open-voice.png"));

            ViewModel.ButBackground = new Avalonia.Media.Imaging.Bitmap(voice);

            // 获取当前麦克风设备

            waveIn = new WaveInEvent();

            waveIn.DeviceNumber = 0; // 选择麦克风设备，0通常是默认设备

            WaveFileWriter writer = new WaveFileWriter("recorded.wav", waveIn.WaveFormat);

            // 设置数据接收事件

            waveIn.DataAvailable += (sender, a) =>

            {

                Console.WriteLine($"接收到音频数据: {a.BytesRecorded} 字节");

                writer.Write(a.Buffer, 0, a.BytesRecorded);

                if (writer.Position > waveIn.WaveFormat.AverageBytesPerSecond * 30)

                {

                    waveIn.StopRecording();

                }

            };

            // 录音结束事件

            waveIn.RecordingStopped += async (sender, e) =>

            {

                writer?.Dispose();

                writer = null;

                waveIn.Dispose();

                await using var fileStream = File.OpenRead("recorded.wav");

                using var wavStream = new MemoryStream();

                await using var reader = new WaveFileReader(fileStream);

                var resampler = new WdlResamplingSampleProvider(reader.ToSampleProvider(), 16000);

                WaveFileWriter.WriteWavFileToStream(wavStream, resampler.ToWaveProvider16());

                wavStream.Seek(0, SeekOrigin.Begin);

                await Dispatcher.UIThread.InvokeAsync(() => { ViewModel.Subtitle = string.Empty; });

                string text = string.Empty;

                await foreach (var result in _processor.ProcessAsync(wavStream))

                {

                    await Dispatcher.UIThread.InvokeAsync(() => { ViewModel.Subtitle += text += result.Text; });

                }

                _channel.Writer.TryWrite(text);

            };

            Console.WriteLine("开始录音...");

            waveIn.StartRecording();

        }

        openVoice = !openVoice;

    }

    private async Task ReadMessage()

    {

        try

        {

            var pluginsDirectory = Path.Combine(Directory.GetCurrentDirectory(), "plugins");

            var chatPlugin = _kernel

                .ImportSemanticFunctionsFromDirectory(pluginsDirectory, "MapPlugin");

            // 循环读取管道中的数据

            while (await _channel.Reader.WaitToReadAsync())

            {

                // 读取管道中的数据

                while (_channel.Reader.TryRead(out var message))

                {

                    // 使用AcquireLatitudeLongitude插件，解析用户输入的地点，然后得到地点的经纬度

                    var value = await _kernel.RunAsync(new ContextVariables

                    {

                        ["input"] = message

                    }, chatPlugin["AcquireLatitudeLongitude"]);

                    // 解析字符串成模型

                    var acquireLatitudeLongitude =

                        JsonSerializer.Deserialize<AcquireLatitudeLongitude>(value.ToString());

                    // 使用MapPlugin插件，定位到用户输入的地点

                    var centerOfLondonOntario = new MPoint(acquireLatitudeLongitude.longitude, acquireLatitudeLongitude.latitude);

                    var sphericalMercatorCoordinate = SphericalMercator

                        .FromLonLat(centerOfLondonOntario.X, centerOfLondonOntario.Y).ToMPoint();

                    // 默认使用15级缩放

                    mapControl.Map.Navigator.ZoomToLevel(15);

                    mapControl.Map.Navigator.CenterOnAndZoomTo(sphericalMercatorCoordinate, mapControl.Map.Navigator.Resolutions[15]);

                }

            }

        }

        catch (Exception e)

        {

            Console.WriteLine(e);

        }

    }

    public class AcquireLatitudeLongitude

    {

        public double latitude { get; set; }

        public double longitude { get; set; }

    }

}

流程讲解：

用户点击了录制按钮触发了Button_OnClick事件，然后在Button_OnClick事件中会打开用户的麦克风，打开麦克风进行录制，在录制结束事件中使用录制完成产生的wav文件，然后拿到Whisper进行识别，识别完成以后会将识别结果写入到_channel
ReadMessage则是一直监听_channel的数据，当有数据写入，这里则会读取到，然后就将数据使用下面的sk执行AcquireLatitudeLongitude函数。

 var value = await _kernel.RunAsync(new ContextVariables

                    {

                        ["input"] = message

                    }, chatPlugin["AcquireLatitudeLongitude"]);

在解析value得到用户的城市经纬度
通过mapControl.Map.Navigator修改到指定经纬度。

完整的操作流程就完成了，当然实际业务会比这个更复杂。

素材

分享总结

讨论总结:

在本次会议中，讨论了如何结合SK、ChatGLM3B、Whisper和Avalonia来实现语音切换城市的功能。具体讨论了创建Avalonia的MVVM项目模板，添加了相关的NuGet依赖，修改了App.cs、ViewModels/MainWindowViewModel.cs以及添加了SK插件的相关配置和文件。

行动项目:

创建Avalonia的MVVM项目模板，项目名称为GisApp。
添加所需的NuGet依赖，包括Mapsui.Avalonia, Microsoft.Extensions.DependencyInjection, Microsoft.Extensions.Http, Microsoft.SemanticKernel, NAudio, Whisper.net和Whisper.net.Runtime。
修改App.cs，OpenAIHttpClientHandler.cs，ViewModels/MainWindowViewModel.cs以及相关的视图文件。
添加SK插件，包括创建相关的配置信息和prompt文件。
实现录制语音、语音识别和切换城市的功能流程。

技术交流群：737776595