Exception generated launching Horde Agent built with 5.6

The source of the problem seems to be using IOptionsSnapshot with a Singleton. Per the MS documentation for IOptionsSnapshot, "Is registered as Scoped and therefore can't be injected into a Singleton service." But, for example, HordeClient uses it during the dependency injection process. I was able to get things working again by converting several IOptionsSnapshot to IOptions. It loses the updating function of course, but it doesn't exception. Is this a known issue? I was a little confused when it happened. It works fine if you run as local because the checking isn't done in that case. But the packaged one breaks. Additional question, we're new to horde - we're not doing something wrong with it are we? (That's causing this for us)

Update from licensee

Looking at some articles and after some testing I did find that changing (in ServiceRun.cs)

IHostBuilder hostBuilder = Host.CreateDefaultBuilder() 
==to==> 
IHostBuilder hostBuilder = Host.CreateDefaultBuilder().UseDefaultServiceProvider(options => options.ValidateScopes = false);

does succeed with the original source code. Not sure what's different that makes our behavior different.

Steps to Reproduce

1. Sync 5.6
2. Run BuildHorde.xml (development configuration)
3. Deploy
4. Install the agent as service
5. (Optionally) Download zip and launch

It seems to install, but on start of the service, it exceptions leading to a timeout starting. If you run the agent after unzip, you can see the exceptions.

Fatal error: System.AggregateException: Some services are not able to be constructed (Error while validating the service descriptor 'ServiceType: EpicGames.Horde.Storage.HttpStorageClient Lifetime: Singleton ImplementationType: EpicGames.Horde.Storage.HttpStorageClient': Cannot consume scoped service 'MS.Extensions.Options.IOptionsSnapshot`1[EpicGames.Horde.HordeOptions]' from singleton 'EpicGames.Horde.Storage.HttpStorageClient'.) (Error while validating the service descriptor 'ServiceType: EpicGames.Horde.HordeClientFactory Lifetime: Singleton ImplementationType: EpicGames.Horde.HordeClientFactory': Cannot consume scoped service 'MS.Extensions.Options.IOptionsSnapshot`1[EpicGames.Horde.HordeOptions]' from singleton 'EpicGames.Horde.HordeClientFactory'.) (Error while validating the service descriptor 'ServiceType: HordeAgent.Services.ISessionFactory Lifetime: Singleton ImplementationType: HordeAgent.Services.SessionFactory': Cannot consume scoped service 'MS.Extensions.Options.IOptionsSnapshot`1[EpicGames.Horde.HordeOptions]' from singleton 'EpicGames.Horde.HordeClientFactory'.) (Error while validating the service descriptor 'ServiceType: HordeAgent.Services.WorkerService Lifetime: Singleton ImplementationType: HordeAgent.Services.WorkerService': Cannot consume scoped service 'MS.Extensions.Options.IOptionsSnapshot`1[EpicGames.Horde.HordeOptions]' from singleton 'EpicGames.Horde.HordeClientFactory'.)
---> System.InvalidOperationException: Error while validating the service descriptor 'ServiceType: EpicGames.Horde.Storage.HttpStorageClient Lifetime: Singleton ImplementationType: EpicGames.Horde.Storage.HttpStorageClient': Cannot consume scoped service 'MS.Extensions.Options.IOptionsSnapshot`1[EpicGames.Horde.HordeOptions]' from singleton 'EpicGames.Horde.Storage.HttpStorageClient'.
---> System.InvalidOperationException: Cannot consume scoped service 'MS.Extensions.Options.IOptionsSnapshot`1[EpicGames.Horde.HordeOptions]' from singleton 'EpicGames.Horde.Storage.HttpStorageClient'.
at MS.Extensions.DependencyInjection.ServiceLookup.CallSiteValidator.VisitScopeCache(ServiceCallSite scopedCallSite, CallSiteValidatorState state)
at MS.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSite(ServiceCallSite callSite, TArgument argument)
at MS.Extensions.DependencyInjection.ServiceLookup.CallSiteValidator.VisitConstructor(ConstructorCallSite constructorCallSite, CallSiteValidatorState state)
at MS.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSiteMain(ServiceCallSite callSite, TArgument argument)
at MS.Extensions.DependencyInjection.ServiceLookup.CallSiteValidator.VisitRootCache(ServiceCallSite singletonCallSite, CallSiteValidatorState state)
at MS.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSite(ServiceCallSite callSite, TArgument argument)
at MS.Extensions.DependencyInjection.ServiceLookup.CallSiteValidator.ValidateCallSite(ServiceCallSite callSite)
at MS.Extensions.DependencyInjection.ServiceProvider.OnCreate(ServiceCallSite callSite)
at MS.Extensions.DependencyInjection.ServiceProvider.ValidateService(ServiceDescriptor descriptor)
— End of inner exception stack trace —
at MS.Extensions.DependencyInjection.ServiceProvider.ValidateService(ServiceDescriptor descriptor)
at MS.Extensions.DependencyInjection.ServiceProvider..ctor(ICollection`1 serviceDescriptors, ServiceProviderOptions options)
— End of inner exception stack trace —
at MS.Extensions.DependencyInjection.ServiceProvider..ctor(ICollection`1 serviceDescriptors, ServiceProviderOptions options)
at MS.Extensions.DependencyInjection.ServiceCollectionContainerBuilderExtensions.BuildServiceProvider(IServiceCollection services, ServiceProviderOptions options)
at MS.Extensions.DependencyInjection.DefaultServiceProviderFactory.CreateServiceProvider(IServiceCollection containerBuilder)
at MS.Extensions.Hosting.Internal.ServiceFactoryAdapter`1.CreateServiceProvider(Object containerBuilder)
at MS.Extensions.Hosting.HostBuilder.InitializeServiceProvider()
at MS.Extensions.Hosting.HostBuilder.Build()
at HordeAgent.Commands.Service.RunCommand.ExecuteAsync(ILogger logger) in C:\p4\dev\AOCUE5\Engine\Source\Programs\Horde\HordeAgent\Commands\Service\ServiceRun.cs:line 99
at EpicGames.Core.CommandHost.RunAsync(CommandLineArguments args, IServiceProvider serviceProvider, Type defaultCommandType, String toolDescription) in C:\p4\dev\AOCUE5\Engine\Source\Programs\Shared\EpicGames.Core\CommandHost.cs:line 306
---> (Inner Exception #1) System.InvalidOperationException: Error while validating the service descriptor 'ServiceType: EpicGames.Horde.HordeClientFactory Lifetime: Singleton ImplementationType: EpicGames.Horde.HordeClientFactory': Cannot consume scoped service 'MS.Extensions.Options.IOptionsSnapshot`1[EpicGames.Horde.HordeOptions]' from singleton 'EpicGames.Horde.HordeClientFactory'.
---> System.InvalidOperationException: Cannot consume scoped service 'MS.Extensions.Options.IOptionsSnapshot`1[EpicGames.Horde.HordeOptions]' from singleton 'EpicGames.Horde.HordeClientFactory'.
at MS.Extensions.DependencyInjection.ServiceLookup.CallSiteValidator.VisitScopeCache(ServiceCallSite scopedCallSite, CallSiteValidatorState state)
at MS.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSite(ServiceCallSite callSite, TArgument argument)
at MS.Extensions.DependencyInjection.ServiceLookup.CallSiteValidator.VisitConstructor(ConstructorCallSite constructorCallSite, CallSiteValidatorState state)
at MS.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSiteMain(ServiceCallSite callSite, TArgument argument)
at MS.Extensions.DependencyInjection.ServiceLookup.CallSiteValidator.VisitRootCache(ServiceCallSite singletonCallSite, CallSiteValidatorState state)
at MS.Extensions.DependencyInjection.ServiceLookup.CallSiteVisitor`2.VisitCallSite(ServiceCallSite callSite, TArgument argument)
at MS.Extensions.DependencyInjection.ServiceLookup.CallSiteValidator.ValidateCallSite(ServiceCallSite callSite)
at MS.Extensions.DependencyInjection.ServiceProvider.OnCreate(ServiceCallSite callSite)
at MS.Extensions.DependencyInjection.ServiceProvider.ValidateService(ServiceDescriptor descriptor)
— End of inner exception stack trace —
at MS.Extensions.DependencyInjection.ServiceProvider.ValidateService(ServiceDescriptor descriptor)
at MS.Extensions.DependencyInjection.ServiceProvider..ctor(ICollection`1 serviceDescriptors, ServiceProviderOptions options)<---