Enable and package the C# language extension on Linux - #110
Enable and package the C# language extension on Linux#110SicongLiu2000 wants to merge 45 commits into
Conversation
SQL Server on Linux expects a .tar.gz file instead of a .zip when using CREATE EXTERNAL LANGUAGE.
We will use the nethost to locate the system's installed hostfxr, as using a local libhostfxr.so breaks the .NET runtime lookup. nethost contains the logic needed to find the .NET root based on the DOTNET_ROOT environment variable or the install_location configuration file, which we will require for .NET runtime lookup on diverging Linux distributions (e.g. RHEL and Debian packages not installing .NET in the same base directories).
- Replace PowerShell build scripts with bash (build, archive, restore) - Remove checked-in libnethost.a; fetch at build time via restore-packages.sh - Add libnethost.a existence guard in build script - Add convert_string() abstraction in DotnetEnvironment - Conditional nethost.h include (Linux only), HOSTFXR_PATH_BUFFER_SIZE constant - Add LOG_ERROR on hostfxr/dlopen/dlsym failures (release-safe, not just assert) - Guard S_OK/E_FAIL macros (#ifndef) on non-Windows - Add Linux test build/run scripts and CMake platform support - Add -fshort-wchar and replace std::wstring with vector<wchar_t>/raw wchar_t* in test code to avoid ABI mismatch under -fshort-wchar on Linux - Add platform-conditional HintPath in test csproj - Add Linux dlopen/dlsym equivalents in test API code; throw on readlink failure (cherry picked from commit 2497aa2)
Bundle the entire .NET runtime (hostfxr, coreclr, shared framework) in the extension tar.gz so it works in containers without a system .NET installation. Changes: - build-dotnet-core-CSharp-extension.sh: dotnet build -> dotnet publish --self-contained -r linux-x64 - restore-packages.sh: add -r linux-x64 to dotnet restore for runtime pack download during network-enabled phase - DotnetEnvironment.cpp: pass assembly_path to get_hostfxr_path() for self-contained hostfxr discovery; default dotnet_root to extension directory when DOTNET_ROOT env var is not set Fixes: Extension Init() failed with -8223344 (hostfxr not found)
Load libhostfxr.so from the extension directory (self-contained deployment) rather than using get_hostfxr_path(), which may find an incompatible older hostfxr (e.g. .NET Core 3.x shipped with SQL Server). This mirrors the Windows approach of loading hostfxr.dll from the extension root. Falls back to nethost discovery for framework-dependent deployments where the bundled libhostfxr.so is not present. Fixes: hostfxr_initialize_for_runtime_config failed with 0x80008093 (Initialization for self-contained components is not supported)
Adds [DIAG] cout messages to load_hostfxr() and get_dotnet() to trace: - m_root_path value - Whether bundled libhostfxr.so is found via access() - Which hostfxr path is actually loaded (bundled vs nethost fallback) - config_path and dotnet_root values passed to hostfxr - hostfxr_initialize_for_runtime_config return code
…d Linux deployment hostfxr_initialize_for_runtime_config rejects self-contained runtimeconfig.json files (with 'includedFrameworks') returning error 0x80008093 (HostIncompatibleConfig). This is by design in .NET - that API only supports framework-dependent components. For self-contained deployments on Linux, use hostfxr_initialize_for_dotnet_command_line instead, which supports both framework-dependent and self-contained apps (.NET 5+). The hdt_load_assembly_and_get_function_pointer delegate works from command-line initialized contexts since .NET 5. Validated locally in WSL with bundled .NET 8.0.25 hostfxr - returns 0x00000000.
…ned .NET hosting - Use hostfxr_initialize_for_dotnet_command_line instead of hostfxr_initialize_for_runtime_config for self-contained deployments - Use hdt_get_function_pointer (type 6, Default ALC) instead of hdt_load_assembly_and_get_function_pointer (type 5, Isolated ALC) to resolve AssemblyLoadContext type cast failures - Add get_function_pointer_fn typedef and hdt_get_function_pointer enum - Dual-path call_managed_method: tries Default ALC first, falls back to Isolated ALC for backward compatibility
…nt runtimeconfig hostfxr_initialize_for_dotnet_command_line returns 0x80008093 (InvalidConfigFile) because .NET does not support self-contained component initialization. Instead: - Post-process runtimeconfig.json: replace includedFrameworks with framework - Create shared/Microsoft.NETCore.App/<version>/ symlink to extension root - Always use hostfxr_initialize_for_runtime_config (works for both modes) - Include shared/ directory in tar.gz - Fix double 0x prefix in to_hex_string
…ction SQL Server's tar extraction does not preserve symbolic links when extracting CREATE EXTERNAL LANGUAGE payloads. The shared/Microsoft.NETCore.App/<version>/ symlink (pointing to ../../../) was silently dropped, causing hostfxr to fail with 0x80008096 (FrameworkMissingFailure) - 'No frameworks were found'. Replace the directory symlink with hard links for each .dll and .so file. Hard links are stored efficiently in tar (as link entries with zero additional tar size) and are reliably extracted by all tar implementations - either as real hard links or as file copies.
SQL Server's CREATE EXTERNAL LANGUAGE tar extraction silently drops both symbolic links AND hard links. Use actual file copies (cp) to ensure framework DLLs/SOs are always present after extraction. Previous attempts: - Symlinks (ln -sfn): dropped by tar extraction -> 0x80008096 - Hard links (ln): also dropped by tar extraction -> 0x80008096 - File copies (cp): regular file entries, always extracted correctly
…debugging Log dotnet_root path, shared/ directory existence, framework version directories, runtimeconfig.json content, and relevant environment variables before calling hostfxr_initialize_for_runtime_config. This helps diagnose the 0x80008096 (FrameworkMissingFailure) error.
hostfxr requires Microsoft.NETCore.App.deps.json to recognize a framework version directory as valid. The build script was only copying *.dll and *.so, missing the .deps.json and .version files that hostfxr needs for framework resolution. This caused error 0x80008096 (FrameworkMissingFailure).
The .deps.json file only exists in the .NET SDK's shared/Microsoft.NETCore.App/ directory, NOT in the self-contained publish output. hostfxr requires this file to recognize a framework version as valid. Without it, hostfxr reports FrameworkMissingFailure (0x80008096) even when all DLLs/SOs are present.
…aged_method The get_dotnet_load_assembly function previously returned immediately after obtaining hdt_get_function_pointer without also obtaining hdt_load_assembly_and_get_function_pointer. This meant that when the Default ALC failed to resolve Microsoft.SqlServer.CSharpExtension by name (as happens in component-host scenarios on Linux), the fallback path in call_managed_method was never available because m_load_assembly_and_get_function_pointer was null. Fix: Always obtain both delegates. The explicit-path delegate (hdt_load_assembly_and_get_function_pointer) serves as a reliable fallback when name-based resolution via hdt_get_function_pointer fails. Also: switch diagnostic logging from LOG() (debug-only) to LOG_ERROR() (always on) for key initialization checkpoints so failures are visible in Release builds.
…xx-init diagnostics
…nd at dlopen on Ubuntu 22.04-based mssql containers)
|
Validation update: internal package pipeline build 178433468 succeeded after adding C# restore/build/archive/staging and nuspec entries. Package data-sql-language-extensions-linux 1.0.0-PR-csharp-linux-package-20260827.2 physically contains both debug/release C# Linux zips. Downstream mssql-server validation is blocked on feed promotion permissions, not source/build correctness: AddPackage is denied on SqlUnified and SQLPAL-Scratch, and CreateFeed is denied for a temporary SQL Linux feed. Required next step is feed-owner promotion of the exact package (184,631,226 bytes; SHA-256 AC796751DD67BDACD9FDE77C05EEE7129A96BEBAC3B8A07074A2E0BD4EBB3337) into SQLPAL-Unified. |
|
Six focused Linux PVS arms on drop 18.0.263-2-1842 all reached C# Setup, then exthost exited immediately before protocol initialization (one Failed Setup + one harness pass per distro; 0x80004004). Mounted Ubuntu 22.04 session 98c782be-6bf0-65c4-3c03-bcdf3768760c confirms the process launches and all child processes exit in the same second; SQL's later (0,0) abort is cleanup. The shipped native C# library has only libc in DT_NEEDED and embeds libstdc++ plus the GCC unwinder because both -static-libstdc++ and -static-libgcc remained after the final build switched to g++-11. The minimal fix removes those obsolete static flags: dynamic g++-11 stays within the RHEL 9/Ubuntu 22.04 GLIBCXX ceiling without loading a second runtime/unwinder into exthost. Bash syntax and diff checks pass; new DSLE artifact validation will verify DT_NEEDED and rerun the same matrix. |
|
Drop 18.0.263-2-1844 reproduced the same deterministic Setup abort on five completed distros after switching to dynamic g++-11. Each run still has Passed=1/Failed=1 (Setup), with no C# cases executing. Mounted Ubuntu 22 session a0d5cfc0-a2b1-2bee-3494-fba29d1cf766 shows exthost reports failure (SQL receives abort 1,1), but no CSharpExt constructor/Init diagnostic is retained, indicating failure before or during RTLD_NOW load. Registration selects libnativecsharpextension.so and all required v3 exports are present. Added a Linux hosted smoke gate that loads the release .so with RTLD_NOW and calls GetInterfaceVersion, Init, and Cleanup; both PR and Official pipelines now run it before packaging. Payload 34c9577, wrapper d7b31d405ed62215983eed7b22f9d257d50fdefd, diagnostic build 178634582. |
|
Hosted smoke build 178634582 passed dlopen, interface v3, bundled hostfxr, managed Init, and Cleanup. This proves package/runtime integrity outside SQL. The remaining PVS-only difference is ambient process environment. DotnetEnvironment's comment required bundled deployments to use their own extension root, but implementation prioritized ambient DOTNET_ROOT even after selecting bundled libhostfxr.so. Fixed precedence so self-contained mode always uses m_root_path; framework-dependent mode still honors ambient DOTNET_ROOT. The smoke test now poisons DOTNET_ROOT with /nonexistent/ambient-dotnet-root, making this invariant executable. Payload 87a74ed, wrapper 6456fa40ae57631d65aff1b4d4cfe6053df28890, validation build 178638295. |
|
Final drop 1845 still fails Setup on five normal distros. Mounted Ubuntu 22 session 5a8c8b42-7a32-0801-9482-bfeef0a51183 proves the library now loads, bundled root is selected, hostfxr initializes, managed Init succeeds, and InstallExternalLibrary succeeds (ErrorCode 0). PVS uses the DEBUG archive: runtimeconfig 258 bytes, managed DLL 62,976 bytes. The hosted smoke tested only RELEASE: runtimeconfig 329 bytes, managed DLL 59,392 bytes. Updated smoke-test-extension.py to require debug|release and pipeline invokes each in a separate process. Payload b4f47bb; wrapper e41e1ce34aaa26ed71e513ef51e2d5ff3922b115; diagnostic build 178738077. |
|
Both Debug and Release load/Init/Cleanup smoke passed in build 178738077. Final PVS logs show the bundled runtime and managed Init succeed; external E2E DLL installation also succeeds. The remaining untested boundary is user assembly loading/execution: PVS script Microsoft.SqlServer.CSharpExtensionE2ETest.E2EPassThroughExecutor runs from an installed DLL. Added a session smoke using the existing Microsoft.SqlServer.CSharpExtensionTest.CSharpTestExecutor: build its managed/native test project, pass that output as publicLibraryPath, then call InitSession, zero-row Execute, CleanupSession, and Cleanup for Debug and Release. Payload 90ca99a; wrapper c1dd2a45350c42671e88b7c896220f9cc76ab839; build 178744132. |
|
Session smoke build 178757020 reproduced the PVS failure in both Debug and Release. Root cause: hostfxr fallback loads Microsoft.SqlServer.CSharpExtension in an IsolatedComponentLoadContext, but DllUtils used Assembly.LoadFrom for user executors, loading them into the default context. The executor's AbstractSqlServerExtensionExecutor therefore had a different assembly identity and cast failed. The same scan also attempted to load PDB/JSON files as assemblies. Fix b6268a9 loads user DLLs through the AssemblyLoadContext owning AbstractSqlServerExtensionExecutor and filters discovery to .dll. Wrapper c2bbda362b5f1a1d8c1cceb553ceabb98e53ff68; direct session-smoke validation build 178764534. |
|
Latest Linux PVS diagnosis: five distros reached C# Execute, then failed in DllUtils.CreateDllList with DirectoryNotFoundException for /home/mssql_satellite/externallibrariessandboxprivatepath. SQL passes the optional private path even when no private library is mounted. Commit 5036691 now skips absent optional library roots and makes the permanent Debug/Release session smoke pass a nonempty nonexistent private path. GitHub checks are green; internal DSLE validation is running as build 178780126. |
|
Final drop 18.0.263-2-1849 results: Ubuntu 22.04, Ubuntu 24.04, RHEL 9, RHEL 10, and Azure Linux 3 x64 each completed 26 PlatformLinux C# E2E records with 26 passed, 0 failed, 0 not-executed, and 0 inconclusive. This validates the installed-library permission fix in real SQL execution. Ubuntu 26.04 failed before SQL startup and executed no C# tests; a matched official-drop control (18.0.263-2-115) reproduced the identical startup timeout. Six-distro qualification is blocked on the Ubuntu 26 image/startup owner, not on this payload. |
|
QUALIFIED on Linux x64. Final chain: payload 4d0809d -> wrapper 8518c24d52d8a8cbc0ea02c4092cfd2f95c4386f -> package 1.0.0-PR-csharp-linux-package-20260829.4 (182523934 bytes, SHA-256 7458DA8EB9C4020CDB2DB117927BE983502015D62E7F053CE99CBFAD4BD1A367) -> consumer a58ccea4b951fd49f4e272743bdb49496818227a -> build 233577302 -> drop 18.0.263-2-1855. Ubuntu 22/24/26, RHEL 9/10, and Azure Linux 3 x64 each executed 26 PlatformLinux C# E2E records: 26 passed, 0 failed, 0 not-executed, 0 inconclusive. |
2 participants
Why
DsMainDev C# Linux E2E was enabled on six x64 distros, but every board failed in
Setupbecause the officialdata-sql-language-extensions-linux 1.0.0-CI-master-20260813.1package contains R/Python/Java/ONNX only. Consequently mssql-server has no/opt/mssql-extensibility/lib/dotnet-core-CSharp-lang-extension-linux.zipto install.This branch rebases the Linux implementation from #64 onto current
mainand carries forward the proven runtime/package fixes fromdev/sicongliu/linuxTestwithout that branch's obsolete v2-host reset.What
nethost/hostfxrdiscovery and framework-dependent runtime layout that survives SQL archive extraction..zipoutput required byCREATE EXTERNAL LANGUAGE, while retaining.tar.gzcompatibility output.SetHostCallbacksimplementation.Validation
main(9a897b70).7992ab3.SetHostCallbacksnative/managed code remains present.dotnet-core-CSharp-lang-extension-linux.zip.Related: #64