PowerShell Invoke-RestMethod Umlauts issues with UTF-8 and Windows-1252

Question

Upon executing Confluence REST API calls I get back a response encoded in UTF-8. However, when I export the results with either Out-File or Export-CSV even with the -Encoding utf8 parameter German Umlauts are not correctly represented. For example, 'ü' is still 'Ã¼'.

From what I could gather it's due to the fact that PowerShell 5.1 natively relies on Windows-1252. I verified that Umlauts are preserved when using PowerShell Core by executing

[psobject].Assembly.GetTypes() | Where-Object { $_.Name -eq 'ClrFacade'} | ForEach-Object { $_.GetMethod('GetDefaultEncoding', [System.Reflection.BindingFlags]'nonpublic,static').Invoke($null, @()) }

Even changing the script file itself to use the encoding UTF-8 with BOM or Windows-1252 does not preserve Umlauts, neither in the PowerShell nor exportet output.

Do you know of any way to tell PowerShell 5.1 to preserve Umlauts while executing the REST call?

I cannot use PowerShell core as further operations require cmdlets which do net yet exist for PowerShell Core.

Thanks!

Do you know which part of your workflow is breaking the encoding? e.g. is it definitely invoke-restmethod, or maybe out-file or the viewer you're checking the file contents in? for example if i do this "ü" | out-file c:\temp\umlaut-utf8.txt -encoding utf8 i get a file with the bytes "EF BB BF C3 BC 0D 0A" which shows "ü" in notepad, but "ï»¿Ã¼" in a hex editor. — mclayton
@AdminOfThings, what you suggested does indeed preserve the Umlauts, that's nice. However, I noticed that it only does so when specifying the .txt extension. .csv does not work here as well. Ultimately, I would want to store REST response details in a CSV-file. — colonel_claypoo
@mclayton, I suspect it's the shell, i.e. PowerShell 5.1. in VSCode (but also powershell.exe in standalone mode behaves the same). I guess "ü" | out-file works because it's not first "internalized" by the shell as when executing a cmdlet. So to generalize, each Umlaut that is processed by a cmdlet in PS 5.1. should exhibit the encoding issue. — colonel_claypoo
do you have a simple repro i can try here? i tried this in a powershell.exe console, but the file shows the umlaut fine in notepad: $xmldoc = Invoke-RestMethod -Uri "https://en.wikipedia.org/wiki/%C3%9C"; $xmldoc.DocumentElement.InnerText | out-file c:\temp\out.txt -Encoding utf8; — mclayton

mclayton mclayton · Accepted Answer · 2019-10-24T13:29:58

As discussed in the comments, it looks like the Confluence API encodes http responses using UTF8, but does not include the "Content-Type" header to indicate that.

The HTTP specification for the charset parameter says that in the absence of this header, the client should assume it's encoded with ISO-8859-1 character set, so what is happening in your request is something like this:

# server (Confluence API) encodes response text using utf8
PS> $text = "ü";
PS> $bytes = [System.Text.Encoding]::UTF8.GetBytes($text);
PS> write-host $bytes;
195 188

# client (Invoke-RestMethod) decodes bytes as ISO-8859-1
PS> $text = [System.Text.Encoding]::GetEncoding("ISO-8859-1").GetString($bytes);
PS> write-host $text;
Ã¼

Given that you can't control what the server sends, you'll either need to capture the raw bytes yourself (e.g. using System.Net.Http.HttpClient) and decode them using UTF8, or modify the existing response to compensate for the encoding mismatch (e.g. below).

PS> $text = "Ã¼"
PS> $bytes = [System.Text.Encoding]::GetEncoding("ISO-8859-1").GetBytes($text)
PS> $text = [System.Text.Encoding]::UTF8.GetString($bytes)
PS> write-host $text
ü

Note that if you use the -Outfile parameter of Invoke-RestMethod it presumably streams the response bytes directly to disk without decoding or encoding them, so the resultant file already contains utf8 $bytes rather than utf8 $bytes -> string decoded using ISO-8859-1 -> file bytes encoded using utf8

PowerShell Invoke-RestMethod Umlauts issues with UTF-8 and Windows-1252

1 Answers