Rails friendly_id removing non ascii characters for url generation

Question

I want to auto generate the slug based on the title and author of the book, which can be in any language. Using friendly_id with rails, and when there are non-ascii characters, it just removes them. I know I can technically include by encoding them like this:

class Resource < ApplicationRecord
  extend FriendlyId
  friendly_id :title_and_author, use: :slugged

  def title_and_author
    "#{URI::encode(title)} by #{URI::encode(author)} #{language.language} book"
  end
end

This produces a long line that isn't user friendly d0-9b-d0-b5-d0-b4-d0-b8-20-d0-9c-d0-b0-d0-ba-d0-b1-d0-b5-d1-82-20-d0-9c-d1-86-d0-b5-d0-bd-d0-ba-d0-be-d0-b3-d0-be-20-d1-83-d0-b5-d0-b7-d0-b4-by-d0-9c-d0-b0-d1-82-spanish-book

I want the URL to have the words in the native script, not the encoded version. How can the slug include Russian, Japanese, Chinese, etc? I also have seen many posts suggesting including a translation gem, but there has to be an easier to just save the characters?? Why is it removing the characters in the first place?

UPDATE 1 Thanks to the comment below, here is the working model:

class Resource < ApplicationRecord
  extend FriendlyId
  friendly_id :title_and_author, use: :slugged

  def title_and_author
    "#{title} by #{author} #{language.language} book"
  end

  def normalize_friendly_id(string)
    string.gsub(" ", "-")
  end
end

UPDATE 2: Okay, Russian and the German B is working fine now, but Chinese characters are now causing issues. In the database, the slug will save as "/哈利波特百科全书-by-J.-K.-Rowling-Chinese-(Mandarin)-book-ed265df6-73d2-4b87-9326-524c2684756d", but when I go to the page, the url says http://localhost:3000/resources/%2F%E5%93%88%E5%88%A9%E6%B3%A2%E7%89%B9%E7%99%BE%E7%A7%91%E5%85%A8%E4%B9%A6-by-J.-K.-Rowling-Chinese-(Mandarin)-book-ed265df6-73d2-4b87-9326-524c2684756d and it throws a No route matches [GET] error. Even if I copy and paste http://localhost:3000/resources/%E5%93%88%E5%88%A9%E6%B3%A2%E7%89%B9%E7%99%BE%E7%A7%91%E5%85%A8%E4%B9%A6-by-J.-K.-Rowling-Chinese-(Mandarin)-book-ed265df6-73d2-4b87-9326-524c2684756d into the browser, I am still getting a no route error.

UPDATE 3 I'm realizing this is a bigger scope than I initially realized. I found articles about people asking how to handle other scripts, such as Chinese, Korean, Arabic, etc. but they all suggest translating that specific language. Is there not gem, or a way to include any language in the URL?

I saw this Babosa gem, but it doesn't have any Asian scripts

max max · Accepted Answer · 2020-04-28T15:43:49

By default, FriendlyId uses Active Support's paramaterize method to create slugs. This method will intelligently replace spaces with dashes, and Unicode Latin characters with ASCII approximations:
movie = Movie.create! :title => "Der Preis fürs Überleben"
movie.slug #=> "der-preis-furs-uberleben"
http://norman.github.io/friendly_id/file.Guide.html#Working_With_Slugs

You can override the slug generation process by overriding the #normalize_friendly_id method:

class User < ActiveRecord::Base
  friendly_id :name

  def normalize_friendly_id(string)
    string
  end
end

Why is it removing the characters in the first place?

Its known as transliteration and is done for compatibility reasons. While modern browsers support UTF-8 characters in urls this has not always been the case and support on the server side has also been spotty.

Rails friendly_id removing non ascii characters for url generation

1 Answers