Get emoji list & aliases from data file instead of symlinks #47

mislav · 2014-06-03T13:05:21Z

Alternative to #44

Previously, emoji name & unicode aliases were determined by following symlinks among images/emoji/*.png. This led to nontrivial code for resolving these aliases, made it tricky for contributors to add new aliases and inspect existing ones, and didn't leave room for adding metadata to emojis such as tags or descriptions from the Unicode spec.

Moreover, the aliases as symlinks led to duplication of image assets in users' applications, with hocho.png and knife.png representing the same emoji but being two separate images. Users were also unsure what to do with unicode/{HEX-NAME}.png files, which would end up among their images after running the :emoji task.

This change removes the symlinks support and creates the list of emojis and their aliases in emoji.txt. A single emoji is now represented with an Emoji::Character instance, which has the image_filename method to determine the path to the corresponding image instead of having to construct it manually.

This is kind of a breaking change, however. Since neither hocho.png nor knife.png images won't exist anymore in people's image assets (only the unicode/{HEX-CODE}.png version of each emoji), simply gsubbing :knife: with <img src="/images/emoji/knife.png"> won't work anymore. Instead, users need to look up the Emoji::Character and use its image_filename method. I believe this is a worthwhile breaking change, since it gives us leverage to shift things around internally in future revisions (such as changing the images naming scheme) and not have to break anything for users again.

/cc @josh @muan

Emoji aliases as symlinks didn't scale.

mislav · 2014-06-03T13:37:43Z

Pros:

Ability to easily add new aliases, tags
Ability for contributors to be able to edit aliases/tags in the GitHub.com web interface
Handling individual emoji as actual Ruby objects
No symlinks means no cross-platform or gem problems
Fewer image files in users' image assets

Cons:

Can't easily replace syntax like :knife: with <img src="/images/emoji/knife.png">. Emoji filters like that in html-pipeline will have to be revisited.

Open questions:

Is the custom format of the emoji.txt file OK? I didn't want to go with YAML because I wanted to display raw emoji in the file, and YAML insists on escaping them.
I've preserved the old APIs such as Emoji.names, Emoji.unicodes, Emoji.unicode_for(...), but if this is going to lead to a major release of gemoji library, I would like to nuke those APIs as they are obsoleted by the new APIs that deal with Emoji::Character instances.
Could we maybe revisit functionality proposed in Add method to add your own custom emoji after loading the library #45?

/cc @javan

josh · 2014-06-03T15:22:58Z

General 👍

I think the directly alias linking is fine to 🔥. Maybe we just want to release this as a new major version.

Is the custom format of the emoji.txt file OK?

Yeah. Can we document what encoding that file is in. Assuming UTF-8, just want to be clear somewhere.

, I would like to nuke those APIs as they are obsoleted by the new APIs that deal with Emoji::Character instances.

🔥 them and lets do a major release.

Could we maybe revisit functionality proposed in 45

Yeah, the api should encompass image name and aliases now.

josh · 2014-06-03T15:25:25Z

db/dump.rb

+  puts "  = #{aliases.join(', ')}" if aliases.any?
+  puts "  ~ #{tags.join(', ')}" if tags.any?
+  puts "  + #{unicodes.map{|u| Emoji::Character.hex_inspect(u) }.join(', ')}" if unicodes.any?
+  # puts "  * #{spec_aliases.join(', ')}" if spec_aliases.any?


Is dump supposed to write back out to the .txt file somewhere?

I used it to create the .txt file via shell redirection, but yeah, it could write back to the .txt file maybe. Or I can just 💣 the dump script? Honestly, I don't think that the .txt file will need being generated programatically anymore because it's supposed to be edited by hand from now on.

josh · 2014-06-03T15:26:58Z

/cc @aroben for emoji too

javan · 2014-06-03T20:22:38Z

Is the custom format of the emoji.txt file OK?

What do you think about using JSON instead since we already store emoji in db/Category-Emoji.json?

mislav · 2014-06-04T03:26:07Z

On Wed, Jun 4, 2014 at 3:22 AM, Javan Makhmali [email protected]
wrote:

What do you think about using JSON instead since we already store emoji in
db/Category-Emoji.json?

I don't like JSON files as a source of data that's supposed to be manually
edited. I'd much rather that users append new aliases to a list such as = foo, bar, baz than to edit arrays like `["foo", "bar", "baz"]. Also, I
find the current text file much more readable and easier to navigate than a
huge JSON array with nested hashes.

We don't actually store emojis in Category-Emoji.json. That's simply a dump
of Apple's own configuration file which we're using in tests to check does
our list cover all known emojis. This dump is not used as source of data at
runtime and is not intended to be edited.

josh · 2014-06-04T03:42:35Z

Question about aliases, do they have to be globally unique? How are we validating that if so?

mislav · 2014-06-04T03:44:49Z

On Wed, Jun 4, 2014 at 10:42 AM, Joshua Peek [email protected]
wrote:

Question about aliases, do they have to be globally unique? How are we
validating that if so?

They are supposed to. We're not enforcing that constraint, though. I'll
think about how to do that technically.

javan · 2014-06-05T12:43:11Z

I'd much rather that users append new aliases to a list such as = foo, bar, baz

True, that is easier, but IMO we don't need to make it easy to add aliases. In fact, I'd rather freeze our current aliases (and possibly allow people to extend them in their app).

I'm more concerned with how it easy it us for us to add new emojis if / when Apple releases them.

josh · 2014-06-05T14:58:41Z

and possibly allow people to extend them in their app

Good point.

I'm more concerned with how it easy it us for us to add new emojis if / when Apple releases them.

Yeah, a set of new emojis are currently being discussed by the unicode community

mislav · 2014-06-06T03:17:45Z

On Thu, Jun 5, 2014 at 7:43 PM, Javan Makhmali [email protected]
wrote:

IMO we don't need to make it easy to add aliases.

Yeah, I agree we shouldn't be going wild with new aliases. I intend to use
tags for completion suggestions, and to add better tags instead of better
aliases in the future.

In fact, I'd rather freeze our current aliases (and possibly allow people
to extend them in their app).

👍 for in-app extensions. Also, an API for adding custom emoji might be
useful as well. I'll think about how that might look like.

So do both of you think we should have a JSON format instead of plaintext
I've invented? It means we would have to add the JSON gem dependency since
JSON is not available by default in Ruby 1.8. Personally, I don't think we
gain anything with JSON format, but I'm open to being convinced.

As for new emojis, that's why we have "Category-Emoji.json" extracted from
OS X and the integrity test. When a new OS X gets released with new emojis,
we'll re-run that task, the build will start failing, and we'll simply
inject a list of new emojis into the old list. I might repurpose the dump
script to do just that, actually. The "tough" part will be getting the new
PNGs (I have no idea how the current ones were obtained)?

javan · 2014-06-06T13:57:42Z

I'm not opposed to the plain text file and I like how readable it is. Just pushing back a little since it's a made up format, which makes it less useful universally. If, for example, http://www.emoji-cheat-sheet.com/ was built with a language other than Ruby, they'd have to write their own parser to consume our list of emojis.

The emoji PNGs were extracted with https://github.com/tmm1/emoji-extractor. It's awesome, but it saves the files like 1.png, 2.png, etc. so they have to be manually renamed to match their code points. Figuring out a way to automate that would be great. Here are a couple leads: tmm1/emoji-extractor#1

mislav · 2014-06-06T14:26:08Z

On Fri, Jun 6, 2014 at 8:57 PM, Javan Makhmali [email protected]
wrote:

If, for example, http://www.emoji-cheat-sheet.com/ was built with a
language other than Ruby, they'd have to write their own parser to consume
our list of emojis.

That's a good point.

josh · 2014-06-06T17:52:04Z

JSON is not available by default in Ruby 1.8.

Don't worry about 1.8. Consider it 1.9+. And if you wanted 1.8, you could just install the json gem as a polyfill.

mislav · 2014-06-20T11:14:11Z

@josh @javan OK the list of emoji is now in JSON. Comma-first style to allow adding/removing specific aliases or tags without affecting surrounding lines.

However, now that I think about it, I should probably configure CI or something to fail if someone tried to remove an existing alias via a PR, since that would constitute a breaking change. Will probably make such and other sanity checks in another PR.

I would merge this PR in an attempt to start moving master branch towards the next major release. Other tasks that I had in mind for separate PRs:

Create a dump script that will help extract new Unicode 7 emojis when Apple releases them
🔥 the old API such as name_for() and unicode_for()
Create new APIs for users to customize the list of emojis in their apps: add new (custom) emoji, add new aliases/tags to existing emoji

aroben · 2014-06-20T12:52:29Z

Seems like we don't need emoji.txt anymore, right?

aroben · 2014-06-20T12:55:27Z

db/emoji.txt

+  = green_heart
+❤️  (2764-fe0f) heavy black heart
+  = heart
+  ~ love


The current release of gemoji also maps a plain U+2764 character to this image. Is it intentional that we're no longer doing that? (Windows shows an emoji for plain U+2764, OS X does not.)

aroben · 2014-06-20T12:57:34Z

Did you generate emoji.json manually? Should we have a script that will generate it?

aroben · 2014-06-20T13:02:52Z

It would be great to sort emoji.txt and emoji.json in ascending code point order.

aroben · 2014-06-20T13:04:25Z

So many great things in this PR! I love the test that the README docs actually work.

aroben · 2014-06-20T20:07:31Z

Looks like emoji.json needs to be added to the gemspec.

mislav · 2014-06-23T04:32:01Z

Thanks @aroben for the review. I'll fix these items and work towards wrapping up a 2.0 release soon

Previously, emoji name & unicode aliases were determined by following symlinks among `images/emoji/*.png`. This led to nontrivial code for resolving these aliases, made it tricky for contributors to add new aliases and inspect existing ones, and didn't leave room for adding metadata to emojis such as tags or descriptions from the Unicode spec. Moreover, the aliases as symlinks led to duplication of image assets in users' applications, with `hocho.png` and `knife.png` representing the same emoji but being two separate images. Users were also unsure what to do with `unicode/{HEX-NAME}.png` files, which would end up among their images after running the `:emoji` task. This change removes the symlinks support and creates the list of emojis and their aliases in `emoji.json`. A single emoji is now represented with an Emoji::Character instance, which has the `image_filename` method to determine the path to the corresponding image instead of having to construct it manually.

This list contains textual descriptions for emoji characters.

This script was used to construct the initial `emoji.json` file. Example: rake db:dump | less -r

Instead of an unfriendly LocalJumpError, raise the Emoji::NotFound exception with a helpful message if the block was not given. If a fallback block was given, yield the value for which the lookup failed.

This brings it up to par with `aliases`, which includes `name` as well.

Ensure that tags accurately describe the original meaning of the emoji (with respect to its definition from Unicode spec).

Get emoji list & aliases from data file instead of symlinks

mislav · 2014-06-27T10:33:51Z

@aroben I've nuked emoji.txt, fixed the gemspec, and adjusted the dump script to write out the JSON format.

aroben · 2014-06-27T14:02:19Z

⚡⚡⚡

On Fri, Jun 27, 2014 at 6:33 AM, Mislav Marohnić [email protected]
wrote:

@aroben https://github.com/aroben I've nuked emoji.txt, fixed the
gemspec, and adjusted the dump script to write out the JSON format.

—
Reply to this email directly or view it on GitHub
#47 (comment).

mislav added 2 commits March 12, 2014 12:16

Simplify db:generate task and have JSON be human-readable

f1d0bfe

🔥 image symlinks

c8c9805

Emoji aliases as symlinks didn't scale.

josh reviewed Jun 3, 2014
View reviewed changes

mislav mentioned this pull request Jun 18, 2014

Adds support for aliases #44

Closed

aroben reviewed Jun 20, 2014
View reviewed changes

mislav added 4 commits June 27, 2014 11:21

Remove duplicate image for shipit squirrel

07e15fd

Add task to download Unicode 6.3 names list

0845c38

This list contains textual descriptions for emoji characters.

Add dump script to inspect Unicode descriptions & aliases

70db910

This script was used to construct the initial `emoji.json` file. Example: rake db:dump | less -r

mislav added 4 commits June 27, 2014 11:21

Improve emoji lookup fallback behavior

5d24012

Instead of an unfriendly LocalJumpError, raise the Emoji::NotFound exception with a helpful message if the block was not given. If a fallback block was given, yield the value for which the lookup failed.

Update usage examples in the README with the new API

2530b24

Have unicode_aliases include emoji's raw representation as well

01394ed

This brings it up to par with `aliases`, which includes `name` as well.

Tweak tags for some emoji

a911209

Ensure that tags accurately describe the original meaning of the emoji (with respect to its definition from Unicode spec).

mislav added a commit that referenced this pull request Jun 27, 2014

Merge pull request #47 from github/no-symlinks

d7b020f

Get emoji list & aliases from data file instead of symlinks

mislav merged commit d7b020f into master Jun 27, 2014

mislav deleted the no-symlinks branch June 27, 2014 10:33

jwheare mentioned this pull request Jun 27, 2014

Use emoji.json from gemoji iamcal/emoji-data#9

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Get emoji list & aliases from data file instead of symlinks #47

Get emoji list & aliases from data file instead of symlinks #47

mislav commented Jun 3, 2014

mislav commented Jun 3, 2014

josh commented Jun 3, 2014

josh Jun 3, 2014

mislav Jun 4, 2014

josh commented Jun 3, 2014

javan commented Jun 3, 2014

mislav commented Jun 4, 2014

josh commented Jun 4, 2014

mislav commented Jun 4, 2014

javan commented Jun 5, 2014

josh commented Jun 5, 2014

mislav commented Jun 6, 2014

javan commented Jun 6, 2014

mislav commented Jun 6, 2014

josh commented Jun 6, 2014

mislav commented Jun 20, 2014

aroben commented Jun 20, 2014

aroben Jun 20, 2014

aroben commented Jun 20, 2014

aroben commented Jun 20, 2014

aroben commented Jun 20, 2014

aroben commented Jun 20, 2014

mislav commented Jun 23, 2014

mislav commented Jun 27, 2014

aroben commented Jun 27, 2014

Get emoji list & aliases from data file instead of symlinks #47

Get emoji list & aliases from data file instead of symlinks #47

Conversation

mislav commented Jun 3, 2014

mislav commented Jun 3, 2014

josh commented Jun 3, 2014

josh Jun 3, 2014

Choose a reason for hiding this comment

mislav Jun 4, 2014

Choose a reason for hiding this comment

josh commented Jun 3, 2014

javan commented Jun 3, 2014

mislav commented Jun 4, 2014

josh commented Jun 4, 2014

mislav commented Jun 4, 2014

javan commented Jun 5, 2014

josh commented Jun 5, 2014

mislav commented Jun 6, 2014

javan commented Jun 6, 2014

mislav commented Jun 6, 2014

josh commented Jun 6, 2014

mislav commented Jun 20, 2014

aroben commented Jun 20, 2014

aroben Jun 20, 2014

Choose a reason for hiding this comment

aroben commented Jun 20, 2014

aroben commented Jun 20, 2014

aroben commented Jun 20, 2014

aroben commented Jun 20, 2014

mislav commented Jun 23, 2014

mislav commented Jun 27, 2014

aroben commented Jun 27, 2014