词嵌入进阶.ipynb

{"cells":[{"cell_type":"markdown","metadata":{"graffitiCellId":"id_rjm21dx","jupyter":{},"tags":[],"slideshow":{"slide_type":"slide"},"id":"A5A4340181EA406E8EF192DFC1A66A96","mdEditEnable":false},"source":"# 词嵌入进阶\n\n在[“Word2Vec的实现”]()一节中，我们在小规模数据集上训练了一个 Word2Vec 词嵌入模型，并通过词向量的余弦相似度搜索近义词。虽然 Word2Vec 已经能够成功地将离散的单词转换为连续的词向量，并能一定程度上地保存词与词之间的近似关系，但 Word2Vec 模型仍不是完美的，它还可以被进一步地改进：\n\n1. 子词嵌入（subword embedding）：[FastText](https://zh.d2l.ai/chapter_natural-language-processing/fasttext.html) 以固定大小的 n-gram 形式将单词更细致地表示为了子词的集合，而 [BPE (byte pair encoding)](https://d2l.ai/chapter_natural-language-processing/subword-embedding.html#byte-pair-encoding) 算法则能根据语料库的统计信息，自动且动态地生成高频子词的集合；\n2. [GloVe 全局向量的词嵌入](https://zh.d2l.ai/chapter_natural-language-processing/glove.html): 通过等价转换 Word2Vec 模型的条件概率公式，我们可以得到一个全局的损失函数表达，并在此基础上进一步优化模型。\n\n实际中，我们常常在大规模的语料上训练这些词嵌入模型，并将预训练得到的词向量应用到下游的自然语言处理任务中。本节就将以 GloVe 模型为例，演示如何用预训练好的词向量来求近义词和类比词。\n\n## GloVe 全局向量的词嵌入\n\n### GloVe 模型\n\n先简单回顾以下 Word2Vec 的损失函数（以 Skip-Gram 模型为例，不考虑负采样近似）：\n\n\n$$\n\n-\\sum_{t=1}^T\\sum_{-m\\le j\\le m,j\\ne 0} \\log P(w^{(t+j)}\\mid w^{(t)})\n\n$$\n\n\n其中\n\n\n$$\n\nP(w_j\\mid w_i) = \\frac{\\exp(\\boldsymbol{u}_j^\\top\\boldsymbol{v}_i)}{\\sum_{k\\in\\mathcal{V}}\\exp(\\boldsymbol{u}_k^\\top\\boldsymbol{v}_i)}\n\n$$\n\n\n是 $w_i$ 为中心词，$w_j$ 为背景词时 Skip-Gram 模型所假设的条件概率计算公式，我们将其简写为 $q_{ij}$。\n\n注意到此时我们的损失函数中包含两个求和符号，它们分别枚举了语料库中的每个中心词和其对应的每个背景词。实际上我们还可以采用另一种计数方式，那就是直接枚举每个词分别作为中心词和背景词的情况：\n\n\n$$\n\n-\\sum_{i\\in\\mathcal{V}}\\sum_{j\\in\\mathcal{V}} x_{ij}\\log q_{ij}\n\n$$\n\n\n其中 $x_{ij}$ 表示整个数据集中 $w_j$ 作为 $w_i$ 的背景词的次数总和。\n\n我们还可以将该式进一步地改写为交叉熵 (cross-entropy) 的形式如下：\n\n\n$$\n\n-\\sum_{i\\in\\mathcal{V}}x_i\\sum_{j\\in\\mathcal{V}}p_{ij} \\log q_{ij}\n\n$$\n\n\n其中 $x_i$ 是 $w_i$ 的背景词窗大小总和，$p_{ij}=x_{ij}/x_i$ 是 $w_j$ 在 $w_i$ 的背景词窗中所占的比例。\n\n从这里可以看出，我们的词嵌入方法实际上就是想让模型学出 $w_j$ 有多大概率是 $w_i$ 的背景词，而真实的标签则是语料库上的统计数据。同时，语料库中的每个词根据 $x_i$ 的不同，在损失函数中所占的比重也不同。\n\n注意到目前为止，我们只是改写了 Skip-Gram 模型损失函数的表面形式，还没有对模型做任何实质上的改动。而在 Word2Vec 之后提出的 GloVe 模型，则是在之前的基础上做出了以下几点改动：\n\n1. 使用非概率分布的变量 $p'_{ij}=x_{ij}$ 和 $q′_{ij}=\\exp(\\boldsymbol{u}^\\top_j\\boldsymbol{v}_i)$，并对它们取对数；\n2. 为每个词 $w_i$ 增加两个标量模型参数：中心词偏差项 $b_i$ 和背景词偏差项 $c_i$，松弛了概率定义中的规范性；\n3. 将每个损失项的权重 $x_i$ 替换成函数 $h(x_{ij})$，权重函数 $h(x)$ 是值域在 $[0,1]$ 上的单调递增函数，松弛了中心词重要性与 $x_i$ 线性相关的隐含假设；\n4. 用平方损失函数替代了交叉熵损失函数。\n\n综上，我们获得了 GloVe 模型的损失函数表达式：\n\n\n$$\n\n\\sum_{i\\in\\mathcal{V}}\\sum_{j\\in\\mathcal{V}} h(x_{ij}) (\\boldsymbol{u}^\\top_j\\boldsymbol{v}_i+b_i+c_j-\\log x_{ij})^2\n\n$$\n\n\n由于这些非零 $x_{ij}$ 是预先基于整个数据集计算得到的，包含了数据集的全局统计信息，因此 GloVe 模型的命名取“全局向量”（Global Vectors）之意。\n\n### 载入预训练的 GloVe 向量\n\n[GloVe 官方](https://nlp.stanford.edu/projects/glove/) 提供了多种规格的预训练词向量，语料库分别采用了维基百科、CommonCrawl和推特等，语料库中词语总数也涵盖了从60亿到8,400亿的不同规模，同时还提供了多种词向量维度供下游模型使用。\n\n[`torchtext.vocab`](https://torchtext.readthedocs.io/en/latest/vocab.html) 中已经支持了 GloVe, FastText, CharNGram 等常用的预训练词向量，我们可以通过声明 [`torchtext.vocab.GloVe`](https://torchtext.readthedocs.io/en/latest/vocab.html#glove) 类的实例来加载预训练好的 GloVe 词向量。"},{"cell_type":"code","execution_count":2,"metadata":{"graffitiCellId":"id_su5pfc7","jupyter":{},"tags":[],"slideshow":{"slide_type":"slide"},"id":"BD67D8BDC6374AFD8056B0587FFC3B7E","collapsed":false,"scrolled":false},"outputs":[{"output_type":"stream","text":"['glove.42B.300d', 'glove.840B.300d', 'glove.twitter.27B.25d', 'glove.twitter.27B.50d', 'glove.twitter.27B.100d', 'glove.twitter.27B.200d', 'glove.6B.50d', 'glove.6B.100d', 'glove.6B.200d', 'glove.6B.300d']\n一共包含400000个词。\n3366 beautiful\n","name":"stdout"}],"source":"import torch\nimport torchtext.vocab as vocab\n\nprint([key for key in vocab.pretrained_aliases.keys() if \"glove\" in key])\ncache_dir = \"/home/kesci/input/GloVe6B5429\"\nglove = vocab.GloVe(name='6B', dim=50, cache=cache_dir)\nprint(\"一共包含%d个词。\" % len(glove.stoi))\nprint(glove.stoi['beautiful'], glove.itos[3366])"},{"cell_type":"markdown","metadata":{"graffitiCellId":"id_a6xttki","jupyter":{},"tags":[],"slideshow":{"slide_type":"slide"},"id":"0DCD1B715DE84E258D948974A5CE7F76","mdEditEnable":false},"source":"## 求近义词和类比词\n\n### 求近义词\n\n由于词向量空间中的余弦相似性可以衡量词语含义的相似性（为什么？），我们可以通过寻找空间中的 k 近邻，来查询单词的近义词。"},{"cell_type":"code","execution_count":3,"metadata":{"graffitiCellId":"id_hnlfnud","jupyter":{},"tags":[],"slideshow":{"slide_type":"slide"},"id":"923F41A77F21415883D370F34EF15D33","collapsed":false,"scrolled":false},"outputs":[{"output_type":"stream","text":"cosine sim=0.856: chips\ncosine sim=0.749: intel\ncosine sim=0.749: electronics\n","name":"stdout"},{"output_type":"stream","text":"\r100%|█████████▉| 398393/400000 [00:30<00:00, 38997.22it/s]","name":"stderr"}],"source":"def knn(W, x, k):\n    '''\n    @params:\n        W: 所有向量的集合\n        x: 给定向量\n        k: 查询的数量\n    @outputs:\n        topk: 余弦相似性最大k个的下标\n        [...]: 余弦相似度\n    '''\n    cos = torch.matmul(W, x.view((-1,))) / (\n        (torch.sum(W * W, dim=1) + 1e-9).sqrt() * torch.sum(x * x).sqrt())\n    _, topk = torch.topk(cos, k=k)\n    topk = topk.cpu().numpy()\n    return topk, [cos[i].item() for i in topk]\n\ndef get_similar_tokens(query_token, k, embed):\n    '''\n    @params:\n        query_token: 给定的单词\n        k: 所需近义词的个数\n        embed: 预训练词向量\n    '''\n    topk, cos = knn(embed.vectors,\n                    embed.vectors[embed.stoi[query_token]], k+1)\n    for i, c in zip(topk[1:], cos[1:]):  # 除去输入词\n        print('cosine sim=%.3f: %s' % (c, (embed.itos[i])))\n\nget_similar_tokens('chip', 3, glove)"},{"cell_type":"code","execution_count":4,"metadata":{"graffitiCellId":"id_q1w9uyd","jupyter":{},"tags":[],"slideshow":{"slide_type":"slide"},"id":"875E409C49E44DDD877BC1D8BDFA3D45","collapsed":false,"scrolled":false},"outputs":[{"output_type":"stream","text":"cosine sim=0.839: babies\ncosine sim=0.800: boy\ncosine sim=0.792: girl\n","name":"stdout"}],"source":"get_similar_tokens('baby', 3, glove)"},{"cell_type":"code","execution_count":5,"metadata":{"graffitiCellId":"id_u76vsiw","jupyter":{},"tags":[],"slideshow":{"slide_type":"slide"},"id":"C20619ABAA0F4B8686AD294528BD87D8","collapsed":false,"scrolled":false},"outputs":[{"output_type":"stream","text":"cosine sim=0.921: lovely\ncosine sim=0.893: gorgeous\ncosine sim=0.830: wonderful\n","name":"stdout"}],"source":"get_similar_tokens('beautiful', 3, glove)"},{"cell_type":"markdown","metadata":{"graffitiCellId":"id_j4l1wko","jupyter":{},"tags":[],"slideshow":{"slide_type":"slide"},"id":"E4F2C7332C054680887DE3E8F90795E7","mdEditEnable":false},"source":"### 求类比词\n\n除了求近义词以外，我们还可以使用预训练词向量求词与词之间的类比关系，例如“man”之于“woman”相当于“son”之于“daughter”。求类比词问题可以定义为：对于类比关系中的4个词“$a$ 之于 $b$ 相当于 $c$ 之于 $d$”，给定前3个词 $a,b,c$ 求 $d$。求类比词的思路是，搜索与 $\\text{vec}(c)+\\text{vec}(b)−\\text{vec}(a)$ 的结果向量最相似的词向量，其中 $\\text{vec}(w)$ 为 $w$ 的词向量。"},{"cell_type":"code","execution_count":6,"metadata":{"graffitiCellId":"id_8ovz8go","jupyter":{},"tags":[],"slideshow":{"slide_type":"slide"},"id":"4DA8D296F91C4B2798C8EBF5D36F405E","collapsed":false,"scrolled":false},"outputs":[{"output_type":"execute_result","metadata":{},"data":{"text/plain":"'daughter'"},"transient":{},"execution_count":6}],"source":"def get_analogy(token_a, token_b, token_c, embed):\n    '''\n    @params:\n        token_a: 词a\n        token_b: 词b\n        token_c: 词c\n        embed: 预训练词向量\n    @outputs:\n        res: 类比词d\n    '''\n    vecs = [embed.vectors[embed.stoi[t]] \n                for t in [token_a, token_b, token_c]]\n    x = vecs[1] - vecs[0] + vecs[2]\n    topk, cos = knn(embed.vectors, x, 1)\n    res = embed.itos[topk[0]]\n    return res\n\nget_analogy('man', 'woman', 'son', glove)"},{"cell_type":"code","execution_count":7,"metadata":{"graffitiCellId":"id_6r0q4tn","jupyter":{},"tags":[],"slideshow":{"slide_type":"slide"},"id":"B75CCBF19161403EB5A5C47817A7AD36","collapsed":false,"scrolled":false},"outputs":[{"output_type":"execute_result","metadata":{},"data":{"text/plain":"'japan'"},"transient":{},"execution_count":7}],"source":"get_analogy('beijing', 'china', 'tokyo', glove)"},{"cell_type":"code","execution_count":8,"metadata":{"graffitiCellId":"id_avf2squ","jupyter":{},"tags":[],"slideshow":{"slide_type":"slide"},"id":"61E15B6E656E49ABAE8E64352D9C9360","collapsed":false,"scrolled":false},"outputs":[{"output_type":"execute_result","metadata":{},"data":{"text/plain":"'biggest'"},"transient":{},"execution_count":8}],"source":"get_analogy('bad', 'worst', 'big', glove)"},{"cell_type":"code","execution_count":9,"metadata":{"graffitiCellId":"id_u115i98","jupyter":{},"tags":[],"slideshow":{"slide_type":"slide"},"id":"F89CC50CA1F14F9586A174E8AFE670FE","collapsed":false,"scrolled":false},"outputs":[{"output_type":"execute_result","metadata":{},"data":{"text/plain":"'went'"},"transient":{},"execution_count":9}],"source":"get_analogy('do', 'did', 'go', glove)"},{"metadata":{"id":"6A8560EF91DC48A88F44A79FE53C5AAE","jupyter":{},"tags":[],"slideshow":{"slide_type":"slide"}},"cell_type":"code","outputs":[],"source":"","execution_count":null}],"metadata":{"kernelspec":{"name":"python3","display_name":"Python 3","language":"python"},"language_info":{"name":"python","version":"3.7.3","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}