{"id":3024,"date":"2025-12-10T00:21:20","date_gmt":"2025-12-09T16:21:20","guid":{"rendered":"https:\/\/www.tiptinker.com\/attention-is-all-you-need-a-visual-guide-to-the-transformer-architecture\/"},"modified":"2025-12-10T03:34:19","modified_gmt":"2025-12-09T19:34:19","slug":"attention-is-all-you-need-a-visual-guide-to-the-transformer-architecture","status":"publish","type":"post","link":"https:\/\/www.tiptinker.com\/ja\/attention-is-all-you-need-a-visual-guide-to-the-transformer-architecture\/","title":{"rendered":"\u3010\u56f3\u89e3\u3011Attention Is All You Need\uff1aTransformer\u30a2\u30fc\u30ad\u30c6\u30af\u30c1\u30e3\u5fb9\u5e95\u30ac\u30a4\u30c9"},"content":{"rendered":"

2017\u5e74\u3001Google Brain\u306e\u7814\u7a76\u8005\u305f\u3061\u304c\u767a\u8868\u3057\u305f\u8ad6\u6587\u300cAttention Is All You Need\u300d\u306f\u3001AI\u306e\u6b74\u53f2\u306b\u304a\u3051\u308b\u8ee2\u63db\u70b9\u3068\u306a\u308a\u307e\u3057\u305f\u3002\u73fe\u5728\u306eChatGPT\u3001Claude\u3001DeepL\u306a\u3069\u306e\u57fa\u76e4\u3068\u306a\u3063\u3066\u3044\u308b\u6280\u8853\u3001Transformer<\/strong>\u306f\u3053\u306e\u8ad6\u6587\u304b\u3089\u751f\u307e\u308c\u307e\u3057\u305f\u3002<\/p>\n

\u5f93\u6765\u306eRNN\uff08\u30ea\u30ab\u30ec\u30f3\u30c8\u30cb\u30e5\u30fc\u30e9\u30eb\u30cd\u30c3\u30c8\u30ef\u30fc\u30af\uff09\u3084LSTM\u304c\u62b1\u3048\u3066\u3044\u305f\u300c\u4e26\u5217\u51e6\u7406\u304c\u3067\u304d\u306a\u3044\u300d\u300c\u9577\u8ddd\u96e2\u306e\u6587\u8108\u3092\u4fdd\u6301\u3067\u304d\u306a\u3044\u300d\u3068\u3044\u3046\u81f4\u547d\u7684\u306a\u30dc\u30c8\u30eb\u30cd\u30c3\u30af\u3092\u3001Transformer\u306f\u3044\u304b\u306b\u3057\u3066\u89e3\u6d88\u3057\u305f\u306e\u304b\uff1f \u672c\u8a18\u4e8b\u3067\u306f\u3001\u305d\u306e\u6838\u5fc3\u3067\u3042\u308b\u300cAttention\u30e1\u30ab\u30cb\u30ba\u30e0\u300d\u3068\u30a2\u30fc\u30ad\u30c6\u30af\u30c1\u30e3\u306e\u69cb\u9020\u3092\u8996\u899a\u7684\u304b\u3064\u6280\u8853\u7684\u306b\u89e3\u8aac\u3057\u307e\u3059\u3002<\/p>\n

Transformer\u304c\u9769\u547d\u7684\u3067\u3042\u308b\u7406\u7531<\/h2>\n

Transformer\u4ee5\u524d\u306e\u81ea\u7136\u8a00\u8a9e\u51e6\u7406\u306f\u3001\u5358\u8a9e\u3092\u5148\u982d\u304b\u3089\u9806\u756a\u306b\u51e6\u7406\u3059\u308b\u5fc5\u8981\u304c\u3042\u308a\u307e\u3057\u305f\u3002\u3057\u304b\u3057\u3001Transformer\u306f**\u300cAttention\uff08\u6ce8\u610f\u6a5f\u69cb\uff09\u300d\u3092\u7528\u3044\u308b\u3053\u3068\u3067\u3001\u6587\u4e2d\u306e\u3059\u3079\u3066\u306e\u5358\u8a9e\u306e\u95a2\u4fc2\u6027\u3092<\/strong>\u540c\u6642\u306b\uff08\u4e26\u5217\u306b\uff09**\u8a08\u7b97\u3059\u308b\u3053\u3068\u3092\u53ef\u80fd\u306b\u3057\u307e\u3057\u305f\u3002<\/p>\n

\u6838\u5fc3\u6982\u5ff5\u306f\u4ee5\u4e0b\u306e3\u70b9\u306b\u96c6\u7d04\u3055\u308c\u307e\u3059\uff1a<\/p>\n

    \n
  1. \u4e26\u5217\u51e6\u7406\uff08Parallelism\uff09\uff1a<\/strong> \u5168\u30c7\u30fc\u30bf\u3092\u4e00\u5ea6\u306b\u51e6\u7406\u3067\u304d\u308b\u305f\u3081\u3001\u5b66\u7fd2\u901f\u5ea6\u304c\u98db\u8e8d\u7684\u306b\u5411\u4e0a\u3002<\/li>\n
  2. \u81ea\u5df1\u6ce8\u610f\u6a5f\u69cb\uff08Self-Attention\uff09\uff1a<\/strong> \u6587\u4e2d\u306e\u3042\u308b\u5358\u8a9e\u304c\u3001\u4ed6\u306e\u3069\u306e\u5358\u8a9e\u3068\u5f37\u304f\u95a2\u9023\u3057\u3066\u3044\u308b\u304b\u3092\u6570\u5024\u5316\u3002<\/li>\n
  3. \u4f4d\u7f6e\u30a8\u30f3\u30b3\u30fc\u30c7\u30a3\u30f3\u30b0\uff08Positional Encoding\uff09\uff1a<\/strong> \u9806\u5e8f\u60c5\u5831\u3092\u6301\u305f\u306a\u3044\u69cb\u9020\u306b\u5bfe\u3057\u3001\u5358\u8a9e\u306e\u4f4d\u7f6e\u60c5\u5831\u3092\u6570\u5b66\u7684\u306b\u4ed8\u4e0e\u3002<\/li>\n<\/ol>\n
    \n

    \u3010\u30a2\u30fc\u30ad\u30c6\u30af\u30c1\u30e3\u56f3\u89e3\u3011\u30c7\u30fc\u30bf\u306e\u6d41\u308c<\/h2>\n

    Transformer\u306e\u69cb\u9020\u306f\u8907\u96d1\u306b\u898b\u3048\u307e\u3059\u304c\u3001\u5927\u304d\u304f\u300c\u30a8\u30f3\u30b3\u30fc\u30c0\u30fc\uff08\u60c5\u5831\u306e\u5727\u7e2e\uff09\u300d\u3068\u300c\u30c7\u30b3\u30fc\u30c0\u30fc\uff08\u60c5\u5831\u306e\u751f\u6210\uff09\u300d\u306b\u5206\u304b\u308c\u307e\u3059\u3002\u4ee5\u4e0b\u306f\u3001\u305d\u306e\u30c7\u30fc\u30bf\u30d5\u30ed\u30fc\u3092\u53ef\u8996\u5316\u3057\u305f\u3082\u306e\u3067\u3059\u3002<\/p>\n

    \n
    graph TD\r\n    subgraph \"Encoder Stack (\u5165\u529b\u51e6\u7406)\"\r\n    A[\"\u5165\u529b\u30c6\u30ad\u30b9\u30c8 (Input Text)\"] --> B[\"\u57cb\u3081\u8fbc\u307f\u5c64 (Input Embedding)\"]\r\n    B --> C[\"\u4f4d\u7f6e\u30a8\u30f3\u30b3\u30fc\u30c7\u30a3\u30f3\u30b0 (Positional Encoding)\"]\r\n    C --> D[\"\u30de\u30eb\u30c1\u30d8\u30c3\u30c9\u30fb\u30a2\u30c6\u30f3\u30b7\u30e7\u30f3 (Multi-Head Attention)\"]\r\n    D --> E[\"\u6b8b\u5dee\u63a5\u7d9a & \u6b63\u898f\u5316 (Add & Norm)\"]\r\n    E --> F[\"\u30d5\u30a3\u30fc\u30c9\u30d5\u30a9\u30ef\u30fc\u30c9\u5c64 (Feed Forward)\"]\r\n    F --> G[\"\u6b8b\u5dee\u63a5\u7d9a & \u6b63\u898f\u5316 (Add & Norm)\"]\r\n    end\r\n\r\n    subgraph \"Decoder Stack (\u51fa\u529b\u751f\u6210)\"\r\n    H[\"\u51fa\u529b (Outputs \/ Shifted Right)\"] --> I[\"\u57cb\u3081\u8fbc\u307f\u5c64 (Output Embedding)\"]\r\n    I --> J[\"\u4f4d\u7f6e\u30a8\u30f3\u30b3\u30fc\u30c7\u30a3\u30f3\u30b0 (Positional Encoding)\"]\r\n    J --> K[\"\u30de\u30b9\u30af\u4ed8\u304d\u30a2\u30c6\u30f3\u30b7\u30e7\u30f3 (Masked Multi-Head Attention)\"]\r\n    K --> L[\"\u6b8b\u5dee\u63a5\u7d9a & \u6b63\u898f\u5316 (Add & Norm)\"]\r\n    G --> M[\"\u30a8\u30f3\u30b3\u30fc\u30c0\u30fc\u30fb\u30c7\u30b3\u30fc\u30c0\u30fc\u30a2\u30c6\u30f3\u30b7\u30e7\u30f3 (Cross Attention)\"]\r\n    L --> M\r\n    M --> N[\"\u6b8b\u5dee\u63a5\u7d9a & \u6b63\u898f\u5316 (Add & Norm)\"]\r\n    N --> O[\"\u30d5\u30a3\u30fc\u30c9\u30d5\u30a9\u30ef\u30fc\u30c9\u5c64 (Feed Forward)\"]\r\n    O --> P[\"\u6b8b\u5dee\u63a5\u7d9a & \u6b63\u898f\u5316 (Add & Norm)\"]\r\n    end\r\n\r\n    P --> Q[\"\u7dda\u5f62\u5c64 (Linear)\"]\r\n    Q --> R[\"\u30bd\u30d5\u30c8\u30de\u30c3\u30af\u30b9 (Softmax)\"]\r\n    R --> S[\"\u6700\u7d42\u51fa\u529b\u78ba\u7387 (Output Probabilities)\"]\r\n\r\n    style A fill:#f9f,stroke:#333,stroke-width:2px\r\n    style S fill:#9ff,stroke:#333,stroke-width:2px\r\n    style D fill:#ff9,stroke:#333,stroke-width:2px\r\n    style M fill:#ff9,stroke:#333,stroke-width:2px\r\n<\/code><\/pre>\n<\/div>\n
    \n
    \u6838\u5fc3\u30b3\u30fc\u30c9\uff1aScaled Dot-Product Attention<\/h2>\n
    Transformer\u306e\u5fc3\u81d3\u90e8\u306f\u3001Query (Q)\u3001Key (K)\u3001Value (V)<\/strong> \u3068\u3044\u30463\u3064\u306e\u30d9\u30af\u30c8\u30eb\u3092\u7528\u3044\u305f\u884c\u5217\u8a08\u7b97\u3067\u3059\u3002\u3053\u308c\u306f\u30c7\u30fc\u30bf\u30d9\u30fc\u30b9\u306e\u691c\u7d22\u306b\u4f3c\u3066\u3044\u307e\u3059\uff08\u30af\u30a8\u30ea\u3092\u6295\u3052\u3001\u30ad\u30fc\u3068\u7167\u5408\u3057\u3001\u5024\u3092\u53d6\u308a\u51fa\u3059\uff09\u3002<\/p>\n
    \u4ee5\u4e0b\u306f\u3001\u8ad6\u6587\u306e\u6570\u5f0f $\\text{Attention}(Q, K, V) = \\text{softmax}(\\frac{QK^T}{\\sqrt{d_k}})V$ \u3092PyTorch\u3067\u5b9f\u88c5\u3057\u305f\u3082\u306e\u3067\u3059\u3002<\/p>\n
    import torch\r\nimport torch.nn.functional as F\r\nimport math\r\n\r\ndef scaled_dot_product_attention(query, key, value, mask=None):\r\n    \"\"\"\r\n    Scaled Dot-Product Attention\u306e\u5b9f\u88c5\r\n    Args:\r\n        query: \u691c\u7d22\u30af\u30a8\u30ea (Batch, Heads, Seq_Len_Q, Depth)\r\n        key: \u7167\u5408\u30ad\u30fc (Batch, Heads, Seq_Len_K, Depth)\r\n        value: \u62bd\u51fa\u3059\u308b\u5024 (Batch, Heads, Seq_Len_K, Depth)\r\n        mask: \u7279\u5b9a\u306e\u5358\u8a9e\u3092\u7121\u8996\u3059\u308b\u305f\u3081\u306e\u30de\u30b9\u30af (\u30aa\u30d7\u30b7\u30e7\u30f3)\r\n    \"\"\"\r\n    d_k = query.size(-1) # \u6b21\u5143\u306e\u30b5\u30a4\u30ba\r\n    \r\n    # 1. Q\u3068K\u306e\u8ee2\u7f6e\u306e\u5185\u7a4d\u3092\u8a08\u7b97\uff08\u95a2\u9023\u5ea6\u30b9\u30b3\u30a2\uff09\r\n    scores = torch.matmul(query, key.transpose(-2, -1))\r\n    \r\n    # 2. \u6b21\u5143\u306e\u30eb\u30fc\u30c8\u3067\u5272\u3063\u3066\u30b9\u30b1\u30fc\u30ea\u30f3\u30b0\uff08\u52fe\u914d\u6d88\u5931\u9632\u6b62\uff09\r\n    scores = scores \/ math.sqrt(d_k)\r\n    \r\n    # 3. \u30de\u30b9\u30af\u9069\u7528\uff08\u5fc5\u8981\u306a\u5834\u5408\u3001\u672a\u6765\u306e\u5358\u8a9e\u306a\u3069\u3092\u96a0\u3059\uff09\r\n    if mask is not None:\r\n        scores = scores.masked_fill(mask == 0, -1e9)\r\n    \r\n    # 4. Softmax\u3067\u78ba\u7387\u5206\u5e03\uff08\u91cd\u307f\uff09\u306b\u5909\u63db\r\n    attention_weights = F.softmax(scores, dim=-1)\r\n    \r\n    # 5. \u91cd\u307f\u306b\u57fa\u3065\u3044\u3066Value\u3092\u52a0\u91cd\u5e73\u5747\r\n    output = torch.matmul(attention_weights, value)\r\n    \r\n    return output, attention_weights\r\n<\/code><\/pre>\n
    \n
    \u30b9\u30c6\u30c3\u30d7\u30d0\u30a4\u30b9\u30c6\u30c3\u30d7\uff1a\u51e6\u7406\u306e\u4ed5\u7d44\u307f<\/h2>\n
    Transformer\u304c\u300cI love AI\u300d\u3068\u3044\u3046\u6587\u3092\u7ffb\u8a33\u30fb\u7406\u89e3\u3059\u308b\u969b\u3001\u5185\u90e8\u3067\u306f\u4ee5\u4e0b\u306e\u51e6\u7406\u304c\u884c\u308f\u308c\u3066\u3044\u307e\u3059\u3002<\/p>\n
      \n
    1. \u30c8\u30fc\u30af\u30f3\u5316\u3068\u57cb\u3081\u8fbc\u307f (Embedding)<\/strong>\n