强曰为道

与天地相似,故不违。知周乎万物,而道济天下,故不过。旁行而不流,乐天知命,故不忧.
文档目录

04 - JavaScript 实践 / JavaScript Implementation

JavaScript 实践 / JavaScript Implementation

本章介绍如何在 JavaScript 和 TypeScript 中使用 MessagePack,涵盖浏览器端、Node.js 端以及二进制数据处理。

This chapter covers using MessagePack in JavaScript and TypeScript, including browser, Node.js, and binary data handling.


📖 库概览 / Library Overview

JavaScript 生态中有多个 MessagePack 实现,推荐使用官方的 @msgpack/msgpack

库 / Library特点 / Features推荐度
@msgpack/msgpack官方推荐,支持流式解码,TypeScript 支持⭐⭐⭐⭐⭐
msgpack-lite老牌库,稳定但不活跃⭐⭐⭐
msgpack5Node.js Buffer 优化⭐⭐⭐
@msgpack/msgpack (ES Module)现代 ESM 支持⭐⭐⭐⭐⭐

安装

# npm
npm install @msgpack/msgpack

# yarn
yarn add @msgpack/msgpack

# pnpm
pnpm add @msgpack/msgpack

# 验证安装
node -e "const { encode, decode } = require('@msgpack/msgpack'); console.log('OK')"

💻 基础使用 / Basic Usage

encode / decode

import { encode, decode } from "@msgpack/msgpack";

// ========== 序列化 ==========
const data = {
  id: 1001,
  name: "Alice",
  scores: [95, 87, 92],
  active: true,
  address: null,
};

const encoded = encode(data);
console.log(encoded);        // Uint8Array [133, 162, 105, 100, ...]
console.log(encoded.length); // 43 bytes (vs JSON ~70 bytes)

// ========== 反序列化 ==========
const decoded = decode(encoded);
console.log(decoded);
// { id: 1001, name: "Alice", scores: [95, 87, 92], active: true, address: null }

// 类型保持
console.log(typeof decoded.id);      // "number"
console.log(typeof decoded.name);    // "string"
console.log(Array.isArray(decoded.scores)); // true

与 JSON 对比

import { encode, decode } from "@msgpack/msgpack";

const user = {
  id: 12345,
  name: "张三",
  email: "[email protected]",
  roles: ["admin", "editor"],
  metadata: { loginCount: 42, lastLogin: "2024-01-15" },
};

// JSON
const jsonStr = JSON.stringify(user);
const jsonSize = new TextEncoder().encode(jsonStr).length;

// MessagePack
const mpEncoded = encode(user);
const mpSize = mpEncoded.length;

console.log(`JSON:       ${jsonSize} bytes`);
console.log(`MessagePack: ${mpSize} bytes`);
console.log(`节省:       ${((1 - mpSize / jsonSize) * 100).toFixed(1)}%`);
// JSON:       158 bytes
// MessagePack: 107 bytes
// 节省:       32.3%

💻 类型映射 / Type Mapping

JavaScript → MessagePack

JavaScript 类型MessagePack 类型说明
nullnil空值
undefinednil也映射为 nil
true/falseboolean布尔
number (整数)int/uint自动选择
number (浮点)float64双精度
bigintint64/uint64大整数
stringstrUTF-8 字符串
Uint8Arraybin二进制数据
Arrayarray数组
Objectmap映射
Date❌ 不支持需自定义
Map❌ 不支持需自定义
Set❌ 不支持需自定义

MessagePack → JavaScript

MessagePack 类型JavaScript 类型
nilnull
booleanboolean
int/uintnumber / bigint
float32/64number
strstring
binUint8Array
arrayArray
mapObject
extExtension 类型

处理 BigInt

import { encode, decode, useBigInt64 } from "@msgpack/msgpack";

// 默认: 大整数可能丢失精度
const bigNum = 9007199254740993n; // 超过 Number.MAX_SAFE_INTEGER

// 使用 useBigInt64 选项
const encoded = encode({ value: bigNum }, { useBigInt64: true });
const decoded = decode(encoded, { useBigInt64: true });

console.log(decoded.value); // 9007199254740993n (bigint)
console.log(typeof decoded.value); // "bigint"

处理二进制数据

import { encode, decode } from "@msgpack/msgpack";

// Uint8Array 自动编码为 MessagePack bin 类型
const binaryData = new Uint8Array([0xde, 0xad, 0xbe, 0xef]);
const encoded = encode({ payload: binaryData });
const decoded = decode(encoded);

console.log(decoded.payload);        // Uint8Array [222, 173, 190, 239]
console.log(decoded.payload instanceof Uint8Array); // true

💻 自定义类型 / Custom Types

ExtensionType 编码器

import { encode, decode, ExtensionCodec } from "@msgpack/msgpack";

// 创建扩展编解码器
const extensionCodec = new ExtensionCodec();

// 注册 Date 类型 (type = 0)
extensionCodec.register({
  type: 0,
  encode: (object) => {
    if (object instanceof Date) {
      // 编码为 ISO 字符串
      const str = object.toISOString();
      return new TextEncoder().encode(str);
    }
    return null; // 不处理
  },
  decode: (data) => {
    const str = new TextDecoder().decode(data);
    return new Date(str);
  },
});

// 注册 Set 类型 (type = 1)
extensionCodec.register({
  type: 1,
  encode: (object) => {
    if (object instanceof Set) {
      return encode([...object], { extensionCodec });
    }
    return null;
  },
  decode: (data) => {
    const arr = decode(data, { extensionCodec });
    return new Set(arr);
  },
});

// 注册 Map 类型 (type = 2)
extensionCodec.register({
  type: 2,
  encode: (object) => {
    if (object instanceof Map) {
      return encode(Object.fromEntries(object), { extensionCodec });
    }
    return null;
  },
  decode: (data) => {
    const obj = decode(data, { extensionCodec });
    return new Map(Object.entries(obj));
  },
});

// 使用
const data = {
  created: new Date("2024-01-15T10:30:00Z"),
  tags: new Set(["js", "msgpack", "binary"]),
  config: new Map([["debug", true], ["verbose", false]]),
};

const encoded = encode(data, { extensionCodec });
const decoded = decode(encoded, { extensionCodec });

console.log(decoded.created instanceof Date);   // true
console.log(decoded.tags instanceof Set);        // true
console.log(decoded.config instanceof Map);      // true
console.log(decoded.tags.has("msgpack"));        // true

简单的 TypedArray 支持

import { encode, decode, ExtensionCodec } from "@msgpack/msgpack";

const extensionCodec = new ExtensionCodec();

// 支持 Float32Array (type = 10)
extensionCodec.register({
  type: 10,
  encode: (object) => {
    if (object instanceof Float32Array) {
      return new Uint8Array(object.buffer);
    }
    return null;
  },
  decode: (data) => new Float32Array(data.buffer),
});

const vertices = new Float32Array([1.0, 2.0, 3.0, 4.0, 5.0, 6.0]);
const encoded = encode({ vertices }, { extensionCodec });
const decoded = decode(encoded, { extensionCodec });

console.log(decoded.vertices); // Float32Array [1, 2, 3, 4, 5, 6]

💻 浏览器端使用 / Browser Usage

基础用法

<!DOCTYPE html>
<html>
<head>
  <title>MessagePack Browser Demo</title>
</head>
<body>
  <script type="module">
    import { encode, decode } from "https://esm.sh/@msgpack/msgpack";
    
    // 或者通过打包工具引入
    // import { encode, decode } from "@msgpack/msgpack";
    
    const data = {
      action: "ping",
      timestamp: Date.now(),
      payload: { message: "Hello from browser" }
    };
    
    // 序列化
    const encoded = encode(data);
    console.log("编码大小:", encoded.length, "bytes");
    
    // 通过 fetch 发送
    const response = await fetch("/api/data", {
      method: "POST",
      headers: { "Content-Type": "application/x-msgpack" },
      body: encoded,
    });
    
    // 解析响应
    const buffer = await response.arrayBuffer();
    const result = decode(new Uint8Array(buffer));
    console.log("响应:", result);
  </script>
</body>
</html>

与 fetch 集成

import { encode, decode } from "@msgpack/msgpack";

// 发送 MessagePack 请求
async function msgpackFetch(url, data) {
  const encoded = encode(data);
  
  const response = await fetch(url, {
    method: "POST",
    headers: {
      "Content-Type": "application/x-msgpack",
      "Accept": "application/x-msgpack",
    },
    body: encoded,
  });
  
  if (!response.ok) {
    throw new Error(`HTTP ${response.status}`);
  }
  
  const buffer = await response.arrayBuffer();
  return decode(new Uint8Array(buffer));
}

// 使用
const result = await msgpackFetch("/api/users", {
  name: "Alice",
  email: "[email protected]"
});
console.log(result);

WebSocket 集成

import { encode, decode } from "@msgpack/msgpack";

class MsgPackWebSocket {
  constructor(url) {
    this.ws = new WebSocket(url);
    this.ws.binaryType = "arraybuffer";
    
    this.ws.onmessage = (event) => {
      const data = decode(new Uint8Array(event.data));
      this.onMessage(data);
    };
  }
  
  send(data) {
    const encoded = encode(data);
    this.ws.send(encoded);
  }
  
  onMessage(data) {
    console.log("收到:", data);
  }
}

// 使用
const client = new MsgPackWebSocket("ws://localhost:8080/ws");
client.send({ type: "subscribe", channel: "updates" });

💻 Node.js 端使用 / Node.js Usage

Buffer 与 Uint8Array

import { encode, decode } from "@msgpack/msgpack";
import { Buffer } from "buffer";

// encode 返回 Uint8Array
const encoded = encode({ hello: "world" });

// Node.js 中可转为 Buffer
const buffer = Buffer.from(encoded);
console.log(buffer.toString("hex"));

// 从 Buffer 解码
const decoded = decode(new Uint8Array(buffer));
console.log(decoded);

文件读写

import { encode, decode } from "@msgpack/msgpack";
import fs from "fs";

// 写入文件
const data = { users: [{ name: "Alice" }, { name: "Bob" }] };
const encoded = encode(data);
fs.writeFileSync("data.msgpack", encoded);

// 读取文件
const buffer = fs.readFileSync("data.msgpack");
const decoded = decode(new Uint8Array(buffer));
console.log(decoded);

流式解码

import { decodeAsync } from "@msgpack/msgpack";
import fs from "fs";

async function readMsgPackStream(filePath) {
  const stream = fs.createReadStream(filePath);
  
  // decodeAsync 可以从 ReadableStream 中解码
  for await (const item of decodeAsync(stream)) {
    console.log("解码:", item);
  }
}

// 使用
await readMsgPackStream("large-data.msgpack");

Express 中间件

import express from "express";
import { encode, decode } from "@msgpack/msgpack";

const app = express();

// 解析 MessagePack 请求体
app.use("/api", (req, res, next) => {
  if (req.headers["content-type"] === "application/x-msgpack") {
    const chunks = [];
    req.on("data", chunk => chunks.push(chunk));
    req.on("end", () => {
      const buffer = Buffer.concat(chunks);
      req.body = decode(new Uint8Array(buffer));
      next();
    });
  } else {
    next();
  }
});

// MessagePack 响应
function msgpackRes(res, data) {
  const encoded = encode(data);
  res.setHeader("Content-Type", "application/x-msgpack");
  res.send(Buffer.from(encoded));
}

app.post("/api/users", (req, res) => {
  console.log("收到:", req.body);
  msgpackRes(res, { status: "ok", id: 1 });
});

app.listen(3000);

💻 TypeScript 类型 / TypeScript Types

基础类型定义

import { encode, decode } from "@msgpack/msgpack";

// MessagePack 支持的类型
type MsgPackValue =
  | null
  | undefined
  | boolean
  | number
  | bigint
  | string
  | Uint8Array
  | MsgPackValue[]
  | { [key: string]: MsgPackValue };

// 使用泛型
interface User {
  id: number;
  name: string;
  email: string;
  roles: string[];
}

const user: User = {
  id: 1,
  name: "Alice",
  email: "[email protected]",
  roles: ["admin"],
};

// 编码
const encoded: Uint8Array = encode(user);

// 解码并断言类型
const decoded = decode(encoded) as User;
console.log(decoded.name); // 类型安全

泛型辅助函数

import { encode, decode, ExtensionCodec } from "@msgpack/msgpack";

// 类型安全的序列化/反序列化
function serialize<T>(data: T, extensionCodec?: ExtensionCodec): Uint8Array {
  return encode(data, { extensionCodec });
}

function deserialize<T>(data: Uint8Array, extensionCodec?: ExtensionCodec): T {
  return decode(data, { extensionCodec }) as T;
}

// 使用
interface Config {
  debug: boolean;
  maxRetries: number;
  timeout: number;
}

const config: Config = { debug: true, maxRetries: 3, timeout: 5000 };
const packed = serialize<Config>(config);
const unpacked = deserialize<Config>(packed);

// 类型安全
console.log(unpacked.debug);     // boolean
console.log(unpacked.maxRetries); // number

扩展类型的 TypeScript 支持

import { encode, decode, ExtensionCodec } from "@msgpack/msgpack";

// 定义扩展类型接口
interface DateExt {
  __ext_type: 0;
  value: string;
}

interface SetExt {
  __ext_type: 1;
  value: unknown[];
}

// 创建类型安全的扩展编解码器
const extensionCodec = new ExtensionCodec();

extensionCodec.register({
  type: 0,
  encode: (object: unknown): Uint8Array | null => {
    if (object instanceof Date) {
      return encode(object.toISOString());
    }
    return null;
  },
  decode: (data: Uint8Array): Date => {
    return new Date(decode(data) as string);
  },
});

// 使用
interface DataPoint {
  timestamp: Date;
  value: number;
}

const point: DataPoint = { timestamp: new Date(), value: 42.5 };
const encoded = encode(point, { extensionCodec });
const decoded = decode(encoded, { extensionCodec }) as DataPoint;

console.log(decoded.timestamp instanceof Date); // true

模块声明

// types/msgpack.d.ts
declare module "@msgpack/msgpack" {
  export function encode(
    data: unknown,
    options?: EncodeOptions
  ): Uint8Array;
  
  export function decode(
    buffer: ArrayLike<number> | BufferSource,
    options?: DecodeOptions
  ): unknown;
  
  export function decodeAsync(
    stream: ReadableStream<Uint8Array>,
    options?: DecodeOptions
  ): AsyncIterable<unknown>;
  
  export interface EncodeOptions {
    extensionCodec?: ExtensionCodec;
    maxDepth?: number;
    initialBufferSize?: number;
    sortKeys?: boolean;
    forceFloat32?: boolean;
    forceIntegerToFloat?: boolean;
    useBigInt64?: boolean;
    ignoreUndefined?: boolean;
  }
  
  export interface DecodeOptions {
    extensionCodec?: ExtensionCodec;
    maxStrLength?: number;
    maxBinLength?: number;
    maxArrayLength?: number;
    maxMapLength?: number;
    maxExtLength?: number;
    useBigInt64?: boolean;
  }
  
  export class ExtensionCodec {
    register(options: {
      type: number;
      encode: (input: unknown) => Uint8Array | null;
      decode: (data: Uint8Array) => unknown;
    }): void;
  }
}

💻 性能优化 / Performance Optimization

选择合适的编码选项

import { encode, decode } from "@msgpack/msgpack";

// 1. 紧凑模式:强制 float32(节省空间)
const compact = encode(3.14, { forceFloat32: true });

// 2. 忽略 undefined 字段
const data = { a: 1, b: undefined, c: 3 };
const clean = encode(data, { ignoreUndefined: true });

// 3. 排序键(确定性输出)
const stable = encode({ b: 2, a: 1 }, { sortKeys: true });

批量处理

import { encode, decode } from "@msgpack/msgpack";

// 高效批量处理
function batchProcess<T>(items: T[]): Uint8Array {
  // 一次编码整个数组,而不是逐个编码
  return encode(items);
}

// 反向:批量解码
function batchDecode<T>(data: Uint8Array): T[] {
  return decode(data) as T[];
}

// 测试
const items = Array.from({ length: 10000 }, (_, i) => ({
  id: i,
  value: `item_${i}`,
}));

const packed = batchProcess(items);
console.log(`10000 条记录编码为 ${packed.length} bytes`);
const unpacked = batchDecode<typeof items[0]>(packed);
console.log(`解码回 ${unpacked.length} 条记录`);

内存管理

// 大数据时注意内存使用
async function processLargeStream(stream: ReadableStream) {
  let count = 0;
  
  // 使用 decodeAsync 流式处理,避免一次性加载全部数据
  for await (const item of decodeAsync(stream)) {
    await processItem(item);
    count++;
    
    // 定期让 GC 回收
    if (count % 10000 === 0) {
      await new Promise(resolve => setTimeout(resolve, 0));
    }
  }
  
  console.log(`处理了 ${count} 条记录`);
}

⚠️ 注意事项 / Pitfalls

1. undefined 处理

import { encode, decode } from "@msgpack/msgpack";

// undefined 默认被编码为 nil
const data = { a: 1, b: undefined, c: 3 };
const encoded = encode(data);
const decoded = decode(encoded);
console.log(decoded); // { a: 1, b: null, c: 3 }
// undefined 变成了 null!

// 如果要忽略 undefined 字段
const clean = encode(data, { ignoreUndefined: true });
const cleanDecoded = decode(clean);
console.log(cleanDecoded); // { a: 1, c: 3 }

2. 数字精度

import { encode, decode } from "@msgpack/msgpack";

// 超过安全整数范围的数字
const big = 9007199254740993; // Number.MAX_SAFE_INTEGER + 1
const encoded = encode(big);
const decoded = decode(encoded);
console.log(decoded); // 9007199254740992 (精度丢失!)

// 使用 BigInt
const bigInt = 9007199254740993n;
const encoded2 = encode(bigInt, { useBigInt64: true });
const decoded2 = decode(encoded2, { useBigInt64: true });
console.log(decoded2); // 9007199254740993n (正确)

3. 循环引用

import { encode } from "@msgpack/msgpack";

// ❌ 循环引用会导致栈溢出
const obj = {};
obj.self = obj;
// encode(obj); // RangeError: Maximum call stack size exceeded

// ✅ 解决:移除循环引用或使用自定义编码器

4. 原型链上的属性

import { encode, decode } from "@msgpack/msgpack";

class User {
  constructor(public name: string) {}
  greet() { return `Hello, ${this.name}`; }
}

const user = new User("Alice");
const encoded = encode(user);
const decoded = decode(encoded) as any;

console.log(decoded.name);      // "Alice"
console.log(decoded.greet);     // undefined — 方法丢失了!
console.log(decoded instanceof User); // false — 原型链丢失了!

5. Date 对象

import { encode, decode } from "@msgpack/msgpack";

// Date 默认不被特殊处理
const data = { created: new Date() };
const encoded = encode(data);
const decoded = decode(encoded) as any;

console.log(decoded.created); // {} — Date 变成了空对象!

// 必须使用扩展类型或自定义编码器

🔗 扩展阅读 / Further Reading

资源链接
@msgpack/msgpack 文档https://github.com/msgpack/msgpack-javascript
npm 页面https://www.npmjs.com/package/@msgpack/msgpack
TypeScript 类型https://www.typescriptlang.org/docs/handbook/2/types-from-types.html
Web Streams APIhttps://developer.mozilla.org/en-US/docs/Web/API/Streams_API

📝 下一章 / Next: 第 5 章 - Go 实践 / Go Implementation — 在 Go 中使用 MessagePack 进行结构体序列化。