04 - JavaScript 实践 / JavaScript Implementation
JavaScript 实践 / JavaScript Implementation
本章介绍如何在 JavaScript 和 TypeScript 中使用 MessagePack,涵盖浏览器端、Node.js 端以及二进制数据处理。
This chapter covers using MessagePack in JavaScript and TypeScript, including browser, Node.js, and binary data handling.
📖 库概览 / Library Overview
JavaScript 生态中有多个 MessagePack 实现,推荐使用官方的 @msgpack/msgpack:
| 库 / Library | 特点 / Features | 推荐度 |
|---|---|---|
@msgpack/msgpack | 官方推荐,支持流式解码,TypeScript 支持 | ⭐⭐⭐⭐⭐ |
msgpack-lite | 老牌库,稳定但不活跃 | ⭐⭐⭐ |
msgpack5 | Node.js Buffer 优化 | ⭐⭐⭐ |
@msgpack/msgpack (ES Module) | 现代 ESM 支持 | ⭐⭐⭐⭐⭐ |
安装
# npm
npm install @msgpack/msgpack
# yarn
yarn add @msgpack/msgpack
# pnpm
pnpm add @msgpack/msgpack
# 验证安装
node -e "const { encode, decode } = require('@msgpack/msgpack'); console.log('OK')"
💻 基础使用 / Basic Usage
encode / decode
import { encode, decode } from "@msgpack/msgpack";
// ========== 序列化 ==========
const data = {
id: 1001,
name: "Alice",
scores: [95, 87, 92],
active: true,
address: null,
};
const encoded = encode(data);
console.log(encoded); // Uint8Array [133, 162, 105, 100, ...]
console.log(encoded.length); // 43 bytes (vs JSON ~70 bytes)
// ========== 反序列化 ==========
const decoded = decode(encoded);
console.log(decoded);
// { id: 1001, name: "Alice", scores: [95, 87, 92], active: true, address: null }
// 类型保持
console.log(typeof decoded.id); // "number"
console.log(typeof decoded.name); // "string"
console.log(Array.isArray(decoded.scores)); // true
与 JSON 对比
import { encode, decode } from "@msgpack/msgpack";
const user = {
id: 12345,
name: "张三",
email: "[email protected]",
roles: ["admin", "editor"],
metadata: { loginCount: 42, lastLogin: "2024-01-15" },
};
// JSON
const jsonStr = JSON.stringify(user);
const jsonSize = new TextEncoder().encode(jsonStr).length;
// MessagePack
const mpEncoded = encode(user);
const mpSize = mpEncoded.length;
console.log(`JSON: ${jsonSize} bytes`);
console.log(`MessagePack: ${mpSize} bytes`);
console.log(`节省: ${((1 - mpSize / jsonSize) * 100).toFixed(1)}%`);
// JSON: 158 bytes
// MessagePack: 107 bytes
// 节省: 32.3%
💻 类型映射 / Type Mapping
JavaScript → MessagePack
| JavaScript 类型 | MessagePack 类型 | 说明 |
|---|---|---|
null | nil | 空值 |
undefined | nil | 也映射为 nil |
true/false | boolean | 布尔 |
number (整数) | int/uint | 自动选择 |
number (浮点) | float64 | 双精度 |
bigint | int64/uint64 | 大整数 |
string | str | UTF-8 字符串 |
Uint8Array | bin | 二进制数据 |
Array | array | 数组 |
Object | map | 映射 |
Date | ❌ 不支持 | 需自定义 |
Map | ❌ 不支持 | 需自定义 |
Set | ❌ 不支持 | 需自定义 |
MessagePack → JavaScript
| MessagePack 类型 | JavaScript 类型 |
|---|---|
| nil | null |
| boolean | boolean |
| int/uint | number / bigint |
| float32/64 | number |
| str | string |
| bin | Uint8Array |
| array | Array |
| map | Object |
| ext | Extension 类型 |
处理 BigInt
import { encode, decode, useBigInt64 } from "@msgpack/msgpack";
// 默认: 大整数可能丢失精度
const bigNum = 9007199254740993n; // 超过 Number.MAX_SAFE_INTEGER
// 使用 useBigInt64 选项
const encoded = encode({ value: bigNum }, { useBigInt64: true });
const decoded = decode(encoded, { useBigInt64: true });
console.log(decoded.value); // 9007199254740993n (bigint)
console.log(typeof decoded.value); // "bigint"
处理二进制数据
import { encode, decode } from "@msgpack/msgpack";
// Uint8Array 自动编码为 MessagePack bin 类型
const binaryData = new Uint8Array([0xde, 0xad, 0xbe, 0xef]);
const encoded = encode({ payload: binaryData });
const decoded = decode(encoded);
console.log(decoded.payload); // Uint8Array [222, 173, 190, 239]
console.log(decoded.payload instanceof Uint8Array); // true
💻 自定义类型 / Custom Types
ExtensionType 编码器
import { encode, decode, ExtensionCodec } from "@msgpack/msgpack";
// 创建扩展编解码器
const extensionCodec = new ExtensionCodec();
// 注册 Date 类型 (type = 0)
extensionCodec.register({
type: 0,
encode: (object) => {
if (object instanceof Date) {
// 编码为 ISO 字符串
const str = object.toISOString();
return new TextEncoder().encode(str);
}
return null; // 不处理
},
decode: (data) => {
const str = new TextDecoder().decode(data);
return new Date(str);
},
});
// 注册 Set 类型 (type = 1)
extensionCodec.register({
type: 1,
encode: (object) => {
if (object instanceof Set) {
return encode([...object], { extensionCodec });
}
return null;
},
decode: (data) => {
const arr = decode(data, { extensionCodec });
return new Set(arr);
},
});
// 注册 Map 类型 (type = 2)
extensionCodec.register({
type: 2,
encode: (object) => {
if (object instanceof Map) {
return encode(Object.fromEntries(object), { extensionCodec });
}
return null;
},
decode: (data) => {
const obj = decode(data, { extensionCodec });
return new Map(Object.entries(obj));
},
});
// 使用
const data = {
created: new Date("2024-01-15T10:30:00Z"),
tags: new Set(["js", "msgpack", "binary"]),
config: new Map([["debug", true], ["verbose", false]]),
};
const encoded = encode(data, { extensionCodec });
const decoded = decode(encoded, { extensionCodec });
console.log(decoded.created instanceof Date); // true
console.log(decoded.tags instanceof Set); // true
console.log(decoded.config instanceof Map); // true
console.log(decoded.tags.has("msgpack")); // true
简单的 TypedArray 支持
import { encode, decode, ExtensionCodec } from "@msgpack/msgpack";
const extensionCodec = new ExtensionCodec();
// 支持 Float32Array (type = 10)
extensionCodec.register({
type: 10,
encode: (object) => {
if (object instanceof Float32Array) {
return new Uint8Array(object.buffer);
}
return null;
},
decode: (data) => new Float32Array(data.buffer),
});
const vertices = new Float32Array([1.0, 2.0, 3.0, 4.0, 5.0, 6.0]);
const encoded = encode({ vertices }, { extensionCodec });
const decoded = decode(encoded, { extensionCodec });
console.log(decoded.vertices); // Float32Array [1, 2, 3, 4, 5, 6]
💻 浏览器端使用 / Browser Usage
基础用法
<!DOCTYPE html>
<html>
<head>
<title>MessagePack Browser Demo</title>
</head>
<body>
<script type="module">
import { encode, decode } from "https://esm.sh/@msgpack/msgpack";
// 或者通过打包工具引入
// import { encode, decode } from "@msgpack/msgpack";
const data = {
action: "ping",
timestamp: Date.now(),
payload: { message: "Hello from browser" }
};
// 序列化
const encoded = encode(data);
console.log("编码大小:", encoded.length, "bytes");
// 通过 fetch 发送
const response = await fetch("/api/data", {
method: "POST",
headers: { "Content-Type": "application/x-msgpack" },
body: encoded,
});
// 解析响应
const buffer = await response.arrayBuffer();
const result = decode(new Uint8Array(buffer));
console.log("响应:", result);
</script>
</body>
</html>
与 fetch 集成
import { encode, decode } from "@msgpack/msgpack";
// 发送 MessagePack 请求
async function msgpackFetch(url, data) {
const encoded = encode(data);
const response = await fetch(url, {
method: "POST",
headers: {
"Content-Type": "application/x-msgpack",
"Accept": "application/x-msgpack",
},
body: encoded,
});
if (!response.ok) {
throw new Error(`HTTP ${response.status}`);
}
const buffer = await response.arrayBuffer();
return decode(new Uint8Array(buffer));
}
// 使用
const result = await msgpackFetch("/api/users", {
name: "Alice",
email: "[email protected]"
});
console.log(result);
WebSocket 集成
import { encode, decode } from "@msgpack/msgpack";
class MsgPackWebSocket {
constructor(url) {
this.ws = new WebSocket(url);
this.ws.binaryType = "arraybuffer";
this.ws.onmessage = (event) => {
const data = decode(new Uint8Array(event.data));
this.onMessage(data);
};
}
send(data) {
const encoded = encode(data);
this.ws.send(encoded);
}
onMessage(data) {
console.log("收到:", data);
}
}
// 使用
const client = new MsgPackWebSocket("ws://localhost:8080/ws");
client.send({ type: "subscribe", channel: "updates" });
💻 Node.js 端使用 / Node.js Usage
Buffer 与 Uint8Array
import { encode, decode } from "@msgpack/msgpack";
import { Buffer } from "buffer";
// encode 返回 Uint8Array
const encoded = encode({ hello: "world" });
// Node.js 中可转为 Buffer
const buffer = Buffer.from(encoded);
console.log(buffer.toString("hex"));
// 从 Buffer 解码
const decoded = decode(new Uint8Array(buffer));
console.log(decoded);
文件读写
import { encode, decode } from "@msgpack/msgpack";
import fs from "fs";
// 写入文件
const data = { users: [{ name: "Alice" }, { name: "Bob" }] };
const encoded = encode(data);
fs.writeFileSync("data.msgpack", encoded);
// 读取文件
const buffer = fs.readFileSync("data.msgpack");
const decoded = decode(new Uint8Array(buffer));
console.log(decoded);
流式解码
import { decodeAsync } from "@msgpack/msgpack";
import fs from "fs";
async function readMsgPackStream(filePath) {
const stream = fs.createReadStream(filePath);
// decodeAsync 可以从 ReadableStream 中解码
for await (const item of decodeAsync(stream)) {
console.log("解码:", item);
}
}
// 使用
await readMsgPackStream("large-data.msgpack");
Express 中间件
import express from "express";
import { encode, decode } from "@msgpack/msgpack";
const app = express();
// 解析 MessagePack 请求体
app.use("/api", (req, res, next) => {
if (req.headers["content-type"] === "application/x-msgpack") {
const chunks = [];
req.on("data", chunk => chunks.push(chunk));
req.on("end", () => {
const buffer = Buffer.concat(chunks);
req.body = decode(new Uint8Array(buffer));
next();
});
} else {
next();
}
});
// MessagePack 响应
function msgpackRes(res, data) {
const encoded = encode(data);
res.setHeader("Content-Type", "application/x-msgpack");
res.send(Buffer.from(encoded));
}
app.post("/api/users", (req, res) => {
console.log("收到:", req.body);
msgpackRes(res, { status: "ok", id: 1 });
});
app.listen(3000);
💻 TypeScript 类型 / TypeScript Types
基础类型定义
import { encode, decode } from "@msgpack/msgpack";
// MessagePack 支持的类型
type MsgPackValue =
| null
| undefined
| boolean
| number
| bigint
| string
| Uint8Array
| MsgPackValue[]
| { [key: string]: MsgPackValue };
// 使用泛型
interface User {
id: number;
name: string;
email: string;
roles: string[];
}
const user: User = {
id: 1,
name: "Alice",
email: "[email protected]",
roles: ["admin"],
};
// 编码
const encoded: Uint8Array = encode(user);
// 解码并断言类型
const decoded = decode(encoded) as User;
console.log(decoded.name); // 类型安全
泛型辅助函数
import { encode, decode, ExtensionCodec } from "@msgpack/msgpack";
// 类型安全的序列化/反序列化
function serialize<T>(data: T, extensionCodec?: ExtensionCodec): Uint8Array {
return encode(data, { extensionCodec });
}
function deserialize<T>(data: Uint8Array, extensionCodec?: ExtensionCodec): T {
return decode(data, { extensionCodec }) as T;
}
// 使用
interface Config {
debug: boolean;
maxRetries: number;
timeout: number;
}
const config: Config = { debug: true, maxRetries: 3, timeout: 5000 };
const packed = serialize<Config>(config);
const unpacked = deserialize<Config>(packed);
// 类型安全
console.log(unpacked.debug); // boolean
console.log(unpacked.maxRetries); // number
扩展类型的 TypeScript 支持
import { encode, decode, ExtensionCodec } from "@msgpack/msgpack";
// 定义扩展类型接口
interface DateExt {
__ext_type: 0;
value: string;
}
interface SetExt {
__ext_type: 1;
value: unknown[];
}
// 创建类型安全的扩展编解码器
const extensionCodec = new ExtensionCodec();
extensionCodec.register({
type: 0,
encode: (object: unknown): Uint8Array | null => {
if (object instanceof Date) {
return encode(object.toISOString());
}
return null;
},
decode: (data: Uint8Array): Date => {
return new Date(decode(data) as string);
},
});
// 使用
interface DataPoint {
timestamp: Date;
value: number;
}
const point: DataPoint = { timestamp: new Date(), value: 42.5 };
const encoded = encode(point, { extensionCodec });
const decoded = decode(encoded, { extensionCodec }) as DataPoint;
console.log(decoded.timestamp instanceof Date); // true
模块声明
// types/msgpack.d.ts
declare module "@msgpack/msgpack" {
export function encode(
data: unknown,
options?: EncodeOptions
): Uint8Array;
export function decode(
buffer: ArrayLike<number> | BufferSource,
options?: DecodeOptions
): unknown;
export function decodeAsync(
stream: ReadableStream<Uint8Array>,
options?: DecodeOptions
): AsyncIterable<unknown>;
export interface EncodeOptions {
extensionCodec?: ExtensionCodec;
maxDepth?: number;
initialBufferSize?: number;
sortKeys?: boolean;
forceFloat32?: boolean;
forceIntegerToFloat?: boolean;
useBigInt64?: boolean;
ignoreUndefined?: boolean;
}
export interface DecodeOptions {
extensionCodec?: ExtensionCodec;
maxStrLength?: number;
maxBinLength?: number;
maxArrayLength?: number;
maxMapLength?: number;
maxExtLength?: number;
useBigInt64?: boolean;
}
export class ExtensionCodec {
register(options: {
type: number;
encode: (input: unknown) => Uint8Array | null;
decode: (data: Uint8Array) => unknown;
}): void;
}
}
💻 性能优化 / Performance Optimization
选择合适的编码选项
import { encode, decode } from "@msgpack/msgpack";
// 1. 紧凑模式:强制 float32(节省空间)
const compact = encode(3.14, { forceFloat32: true });
// 2. 忽略 undefined 字段
const data = { a: 1, b: undefined, c: 3 };
const clean = encode(data, { ignoreUndefined: true });
// 3. 排序键(确定性输出)
const stable = encode({ b: 2, a: 1 }, { sortKeys: true });
批量处理
import { encode, decode } from "@msgpack/msgpack";
// 高效批量处理
function batchProcess<T>(items: T[]): Uint8Array {
// 一次编码整个数组,而不是逐个编码
return encode(items);
}
// 反向:批量解码
function batchDecode<T>(data: Uint8Array): T[] {
return decode(data) as T[];
}
// 测试
const items = Array.from({ length: 10000 }, (_, i) => ({
id: i,
value: `item_${i}`,
}));
const packed = batchProcess(items);
console.log(`10000 条记录编码为 ${packed.length} bytes`);
const unpacked = batchDecode<typeof items[0]>(packed);
console.log(`解码回 ${unpacked.length} 条记录`);
内存管理
// 大数据时注意内存使用
async function processLargeStream(stream: ReadableStream) {
let count = 0;
// 使用 decodeAsync 流式处理,避免一次性加载全部数据
for await (const item of decodeAsync(stream)) {
await processItem(item);
count++;
// 定期让 GC 回收
if (count % 10000 === 0) {
await new Promise(resolve => setTimeout(resolve, 0));
}
}
console.log(`处理了 ${count} 条记录`);
}
⚠️ 注意事项 / Pitfalls
1. undefined 处理
import { encode, decode } from "@msgpack/msgpack";
// undefined 默认被编码为 nil
const data = { a: 1, b: undefined, c: 3 };
const encoded = encode(data);
const decoded = decode(encoded);
console.log(decoded); // { a: 1, b: null, c: 3 }
// undefined 变成了 null!
// 如果要忽略 undefined 字段
const clean = encode(data, { ignoreUndefined: true });
const cleanDecoded = decode(clean);
console.log(cleanDecoded); // { a: 1, c: 3 }
2. 数字精度
import { encode, decode } from "@msgpack/msgpack";
// 超过安全整数范围的数字
const big = 9007199254740993; // Number.MAX_SAFE_INTEGER + 1
const encoded = encode(big);
const decoded = decode(encoded);
console.log(decoded); // 9007199254740992 (精度丢失!)
// 使用 BigInt
const bigInt = 9007199254740993n;
const encoded2 = encode(bigInt, { useBigInt64: true });
const decoded2 = decode(encoded2, { useBigInt64: true });
console.log(decoded2); // 9007199254740993n (正确)
3. 循环引用
import { encode } from "@msgpack/msgpack";
// ❌ 循环引用会导致栈溢出
const obj = {};
obj.self = obj;
// encode(obj); // RangeError: Maximum call stack size exceeded
// ✅ 解决:移除循环引用或使用自定义编码器
4. 原型链上的属性
import { encode, decode } from "@msgpack/msgpack";
class User {
constructor(public name: string) {}
greet() { return `Hello, ${this.name}`; }
}
const user = new User("Alice");
const encoded = encode(user);
const decoded = decode(encoded) as any;
console.log(decoded.name); // "Alice"
console.log(decoded.greet); // undefined — 方法丢失了!
console.log(decoded instanceof User); // false — 原型链丢失了!
5. Date 对象
import { encode, decode } from "@msgpack/msgpack";
// Date 默认不被特殊处理
const data = { created: new Date() };
const encoded = encode(data);
const decoded = decode(encoded) as any;
console.log(decoded.created); // {} — Date 变成了空对象!
// 必须使用扩展类型或自定义编码器
🔗 扩展阅读 / Further Reading
| 资源 | 链接 |
|---|---|
| @msgpack/msgpack 文档 | https://github.com/msgpack/msgpack-javascript |
| npm 页面 | https://www.npmjs.com/package/@msgpack/msgpack |
| TypeScript 类型 | https://www.typescriptlang.org/docs/handbook/2/types-from-types.html |
| Web Streams API | https://developer.mozilla.org/en-US/docs/Web/API/Streams_API |
📝 下一章 / Next: 第 5 章 - Go 实践 / Go Implementation — 在 Go 中使用 MessagePack 进行结构体序列化。