北野

那年江湖，遇见你

Building a message queue with only two UNIX signals

Have you ever asked yourself what if we could replace any message broker with a very simple one using only two UNIX signals? Well, I’m not surprised if you didn’t. But I did. And I want to share my journey of how I achieved it.

If you want to learn about UNIX signals, binary operations the easy way, how a message broker works under the hood, and a bit of Ruby, this post is for you.

And if you came here just because of the clickbait title, I apologize and invite you to keep reading. It’ll be fun, I promise.

image

It’s all about UNIX

A few days ago, I saw some discussion on the internet about how we could send messages between processes. Many people think of sockets, which are the most common way to send messages, even allowing communication across different machines and networks. Some don’t even realize that pipes are another way to send messages between processes:

$ echo 'hello' | base64
aGVsbG8K

Kitex TTHeader协议

参考 Thrift THeader 协议 ，Kitex设计了 TTheader 协议。

协议编码

 0 1 2 3 4 5 6 7 8 9 a b c d e f 0 1 2 3 4 5 6 7 8 9 a b c d e f
+----------------------------------------------------------------+
| 0|                          LENGTH                             |
+----------------------------------------------------------------+
| 0|       HEADER MAGIC          |            FLAGS              |
+----------------------------------------------------------------+
|                         SEQUENCE NUMBER                        |
+----------------------------------------------------------------+
| 0|     HEADER SIZE             | ...
+---------------------------------

                  Header is of variable size:
                   (and starts at offset 14)

+----------------------------------------------------------------+
| PROTOCOL ID  |NUM TRANSFORMS . |TRANSFORM 0 ID (uint8)|
+----------------------------------------------------------------+
|  TRANSFORM 0 DATA ...
+----------------------------------------------------------------+
|         ...                              ...                   |
+----------------------------------------------------------------+
| INFO 0 ID (uint8)|       INFO 0  DATA ...
+----------------------------------------------------------------+
|         ...                              ...                   |
+----------------------------------------------------------------+
|                                                                |
|                              PAYLOAD                           |
|                                                                |
+----------------------------------------------------------------+

Thrift THeader协议

Header format for the THeader.h

      0 1 2 3 4 5 6 7 8 9 a b c d e f 0 1 2 3 4 5 6 7 8 9 a b c d e f
    +----------------------------------------------------------------+
    | 0|                          LENGTH                             |
    +----------------------------------------------------------------+
    | 0|       HEADER MAGIC          |            FLAGS              |
    +----------------------------------------------------------------+
    |                         SEQUENCE NUMBER                        |
    +----------------------------------------------------------------+
    | 0|     Header Size(/32)        | ...
    +---------------------------------

                      Header is of variable size:
                       (and starts at offset 14)

    +----------------------------------------------------------------+
    |         PROTOCOL ID  (varint)  |   NUM TRANSFORMS (varint)     |
    +----------------------------------------------------------------+
    |      TRANSFORM 0 ID (varint)   |        TRANSFORM 0 DATA ...
    +----------------------------------------------------------------+
    |         ...                              ...                   |
    +----------------------------------------------------------------+
    |        INFO 0 ID (varint)      |       INFO 0  DATA ...
    +----------------------------------------------------------------+
    |         ...                              ...                   |
    +----------------------------------------------------------------+
    |                                                                |
    |                              PAYLOAD                           |
    |                                                                |
    +----------------------------------------------------------------+

The LENGTH field is 32 bits, and counts the remaining bytes in the packet, NOT including the length field. The header size field is 16 bits, and defines the size of the header remaining NOT including the HEADER MAGIC, FLAGS, SEQUENCE NUMBER and header size fields. The Header size field is in bytes/4.

妙用Go类型定义避免循环引用

类型定义（Type Definition）

type MyInt int
  • 创建一个新类型，与底层类型（underlying type）不同。
  • 新类型拥有独立的类型集，不会继承底层类型的方法。
  • 需要进行显式类型转换才能与底层类型互用。

巧妙使用类型定义来避免对象实例方法循环调用

// file: github.com/powerman/rpc-codec/blob/master/jsonrpc2/client.go

type clientResponse struct {
	Version string           `json:"jsonrpc"`
	ID      *uint64          `json:"id"`
	Result  *json.RawMessage `json:"result,omitempty"`
	Error   *Error           `json:"error,omitempty"`
}

func (r *clientResponse) reset() {
	r.Version = ""
	r.ID = nil
	r.Result = nil
	r.Error = nil
}

func (r *clientResponse) UnmarshalJSON(raw []byte) error {
	r.reset()
	type resp *clientResponse // 非常关键的一个动作，要身不要心
	if err := json.Unmarshal(raw, resp(r)); err != nil {
		return errors.New("bad response: " + string(raw))
	}

	var o = make(map[string]*json.RawMessage)
	if err := json.Unmarshal(raw, &o); err != nil {
		return errors.New("bad response: " + string(raw))
	}

	// ...other logic...

	return nil
}

"兄弟"

李奔腾： “我现在…自身难保”

李奔腾： “跟我一起创业的老兄弟不会放过我的”

李奔腾： “我有可能…”

李奔腾： “要被迫放弃麒麟”

李奔腾： “不过也挺好的，一无所有也是一个新的开始 ”

罗 维： “所以…对于你来说，到底是生意更重要，还是兄弟更重要？”

李奔腾： “ 不做一个梦，就不算兄弟 ”

李奔腾： “生意，一点都不重要”

……

git-sync同步所有git库

同步指定目录中所有git库的一个简单shell脚本。

$ git sync -maxdepth 3
> Found 89 repositories.

> Synching gin 1% (1/89)
Already up to date.
> Synching gin finish.

> Synching yj 2% (2/89)
Already up to date.
> Synching yj finish.

> Synching go-cmp 3% (3/89)
Already up to date.
> Synching go-cmp finish.

> Synching opentelemetry-go-contrib 4% (4/89)
remote: Enumerating objects: 775, done.
remote: Counting objects: 100% (363/363), done.
remote: Compressing objects: 100% (117/117), done.
remote: Total 775 (delta 264), reused 307 (delta 240), pack-reused 412 (from 3)
Receiving objects: 100% (775/775), 351.23 KiB | 636.00 KiB/s, done.
Resolving deltas: 100% (448/448), completed with 79 local objects.

TL在Go如何实现oneof语义

TL的Type是一个总类型Class包含不同的结构子类型实例Object，类似于Protubuf中Message与Oneof的关系。

Telegram的RPC TL片段1：

peerUser#9db1bc6d user_id:int = Peer;
peerChat#bad0e5bb chat_id:int = Peer;
peerChannel#bddde532 channel_id:int = Peer;

folderPeer#e9baa668 peer:Peer folder_id:int = FolderPeer;

在go库包中包含main包的方式 

// file: golang.org/x/tools/go/cfg/cfg.go
package cfg

import (
	"bytes"
	"fmt"
	"go/ast"
	"go/format"
	"go/token"
)

// A CFG represents the control-flow graph of a single function.
//
// The entry point is Blocks[0]; there may be multiple return blocks.
type CFG struct {
	Blocks []*Block // block[0] is entry; order otherwise undefined
	noreturn bool // function body lacks a reachable return statement
}
// ...