I wanted to run an experiment: can you ship a publishable desktop app by relying purely on vibe coding in a tech stack you’ve never touched?

The key phrase here is “never touched.” I don’t know C#, I don’t know Swift (the original codebase), I don’t know XAML, and I’ve never built a Windows desktop app. I wanted to strip away all my domain expertise and experience vibe coding — AI agent-driven development — in its purest form.

The result: 24 days, from zero to Windows Store listing, 19 translation services, 715 test cases.

Tools

Only two AI tools were used throughout the entire project:

• Claude Code — 100% of the coding work (writing code, research, debugging, refactoring, testing, documentation)
• GitHub Copilot Review — PR-level code review

No Cursor, no Windsurf, no ChatGPT. The editor was VS Code, but I barely touched the code manually.

The split was roughly 90% AI : 10% human. My 10% went to directional decisions (WinUI 3 vs. Tauri, feature prioritization), having Claude Code explain code and approaches then picking the reasonable one, final review, and providing hints when the AI got stuck.

Initially, I ran Claude Code on WSL. The experience was quite fragmented: the AI would modify code in WSL, then I’d sync it to a Windows directory and compile in VS Code. Switching between the two environments constantly was inefficient. Later, I installed Claude Code directly on Windows, and the entire workflow became much smoother — edit, compile, and run in one place. This was an important factor in the productivity gains that followed.

Origin and the Decision

Easydict is a popular translation dictionary app on macOS. I wanted to port it to Windows.

The initial idea was simple: keep the Swift backend code and only build a Windows UI frontend. I used vibe coding to evaluate several approaches and narrowed it down to two, developed in parallel: WinUI 3 (Microsoft’s official Windows desktop UI framework) and Tauri (a cross-platform framework based on Rust + Web technologies).

After pushing both approaches forward, a common problem quickly emerged: the backend code also had to be fully ported. Keeping the Swift backend was impractical — either the dependency chain became extremely complex, or the results on Windows were poor.

Then the Tauri approach hit a wall first: the build environment was too complex, and I could never get a POC sample running. Meanwhile, WinUI 3 had already produced a working demo. So I dropped Tauri and went all-in on WinUI 3.

For the backend, I chose .NET 8 (an LTS version), which felt very smooth on Windows. The whole C# + WinUI 3 + .NET 8 stack is truly a first-class citizen in the Windows ecosystem.

During the POC phase, I created a subdirectory directly inside the Easydict macOS source repository¹. Once the POC was running and the approach was validated, I created a completely independent project, detaching from the original source repository.

Prototyping and Building Features

I gave the AI agent almost full control over the UI design. My only requirements were: reuse the original Easydict icon assets where possible, keep the interface clean and minimal, and support Dark/Light themes. The result was satisfactory — what you see today. As someone who knows absolutely nothing about XAML, the outcome exceeded my expectations.

After the prototype was running, I started stacking features one by one. The key strategy at this stage: commit each feature individually with Git. One commit per feature, preventing later changes from breaking earlier ones. This is basic instinct for a backend developer, but it’s especially important in vibe coding — the AI sometimes accidentally breaks existing functionality when adding new features.

Once two or three features were working properly, I started adding test coverage. On 2026-01-20, I added the first unit test framework (xUnit + FluentAssertions).

Parallel PR Development

After pushing the code to GitHub, I started using a PR workflow to develop features in parallel. That’s when I discovered an unexpectedly good combination: GitHub Copilot Review + Claude Code work remarkably well together.

My development loop looked like this:

Claude Code writes code and submits PR
    → GitHub Copilot Review finds issues, posts review comments
        → Claude Code reads review comments and resolves them
            → Push
                → Another round of review
                    → Loop until approved

Code quality was generally well-maintained through this loop. And because multiple PRs were running in parallel, while waiting for Copilot review or Claude Code to resolve comments, I’d switch to the next feature’s PR. Very efficient.

Not much domain knowledge was needed throughout this process. I had zero knowledge of C# and the original Swift code. I’d have Claude Code explain things to me and pick the approach that sounded reasonable. Occasionally I’d apply general software engineering intuition — “this design is too tightly coupled” or “this should have an interface abstraction.”

Release Journey

v0.1.0 — First Release (Day 5)

5 days after the project started, the first usable version was released. On the same day, I set up CI/CD (GitHub Actions), merged the first PR (#1), and had a basic automated build and test pipeline running. This pace would be hard to imagine in traditional development — I still didn’t know C#’s basic syntax.

v0.2.0 — The Turning Point (Day 12)

12 days after the project started, I submitted release builds to two distribution channels simultaneously: Windows Store (MSIX format) and WinGet (Portable ZIP format).

This version was a major turning point. It meant the app was on a real trajectory — end users could install it through normal channels, and I could start collecting real user feedback instead of building a hardcore tool that required developers to compile locally.

But the release process was far from smooth. This version went through 17 Release Candidates (RCs), from rc1 to rc17, spanning 2 days. The main pain points were in MSIX manifest configuration: MinVersion requirements, architecture support configuration², icon asset specification requirements, and Store submission compliance checks. These issues surfaced one after another, and each small fix required repackaging, testing, and resubmitting — hence the 17 RCs.

The Blurry MSIX Icon Saga

After packaging as MSIX, a stubborn issue appeared: the Start Menu icon was extremely blurry. Oddly, the .zip portable version had no such problem — only the MSIX-installed version was affected. It took multiple version iterations to resolve. The root cause was that MSIX packages have special size and format requirements for icon assets, different from how regular .exe icons are handled.

Testing and Performance

To improve stability, a multi-layer testing system was gradually established: unit tests (Day 4), integration tests (Day 12), UI automation screenshots (Day 14), and UI regression tests (Day 15). With these in place, my local verification workload dropped significantly — I could just look at automated screenshots in CI to determine whether a change caused visual regression, instead of manually running the app and checking each feature.

In the later stages, I started focusing on performance. I added performance regression tests to monitor the latency of hot paths like mouse-select-to-translate and application startup. However, the test framework can’t yet intuitively prove that a given change hasn’t impacted performance — benchmark stability in CI environments has too much variance. This needs more research.

Where AI Exceeded Expectations

Two aspects of the AI’s performance clearly exceeded my expectations.

The first was architecture design. The overall project structure proposed by the AI was remarkably sound: the solution was split into Easydict.WinUI (UI layer), Easydict.TranslationService (business logic layer), and Easydict.SidecarClient (IPC layer); the streaming translation architecture used SSE + IAsyncEnumerable<string>, the idiomatic C# approach for streaming data³; the Translation Service inheritance hierarchy was cleanly designed: ITranslationService → BaseTranslationService → BaseOpenAIService, where adding a new OpenAI-compatible service only requires extending BaseOpenAIService and providing Endpoint/ApiKey/Model. Even if I knew C#, these architectural decisions would probably have taken me several refactoring cycles to reach this level of design quality.

The second was debugging ability. When bugs appeared, the AI could locate and fix issues on its own with a surprisingly high success rate. This was especially true for WinUI 3-specific pitfalls (thread dispatching, COM object lifecycles) — issues that would have cost me significant time searching through documentation and Stack Overflow.

Toolchain Pain Points

Beyond the code itself, the toolchain caused more pain than the code.

The entire MSIX packaging, signing, and Store submission pipeline was extremely complex. Every configuration item in the manifest could potentially cause a packaging failure or Store review rejection. Most of the effort behind v0.2.0’s 17 RCs was spent here.

The GitHub Actions Windows runner environment also had many subtle differences from the local development environment: different Windows App SDK versions and installation states, WinUI tests requiring a graphical environment that mostly fails on headless CI, and MSIX packaging on CI requiring additional toolchain configuration. The AI could help resolve these issues, but it often took multiple attempts. The AI’s solutions are based on documentation and general knowledge, while the actual CI environment is a black box — you can only debug through the push → check logs → fix → push again cycle.

Conclusions and Advice

What is vibe coding best for? Exploring unfamiliar technology domains and rapidly prototyping. This is vibe coding’s greatest value today. Going from zero to Store listing in 24 days — without AI, given my zero-knowledge baseline in C#/WinUI, this timeline would have been at least 5-10x longer.

But foundational engineering skills are still essential. Even though 90% of the code was written by AI, the remaining 10% of human intervention was still critical. That 10% wasn’t about writing code — it was Git workflow (per-feature commits, parallel PRs, branch management), CI/CD mindset (establishing automated pipelines early), testing strategy (knowing when to add tests and at what level), release strategy (shipping to users early to gather feedback), and architectural judgment (choosing the more reasonable approach among the AI’s proposals). These are all universal software engineering skills, independent of any specific language or framework. Vibe coding lowers the barrier of the tech stack, but it doesn’t lower the barrier of engineering discipline.

A few pieces of advice for developers wanting to try vibe coding:

1. Pick a domain you don’t know but want to learn — vibe coding’s greatest value isn’t writing code you already know how to write, but helping you do things you can’t
2. Establish Git + CI + tests early — this is your only safety net. AI makes mistakes, and those mistakes are often hard to spot
3. Use PR review tools as quality gates — AI-written code needs another AI to review it; humans make the final call
4. Ship early — don’t wait for “perfect.” Deliver a minimum viable version first
5. Maintain your sense of direction — AI can do 90% of the execution, but only you can make that 10% of directional decisions

Project Data

Key Metrics

Code by Module

Milestone Timeline

2026-01-16  Day 0   Project started, first commit
2026-01-20  Day 4   Unit test framework added (xUnit + FluentAssertions)
2026-01-21  Day 5   v0.1.0 released + CI/CD + first PR (#1)
2026-01-25  Day 9   MSIX packaging + WinGet release workflow
2026-01-26  Day 10  Windows Store submission begins (17 RC iterations)
2026-01-28  Day 12  v0.2.0 released — turning point (Store + WinGet dual-channel)
2026-01-30  Day 14  UI automation tests + screenshot verification
2026-02-01  Day 16  v0.2.2 + UI regression tests
2026-02-02  Day 17  v0.2.3 + mouse-select-to-translate + Copilot Review Loop
2026-02-03  Day 18  Performance benchmarks
2026-02-04  Day 19  v0.3.0 (Inno Setup EXE installer + GitHub Pages website)
2026-02-06  Day 21  v0.3.1 / v0.3.2
2026-02-08  Day 23  v0.3.3 / v0.3.4

19 Translation Services

Google Translate, Google Web, Bing, DeepL, OpenAI, DeepSeek, Gemini, Groq, Zhipu AI, GitHub Models, Doubao, Volcano, Caiyun, NiuTrans, Linguee, Youdao, Ollama, Built-in AI, Custom OpenAI

Containerized applications often spend a lot of time downloading packages during build (sometimes encountering network issues, requiring multiple builds for one version to succeed), yet most dependency packages rarely change. So I had this optimization idea for building images:

Use a minimal container to pre-download required packages locally, then build these packages into local repositories. In containers that need to be built, replace the software sources with local repositories to save container build time (by orders of magnitude).

Another reason I’m writing this blog post: although everything used here is existing tools and software, besides their own manuals and --help, other helpful documentation is really scattered. And from my searching, these existing tools always have small “pitfalls” not mentioned in documentation, or even rarely encountered on sites like Stack Overflow - solving these “pitfalls” is what takes the most time.

TL;DR

I’ll later open source part of this on GitHub, placed here (TODO)

Currently tested compatible distributions:

• centos 6 / 7 / 8
• fedora 31 / 32 / 33
• amazonlinux 1 / 2
• ubuntu trusty (14.04) / xenial (16.04) / bionic (18.04) / focal (20.04)
• debian jessie (8) / stretch (9) / buster (10)
• opensuse leap 15

Process

1. Based on a minimal container for the distribution, add some software sources that will be needed.
2. For different distributions, use the corresponding package management tool to download all packages in the required package list plus their dependencies
3. Place these packages in corresponding directories by distribution, use repository creation commands in the container to build local repositories
4. Use a simple static web server, listening on a local port. This sets up a local http software repository
5. Add the local software source to the Dockerfile of containers that need frequent rebuilds. Note that local software repositories generally don’t have signature verification or https, so you need to manually add trust.

Here I’ll only explain the more tedious steps

0x02. Package Download

yum / dnf

$ cd /path/to/dir \
    && yumdownloader --resolve pkg-1 pkg-2 ...

• yumdownloader is preferred here. The previous approach tried dnf install --downloadonly, and found quite a few unknown pitfalls. One of them is that after downloading, packages already downloaded locally occasionally get deleted - feels like dnf / yum has some storage optimization strategies.
• The --resolve option tells yumdownloader to download dependencies for the specified packages
• --installroot Not recommended to use this option to specify download path. After using this option, macros (variables) in software source config files won’t be automatically resolved. For example, the common $releasever variable would need extra manual specification.
• yumdownloader downloads packages directly to the working directory, just use cd to switch the working directory first

apt-get

$ cd /path/to/dir \
    && apt-get download \
    $(apt-cache depends --recurse --no-recommends --no-suggests \
        --no-conflicts --no-breaks --no-replaces --no-enhances \
        pkg-1 pkg-2 ... | grep "^\w")

• If using apt-get install --donwload-only --reinstall to download packages, dependency packages that already exist in the current container won’t be downloaded again.

For example, if the downloader-container already has ca-certificates and openssl packages installed, when executing the following command, the result is: due to the --reinstall option ca-certificates will be downloaded, but openssl as a dependency of ca-certificates will be ignored.

$ apt-get download \
        $(apt-cache depends --recurse --no-recommends --no-suggests \
        --no-conflicts --no-breaks --no-replaces --no-enhances \
        ca-certificates | grep "^\w")

• Here we use apt-get download rather than apt-get --install --donwload-only, mainly because in the subcommand apt-cache depends, the queried dependencies will have preferred and alternative choices, and these two are often conflicting. Even if apt-get install uses --donwload-only, it will cause package download failure because conflicts can’t be resolved.

Below is an example of apt-cache depends output, where pinentry-curses is the more preferred choice over <pinentry:i386>. Detailed explanation can be found at https://www.thecodeship.com/gnu-linux/understanding-apt-cache-depends-output/

$ apt-cache depends --recurse --no-recommends \
    --no-suggests --no-conflicts --no-breaks \
    --no-replaces --no-enhances --no-pre-depends \
    gnupg2 | grep -E '^gnupg-agent:i386' -A10

gnupg-agent:i386
 |Depends: pinentry-curses:i386
  Depends: <pinentry:i386>
    mew-beta-bin:i386
    mew-bin:i386
    pinentry-curses:i386
    pinentry-gnome3:i386
    pinentry-gtk2:i386
    pinentry-qt:i386
    pinentry-tty:i386
  Depends: libassuan0:i386

• apt-get download also downloads packages directly to the current directory, so just use the cd command to switch working directory first

zypper

$ zypper --no-gpg-checks --non-interactive \
    --pkg-cache-dir /path/to/dir \
    install -y -f --download-only \
    pkg-1 pkg-2 ...

• --non-interactive is mainly for scripts, preventing zypper from waiting for user input until timeout
• --pkg-cache-dir specifies the download directory
• -f forces downloading already installed packages. This actually encounters the same problem as apt-get install --download-only, where dependency packages won’t be downloaded if already installed. Currently I write it this way, adding missing base packages manually.
• For zypper, distinguish between global arguments and subcommand arguments. Specifically for this command, before install are global arguments, and after are subcommand arguments

0x03. Directory Structure

yum

The yum repository directory structure is as follows:

base/
├── amazonlinux-1
│   └── x86_64
|       ├── audit-libs-2.6.5-3.28.amzn2.i686.rpm
|       ├── ...
│       └── repodata
...

Note: yum repository structure is relatively simple. Under distribution subdirectory -> CPU architecture directory, store downloaded rpm packages, then create local repository index in the same directory.

Command to create yum repository index:

cd /path/to/dir \
    && createrepo --update ./

There’s also a C version createrepo_c which is faster with the same usage. Recommended for newer distributions, like centos 8 / fedora 31+ / amazonlinux

cd /path/to/dir \
    && createrepo_c --update ./

Newer distributions have some packages built with modularity¹. If you want to build local repositories for these packages, extra commands are needed:

Documentation at: https://docs.fedoraproject.org/en-US/modularity/hosting-modules/

cd /path/to/dir \
    && createrepo_c --update ./ \
    && repo2module -s stable -n REPO_NAME -d ./ ./repodata/modules \
    && modifyrepo_c --mdtype=modules ./repodata/modules.yaml ./repodata

Where REPO_NAME is the local repository name

A noteworthy command here is repo2module (from https://github.com/rpm-software-management/modulemd-tools), because the above documentation doesn’t mention how to generate the modules.yaml file.

fedora or centos 8 (needs additional epel repository) can install the repo2module command via dnf install -y python3-gobject modulemd-tools

apt

The apt repository directory structure is as follows:

ubuntu/
├── dists
│   ├── bionic
│   │   └── base
│   │       └── main
│   │           └── binary-amd64
|  ...
└── pool
    ├── bionic
    │   └── base
    │       └── main
    │           └── binary-amd64
   ...

Note: apt repository has two subdirectories dists/ and pool/. dists/ subdirectory stores indexes, pool/ subdirectory stores packages.

Command to create apt repository index:

Here the local repository won’t use gpg signed Release, full command at: https://medium.com/sqooba/create-your-own-custom-and-authenticated-apt-repository-1e4a4cf0b864#35dd

cd /path/to/dir

apt-ftparchive --arch amd64 packages \
    pool/bionic/base/main/binary-amd64 \
    > dists/base/main/binary-amd64/Packages

gzip -k -c \
    -f dists/base/main/binary-amd64/Packages \
    > dists/base/main/binary-amd64/Packages.gz

apt-ftparchive release dists/bionic/base > dists/bionic/Release

Where base is a custom repository subdirectory, convenient for future expansion. The apt-ftparchive command can be installed via apt-get install -y dpkg-dev.

0x05. Adding Local Repository

The host.docker.internal below is a hostname added via docker build’s --add-host, 4891 is the port openresty listens on locally

yum

printf "[local-base]\n\
name=Local Base Repo\n\
baseurl=http://host.docker.internal:4891/base/centos-7/x86_64/\n\
skip_if_unavailable=True\n\
gpgcheck=0\n\
repo_gpgcheck=0\n\
enabled=1\n\
enabled_metadata=1" > /etc/yum.repos.d/local-base.repo

zypper

printf "[local-base]\n\
name=Local Base Repo\n\
baseurl=http://host.docker.internal:4891/base/sles-12/x86_64/\n\
skip_if_unavailable=True\n\
gpgcheck=0\n\
repo_gpgcheck=0\n\
enabled=1\n\
enabled_metadata=1" > /root/local-base.repo \
    && zypper -n ar --check --refresh -G file:///root/local-base.repo \
    && zypper -n mr --gpgcheck-allow-unsigned-repo local-base \
    && zypper -n mr --gpgcheck-allow-unsigned-package local-base \
    && rm -f /root/local-base.repo

apt

echo "deb [trusted=yes] http://host.docker.internal:4891/ubuntu bionic/base main" > /etc/apt/sources.list

The affected kernel versions were:

• 3.10.0-123
• 3.10.0-229

TL;DR

This is actually a compatibility bug between AMD Family-17h architecture CPUs and older kernel versions. The bug has been fixed in newer kernel versions:

https://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git/commit/?id=e40ed1542dd779e5037a22c6b534e57127472365

Troubleshooting Process

LVM?

After the OS in KVM failed to boot, I connected to the machine via VNC and found this error:

dracut-initqueue timeout and could not boot – warning /dev/centos/root-lv does not exist

I’ve seen this error many times, and searching for keywords brings up numerous solutions. Most cases are caused by UUID mismatches, but since I was using LVM, I suspected the kernel wasn’t recognizing the LVM partition during boot, preventing system startup.

I then tried using standard partitions instead of LVM and reinstalled the system. The error persisted - still couldn’t boot.

XFS/EXT4/EXT2?

At this point, I began to suspect the boot partition filesystem. The reinstalled system was using XFS, and I recalled that very old kernel versions only support ext2/ext3/ext4 filesystems. So I reinstalled the OS twice more, using ext4 and ext2 for the /boot partition respectively. Still no luck.

GRUB2?

Then I started suspecting GRUB 2 parameters. After switching to standard partitions, the boot disk specification became linux16 ... root=UUID=xxxx. Could this root=UUID=xxx format also be unsupported by older kernel versions? So I added GRUB_DISABLE_LINUX_UUID=true to /etc/default/grub, then ran:

# Note: KVM wasn't using EFI boot
grub2-mkconfig -o /boot/grub2/grub.cfg

After updating, I tried rebooting again…

Still didn’t work.

VIRTIO?

I started suspecting the kernel couldn’t find the device at all, so I began investigating KVM’s virtio. Virtio is the disk bus provided by KVM, which should require corresponding drivers. I ran the lsinitrd command to check for the drivers:

# sudo lsinitrd /boot/initramfs-3.10.0-229.7.2.el7.x86_64.img | grep virtio

-rw-r--r--   1 root     root        27885 Jun 24  2015 usr/lib/modules/3.10.0-229.7.2.el7.x86_64/kernel/drivers/block/virtio_blk.ko
-rw-r--r--   1 root     root        52861 Jun 24  2015 usr/lib/modules/3.10.0-229.7.2.el7.x86_64/kernel/drivers/char/virtio_console.ko
-rw-r--r--   1 root     root        50501 Jun 24  2015 usr/lib/modules/3.10.0-229.7.2.el7.x86_64/kernel/drivers/net/virtio_net.ko
-rw-r--r--   1 root     root        29125 Jun 24  2015 usr/lib/modules/3.10.0-229.7.2.el7.x86_64/kernel/drivers/scsi/virtio_scsi.ko
drwxr-xr-x   2 root     root            0 Nov 13 16:19 usr/lib/modules/3.10.0-229.7.2.el7.x86_64/kernel/drivers/virtio
-rw-r--r--   1 root     root        15797 Jun 24  2015 usr/lib/modules/3.10.0-229.7.2.el7.x86_64/kernel/drivers/virtio/virtio.ko
-rw-r--r--   1 root     root        21253 Jun 24  2015 usr/lib/modules/3.10.0-229.7.2.el7.x86_64/kernel/drivers/virtio/virtio_pci.ko
-rw-r--r--   1 root     root        25541 Jun 24  2015 usr/lib/modules/3.10.0-229.7.2.el7.x86_64/kernel/drivers/virtio/virtio_ring.ko

Result: drivers were present, no difference from newer kernels that boot successfully.

AMD Family-17h!

There was still one clue - a message that flashed by quickly during boot (I had to take screenshots after many reboots to finally read it): core perfctr but no constraints; unknown hardware!

Googling this keyword, everything became clear: this is a compatibility issue between AMD Family-17h and older kernel versions. Here’s the explanation:

In family-17h, there is no PMC-event constraint. All events, irrespective of the type, can be measured using any of the six generic performance counters.

AMD Family-17h corresponds to the Zen/Zen+/Zen2 architecture CPUs, which matches the host machine running this VM. This also explains why the same kernel version worked fine on Intel platforms.

The solution is simply to upgrade the kernel - this fix has been applied by major operating systems years ago.

Overall I’m very satisfied with it. Although I did encounter some issues during use, I basically solved them with Google.

At the time I also planned to write a related troubleshooting tutorial, but unfortunately I kept putting it off. This article basically covers most issues you’ll encounter: https://mechanus.io/ke-neng-shi-zui-hao-de-yubikey-gpg-ssh-zhi-neng-qia-jiao-cheng/

This article mainly discusses the pitfalls of getting Yubikey and polkit tools to work together:

Reproduction

Environment: fedora 30 / 31 (encountered on two virtual machines)

After inserting the Yubikey, running gpg2 --card-status as a non-root user shows no device found, with the prompt:

gpg: selecting card failed: No such device
gpg: OpenPGP card not available: No such device

Running journalctl -xe shows logs similar to the following:

pcscd[13141]: 00000000 ifdhandler.c:150:CreateChannelByNameOrChannel() failed
pcscd[13141]: 00000071 readerfactory.c:1105:RFInitializeReader() Open Port 0x200000 Failed (usb:1050/0407:libudev:0:/dev/bus/usb/003/006)
pcscd[13141]: 00000004 readerfactory.c:376:RFAddReader() Yubico YubiKey OTP+FIDO+CCID init failed.
pcscd[13141]: 00004720 ifdhandler.c:150:CreateChannelByNameOrChannel() failed
pcscd[13141]: 00000023 readerfactory.c:1105:RFInitializeReader() Open Port 0x200000 Failed (usb:1050/0407:libudev:1:/dev/bus/usb/003/006)
pcscd[13141]: 00000004 readerfactory.c:376:RFAddReader() Yubico YubiKey OTP+FIDO+CCID init failed.
pcscd[13141]: 00143849 auth.c:135:IsClientAuthorized() Process 13120 (user: 1000) is NOT authorized for action: access_pcsc
pcscd[13141]: 00000140 winscard_svc.c:335:ContextThread() Rejected unauthorized PC/SC client

Analysis

Carefully analyzing this log, you can see this is an exception call stack output, and the crux of the matter is that user: 1000 doesn’t have access_pcsc permission.

By reviewing the polkit documentation, we know that polkit is a permission management toolset designed to enable communication between low-priority processes (pcscd) and high-priority processes (drivers).

Solution

To grant permissions, you need to add a configuration file 051-org.debian.pcsc-lite.rules (filename can be customized, just needs to end with .rules) to the /etc/polkit-1/rules.d/ directory. 050 is the default rule, after 050 are custom rules, before 050 are supplements to the default rules

Configuration file as follows:

polkit.addRule(function(action, subject) {
    // Here I granted access_pcsc permission to all users in the wheel group
    if (action.id == "org.debian.pcsc-lite.access_pcsc" &&
        subject.isInGroup("wheel")) {
        return polkit.Result.YES;
    }
});

polkit.addRule(function(action, subject) {
    // ditto ...
    if (action.id == "org.debian.pcsc-lite.access_card" &&
        subject.isInGroup("wheel")) {
        return polkit.Result.YES;
    }
});

Debugging Tips

The polkit configuration file syntax is very similar to JS. You can use polkit.log for debug output. Other methods and explanations are clearly described in the polkit documentation¹.

During debugging, pay close attention to logs. If you see keywords like Error compiling script, it means there’s a compilation error and this change won’t take effect.

After learning freestyle, my Xiaomi band gloriously malfunctioned after only 8 uses.

Options

So I started looking for reliable alternatives on JD and Taobao:

1. Moov Now Swimming Smart Band: ¥399.00
2. Swimovate PoolMate2 Standard Edition: ¥790.00
3. Swimovate PoolMate Live Agile Edition + Swimovate PoolMate Live Agile Edition USB Data Clip: ¥945.00 + ¥335.00
4. GARMIN Forerunner735: ¥2280.00
5. COUNTU 2018: ¥288.17
6. SportCount: ¥238.00

The final choice, as the title suggests, was COUNTU 2018

Reasoning

Moov Now was my first choice, but after reading several reviews, I found its measurement accuracy was still mediocre. Having learned from the Mi band experience, I passed on it.

The two Swimovate models are listed separately because of price and features. In theory, these two watches achieve professional-level measurement accuracy. The Live version has an additional data export feature, but requires buying a USB data clip. If only considering measurement accuracy, buying the Swimovate PoolMate2 Standard Edition is sufficient. Pending.

GARMIN Forerunner735 is a professional triathlon watch, but the price isn’t pretty, so I passed.

Two remaining options are non-band form factors: COUNTU and SportCount. These are actually the only two finger-ring type counters for swimming that can be purchased online. The SportCount’s operation and display are reportedly quite dumb, while COUNTU as a domestic brand has made many user-friendly optimizations.

In the end, I chose the more hardcore and cheaper option between Swimovate and COUNTU: COUNTU.

Summer is here, and I plan to maintain my daily exercise routine through swimming outside of work.

Of course, this choice isn’t completely out of the blue. When I was still working in Beijing, my leader at the time applied for swimming cards and took us to the swimming pool next to the company once a week. Every time I think about it, I can still remember how we’d eat some jianbing guozi (Chinese crepe) behind the company after swimming and then go back to work.

Background

Here it’s necessary to mention my swimming experience:

• After the college entrance exam, relatives bought me some local swimming tickets. I probably went about a dozen times that summer
• Sophomore year swimming class, got a yearly card for the school’s swimming pool (later mainly used for showering)
• After working in Beijing, leader took us to the swimming pool for exercise

As you can see, I started swimming quite late, and going to the pool was basically just messing around. I’m still learning in the shallow end to this day.

My current company reimburses a certain amount of fitness expenses, so I signed up for a swimming class near my home. Next, I’ll talk about my experience in the swimming class and my practice afterwards.

Learning

I signed up for an adult breaststroke class at the swimming pool, with classes from Monday to Friday evenings from 7:30 to 8:30. My planned learning schedule was after work on Monday, Tuesday, and Thursday.

First week, went to class with a sense of unfamiliarity. Told the coach about my level and swam a short stretch of “breaststroke” in front of the coach. The coach shook his head: You should learn freestyle instead, your leg kicks look like butterfly stroke.

So I began learning freestyle:

1. Holding the edge, kicking
2. Holding a kickboard, kicking
3. Holding kickboard, kicking + single-arm stroke
4. Holding kickboard, kicking + stroke + high elbow recovery
5. Holding kickboard, kicking + single-arm stroke + breathing
6. Holding kickboard, kicking + stroke + breathing
7. Kickboard + coordination
8. No board coordination practice 9-12. Coordination practice

Besides learning in the swimming class, I also looked for video tutorials on video sites. The one I’m currently watching is “Mengjue Teaches Swimming – ‘Freestyle Introduction Version 2’ (with subtitles)” which has been very helpful.

A few key knowledge points that were important for me:

1. Side body kicking
2. Breathing
3. Training plan

Let me explain the “training plan” knowledge point - it’s very helpful for improving swimming level: before getting in the water each time, plan out the training for this session. For example, my current routine is 200 meters kicking practice + technique drills + 1.5km - 2km coordination practice.

This kind of planned training is much more efficient than simply practicing coordination swimming or sprint swimming each time. It’s also less boring, allowing for visible improvement in skill level.

Results

Currently swimming twice a week, about 1500-2000 meters of coordination practice each time. Due to breathing still not being smooth enough (gets distracting), swimming more than 100 meters continuously is still difficult. But breathing issues like choking on water and fake breathing have improved, and swimming 50 meters continuously is no longer an obstacle :)

Openresty is a high-performance web platform based on Nginx and Lua. Its main components are:

1. Nginx¹
2. Lua virtual machine
3. lua-nginx-module: A project that embeds the Lua virtual machine into Nginx and provides Nginx APIs for Lua to call. At the Lua level, you can achieve non-blocking effects by using these APIs, mainly thanks to cosocket and the nginx event model
4. stream-lua-nginx-module: Similar functionality to lua-nginx-module, the difference is that lua-nginx-module provides APIs for nginx’s http module, while stream-lua-nginx-module provides APIs for nginx’s stream module.
5. lua-resty-core: Uses FFI to provide a series of common APIs at the Lua level
6. lua-resty-*: Based on the above modules, a series of commonly used service modules are encapsulated. Such as: lua-resty-redis/lua-resty-mysql/lua-resty-http

Development

In Openresty-based development, you need to understand the working principles of both Nginx and Lua, then further understand the principles of lua-nginx-module. This way you can have a project with a good balance between performance and maintainability.

1. Nginx

In Openresty-based development, you first need to recognize Nginx’s role in the entire project. Typical roles include:

1. Reverse proxy server
2. Web Server itself
3. Forward proxy server

Considering that projects developed with Openresty generally lean toward the server side, we’ll only discuss the first two cases here

1.1 Reverse Proxy

This is Nginx’s typical usage, i.e., acting as a gateway or load balancer frontend, directly receiving external requests, doing simple processing, then using the upstream feature to proxy traffic to the backend.

+---------+---------+    +------------------------+
|         |         |    |                        |
|         |         |    |    Web server          |
|         |         |    |                        |
|         |         |    +------------------------+
|         |         |
|         |         |    +------------------------+
|         |         |    |                        |
|   Lua   |Upstream |    |    Web server          |
|         |         |    |                        |
|         |         |    +------------------------+
|         |         |
|         |         |    +------------------------+
|         |         |    |                        |
|         |         |    |    Web server          |
|         |         |    |                        |
+---------+---------+    +------------------------+

Here’s a typical architectural diagram. The Openresty part is abstracted into the gateway logic layer handled by Lua, and the Upstream functionality layer handled by Nginx.

Here we’ll focus on Nginx’s role in this architecture:

1. Receiving requests
   1. If nginx works in master-worker multi-process mode, multiple worker processes listening on the same port will face a “thundering herd” problem: multiple processes are awakened simultaneously by a connection event, but only one process actually handles the connection. Nginx uses the ngx_accept_mutex synchronization lock mechanism to solve this problem.²
   2. Load balancing between multiple processes: Uses load thresholds to represent process load conditions, thus dynamically balancing load between multiple processes
2. Simple routing functionality: This mainly refers to multiple levels of keywords set in nginx configuration files, such as http/server/location, providing simple routing functionality (of course complex routing can be set up, but I tend to hand off complex routing to Lua code)
3. Providing hook points for lua-nginx-module to load lua virtual machine and code: These are Nginx’s module mount points. These mount points subdivide the request lifecycle into multiple phases, each phase with clear purposes, facilitating modular program management.³ The above phase subdivision is provided to nginx module developers. Before developing specific functionality for a nginx module, there are roughly several things to do: initialize configuration directive data structures, module context, handle configuration directive conflicts, register the module, and only then can you start developing the module’s specific functionality. After openresty provides multiple hook points and exposes appropriate APIs at corresponding hook points, development costs are significantly reduced
4. Upstream functionality

1.2 Web Server

In this architecture, my general approach is:

Hand off the simple routing part to Nginx configuration files, then run Lua programs that respond to user requests through mount points provided by lua-nginx-module

server {
    listen 80;
    server_name example.com;

    location /backend/ {
        content_by_lua_block {
            local handler = require "handler"
            handler.go()
        }
    }
}

If you need to use an existing web framework here, you might see frameworks like Vanilla.

Personally, I prefer combining multiple lightweight modules:

1. Use simple lua tables for routing
2. Use openresty/lemplate for rendering pages or results
3. Use lua-resty-* series modules for accessing databases or making http requests

The advantages of web servers developed this way:

1. Easy packaging, no need for luarocks tool (actually openresty officially provides the opm tool to solve lua-resty-* dependencies)
2. Easy deployment
3. Simple modularization

2. Lua

There are simple analyses of Lua source code here, so I won’t elaborate further.

Lua source code series articles:

• Lua Source Code Reading Plan
• Lua Source Code Reading (Part 1)
• Lua Source Code Reading (Part 2)
• Lua Source Code Reading (Part 3)
• Lua Source Code Reading (Part 4)
• Lua Source Code Reading (Part 5)
• Lua Source Code Reading (Part 6)

1. Use table.new to allocate lua tables of known size, because table.new calls the lua_createtable() function which can be optimized in LuaJIT. Another advantage is pre-allocating table size, preventing resource consumption during table growth.
2. Use local variables to cache results returned by APIs provided by lua-nginx-module or lua-resty-core. This reduces consumption by reducing unnecessary stack operations.
3. Note the application of lua’s __gc metamethod in some modules to prevent strange bugs caused by triggering __gc⁴
4. Note that in openresty, lua level code should avoid blocking IO: such as using lua’s native os library to read/write local files or system calls, which will affect the entire nginx worker’s operation

3. lua-nginx-module

This section mainly discusses some important configurations in lua-nginx-module.

• lua_code_cache: on|off: Turning off code caching means each request runs a separate Lua VM instance, i.e., changes to lua code take effect immediately. This feature is recommended only during debugging. Some module functionality may depend on code caching
• lua_package_path / lua_package_cpath These two directives directly determine whether nginx can find the lua modules you want to reference, so they are very important

Other directive documentation can also be clearly found in the official openresty/lua-nginx-module documentation.

4. Tools

1. openresty/openresty-devel-utils: This repository has many convenient small tools to use during openresty-related development, such as:
   1. lua-releng: A wrapper around the luac command line, bringing multiple global variables provided by openresty into the correct scope
   2. reindex: Mainly a syntax format check for test files based on the Test::Nginx module
2. spacewander/luacov-console: Combined with the luacov tool, generates colored code coverage in the terminal, and with slight processing can be used as a code coverage data source in the CI tool chain
3. Test::Nginx: The data-driven testing framework officially used by openresty

See Also

1. OpenResty Best Practices
2. openresty/lua-nginx-module / openresty/stream-lua-nginx-module / openresty/lua-resty-core official documentation

A few days ago, I wanted to implement a feature that adds a tag in the blog’s front-matter to distinguish lifestyle articles from technical articles, with the homepage displaying different link colors.

Problem

While searching for solutions on the Hexo website and Google, I found several issues:

1. Hexo’s default template paularmstrong/swig is no longer maintained¹
2. The documentation on the Hexo website was still quite sparse

As I mentioned in hexo + gitlab serving hidden static files:

Having previous experience with hexo, plus the exceptionally thriving node ecosystem for personal blogs (lots of third-party node-related plugins available), I decided to migrate over.

This is actually one of the causes of problem one. Problem one has been discussed in hexo’s GitHub issues: Why not totally replace Swig with Nunjucks? #1593, where mozilla/nunjucks² as an upgrade to the Swig template, can serve as a replacement after Swig is no longer maintained. But the hexo project wasn’t so quick to replace the default template.³

Problem 2 has been an issue for a long time, being quite unfriendly to theme developers. Here you can see one theme developer’s feelings https://blessing.studio/get-hexo-posts-by-category-or-tag/:

Today while porting my blog theme to Hexo, I wanted to get all posts under a certain category or tag (more accurately, I wanted to get the total number of posts). When searching with Chinese keywords, I didn’t get any useful information (perhaps my search technique was wrong). After switching to the English keyword “hexo category all posts”, I found the information I needed, so I decided to write a post to document this, hoping to help others later.~~~~

I have to complain here - Hexo’s documentation is really terrible, just terrible. Writing a theme, sometimes wanting to implement a feature requires frantically reading Hexo source code, I’m speechless.

Solution

My solution options were:

• Option 1: Keep the swig template unchanged, search for similar implementation methods.
• Option 2: Use a nunjucks plugin to enable hexo to support that template, then implement the feature.
• Option 3: Switch to other template engines supported by Hexo, such as hexojs/hexo-renderer-ejs, https://github.com/hexojs/hexo-renderer-haml, hexojs/hexo-renderer-jade, re-implement the theme, and implement the feature along the way
• Option 4: Switch to another static blog generation platform

My solution process happened to follow the order I listed

Option 1

Since hexo’s custom front-matter parts require hexo import scripts, and I didn’t want to spend too much time on the hexo framework itself. So I gave up. During the process, I used another method: using a rarely-used front-matter field to mark link colors (I used the layout variable). The actual effect achieved a similar result, but the code looks confusing.⁴

Option 2

nunjucks only has a few hexo-related plugins, hexo-renderer-nunjucks and hexo-nunjucks, and opening them shows these two projects’ last update time was locked to three years ago. So I gave up.

Option 3

These template engines are directly supported by Hexo official, but when actually using them, I still felt the outdated swig was slightly better (mainly because it matched my impression of templates). So I gave up.

Option 4

First, let me list the alternatives:⁵

• JavaScript: Next.js & Gatsby (for React), Nuxt & VuePress (for Vue), Hexo, Eleventy, GitBook, Metalsmith, Harp, Spike.
• Python: Pelican, MkDocs, Cactus.
• Ruby: Jekyll, Middleman, Nanoc, Octopress.
• Go: Hugo, InkPaper.
• .NET: Wyam, pretzel.

I finally chose Hugo from these, mainly because I was learning Go recently. And Hugo’s “text/template” is also a standard extension module for Golang, so it shouldn’t have the jumping between multiple templates like Hexo^(problem 1)^. Hugo’s official documentation is also visibly more extensive^(problem 2)^.

Implementation

Having determined the solution, let me organize what needs to be done after choosing this solution:

1. Article migration
2. Template migration⁶
3. Feature implementation
4. Deployment plan

1. Article Migration

Since both use articles written in markdown, migration is basically just a cp command. I won’t elaborate here.

2. Template Migration

For the template part, it was basically pixel-level COPY: implementing swig template functionality line by line using text/template. Of course, the syntax style follows Hugo’s official documentation. I still encountered a few small challenges during the process, and I’ll post the solutions here:

1. When implementing the /tags/ page, I needed to first group articles by tag, then iterate through articles in each tag group. In text/template, variable scoping is very strange, code as follows:

{{ $v := "init" }}
{{ if true }}
    {{ $v := "changed" }}
{{ end }}
v: {{ $v }} {{/* => init */}}

Intuitively judging this code, $v should output "changed". However, the actual result is surprising. My understanding of this situation: in text/template’s implementation, each code block has an independent scope, and this scope doesn’t inherit when nesting occurs. This is probably the cleanest and simplest code implementation. In such cases, the official recommendation is to use .Scratch to create a page-level scope readable-writable variable, but this is a bit heavy for theme templates.

With Google’s help, I found this gist: https://gist.github.com/Xeoncross/203d8b1459463a153a3c734c98b342a9

         <ul class="tags">
            {{ range $name, $taxonomy := .Site.Taxonomies.tags }}
              <li><a style="text-transform: capitalize" href="#{{ $name | urlize}}">{{ $name }}</a>
                <!-- <span>({{ len $taxonomy }})</span> -->
              </li>
            {{ end }}
          </ul>

2. In articles with a table of contents, I found the automatically rendered TOC would have empty · appearing.

In some articles, my subheadings are marked with <h3>, but the auto-generated TOC template didn’t automatically remove unused <h1> and <h2>

·    ·    · Heading level 3 1
          · Heading level 3 2

This is also reflected in Hugo’s GitHub Issues: Heading levels in Markdown table of contents #1778, and this involves another issue - the markdown rendering template russross/blackfriday used by Hugo is still v1 version in Hugo, and v1 version outputs an HTML fragment after parsing markdown. Because of this, there’s this ugly inefficient code when generating .TableOfContents: https://github.com/gohugoio/hugo/blob/master/helpers/content.go#L416

func ExtractTOC(content []byte) (newcontent []byte, toc []byte) {
    if !bytes.Contains(content, []byte("<nav>")) {
        return content, nil
    }
    origContent := make([]byte, len(content))
    copy(origContent, content)
    first := []byte(`<nav>
<ul>`)

    last := []byte(`</ul>
</nav>`)

    replacement := []byte(`<nav id="TableOfContents">
<ul>`)

    startOfTOC := bytes.Index(content, first)

    peekEnd := len(content)
    if peekEnd > 70+startOfTOC {
        peekEnd = 70 + startOfTOC
    }

    if startOfTOC < 0 {
        return stripEmptyNav(content), toc
    }
    // Need to peek ahead to see if this nav element is actually the right one.
    correctNav := bytes.Index(content[startOfTOC:peekEnd], []byte(`<li><a href="#`))
    if correctNav < 0 { // no match found
        return content, toc
    }
    lengthOfTOC := bytes.Index(content[startOfTOC:], last) + len(last)
    endOfTOC := startOfTOC + lengthOfTOC

    newcontent = append(content[:startOfTOC], content[endOfTOC:]...)
    toc = append(replacement, origContent[startOfTOC+len(first):endOfTOC]...)
    return
}

Here, string processing methods are used to parse content in .Content, then piece together the content to be generated and add it to the original content. This problem is solved in blackfriday.v2, which outputs an AST for other programs to process, which also ensures compatibility with subsequent versions. But in Hugo, the author has repeatedly postponed this feature’s milestone Upgrade to Blackfriday v2 #3949⁷.

But in the Issue discussion, various experts proposed their own solutions - you can click into Heading levels in Markdown table of contents #1778 to see details. I adopted the template solution from there:

            {{ $toc := .TableOfContents }}
            {{ $toc := (replace $toc "<ul>\n<li>\n<ul>" "<ul>") }}
            {{ $toc := (replace $toc "<ul>\n<li>\n<ul>" "<ul>") }}
            {{ $toc := (replace $toc "<ul>\n<li>\n<ul>" "<ul>") }}
            {{ $toc := (replace $toc "</ul></li>\n</ul>" "</ul>") }}
            {{ $toc := (replace $toc "</ul></li>\n</ul>" "</ul>") }}
            {{ $toc := (replace $toc "</ul></li>\n</ul>" "</ul>") }}
            <!-- count the number of remaining li tags -->
            <!-- and only display ToC if more than 1, otherwise why bother -->
            {{ if gt (len (split $toc "<li>")) 2 }}
              {{ safeHTML $toc }}
            {{ end }}
          {{ end }}

This solution can be easily removed from the template after future feature version merges. It’s one of the simpler approaches.

3. Feature Implementation

After template migration was complete, I started implementing the feature I originally wanted. In Hexo, front-matter can include user-defined fields (documentation at: https://gohugo.io/variables/page/#page-level-params). I chose to use the linkcolor field here.

---
title: test
linkcolor: #7076c7
---

Then add the variable judgment where the homepage iterates through titles:

{{ if .Params.linkcolor }}
{{ $color := .Params.linkcolor }}
<a href="{{ .Permalink }}" class="post-list-item" style="color:{{ $color }};">
{{ else }}
                    <a href="{{ .Permalink }}" class="post-list-item">
{{ end }}

I felt that writing an abstract color #7076c7 each time didn’t look good, so I added the following content to the data/color.toml directory (documentation at: https://gohugo.io/templates/data-templates/)

[link]
blue = "#7076c7"

Modified the homepage template

{{ if .Params.linkcolor }}
{{ $color := index .Site.Data.color.link .Params.linkcolor }}
<a href="{{ .Permalink }}" class="post-list-item" style="color:{{ $color }};">
{{ else }}
                    <a href="{{ .Permalink }}" class="post-list-item">
{{ end }}

This way, in front-matter you only need to write linkcolor: blue to achieve the same effect. Future color-related extension features can also be conveniently implemented.

4. Deployment Plan

The official documentation covers this in detail. I use the Netlify platform to publish my blog, documentation at https://gohugo.io/hosting-and-deployment/hosting-on-netlify/ Other commonly used platforms are also covered in the documentation.

The main focus here is migrating commit history from the old blog. I put the Hexo blog files and directories into a separate directory hexo_archive, and put Hugo platform code in the project root directory. This way, previous commit history and files are preserved, while leaving a relatively clean directory for Hugo.

See Also

• https://gist.github.com/Xeoncross/203d8b1459463a153a3c734c98b342a9: Various situations you’ll encounter with Hugo templates
• https://epatr.com/blog/2017/hugo-and-netlify/: Using Hugo with Netlify

This mainly uses text narration to try to explain this process, used to organize knowledge points and identify gaps. If any descriptions are inaccurate, please forgive me.

Q: From entering an address in the browser and pressing enter, to receiving the response - describe the details of this process.

A:

Premise

Complex programs need to be layered.

Overview

Explaining this process according to the OSI seven-layer protocol, focusing on the performance in OSI layer 2 - Data Link Layer (hereafter MAC layer), layer 3 - Network Layer (hereafter IP layer), layer 4 - Transport Layer, and the Application Layer. After reaching the Application Layer, I’ll discuss CDN principles and common architectures. Then briefly cover data center architecture.

OSI Seven Layers vs TCP/IP Four Layers

The OSI seven-layer model is a reference model for open systems interconnection, while TCP/IP protocol suite is a set of communication protocols for implementing network interconnection. The seven layers of OSI are: Physical Layer, Data Link Layer, Network Layer, Transport Layer, Session Layer, Presentation Layer, Application Layer. Memory mnemonic: All People Seem To Need Data Processing

The four layers of TCP/IP are: Interface Layer, Internet Layer, Transport Layer, Application Layer. The Interface Layer in TCP/IP corresponds to Physical and Data Link Layers, while Application Layer corresponds to Session, Presentation, and Application Layers.

Data Link Layer

Every networked device has a unique hardware address, commonly called a MAC address. MAC addresses are used to find corresponding devices within a broadcast domain. Although MAC addresses are unique, they’re not locationable, so IP protocol is used for communication outside the broadcast domain.

Layer 2 devices can cache the correspondence between MAC addresses and port devices, caching devices within the same broadcast domain. The correspondence between MAC addresses and IP addresses can be cached in local routing devices, called the routing table. When IP is known but MAC address is unknown, an ARP request can be sent to query the MAC address.

Internet/Network Layer

What’s entered in the browser is usually a domain name, which is resolved to an IP through DNS.

First determine if the corresponding IP is in the same network segment through CIDR. If not, the request goes through the gateway using routing protocols to find the corresponding network and device. When passing through intermediate routing devices (layer 3), first compare if MAC address matches, then if IP address matches, to see if it’s a packet for itself or needs forwarding.

Routing Protocols

Routing is divided into dynamic and static routing. The routing protocols here refer to dynamic routing protocols.

Distance Vector Routing Protocol

Based on Bellman-Ford algorithm, routers transmit part or all of their routing table to adjacent routers

RIP: 1. Uses hop count as metric 2. Maximum hop count is 15 3. RIP v1 periodically synchronizes the entire routing table

BGP: BGP can be seen as an advanced distance vector routing protocol. In BGP systems, networks can be divided into multiple autonomous systems. Within autonomous systems, iBGP is used to synchronize routing information, while eBGP broadcasts routes between autonomous systems. Autonomous System: All IP networks and routers under the management of one (or more) entities. IANA assigns an ASN (Autonomous System Number) to autonomous systems, enabling BGP protocol to run between ISPs on the Internet. ASN: A 16-bit number, now with 32-bit notation: <high 16 bits decimal>.<low 16 bits decimal>

Autonomous System Classification: 1. Multihomed AS: Autonomous system with more than one connection. This type doesn’t allow other autonomous systems to pass through it to access another autonomous system. 2. Stub AS: Autonomous system connected to only one other autonomous system. 3. Transit AS: An autonomous system that provides connectivity between separate networks. This is the essence of ISPs.

BGP Usage Conditions: 1. Need routers that support storing large routing tables 2. Need multiple connections 3. Have sufficient bandwidth to transmit required data (including routing tables)

Shortest Path First Algorithm

Based on Dijkstra algorithm, routers transmit link state information to all routers in the same area

OSPF: 1. Uses multicast to send link state updates, uses triggered updates on link state changes, improving bandwidth utilization 2. No maximum hop count limit, uses delay and cost as metrics

Difference and Connection Between IGP and IBGP

IGP includes protocols like OSPF/RIP, used within autonomous systems, mainly for route discovery and calculation. IBGP is also used within routing systems. The differences are: 1. IBGP delegates route discovery entirely to IGP, focusing on route control itself. 2. IGP has poor capability handling large routing tables, while IBGP can handle them hierarchically. 3. If BGP routing information is given directly to IGP, route attributes are lost, creating routing loop risks. IBGP can handle these route attributes (point 1)

Gateway

Type one: After leaving the gateway, the destination MAC address changes to the next hop device’s MAC address, while source and destination IP remain unchanged until reaching the specified device. This method is suitable for scenarios without IP address conflicts. For overlapping IP addresses, NAT gateway is needed.

Type two: NAT gateway. After the request leaves the gateway, both source IP and MAC become the gateway’s, while destination IP remains unchanged, reaching the next hop. When the response arrives, destination IP and MAC are mapped back.

DNS

DNS is a distributed data query system for domain name to IP address translation. When a client initiates a DNS query, it first queries the local DNS server (the DNS server configured on the ISP or router). If no record exists, it queries root name servers, which return top-level domain server addresses, which in turn return authoritative name server addresses. The client’s query to local server is recursive, while local DNS server’s upward queries are iterative.

Transport Layer

The IP packet header identifies the transport layer protocol type, commonly UDP and TCP.

UDP

UDP is a connectionless protocol. It inherits most of IP protocol’s characteristics: packet-based, stateless, unordered, no congestion control. Simply put, it’s a stateless transport protocol. In UDP headers, only source and destination port numbers identify the connection. Can be used in simple environments, intranets, and scenarios where packet loss is acceptable. Application layer can also implement state maintenance, making it a reliable connection.

TCP

TCP is a connection-oriented protocol. A connection here means a series of state transitions. After maintaining a complex state machine, connections become ordered, reliable, with congestion control, etc. But TCP’s underlying IP protocol is connectionless and unordered, so TCP heavily uses retransmission and congestion control algorithms to implement these features.

Three-way handshake: Requester sends SYN packet to establish connection (SYN_SENT), receiver returns ACK + SYN packet (SYN_RCVD), requester receives ACK packet (ESTABLISHED), then responds to peer’s SYN with ACK. When peer receives ACK, state becomes ESTABLISHED. At this point, both sides have completed one send-receive cycle, state is ESTABLISHED, connection established.

Four-way handshake (termination): Requester sends FIN packet to terminate connection (FIN_WAIT_1), receiver returns ACK after receiving FIN (CLOSE_WAIT), initiator enters (FIN_WAIT_2) after receiving ACK(seq=k+1). When receiver finishes upper layer logic, returns FIN + ACK packet (seq=k+1), initiating termination (LAST_ACK). Sender responds with ACK after receiving these packets, enters TIME_WAIT state, waits two MSL periods then closes connection. Receiver also closes after receiving ACK. TIME_WAIT state is to prevent receiver not receiving the last ACK, triggering FIN + ACK retry.

Application Layer

HTTP

Viewing web pages generally uses HTTP protocol for data transmission. HTTP is a protocol built on TCP. In HTTP, both request and response messages are plaintext. Request messages consist of: request line, request headers, request body, where request line can be subdivided into request method and request URL. Response messages consist of: response status, response headers, and response body. Response status is divided into status code and reason.

Common request methods: GET/POST/PUT/DELETE/OPTION Common response codes: 200 OK/201 Created/301/302/403 Forbidden/404 Not Found/405 Not Allowed/500 Internal Error/502 Bad Gateway/503 Service Unavailable/504 Gateway Timeout

Keepalive

TCP protocol has a keepalive concept, and HTTP also has a keepalive concept which is enabled by default after HTTP/1.1. HTTP’s keep-alive allows clients to send multiple requests over the same TCP connection, while TCP’s keep-alive is a mechanism to keep TCP connections alive through heartbeats. The two are not directly related.

HTTPS

Since HTTP request and response messages are plaintext, it’s not suitable for high-security scenarios. This introduces HTTPS. HTTPS uses asymmetric encryption to exchange keys, then symmetric encryption after key exchange, based on HTTP protocol. The handshake process is:

1. Client initiates client hello, mainly to negotiate encryption protocol version, compression algorithm, random number c1, and SNI info
2. Server responds with server hello, informing client of adopted encryption protocol version, compression algorithm, and random number s1 generated on server side. At this point, client-generated c1 is stored for later use
3. Server also responds with server certificate for client verification
4. Finally responds with server hello done to tell client hello info is complete
5. After client verifies certificate is legitimate (hash certificate info and compare with CA signature decrypted with CA public key), generates pre-master-key for symmetric encryption
6. Transmits random pre-master-key to server, the client key exchange
7. Client initiates change cipher spec, changing from asymmetric to symmetric encryption
8. Client initiates encrypted handshake message, transmitting info encrypted with c1 + s1 + pre-master-key to server
9. Server also initiates change cipher spec
10. Server similarly initiates encrypted handshake message At this point, both sides’ SSL handshake is complete. Encrypted data transmission begins.

Finding an Apartment

Almost as soon as I started my unemployment, I began paying attention to apartment hunting. My general requirements were:

• Larger room, a big one-bedroom or small two-bedroom
• Normal layout
• Quiet community
• Within two kilometers of subway or served by shuttle bus
• Budget about 100 yuan more than last year’s rent
• Direct contract with landlord, and landlord should be easy to communicate with

Following these requirements, I found three candidates in three days:

• Apartment A: Pros are quiet community and cheap rent. Cons are 2.6km from subway, and no commercial area nearby.
• Apartment B: Pros are newer apartment. Cons are empty without appliances - adding everything would add about 1k monthly rent.
• Apartment C: Pros are quiet community, large room, and landlord is easy to communicate with. Cons are older building, no elevator, and some furniture is seriously worn.

After various trade-offs, I chose Apartment C.

Moving

Looking back at the entire moving process, it took exactly one week.

Monday: Rented a car to move simple clothes, mattress, bedding, and cleaning supplies. Aired out the mattress and bedding, set aside. Tuesday: Organized items at home, ready to move. Wednesday: Moved pots, pans, bowls, and other fragile items. Thursday: Moved some small items. Washed and dried sofa covers. Arranged broadband transfer. Friday: Booked moving company, packed and organized all items. Cleared furniture. Cleaned and disinfected Apartment C. Saturday: Morning followed moving company to move items to Apartment C. Afternoon rented a car-share vehicle, moved remaining items in three trips. Negotiated a price for the wardrobe with a shady furniture buyer, and between the second and third trips arranged for someone to mount the TV on the wall. Finished all three trips after 1 AM. Sunday: Morning made another trip for kitchen items and refrigerator contents, sold the wardrobe. Replaced bathroom shower head, showered and rested. Had lobster for dinner, done.

Lessons Learned

Apartment hunting: If your requirements are clear, quickly filter down to two or three suitable options, then make trade-offs among them. Driving: I rarely drove after getting my license and was timid about it. But this move involved seven round trips by car, navigating through the harsh environment of the residential complex. The timidity about driving is gone - I can even listen to the radio to relax in congested traffic now. Wonderful. Organization: Prepare plenty of boxes. For walk-up buildings, medium-sized boxes are better - a heavy box carried up several floors will make your back ache.

Moving: Observing the moving crew, I suddenly appreciated the importance of work.

I had moved things up and down the stairs several times before - compared to the efficiency of the moving crew, I was way behind.

On one hand, it’s the use of various tools, and on the other, the difference in physical fitness. Only the concept of “work” can gather such a group of professionals together, and through the process of repeated work, they’ll summarize many techniques to improve efficiency.

The driver in charge of the moving truck only drove the truck in and out and operated the cargo lift, but didn’t help with the actual moving.

Different division of labor allows people to spend more time deepening expertise in vertical domains, avoiding dispersed learning effort.

I also want to sound a warning to friends who are job hopping - even if you hate your current job, don’t just jump into another one casually. You must understand why you’re jumping, and think about what you want. If you really want to leave but truly haven’t found something suitable, it’s better to quit without a backup, be poor for a little while, than to enter a new pit. Because a new pit will make you feel even more desperate, and also damages your career stability.

I hope I can maintain my original intentions in these tides and wait for the truly suitable [pit] for me.

Here’s a list of what I’m doing during this vacation:

1. Summarize knowledge points from previous work, distill them into writing. And clarify the unclear parts through learning.
2. Look for a place to rent: The current apartment lease is expiring, hoping to find a comfortable place to live.
3. Rest :)

Haha, the blog post from thirteen years ago was remembering things from three years before that. So even just for this coolness, you should persist in blogging.

I’ve moved between different blog platforms¹ multiple times, and also abandoned maintaining some posts due to domain and hosting issues. The earliest traceable posts are actually from 2013-2014, during the period when I built a blog with Jekyll on GitHub. It’s been 5 years already.

I decided to migrate those posts over to add some historical weight to the blog - you can see since 2013 in the footer now :)

Table of Contents

Overview

To get true multi-color support, you need support from terminal emulator + tmux + vim. Here’s my programming environment: Windows VM (Ubuntu 18.04) + SecureCRT + Tmux + Vim 8.1. So what I need to do is enable True-Color support in each of these components.

Terminal Emulator

My preferred terminal emulator on Windows is SecureCRT, but when checking the True-Color support list at https://gist.github.com/XVilka/8346728#now-supporting-truecolour, I found SecureCRT wasn’t on it. After trying several Windows-supported terminal emulators, I settled on MobaXTerm.

The reasons are its similar operation logic to SecureCRT, and it provides a Personal Edition.

As for color preferences, I still chose the dark green background solarized theme.²

tmux

The conventional method didn’t work here. After much searching, I found http://lists.gnu.org/archive/html/emacs-devel/2017-02/msg00635.html - a method to modify terminfo to enable Tc (True-Color) for tmux. (I used the latest tmux source code compiled from git)

Conventional method - modify $HOME/.tmux.conf file:

# !!!important!!! Enable 24-bit color, other methods don't work
set -g default-terminal "tmux-256color"
set -ga terminal-overrides ",*256col*:Tc"

Modify Terminfo method (tested and working):

/usr/bin/infocmp > /tmp/.infocmp
echo "  Tc," >> /tmp/.infocmp
/usr/bin/tic -x /tmp/.infocmp

Verify True-Color:

awk 'BEGIN{
    s="/\\/\\/\\/\\/\\"; s=s s s s s s s s;
    for (colnum = 0; colnum<77; colnum++) {
        r = 255-(colnum*255/76);
        g = (colnum*510/76);
        b = (colnum*255/76);
        if (g>255) g = 510-g;
        printf "\033[48;2;%d;%d;%dm", r,g,b;
        printf "\033[38;2;%d;%d;%dm", 255-r,255-g,255-b;
        printf "%s\033[0m", substr(s,colnum+1,1);
    }
    printf "\n";
}'

Screenshot:

vim

Three main changes needed here:

1. Modify .vimrc file according to documentation

if has("termguicolors")
    " fix bug for vim
    let &t_8f = "\<Esc>[38;2;%lu;%lu;%lum"
    let &t_8b = "\<Esc>[48;2;%lu;%lu;%lum"

    " enable true color
    set termguicolors
endif

2. Install a suitable vim theme that supports true color. I chose solarized8 with matching colors

Plugin 'lifepillar/vim-solarized8'

" Enhanced contrast
set background=dark
colorscheme solarized8_high
let g:solarized_extra_hi_groups=1

3. After entering vim’s VISUAL mode, I found that selected lines couldn’t reverse display - they just lost highlighting. This makes it hard to see the exact selected characters (for example, comments also have no highlighting, making it hard to find selection boundaries). The search result was a terminal support issue - it can’t recognize the hi Visual gui=reverse reverse command. So I manually adjusted the selected text colors.

hi Visual gui=reverse guifg=Black guibg=Grey

(Still in .vimrc, specifically right after theme colors)

before:

after:

Both images show selecting lines 358-363

Addendum

During use, I also found a conflict between termdebug supported in vim 8.1 and true-color: if you set termguicolors after packadd termdebug, the currently executing line won’t be highlighted during actual debugging. The correct approach is to put packadd termdebug after set termguicolors.

Significance

From 256 color support to 160,000 color support, the literal meaning is smoother color transitions. In vim, it means code looks more comfortable (I wonder if there are corresponding True-color extensions for code syntax highlighting)

Migraines

A migraine is a chronic condition characterized by recurring moderate to severe headaches, usually accompanied by various autonomic nervous system symptoms. The English word “Migraine” comes from the Greek ἡμικρανία (hemikrania), meaning “pain on one side of the head,” where ἡμι- (hemi-) means “half,” and κρανίον (kranion) means “skull.”

Typically this headache is unilateral (involving only one side of the head) and pulsating, lasting 2-72 hours. Related symptoms may include nausea, vomiting, increased sensitivity to light and sound, and physical activity worsening the pain. One-third of migraine sufferers experience an aura: brief visual, sensory, speech, or motor disturbances that signal an impending headache. (Quoted from https://en.wikipedia.org/wiki/Migraine)

Rizatriptan

Rizatriptan is a triptan 5-HT1 agonist developed by Merck for treating migraines, sold under the brand name Maxalt. (Quoted from https://en.wikipedia.org/wiki/Rizatriptan)

My Experience

Under the bombardment of irregular “discomfort”¹ including headaches, stomachaches, photophobia, chest tightness, shortness of breath, and rhinitis attacks, I went to the hospital almost once a month for checkups. Of course, these checkups were just routine examinations in neurology, ENT, ophthalmology, and gastroenterology. After these checkups, doctors would prescribe medication for the corresponding symptoms, and some doctors would even prescribe “Chinese patent medicines”². After episode after episode, I also felt the futility of these methods.

On the internet outside the hospital, on Wikipedia, I saw descriptions of migraine symptoms. Comparing my symptoms with them, I found a high degree of overlap. But as someone not in the medical profession, I naturally wouldn’t self-diagnose based on a description, as this could be a form of “hypochondria”³.

During one of my many medical visits, a doctor suggested I find out when a vertigo specialist was available. Eventually, the specialist confirmed my migraine diagnosis. The doctor mentioned rizatriptan, an internationally recognized migraine treatment drug. Although the hospital didn’t have this prescription drug at the time, at my request, the doctor wrote the drug name on a piece of paper for me to find.

Again with the help of the internet, I made an appointment on JD Health, got a prescription, and bought rizatriptan.

Results

During my most recent migraine attack (symptoms had appeared but were still mild), I took rizatriptan. Without ruling out the placebo effect, the symptoms disappeared.

Having relied on alternative medicine⁴ for years, I finally managed to overcome this chronic condition⁵ using modern scientific methods. Enlightening.

While searching for vim termdebug issues, I accidentally discovered https://fzheng.me’s blog. Its minimalist approach and beautiful fonts made me want to try the related theme. I checked the related source code on GitHub - it’s a Jekyll template. I spent some time converting it to a hexo template, and during the conversion, I ported over a few extension features I was comfortable with from the NexT theme (since there are only three page templates, the workload was small).

The ported features are:

• Disqus comments. I modified it - if you choose lazyload, it’s not triggered by scrolling, but by clicking a button (forgive me for not knowing how to do button hover effects)
• Baidu / Google / Bing website verification and analytics features
• Social network buttons
• Website license
• Custom website footer

Theme project at: https://github.com/wukra/izhengfan.github.io.git

open FILEHANDLE, '/etc/hosts' or die $!;
my $string = do { local $/; <FILEHANDLE> };
print($string);

The output of this Perl code is roughly:

127.0.0.1       localhost

# The following lines are desirable for IPv6 capable hosts
::1     localhost ip6-localhost ip6-loopback
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters

That is, the entire contents of /etc/hosts.

Q&A

The difficult part to understand is line 2. In conventional Perl, reading a file’s complete contents would use a while loop to read line by line:

open FILEHANDLE, '/etc/hosts' or die $!;
my $string;
while (<FILEHANDLE>) {
    $string .= $_;
}

Here’s a detailed explanation of line 2:

• do { <block> } is a code block in Perl. The function’s return value is the last statement in <block>.
• <FILEHANDLE> is the last statement in the above do { <block> }, thus the return value. In Perl, <FILEHANDLE>’s return value has two different results depending on context:

my $scalar = <FILEHANDLE>;    # When left side is scalar, returns single line of file
my @array  = <FILEHANDLE>;    # When left side is array, returns entire file (each line maps to array element)

• The $/ variable setting is key to this syntax sugar. $/ is the input record separator, which defaults to newline. That is, <FILEHANDLE>’s default behavior of returning single-line content in scalar context is determined by the $/ variable. And local $/ is equivalent to local $/ = undef
• Difference between local and my: my creates a new variable, while local temporarily changes a variable’s value (within scope)

Overall, this creates a temporary scope through do {}, changes the value of $/ within the scope, changes the separator for <FILEHANDLE> in scalar context (from \n to undef), achieving the goal of reading the entire file content.

References

• The difference between my and local https://www.perlmonks.org/?node_id=94007
• Perl Idioms Explained - my $string = do { local $/; }; https://www.perlmonks.org/?node_id=287647

Zone

A DNS zone consists of two parts: Resource Records (RRs) and Directives

; zone file for example.com
$TTL 2d    ; 172800 secs default TTL for zone
$ORIGIN example.com.
@             IN      SOA   ns1.example.com. hostmaster.example.com. (
                        2003080800 ; se = serial number
                        12h        ; ref = refresh
                        15m        ; ret = update retry
                        3w         ; ex = expiry
                        3h         ; min = minimum
                        )
              IN      NS      ns1.example.com.
              IN      MX  10  mail.example.net.
joe           IN      A       192.168.254.3
www           IN      CNAME   joe

1. A DNS zone file consists of comments, directives, and records (RRs)
2. Comments start with ; and continue to the end of the line
3. Directives start with $. $ORIGIN and $INCLUDE are defined in RFC 1035, while $GENERATE is a non-standard directive provided by BIND.
4. The $TTL directive must appear before the first RR
5. The first RR must be SOA (Start of Authority)

DNS Message

The message here refers to the message protocol between Resolver and the DNS system.

Format:

    +---------------------+
    |        Header       |
    +---------------------+
    |       Question      | the question for the name server
    +---------------------+
    |        Answer       | RRs answering the question
    +---------------------+
    |      Authority      | RRs pointing toward an authority
    +---------------------+
    |      Additional     | RRs holding additional information
    +---------------------+

(Diagram from RFC-1035)

Header

                                    1  1  1  1  1  1
      0  1  2  3  4  5  6  7  8  9  0  1  2  3  4  5
    +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
    |                      ID                       |
    +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
    |QR|   Opcode  |AA|TC|RD|RA|   Z    |   RCODE   |
    +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
    |                    QDCOUNT                    |
    +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
    |                    ANCOUNT                    |
    +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
    |                    NSCOUNT                    |
    +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
    |                    ARCOUNT                    |
    +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+

(Diagram from RFC-1035)

• ID: A 16-bit request ID, returned as-is in the response, used to identify the uniqueness of requests
• QR: Request/response identifier bit. Set to 0 for request, 1 for response
• OPCODE:
  • 0: QUERY. Standard query
  • 1: IQUERY. Inverse query (optional support)
  • 2: STATUS. DNS status
  • 3-15: Reserved
• AA(res only): Authoritative Answer. Responses from zone owners are authoritative answers, while responses from other DNS servers based on cache are non-authoritative
• TC: Truncated. Message is truncated when the message body exceeds maximum transmittable size
• RD: Recursion Desired. Requests recursive query in request message; if server supports recursive queries, set to 1, otherwise 0
• RA(res only): Recursion Available. Whether this NS server supports recursive queries
• Z: Reserved bits, must be 0 in both request and response
• RCODE(res only): Identifies server response type (similar to error codes)
  • 0: No error
  • 1: Format error: Server cannot parse request
  • 2: Server error: Server failed due to some reason, temporarily unable to respond
  • 3: Response when authority-only server (no recursive support) doesn’t find the domain
  • 4: Not implemented: Current query type not supported
  • 5: Refused: Service denied due to policy or other reasons
• QDCOUNT: 16-bit unsigned integer, indicates number of Question Section entries
• ANCOUNT: 16-bit unsigned integer, indicates number of RR entries in Answer Section
• NSCOUNT: 16-bit unsigned integer, indicates number of Authority Section entries
• ARCOUNT: 16-bit unsigned integer, indicates number of Additional Section entries

Question

                                    1  1  1  1  1  1
      0  1  2  3  4  5  6  7  8  9  0  1  2  3  4  5
    +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
    |                                               |
    /                     QNAME                     /
    /                                               /
    +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
    |                     QTYPE                     |
    +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
    |                     QCLASS                    |
    +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+

(Diagram from RFC-1035)

Note: Each request usually has only one Question Section, but you can actually specify any number of Question Sections through QDCOUNT

• QNAME: The queried domain name. Format: no. of chars domain name no. of chars domain name … Where no. of chars is the string length of adjacent domain name Example:

08 6D 79 64 6F 6D 61 69 6E 03 63 6F 6D 00
// printable
 !  m  y  d  o  m  a  i  n  !  c  o  m  !
// note ! = unprintable

(Diagram from zytrax.open)

• QTYPE: Query type. Corresponds to RR’s TYPE
• QCLASS: Query class. Most common value is x'0001 representing IN or Internet

Answer

Answer / Authority / Additional Section / RR all use the same format

RR / Answer format:

                                    1  1  1  1  1  1
      0  1  2  3  4  5  6  7  8  9  0  1  2  3  4  5
    +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
    |                                               |
    /                                               /
    /                      NAME                     /
    |                                               |
    +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
    |                      TYPE                     |
    +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
    |                     CLASS                     |
    +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
    |                      TTL                      |
    |                                               |
    +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
    |                   RDLENGTH                    |
    +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--|
    /                     RDATA                     /
    /                                               /
    +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+

(Diagram from RFC-1035)

• NAME: Response domain name.
  • Format 1: label format. Same as QNAME above
  • Format 2: Pointer format. Compressed data format. A 16-bit value: first two bits fixed as 1 (distinguished from label format due to label format’s max value limit of 63), OFFSET bit value is offset relative to message start. Where 0 represents the first bit of ID.

    +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
    | 1  1|                OFFSET                   |
    +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+

(Diagram from RFC-1035)

• TYPE: RR type.
  • x'0001(1): A record
  • x'0002(2): NS record
  • x'0005(5): CNAME record
  • x'0006(6): SOA record
  • x'000B(11): WKS record – Well Known Source used to describe common services (like SMTP) using specific protocols (like TCP(6)) on the Internet RFC1010
  • x'000C(12): PTR record. Reverse record for A and AAAA records (IP points to domain)
  • x'000F(15): MX record. Domain used by SMTP Agent to receive mail
  • x'0021(33): SRV record. RFC 2782 MX record is a special case. SRV record is a record field used by other specific services (like OpenLDAP)
  • x'001C(28): AAAA record. IPv6 address
• CLASS: RR class. e.g.: Internet Chaos
• TTL: Time (seconds) the record should be cached
• RDLENGTH: Length of RDATA
• RDATA: Each different type of RR data has a specific format.
  • SOA: SOA record controls domain record update policy

    +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
    /                     MNAME                     /
    /                                               /
    +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
    /                     RNAME                     /
    +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
    |                    SERIAL                     |
    |                                               |
    +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
    |                    REFRESH                    |
    |                                               |
    +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
    |                     RETRY                     |
    |                                               |
    +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
    |                    EXPIRE                     |
    |                                               |
    +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
    |                    MINIMUM                    |
    |                                               |
    +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+

	- Primary NS: Primary NS server. Variable length. label/pointer/mixed
	- Admin MB: Administrator mailbox. Variable length. label/pointer/mixed
	- Serial Number: Serial number. 32-bit unsigned integer. Format is "YYYYMMDDnn"
	- Refresh interval: 32-bit unsigned integer. Interval for secondary NS servers to check zone file updates
	- Retry interval: 32-bit unsigned integer. Retry interval when primary NS server is unreachable
	- Expiration Limit: 32-bit unsigned integer. How long DNS resolver can cache; for some DNS servers, it's the cache duration for responses to resolvers
	- Minimum TTL: 32-bit unsigned integer. Field meaning depends on NS implementation, with three possibilities:
		- Minimum cache duration for the domain by NS, rarely used by servers (officially deprecated)
		- Default TTL value. (Used when no TTL record exists)
		- Defines cache duration when domain has no records (distinct from TTL cache duration when records exist) [RFC 2308](https://www.ietf.org/rfc/rfc2308.txt) (officially recommended)
- MX:

    +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
    |                  PREFERENCE                   |
    +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+
    /                   EXCHANGE                    /
    /                                               /
    +--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+--+

	- PREFERENCE: Priority. Lower value = higher priority. Generally use `0`(max) for mail server records, use `10` for domain ownership verification
	- Mail Exchanger: Domain providing service. Variable length. label/pointer/mixed
- A: 32-bit unsigned integer, IP address
- AAAA: 16 octets, IPv6 address
- PTR, NS: Address. label/pointer/mixed

• Authority: (res only) Value is 0 in requests. Same format as Answer. Data is usually NS type RR

• Additional: (res only) Value is 0 in requests. Same format as Answer. Theoretically, any type of RR is valid. In practice, this field provides A or AAAA records corresponding to NS domains mentioned in Authority Section

  Note: (res only) means fields only valid in DNS responses

References:

• Chapter 8. DNS Resource Records (RRs) http://www.zytrax.com/books/dns/ch8/
• Chapter 15 DNS Messages http://www.zytrax.com/books/dns/ch15/
• Chaosnet https://en.wikipedia.org/wiki/Chaosnet
• DNS: Understanding The SOA Record http://www.peerwisdom.org/2013/05/15/dns-understanding-the-soa-record/
• MX record https://en.wikipedia.org/wiki/MX_record#Priority

Side note about LuaRocks: to use scripts installed by LuaRocks, you need to have LuaRocks installed on the machine. So even if you resolve dependencies when packaging by calling LuaRocks, it won’t work unless you add LuaRocks to the dependencies. This is very unfriendly for someone like me with a standalone obsession.

I rewrote the Perl version of auto-indent into a Lua version. The regex uses ngx.re from OpenResty, and it needs to be started using the resty command line or within openresty (two startup methods).

https://github.com/xiaocang/lua-resty-luatidy

Since https://bitcron.com/, where I had been hosting my blog, couldn’t update certificates, and their Let’s Encrypt auto-renewal script was broken (is this site going to shut down? (whispers)), I decided to take some time to migrate my blog to a pure static blog.

I had previous experience with hexo, plus the Node ecosystem is exceptionally rich for personal blogs (lots of third-party plugins available), so I decided to migrate.

After Github was acquired by Microsoft, although its CEO immediately became a billionaire, my goodwill towards it dropped to rock bottom. I only kept a few small open source projects and previously forked projects there, and stopped the paid plan for private repos. Now I’ve moved most of my code and private repos to GitLab. So the initial tech choice of GitLab Pages + hexo was settled.

Smooth Progress

First, following hexo’s official documentation step by step, I set up the main framework. Combined with GitLab CI for automatic static page generation and deployment, everything seemed surprisingly simple.

Problem Encountered

In GitLab’s pages options, it’s easy to find options for binding domains and setting up certificates. I installed Let’s Encrypt’s CLI tool certbot on my VPS and followed the official documentation:

certbot certonly --manual -d www.ogura.io

Next, next, until it stopped at the step requiring website ownership verification:

Access http://www.ogura.io/.well-known/acme-challenge/JDbJQP50-rI3LgKGLy7U7EhOkSxr73P0bP2UeTIh1yE to get a static file with specified content.

Serving static files on my own VPS couldn’t be simpler, but how do you do it with GitLab + hexo?

After experimentation, I learned that files in the source/ folder are directly placed in the public/ folder, but the special case is that .well-known is a hidden folder, and hexo generate ignores hidden files and folders when generating static files.

GitLab pages gave an example solution for Jekyll using an xxx.md file with content:

---
layout: null
permalink: /.well-known/acme-challenge/5TBu788fW0tQ5EOwZMdu1Gv3e9C33gxjV58hVtWTbDM
---

5TBu788fW0tQ5EOwZMdu1Gv3e9C33gxjV58hVtWTbDM.ewlbSYgvIxVOqiP1lD2zeDKWBGEZMRfO_4kJyLRP_4U

But in hexo (at least with the next theme, there’s no null layout), and the generated link has a .html suffix, which doesn’t pass verification.

After messing around with custom layouts for a long time without success, I finally added a manual copy command in .gitlab-ci.yml, and it worked. (Wasted half an hour on layout templates = =)

# This file is a template, and might need editing before it works on your project.
# Full project: https://gitlab.com/pages/hexo
image: node:6.10.0

pages:
  script:
  - npm install
  - ./node_modules/hexo/bin/hexo generate
  - /bin/cp -rv static/.well-known public/
  - ./node_modules/.bin/hexo algolia
  artifacts:
    paths:
    - public
  cache:
    paths:
      - node_modules
    key: project
  only:
  - master

Conclusions are based on actual runs of Lua 5.1.5 (Ubuntu official distribution) on Ubuntu 18.04

• A Lua file cannot exceed 262144 constants
• A control structure cannot exceed 32895 stacks
• The number of upvalues cannot exceed 60
• Each function in Lua cannot have more than 200 local variables

This article uses tt2 technology, and uses the command line tpage provided by Template::Tools::tpage to generate Lua code

A Lua file cannot exceed 262144 constants

Where 262144 = 2^18^

tt2 code

[% SET array = [1..262144] %]

local a = {
    "[% array.join("\", \"") %]"
}

Lua code

Generated Lua code gist link (large file): https://gist.github.com/xiaocang/cd947e57cdb6d16b83d7bdc9c4d0cecd#file-too_much_constant-lua

Execution result

$ lua too_much_constant.lua
lua: constant table overflow

Source code location

TODO

A control structure cannot exceed 32895 stacks

Where 32895 = 2^15^ + 2^7^ - 1

tt2 code

[% DEFAULT max = 32895 %]

if
[% FOREACH i IN [1..max] -%]
    ([% i %] > 0) [% IF i < max %] and [% END %]
[%- IF i % 10 == 0 %]
[% END -%]
[% END %]
then
    print("ok")
end

Lua code

Generated Lua code gist link (large file): https://gist.github.com/xiaocang/99a5b636694bab0fd870c17ef97fa472#file-control_pattern_too_long-lua

Execution result

$ lua control_pattern_too_long.lua
control_pattern_too_long.lua:3297: control structure too long near '<eof>'

Source code location

TODO

The number of upvalues cannot exceed 60

tt2 code

[% FOREACH i IN [1..200] -%]
local a_[% i %]
[% END %]

function closure()
    [% FOREACH i IN [1..61] -%]
    a_[% i %] = 0
    [% END %]
end

Lua code

Generated Lua code gist link (large file): https://gist.github.com/xiaocang/baf52c6e7fffa31d684ad55ddfc47867#file-too_much_upvalue-lua

Execution result

$ lua too_much_upvalue
lua: stdin:264: function at line 203 has more than 60 upvalues

Source code location

TODO

Each function in Lua cannot have more than 200 local variables

tt2 code

[% FOREACH i IN [1..201] -%]
local a_[% i %]
[% END %]
print("ok")

Lua code

Generated Lua code gist link (large file): https://gist.github.com/xiaocang/f4fd0b56bbdec00e4794e661b0ffd994#file-variables_in_function-lua

Execution result

$ lua variables_in_function.lua
main function has more than 200 local variables

Source code location

from lua5.2.2 src/lparser.c line 30-32

/* maximum number of local variables per function (must be smaller
   than 250, due to the bytecode format) */
#define MAXVARS     200

There aren’t many self-hosted image hosting solutions. I first used Lychee – developed in the world’s best language (PHP). After using it, I felt the features were too heavy: a bunch of features, only a few actually useful. I also had to install a bunch of image processing plugins for image scaling (which I didn’t even need).

Currently I’m using uppy. The advantage is easy deployment - one npm install solves it. The backend uses the Go version of tusd, which is even easier to deploy - just throw a binary file on the server.

Confusion & Solution

After choosing uppy, I read the official documentation for a long time but couldn’t find how to deploy uppy. I even thought uppy was the backend for uppy server. When I was at a dead end, I went to check uppy’s official example page and found that deploying uppy only requires a static page - all options are written in the <script> tag on the page.

Example page:

<html>
<head>
    <link href="/static/uppy.min.css" rel="stylesheet">
</head>
<body>
    <div>
      <div>
        <div id="dashborad-container"></div>
      </div>
    </div>
    <script src="/static/uppy.min.js"></script>
    <script>
      var uppy = Uppy.Core({ autoProceed: true })
      uppy.use(Uppy.Dashboard, { target: '#dashborad-container', inline: true, replaceTargetContent: true })
      uppy.use(Uppy.Tus, { endpoint: '/files/', uploadUrl: '/files/', overridePatchMethod: true, resume: true})
      uppy.run()

      uppy.on('complete', result => {
        console.log('successful files:', result.successful)
        console.log('failed files:', result.failed)
      })
    </script>
</body>
</html>

Screenshots:

Very simple functionality: upload/generate link

The proposal is to introduce OpenResty as a proxy web frontend, using OpenResty’s timer to monitor backend HTTP services (and various network services), then collect information and send unified service registration and heartbeat requests to the etcd server.

Diagram

Project at: https://github.com/xiaocang/lua-resty-etcd-discovery-client

Lua source code series:

• Lua Source Code Reading Plan
• Lua Source Code Reading (Part 1)
• Lua Source Code Reading (Part 2)
• Lua Source Code Reading (Part 3)
• Lua Source Code Reading (Part 4)
• Lua Source Code Reading (Part 5)
• Lua Source Code Reading (Part 6)

Chapter 6 of “Lua Source Code Appreciation” covers content related to Lua coroutines.

Contents

Lua Objects

From the C level, Lua’s state is a lua_State. Within the same Lua VM, multiple lua_States share one global_State. The lua_State should not be viewed as a simple static data structure, but as a state machine in a Lua “thread”, containing information about the current “thread’s” execution state, data stack, call stack, etc.

Lua Data Stack

Lua data can be divided into value types and reference types. The union Value in lstate.h is used to represent them.

union Value {
    GCObject *gc;    /* collectable objects */
    void *p;         /* light userdata */
    int b;           /* booleans */
    lua_CFunction f; /* light C functions */
    numfield         /* numbers */
};

As you can see, reference types use a GCObject pointer for indirect reference, while other value types are stored directly in the union. To distinguish the types in the union, an additional type field is also needed.

In Lua, the data stack size is 2 * LUA_MINSTACK, while in Lua’s C calls, the data stack size is only LUA_MINSTACK. (LUA_MINSTACK defaults to 20)

So when making Lua C calls, you need to explicitly extend the stack size using the luaD_growstack function in ldo.c. Each call extends by at least double the size. When extending the data stack, value type data can be copied directly, while reference type data needs to call correctstack for correction.

Call Stack

Lua’s call stack is in a CallInfo structure, stored as a doubly linked list in the “thread” object.

# define next_ci (L) (L->ci = (L->ci -> next ? L->ci -> next : luaE_extendCI (L)))

As you can see, in the Lua 5.2 implementation, the call stack is encapsulated as an infinitely extensible stack, with unused linked list nodes only cleaned up during GC.

Thread Execution and Interruption

Lua, as an embedded language, uses C’s longjmp mechanism uniformly for interruption and exception handling to implement coroutines not bound to hardware. When embedded in C++, it uses the try / catch mechanism instead. These are switched through macros.

Function Calls

Lua’s pcall is implemented as a function rather than a language mechanism. Implementing pcall involves saving and restoring state at the C level stack.

In pure Lua function calls, C functions are generally not involved. The process is:

Generate a new CallInfo, adjust the data stack, then jump the bytecode execution position to the beginning of the called function. Lua’s return operation does the opposite - restore the data stack, pop the CallInfo, modify the bytecode execution position, and restore to the original execution sequence.

In Lua’s underlying API, there are luaD_precall and luaD_poscall, because:

• In Lua calls, luaD_precall needs to be executed first to specify the bytecode execution position, then bytecode is executed in luaD_poscall and state is restored.
• In C calls, there’s no need to execute bytecode to restore state, only luaD_precall needs to be executed.

C Tricks

TValue uses the NaN Trick to save memory (Lua 5.2 feature). According to IEEE754, when exponent bits are all 1 and mantissa bits are all 0 (0xfff8000000000000), it represents “not a number”. It’s used to represent infinity and the result of division by 0. When a double-precision floating point number is greater than 0xfff8000000000000, it can be considered a deliberately crafted number for special purposes. NaN Trick exploits this situation. On 32-bit machines, the mantissa bits (52 bits) of a double-precision floating point can be used to store types other than numbers (32 bits) and value types (int), which is sufficient. This saves 4 bytes of memory per value.

Reference

1. IEEE_754
2. float uses IEEE R32.24, double uses IEEE R64.53

float

double

https://github.com/xiaocang/lua-5.2.2_with_comments/releases/tag/data_n_call_stack

Lua source code series:

• Lua Source Code Reading Plan
• Lua Source Code Reading (Part 1)
• Lua Source Code Reading (Part 2)
• Lua Source Code Reading (Part 3)
• Lua Source Code Reading (Part 4)
• Lua Source Code Reading (Part 5)
• Lua Source Code Reading (Part 6)

Chapter 5 of “Lua Source Code Appreciation” covers content related to Lua closures.

Contents

Lua Closure Concept

In Lua, binding a set of data to a specific function is called a closure. The bound data are called upvalues, also known as non-local variables (as distinct from local and global variables).

Closure Implementation and Classification

Lua functions differ from C functions in that Lua functions can be dynamically generated during VM runtime - functions can be created within Lua functions. Binding a Lua function prototype with a set of upvalues - binding the Proto structure with UpVal together - is how closures are implemented in Lua.

Lua has two types of closures: Lua closures and C closures. Lua closures are more complex.

1. Closures generated during VM runtime: Before a function exits, an index on the data stack is used to map local variables. At this point, the upval is called “open”. When the outer function returns and the data stack space shrinks, luaF_close is called to close it, and the previous pointer is redirected to a safe place for storage.
2. Closures generated from loading external source code (binary or text): Instead of using local variables on the data stack as upvalues, new upvalues are created. Additional note: When Lua loads external code, Lua code is compiled into function prototypes, but Lua’s external calls don’t return function prototypes - instead, they convert the function prototype into a closure.

C closures are simpler than Lua closures. Since C closures don’t reference external upvalues, upvalues are all in a closed state. Just bind the upvalues to the Closure structure. C closures also have a special case with no upvalues, called light C functions. Light functions don’t need the UpVal structure and don’t need GC management.

C Tricks

In Lua source code, the UNUSED() macro is used to avoid compiler unused variable warnings. The source code is:

#define UNUSED(x) ((void)(x))

https://github.com/xiaocang/lua-5.2.2_with_comments/releases/tag/lua_closure_06

Lua source code series:

• Lua Source Code Reading Plan
• Lua Source Code Reading (Part 1)
• Lua Source Code Reading (Part 2)
• Lua Source Code Reading (Part 3)
• Lua Source Code Reading (Part 4)
• Lua Source Code Reading (Part 5)
• Lua Source Code Reading (Part 6)

Content from Chapter 4.2.2 to the end of Chapter 4 of “Lua Source Code Appreciation” still mainly covers source code related to Lua tables.

Contents

Hashing of Numeric Types

Regarding hashing of numeric types, several version iteration improvements were mentioned.

Lua 5.1 directly reads and adds data from corresponding memory blocks for hashing. This algorithm could produce a bug on 64-bit systems where the same number produces different hash results:

On 64-bit systems, long double is considered 16 bytes instead of 12 bytes on 32-bit systems to maintain alignment. But the value itself is still 12 bytes, with the other 4 bytes filled with random garbage data

Lua 5.2 improved this algorithm with two user-configurable algorithms. One uses a union to calculate number hashes (conditions: number type is double, target machine uses IEEE574 standard floating point). The other has lower performance but uses a more general and standard hash algorithm.

Table Iteration (next)

Table iteration is frequently used in Lua. After carefully reading the source code, I understood the principle behind checking whether a table structure is empty that I had seen online.

First, let me explain Lua’s definition of table length: if t[n] is non-empty and t[n + 1] is empty, then n is the table’s length. Additionally, when the array part of the table is full or empty, the hash table part’s length is also calculated.

Another point I hadn’t noticed before: in other languages, any operations on a table are prohibited during iteration. But in Lua, during table iteration, you can modify or even delete existing values (assign to nil) without affecting the iteration result.

When deleting elements from a table being iterated, if GC happens to occur, key-value pairs set to nil will be marked as dead, and the findindex function handles this situation. If you create new elements in a table being iterated, the new key might not be traversed, or if the creation triggers a rehash action, it may cause duplicate traversal issues.

Lua’s next(table [,index]) function accepts two parameters. I always use it like this to check if a table is empty:

local tal_empty = next(t)

Metamethod Optimization

Lua’s implementation of complex data structures heavily relies on attaching a metatable to a table

Therefore, operations on metatables are hotspots in Lua, and the necessity of optimizing hotspots goes without saying.

From the moment Lua creates a table, these metamethods are directly attached to the Lua table, and using the flags bits in the Table structure to mark metamethods can greatly improve metamethod query speed. In actual queries, you can directly query the corresponding metamethod using the corresponding bits. When Lua creates metatables, I noticed that Lua uses luaS_fix to mark data structures in the state machine. When marked as FIXEDBIT, the structure won’t be garbage collected.

https://github.com/xiaocang/lua-5.2.2_with_comments/releases/tag/lua_table_metatable_04

Lua source code series:

• Lua Source Code Reading Plan
• Lua Source Code Reading (Part 1)
• Lua Source Code Reading (Part 2)
• Lua Source Code Reading (Part 3)
• Lua Source Code Reading (Part 4)
• Lua Source Code Reading (Part 5)
• Lua Source Code Reading (Part 6)

Content from Chapter 4 to 4.2.1 of “Lua Source Code Appreciation” mainly covers source code related to Lua tables.

Contents

Lua Table

A Lua table structure consists of at most three contiguous memory blocks: a Table structure, an array storing consecutive integer indices, and a hash table with size as a power of 2. The size of each part is reallocated by the computesize function when calling the rehash function, ensuring utilization is above 50%. The hash table uses closed hashing for collision resolution. When a collision occurs, the two colliding keys are linked together with a singly linked list.

Hash Algorithm Details

Hash

During table queries, lazy hash computation for long strings is also encountered. Yun Feng explains the reason for this optimization in his book:

In most applications, long strings are objects for text processing rather than comparison operations, and internal uniquification would bring additional overhead

However, string optimization was only added after Lua 5.2.0. Whether this is also present in the LuaJIT that comes with OpenResty is still a question mark.

https://github.com/xiaocang/lua-5.2.2_with_comments/releases/tag/lua_table_04

Wiki original https://en.wikipedia.org/wiki/Vienna_Development_Method

The Vienna Development Method (VDM) is one of the longest-established formal methods¹ for the development of computer systems. Its origins were in work done at the IBM Vienna Laboratory in the 1970s. VDM has since evolved into a family of techniques and tools based on a formal specification language—the VDM Specification Language (VDM-SL). VDM also has an extended form: VDM++, which supports object-oriented and concurrent systems. Support for VDM includes tools for analyzing models in commercial and academic contexts, as well as testing and verification of models and code generation from VDM models. VDM has a history of industrial applications and research that continues to evolve, making notable contributions to the engineering of critical systems, compilers, concurrent systems, and logic in computer science.

Philosophy

In VDM-SL, computing systems can be modeled at a higher level of abstraction than programming language implementations, allowing analysis of designs and identification of key characteristics and flaws in the early stages of system development. Validated models can be transformed into detailed system designs through a refinement process. The language has a formal semantics, allowing properties of models to be proved at higher levels of abstraction. It also has an executable subset, enabling models to be analyzed through testing and displayed through graphical interfaces so that models can be evaluated by domain experts who are not familiar with the modeling language itself.

api=>start: API
route=>condition: Route
common_route=>subroutine: Common Route
custom_route=>subroutine: Custom Route
common_controller=>condition: Common Controller
custom_controller=>subroutine: Custom Controller
hacked_code=>operation: hacked code
hacked_middleware=>operation: hacked middleware
common_io=>inputoutput: Common io/db process
hacked_user_io=>inputoutput: Hacked io/db process
render=>end: Render

api->route(no)->custom_route->custom_controller->render
api->route(yes)->common_route->common_controller->common_io->render
api->route(yes)->common_route->common_controller(no)->hacked_code->hacked_user_io->render
api->route(yes)->common_route->common_controller(yes)->hacked_middleware->hacked_user_io->render

I haven’t drawn the Model part of this structure here, because it’s also a mess.

Let me briefly explain this misused structure.

Ideal

Routes are mainly divided into two parts: one for common feature API routes, and one for customer-customized API routes. This is the ideal situation: when customers don’t have customization needs, they just use the common route’s API; when customization is needed, new paths are added separately in the custom routes.

Reality

In practice, I found this isn’t really a universal solution. Because some customers using the common API sometimes have customization needs, such as:

• During POST operations, instead of calling the common random function to generate strings, use customer-customized rules to generate strings before database operations. So I added ugly uid judgment logic in the code - if conditions are met, perform operation A, otherwise perform the common operation. After multiple similar code hacks, I added a middleware for uid judgment, but it’s still ugly.
• In the Common Route section, there’s an authentication middleware to ensure legitimacy. Here again, there are customer-customized authentication schemes to implement.
• In customer-customized features, there might be new data to store, so I would create new tables or fields in the Model, but these customized fields can’t have good access control.

In summary, in this architecture, some modules or structures were fixed at design time, but in actual use, almost every module is required to be customizable and replaceable (at the user level).

Solution (To be continued)

Lua source code series:

• Lua Source Code Reading Plan
• Lua Source Code Reading (Part 1)
• Lua Source Code Reading (Part 2)
• Lua Source Code Reading (Part 3)
• Lua Source Code Reading (Part 4)
• Lua Source Code Reading (Part 5)
• Lua Source Code Reading (Part 6)

After reading Yun Feng’s section on Lua strings, I have a corresponding understanding of Lua’s string handling.

Contents

String

Lua strings are internally divided into long strings and short strings. This classification is transparent at the Lua level and is likely just an optimization within the Lua interpreter.

Addendum¹: The maximum length of short strings is determined by the LUAI_MAXSHORTLEN macro. Long strings call createstrobj to create.

Long and short strings are handled differently during string creation and comparison.

When creating strings, short strings are directly internalized, and the extra bit in the TString structure marks whether it’s an internal reserved field. When creating long strings, memory is directly copied, and the extra bit is marked for lazy hashing during comparison or internalization.

When comparing strings, short strings directly compare pointer addresses. Long strings perform character-by-character comparison.

Addendum²: The long/short string optimization was added starting from Lua 5.2.1. Before that, all strings were stored in a global hash table.

Addendum³: When concatenating strings, using .. is inefficient as it involves memory allocation and memory copying. It’s recommended to use table.concat⁴ or string.format⁵.

userdata

Lua also has a UData structure called userdata. This data structure is mainly used when Lua interacts with C and C++, handing C data structures to Lua’s GC for management. Specifically, in C you can use lua_newuserdata() which creates a data structure in Lua and returns a pointer. This is similar to calling malloc, but the difference is you don’t need to manually call free to release memory - just leave it to Lua’s GC. A good example is Lua’s io library, which puts the C data structure FILE * into Lua’s userdata. By implementing a __gc metamethod, file handles can be automatically closed during GC.

https://github.com/xiaocang/lua-5.2.2_with_comments/releases/tag/lua_string_02

References:

Source Code Implementation of String Type in Lua

• Open hash table: Uses linked list storage, doesn’t produce clustering, but increases space overhead due to additional pointer fields.
• Closed hash table: Uses sequential storage, more storage-efficient, but prone to clustering, harder to implement lookups, requires secondary probing.
• Overflow table: A combination of open and closed hashing - the first sequential table stores pointer-like fields, the second stores overflow.

For detailed descriptions of specific algorithms for open hashing, closed hashing, etc., please refer to the original article.

Original article: http://blog.csdn.net/u014613043/article/details/50726630

Project at: https://github.com/xiaocang/perl-balancer

TODO: No XS version implementation yet

The phpUnit documentation is at https://phpunit.de/manual/current/en/index.html

create

# Create a test case
php artisan make:test XXControllerTest

# Create a unit test
php artisan make:test XXUnitTest --unit

usage

HTTP

• json
• get
• post
• put
• delete

ex:

public function testHttpSample()
{
    array $json = [];

    $this->json("PUT", $url, $json)
        ->assertExactJson(['errno' => 0]);
}

Available Assertions

Database

The database section is very detailed in the official documentation. The main assertions check whether a certain entry exists in the database, whether it doesn’t exist, and soft delete checks.

public function testDbSample()
{
    $this->assertDatabaseHas('sample_table', [
        'id' => '100',
        'data' => 'sample'
    ]);
}

Available Assertions

Mocking

The official documentation only provides a few examples in this section. In practical applications, I have more useful usages, such as the Log object:

public function testLogSample
{
    // This method can be used to simulate that after an error is triggered,
    // the error log printed in the code matches expectations,
    // and from this we can infer whether the expected error was triggered
    Log::shouldRecive('error')->once()->with('error happened');

    // run test...
    // ...
}

Guzzle

Guzzle, as an important module for making HTTP sub-requests, should also be covered by test cases in Laravel. Guzzle provides its own Mock method:

public function testGuzzleSample
{
    $mockResponses = [
        [
            200, // code
            ['Content-Type' => 'application/json'], // headers
            '{"errno": 0}' // body
        ]
        [
            500, // code
            ['Content-Type' => 'application/json'], // headers
            '{"errno": 99}' // body
        ]
    ];
    $mockHandler = new Client([
        'handler' => HandlerStack::create(
            (new MockHandler($mockResponses))
        )
    ]);
    $this->app->instance('GuzzleHttp\Client', $mockHandler);

    // run tests...
    // ...
    // During testing, guzzle will return the predefined responses in order to test expected cases.
}

Extended usage: To specifically test the input and output of sub-requests, you can write a simple MVC class that takes the expected sub-request input (url, method, header, body) to dynamically generate the $mockResponses array in the example.

env

In the Laravel project directory, there’s a phpunit.xml file where you can specify some environment variables to be used during test runs.

<?xml version="1.0" encoding="UTF-8"?>
<phpunit backupGlobals="false"
         backupStaticAttributes="false"
         bootstrap="bootstrap/autoload.php"
         colors="true"
         convertErrorsToExceptions="true"
         convertNoticesToExceptions="true"
         convertWarningsToExceptions="true"
         processIsolation="false"
         stopOnFailure="false">
    <testsuites>
        <testsuite name="Feature">
            <directory suffix="Test.php">./tests/Feature</directory>
        </testsuite>

        <testsuite name="Unit">
            <directory suffix="Test.php">./tests/Unit</directory>
        </testsuite>
    </testsuites>
    <filter>
        <whitelist processUncoveredFilesFromWhitelist="true">
            <directory suffix=".php">./app</directory>
        </whitelist>
    </filter>
    <php>
        <env name="APP_ENV" value="testing"/>
        <env name="CACHE_DRIVER" value="array"/>
        <env name="SESSION_DRIVER" value="array"/>
        <env name="QUEUE_DRIVER" value="sync"/>
        <env name="DB_HOST" value="127.0.0.1"/>
        <env name="DB_PORT" value="3306"/>
        <env name="DB_DATABASE" value="lavaral_test"/>
    </php>
</phpunit>

I forced the specification of the database address and name used during testing in the project to prevent someone from running this script in production with the .env file settings and operating on the production database.

Of course, the official configuration documentation mentions that during test runs, if a .env.testing file exists, when using the --env=testing parameter, .env.testing will override the .env file.

run

Running test cases is very simple - just execute the phpunit file in the project directory.

./vendor/bin/phpunit

phpunit also has some options available. The one I mainly use is --filter, to execute a specific test case or batch of test cases.

./vendor/bin/phpunit --filter=testHttp

Lua source code series:

• Lua Source Code Reading Plan
• Lua Source Code Reading (Part 1)
• Lua Source Code Reading (Part 2)
• Lua Source Code Reading (Part 3)
• Lua Source Code Reading (Part 4)
• Lua Source Code Reading (Part 5)
• Lua Source Code Reading (Part 6)

Contents

Memory Management

I just finished reading Yun Feng’s understanding of Lua’s memory management section, and I’ve also read and commented on the corresponding code myself. The main characteristics of Lua’s memory management are:

1. When the Lua VM initializes, it saves the main thread and global state on the first allocated memory block, which helps avoid memory fragmentation later
2. Lua uses wrapped memory management functions that always have the exact original memory size when allocating, resizing, or freeing memory, thus saving memory that would otherwise be used for cookies like in the standard library
3. Lua has wrapped memory management functions that can handle variable-length arrays, providing flexibility for data structures in the Lua language

https://github.com/xiaocang/lua-5.2.2_with_comments/releases/tag/lmem_01

Initialization

The global state section involves many subsequent parts like gc, string, table, etc., so I can only understand a little bit. Key points to understand:

• During Lua initialization, the main thread’s data stack is created differently from other threads’ data stacks
• The buff in the global state is used to handle string operations in Lua
• Lua implements a version check function to prevent multiple loading issues, which also shows that Lua was designed as an embedded language from the beginning
• For VM robustness, Lua needs to check whether memory allocation succeeds each time

https://github.com/xiaocang/lua-5.2.2_with_comments/releases/tag/global_state_02

1. Write separate GET/POST/PUT/DELETE endpoints for operations on each type of object, then add four corresponding endpoints for each new object type.
2. Have unified GET/POST/PUT/DELETE endpoints for all objects. If objects are related, use glue code to wrap these four endpoints for customized functionality.

Both design approaches are close to extremes. The Type 1 API that I mainly work on has a terrifying amount of work - I have to make thousands of lines of changes for each new requirement. This is “terrifying” level workload for me, while the API implementation looks simple and non-technical to others, and the deadlines are extremely tight.

So I started looking for a middle ground between these two API approaches. (To be continued)

Lua source code series:

• Lua Source Code Reading Plan
• Lua Source Code Reading (Part 1)
• Lua Source Code Reading (Part 2)
• Lua Source Code Reading (Part 3)
• Lua Source Code Reading (Part 4)
• Lua Source Code Reading (Part 5)
• Lua Source Code Reading (Part 6)

Using Lua in OpenResty projects is both enjoyable and frustrating. Lua’s minimalist design allowed me to build a CDN control system in just a few days two years ago, but problems emerged as the business system gradually took shape. Now I have two solutions before me:

1. Use a state machine - implement the core part of the framework with a state machine, changing the previous control flow into individual states
2. Use sandboxing - abstract new phases. Use preset sandbox environments to do specified things in each phase.

If using a state machine, it might greatly complicate the core part of the system, making it unmaintainable later. For sandboxing, the problem is that I only half-understand Lua’s underlying implementation. After seeing Yun Feng’s blog, I decided to take time to read through Lua’s source code to deepen my understanding of Lua. During the reading, I also hope to come up with an ideal Lua architecture for OpenResty projects.

Here’s Yun Feng’s Lua Source Code Appreciation

I’ll update here with some thoughts and knowledge gained after reading the Lua source code.

This is my project on GitHub with Chinese comments added to the lua-5.2.2 source code: https://github.com/xiaocang/lua-5.2.2_with_comments

Notes added on 2018-10-10

$ mencoder -really-quiet \
    /root/video/2159034.mp4 \
    -ss 0 -endpos 60 -of rawvideo \
    -ovc raw -nosound  -ofps 23  \
    -vf-add scale=404:720 \
    -vf-add format=i420 \
    -vf-add hqdn3d \
    -vf-add harddup \
    -o - 2>/dev/null | x264 \
    --no-progress --preset slower \
    --tune film --weightp 2 \
    --keyint 250 --min-keyint 25 \
    --non-deterministic --bframes 3 \
    --no-fast-pskip --qcomp 0.6 \
    --merange 24 --threads auto \
    --input-csp i420  --level 3.1 \
    --profile high --crf 20 \
    --input-res 404x720 \
    --fps 23 - --pass 1 \
    --stats \
    /tmp/video/1415072454_part1/3000/part_12159034/1415072454.passLog \
    -o /dev/null

$ mencoder -really-quiet \
    /root/video/2159034.mp4 \
    -ss 0 -endpos 60 -of rawvideo \
    -ovc raw -nosound  -ofps 23 \
    -vf-add scale=404:720 \
    -vf-add format=i420 \
    -vf-add hqdn3d \
    -vf-add harddup \
    -o - 2>/dev/null |x264 \
    --no-progress --preset slower \
    --tune film --weightp 2 \
    --keyint 250 --min-keyint 25 \
    --non-deterministic --bframes 3 \
    --no-fast-pskip --qcomp 0.6 \
    --merange 24 --threads auto \
    --input-csp i420  --level 3.1 \
    --profile high -B 3000 \
    --input-res 404x720 --fps 23 - --pass 2 \
    --stats /tmp/video/1415072454_part1/3000/part_12159034/1415072454.passLog \
    -o part1.264

However, I encountered a problem during transcoding. The individual .264 output files with MP4 container could play normally. But after merging the .264 files into one, seeking would cause visual artifacts.

$ MP4Box -add part1.264#video -cat part2.264 -fps 29.97 -add video.m4a#audio -new result.mp4

After re-transcoding multiple times with the same result, it became clear that simply appending files was causing the artifacts.

After Googling, I finally discovered that x264 by default optimizes the header of the output .264 file, which causes errors when merging .264 files. Simply adding the --stitchable option prevents the artifacts.

--stitchable        Don't optimize headers based on video content
                    Ensures ability to recombine a segmented encode

The result is as follows:

$ mencoder -really-quiet \
    /root/video/2159034.mp4 \
    -ss 0 -endpos 60 -of rawvideo \
    -ovc raw -nosound  -ofps 23  \
    -vf-add scale=404:720 \
    -vf-add format=i420 \
    -vf-add hqdn3d \
    -vf-add harddup \
    -o - 2>/dev/null | x264 \
    --no-progress --stitchable \
    --preset slower \
    --tune film --weightp 2 \
    --keyint 250 --min-keyint 25 \
    --non-deterministic --bframes 3 \
    --no-fast-pskip --qcomp 0.6 \
    --merange 24 --threads auto \
    --input-csp i420  --level 3.1 \
    --profile high --crf 20 \
    --input-res 404x720 \
    --fps 23 - --pass 1 \
    --stats \
    /tmp/video/1415072454_part1/3000/part_12159034/1415072454.passLog \
    -o /dev/null

$ mencoder -really-quiet \
    /root/video/2159034.mp4 \
    -ss 0 -endpos 60 -of rawvideo \
    -ovc raw -nosound  -ofps 23 \
    -vf-add scale=404:720 \
    -vf-add format=i420 \
    -vf-add hqdn3d \
    -vf-add harddup \
    -o - 2>/dev/null |x264 \
    --no-progress --stitchable \
    --preset slower \
    --tune film --weightp 2 \
    --keyint 250 --min-keyint 25 \
    --non-deterministic --bframes 3 \
    --no-fast-pskip --qcomp 0.6 \
    --merange 24 --threads auto \
    --input-csp i420  --level 3.1 \
    --profile high -B 3000 \
    --input-res 404x720 --fps 23 - --pass 2 \
    --stats /tmp/video/1415072454_part1/3000/part_12159034/1415072454.passLog \
    -o part1.264

You are invited to participate in the Windows Azure beta free trial operated by 21Vianet

I was lucky enough to get a Windows Azure beta trial VPS, and they didn’t specify an expiration date, so it looks like I can use it for free for at least three months.²

Of course, this VPS is on the domestic network, so “setting up a ladder” (VPN) is out of the question. But I can still host a blog on it.

So I migrated my Jekyll blog from GitHub Pages³ (at least I don’t need to start a VM to write blog posts - I can just log into the VPS directly).

Setting Up a Jekyll Environment

When creating the virtual machine on Windows Azure, I chose CentOS (mainly because I use it at work), then next, next… and the VM was created.

Then I opened an SSH terminal, entered the hostname, port, username, and password that I filled in when creating the VM, and logged into the virtual machine.

The first thing after logging in was to edit the /etc/yum.conf file:

[main]
cachedir=/var/cache/yum/$basearch/$releasever
keepcache=0
debuglevel=2
logfile=/var/log/yum.log
exactarch=1
obsoletes=1
gpgcheck=1
plugins=1
installonly_limit=5
bugtracker_url=http://bugs.centos.org/set_project.php?project_id=16&ref=http://bugs.centos.org/bug_report_page.php?category=yum
distroverpkg=centos-release
# exclude=kernel*
#  This is the default, if you make this bigger yum won't see if the metadata
# is newer on the remote and so you'll "gain" the bandwidth of not having to
# download the new metadata and "pay" for it by yum not having correct
# information.
#  It is esp. important, to have correct metadata, for distributions like
# Fedora which don't keep old packages around. If you don't like this checking
# interupting your command line usage, it's much better to have something
# manually check the metadata once an hour (yum-updatesd will do this).
# metadata_expire=90m

# PUT YOUR REPOS HERE OR IN separate files named file.repo
# in /etc/yum.repos.d

Comment out exclude=kernel*, otherwise you’ll get errors when installing packages.

Since Jekyll requires a Ruby environment, execute the following command in the terminal to install Ruby:

$ curl -L https://get.rvm.io | bash -s stable --ruby

For Jekyll installation, as stated in the official Jekyll documentation, execute in the terminal:

$ gem install jekyll

But when I ran this command, I found that gem wasn’t responding. I then ran:

$ gem install jekyll -V

I discovered that the lack of response was because the gem source was inaccessible due to domestic network restrictions (you know what I mean).

I immediately followed the documentation at http://ruby.taobao.org/, removed the inaccessible source, and updated to Taobao’s mirror.

$ gem sources --remove https://rubygems.org/
$ gem sources -a http://ruby.taobao.org/
$ gem sources -l
*** CURRENT SOURCES ***

http://ruby.taobao.org

Then I ran:

$ gem install jekyll

It was fast and smooth. ^_^

Postscript

But I still don’t know if I can point a domain without ICP registration to it… so I’m still using that long, ugly, and hard-to-remember xxxxxx.chinacloudapp.cn⁴… =w=

Notes added on 2018-10-10

So I found this site: bootsnipp¹

Notes added on 2018-10-10

The days without even WeChat² are finally gone forever, but is that really surprising?

Notes added on 2018-10-10

git fetch --all
git reset --hard origin/master

And I would get the result I wanted.

But today, when I repeated the same operation, an accident happened. A lot of useful data was deleted after this operation and couldn’t be recovered.

My Git environment was like this:

This was a system running in production that generates a lot of historical data in real-time in the same directory. However, due to the large data volume, it wasn’t added to git tracking. But the .gitignore file wasn’t properly written. And since it was a multi-person project, someone had executed git add --all at some point.

Just like that… precious historical data was deleted after an irreversible git reset --hard operation.

The data was never recovered…¹

So when managing projects with git, keep these points in mind:

1. Use git reset --hard with caution
2. Write a proper .gitignore file
3. Back up production data
4. Don’t add untracked files carelessly

Notes added on 2018-10-10

Use Case

After making a commit, you immediately notice that a line or two of code needs to be changed. After making the change, you commit again, then find something else that’s not right…

[root@addcp gitdemo]# vim test.txt
[root@addcp gitdemo]# git commit -am "add test.txt"

Found some code that needs minor changes:

[root@addcp gitdemo]# vim test.txt
[root@addcp gitdemo]# git commit -am "update test.txt"
...
...

Found issues for the Nth time:

[root@addcp gitdemo]# vim test.txt
[root@addcp gitdemo]# git commit -am "update test.txt again and again"
...

The Problem

After doing this multiple times, the git commit history becomes messy - a bunch of small changes taking up many commit entries:

[root@addcp gitdemo]# git log
commit abe26fe0f5b1788e8f7b1949082e1477c5337aa0
Author: xiaocang <xiaocang@addcp.com>
Date:   Sat Sep 7 01:29:48 2013 +0800

    update test.txt again and again

commit c3633d5153bacabeceaeaeb552484e7f9b6a2b9c
Author: xiaocang <xiaocang@addcp.com>
Date:   Sat Sep 7 01:29:21 2013 +0800

    update test.txt again

commit c8bc032daac7d0859a0e73cd83a464cee0f59625
Author: xiaocang <xiaocang@addcp.com>
Date:   Sat Sep 7 01:28:51 2013 +0800

    update test.txt

commit 0e80061e90503189dd7f189297aed809953e5e8b
Author: xiaocang <xiaocang@addcp.com>
Date:   Sat Sep 7 01:28:22 2013 +0800

    add test.txt
......

Rebase Example and Usage

Using git rebase to solve this problem is very easy.

The git rebase syntax is:

git rebase [--interactive | -i] [-v] [--force-rebase | -f] [--no-ff] [--onto <newbase>] [<upstream>|--root] [<branch>] [--quiet | -q]

Here we use the -i option, which is interactive rebase:

[root@addcp gitdemo]# git -i HEAD~3

Where HEAD~3 refers to the last three commits of this branch. You can also add a <branch> parameter to specify the branch to rebase.

pick c8bc032 update test.txt
f c3633d5 update test.txt again
f abe26fe update test.txt again and again

# Rebase 0e80061..abe26fe onto 0e80061
#
# Commands:
#  p, pick = use commit
#  r, reword = use commit, but edit the commit message
#  e, edit = use commit, but stop for amending
#  s, squash = use commit, but meld into previous commit
#  f, fixup = like "squash", but discard this commit's log message
#
# If you remove a line here THAT COMMIT WILL BE LOST.
# However, if you remove everything, the rebase will be aborted.

Where pick commits as-is.

reword also commits, but lets you edit the commit message before committing.

edit is similar to reword, but after editing the commit message, it waits for you to git commit --amend, meaning it waits for you to add some files and changes. Then use git rebase --continue to complete the entire rebase.

squash merges this commit into the previous one, but combines the log content of this commit into the previous one.

fixup differs from squash in that it automatically ignores this commit’s log content. This is exactly what I needed for this commit cleanup.

Rebase Result

As shown above, I get a log like this:

[root@addcp gitdemo]# git log
commit 3e5d79d149942f7dc0741a9a42ce43d98beafcd3
Author: xiaocang <xiaocang@addcp.com>
Date:   Sat Sep 7 01:28:51 2013 +0800

    update test.txt

commit 0e80061e90503189dd7f189297aed809953e5e8b
Author: xiaocang <xiaocang@addcp.com>
Date:   Sat Sep 7 01:28:22 2013 +0800

    add test.txt

Now everything is clean and tidy.

Note

Anyone who has read the official git documentation should know:

Do not rebase commits that have been pushed to a public repository

Because changing the commit history on a public repository is not good for a multi-person collaborative project.

Other people on the project will be confused because a commit that previously existed suddenly disappears.

So rebasing is mainly a method for cleaning up commit history before pushing to a public repository.

Notes added on 2018-10-10

In previous years, I would feel a bit sad because the vacation was ending.¹

This year, having just graduated, I still feel a bit sad though I don’t know why.²

Notes added on 2018-10-10

Plugins

After a quick Google search, I found relevant plugins:

• jekyll-only_first_p: A plugin that outputs only the first paragraph requires Nokogiri support
• excerpt.rb: A plugin that recognizes <!-- more -->

However, if your Jekyll is also hosted on GitHub, you can’t use plugins. For security reasons, GitHub runs Jekyll with the --safe parameter, rendering all third-party plugins ineffective.

Liquid

Just when I was feeling a bit disappointed, I found this article: Post excerpts in Jekyll²

You can achieve this using a filter in the Liquid template language:

{{ post.content | split: "<!-- more -->" | first }}

Then add a Read More link after the truncated article:

<a href="#more>Read More →</a>

This way, even Jekyll hosted on GitHub can have truncated output.³

Notes added on 2018-10-10