月度归档： 2014年12月

Freeshell 添加 IPv6 域名反解析

现在 ping/traceroute freeshell 的 IPv6 地址，可以反解析出域名了。

$ traceroute servmon.6.freeshell.ustc.edu.cn
traceroute to servmon.6.freeshell.ustc.edu.cn (2001:da8:d800:71::987), 30 hops max, 80 byte packets
1 2001:da8:d800:f001::1 (2001:da8:d800:f001::1) 0.662 ms 0.778 ms 0.892 ms
2 2001:da8:d800:ffff::4 (2001:da8:d800:ffff::4) 1.127 ms 1.181 ms 1.409 ms
3 s7.freeshell.ustc.edu.cn (2001:da8:d800:71:1::7) 0.648 ms 0.619 ms 0.609 ms
4 servmon.6.freeshell.ustc.edu.cn (2001:da8:d800:71::987) 0.722 ms 0.714 ms 0.705 ms

这个功能的主要用途是确定 freeshell 某个 IPv6 地址的主机名，比如这样：

C:\Users\Bojie>tracert 2001:da8:d800:71::2081

Tracing route to wheezy.6.freeshell.ustc.edu.cn [2001:da8:d800:71::2081]
over a maximum of 30 hops:

1 2 ms 2 ms 5 ms 2001:da8:d800:f001:176:99:0:1
2 2 ms 4 ms 2 ms 2001:da8:d800:f001::1
3 3 ms 3 ms 2 ms 2001:da8:d800:ffff::4
4 3 ms 4 ms 3 ms 2001:da8:d800:70:202:38:70:5
5 2 ms 3 ms 3 ms wheezy.6.freeshell.ustc.edu.cn [2001:da8:d800:71::2081]

Trace complete.

如何修复受损的 MySQL InnoDB 数据库

Blog 服务器文件系统损坏后，由于日志与数据库文件不一致，数据库无法启动。事实上，当时有一个简单粗暴的方法可以启动数据库：修改 /etc/mysql/my.cnf，增加如下的配置：

[mysqld]
innodb_force_recovery = 6

启动后的数据库是只读模式，可以用 mysqldump 命令把数据库 dump 出来，绝大部分数据是完好的。也就是说，文件系统损坏后的数据库，以及使用 rsync 在线备份出来的数据库，尽管都是处于不一致状态的，但是可以忽略日志部分，恢复其大部分数据的。

恢复 blog 的时候，我也查到了 innodb_force_recovery 选项，但只看到了设置它为 1（文档的开头部分），发现启动不起来就没有办法了。事实上这个选项可以设置为 1~6 的不同级别，表示跳过 innodb 数据库恢复的不同步骤。经测试，该值设为 1~5 都不行，只有设置为 6，即完全忽略 redo 和 undo 日志，才能成功启动数据库，也就是 rsync 造成的不一致是比较严重的。

参见 MySQL 官方文档：Forcing InnoDB Recovery。

教训：

  1. 看文档要看完整。
  2. 数据库不能使用 rsync 来同步，而要使用 mysqldump 全量导出、DRBD（块设备级别的同步）或者数据库主从复制。

Gitlab升级至7.6.1

本次维护历时42分钟，服务共中断26分钟。

官方提供的升级程序没有考虑二进制依赖问题，导致bundle install时出错，因此手动安装了依赖包。然而，重新运行升级程序时，检测到已是新版本（虽然没有更新完），abort了，只好手动进行更新操作，耽误了不少时间。

7.6.1 主要特性：

  • 支持将项目Fork到某个组中
  • Monokai高亮更加贴近原始配置
  • 允许为已关闭的issue创建Milestones或Labels
  • 移动端UI改进
  • 多项细节改进
  • 多项bug修正

本次维护，为GitLab服务器添加了VPN线路，以加速外部项目的导入速度。

LUG 权威 DNS 修复 IPv6

12 月 25 日下午，网络中心把 LUG 权威 DNS 服务器的上游网关设备从西区迁移到了东区，网关的 link-local IPv6 地址和 MAC 地址发生了改变。但由于该服务器的 IPv6 地址是静态配置的，使用了 link-local 地址作为网关，该地址已经不可用，因此权威 DNS 的 IPv6 网关中断。

iface eth0 inet6 static
        address 2001:da8:d800:f001::5353
        gateway fe80::212:44ff:fe66:2000
        netmask 64

获取不到网关 MAC 地址的情况：

boj@dns:~$ ip -6 neigh
fe80::250:56ff:fe9f:4 dev eth0 lladdr 00:50:56:9f:00:04 STALE
2001:da8:d800:f001:250:56ff:fe9f:4 dev eth0 lladdr 00:50:56:9f:00:04 STALE
fe80::72ca:9bff:fece:31a2 dev eth0 lladdr 70:ca:9b:ce:31:a2 STALE
fe80::212:44ff:fe66:2000 dev eth0  router FAILED

现在 /etc/network/interfaces 中修改了 IPv6 gateway 为 2001:da8:d800:f001::1。并执行如下命令（远程机器不要轻易执行 service networking restart，网络起不来就悲剧了）：

ip -6 route add default via 2001:da8:d800:f001::1
ip -6 route del default via fe80::212:44ff:fe66:2000

几秒钟后就通过 ICMPv6 协议获取到了网关的 MAC 地址，服务器的 IPv6 连接恢复正常。

boj@dns:~$ ip -6 neigh
2001:da8:d800:f001:225:90ff:fe32:a676 dev eth0 lladdr 00:25:90:32:a6:76 REACHABLE
fe80::250:56ff:fe9f:4 dev eth0 lladdr 00:50:56:9f:00:04 STALE
2001:da8:d800:f001:250:56ff:fe9f:4 dev eth0 lladdr 00:50:56:9f:00:04 STALE
fe80::212:44ff:fe66:2000 dev eth0  router FAILED
fe80::225:90ff:fe32:a676 dev eth0 lladdr 00:25:90:32:a6:76 REACHABLE
2001:da8:d800:f001::1 dev eth0 lladdr 5c:dd:70:91:72:e2 router REACHABLE

此外，之前 /etc/resolv.conf 的内容如下，当 IPv6 不通时，前两个 nameserver 都会解析失败，这导致 LDAP 客户端无法解析 ldap.lug.ustc.edu.cn 的域名，也就是用不了 sudo 了。

search lug.ustc.edu.cn
nameserver 2001:da8:d800::1
nameserver 2001:da8:d800::56
nameserver 202.38.64.1

采用本地账号 sudo，把 nameserver 202.38.64.1 移到了 IPv6 之前，以免 IPv6 再出现不稳定的情况。

Freeshell HTTP Proxy 支持 WebSocket

Freeshell HTTP Proxy 支持 WebSocket 的反向代理了，只需在控制面板中启用 WebSocket，如下图所示：

启用 WebSocket 后，正常的 HTTP(S) 反向代理仍然能使用。但 freeshell 内的 Web 服务器必须支持 HTTP/1.1 版本（绝大多数 Web 服务器都支持的），HTTP/1.0 将不能工作。

事实上昨天晚上 freeshell 就已经上线了这个功能，但由于配置错误不能工作。今天发现，问题是 proxy_set_header Upgrade 和 proxy_set_header Connection 写在了 location / 外面。我本来指望 location 里会把里面和外面的配置 “合并” 起来，但事实上里面的配置会覆盖掉外面的配置，如官方文档所述：

These directives are inherited from the previous level if and only if there are no proxy_set_header directives defined on the current level.

感谢 Zihao Wang 的建议。

Merry Christmas!

GitLab服务器维护说明

由于最近半年以来GitLab服务器压力一直比较大，近几个月我们被迫关闭了HTTP打包功能。在boj的积极联系下，GitLab服务器增加到4核心，服务器响应时间明显缩短。现在已经重新开启HTTP打包服务，同时调整了部分服务的配置。

此次维护修改了以下配置：

  • 重新开启GitLab HTTP打包下载功能
  • 修改nginx工作进程数为4（原始值为1）
  • 修改unicorn工作进程数为4（原始值为1）

为了使新配置生效，2014年12月18日重启了nginx、unicorn。

 

USTC Blog is back online after two days’ downtime

Welcome back,

Due to a file system corruption at 01:30 AM Thursday (2014-12-18), the USTC Blog server is down for maintainence. We have investigated into this issue and restored the last good backup at 02:00 AM Wednesday (2014-12-17). We are sorry that all changes to your blog since Wednesday has been rolled back and would never be replayed. The good news is, your blog is back online now (that’s the point of this email).

We sincerely apologize for the inconvenience this service downtime has brought to you in the past 48 hours, which is the longest downtime throughout USTC Blog history. We will file a report on this incident on https://servers.blog.ustc.edu.cn/ in a few days.

Should you have any problem with the reborn blog system, please feel free to contact us: support (at) blog.ustc.edu.cn

Enjoy USTC Blog!

USTC Freeshell is back online after two days’ downtime

Welcome back,

Due to a file system corruption at 01:30 AM Thursday (2014-12-18), the USTC Freeshell Controller server is down for maintainence. We have investigated into this issue and restored the last good backup at 02:00 AM Wednesday (2014-12-17). The good news is, the freeshell control panel is back online since 02:00 AM Saturday (2014-12-20). (That’s the point of this email.)

During this major service corruption, actually your freeshell is still running and can be accessed through IPv6. However, the following components stopped working in the past 48 hours:

  • Web control panel (https://freeshell.ustc.edu.cn)
  • Public endpoints (IPv4 port mapping, including IPv4 SSH)
  • HTTP(S) proxy
  • Outbound connections from freeshell (fixed on Friday afternoon, a bit earlier)

We sincerely apologize for the inconvenience this service downtime has brought to you in the past 48 hours, which is the longest downtime throughout USTC Freeshell history. We will file a report on this incident on https://servers.blog.ustc.edu.cn/ in a few days.

Should you have any problem with the reborn freeshell system, please feel free to contact us: support (at) freeshell.ustc.edu.cn

Enjoy USTC Freeshell!

LUG 邮件服务器添加 DKIM 验证

为了减少 LUG 发出的邮件被误判为垃圾邮件，参考 How To Install and Configure DKIM with Postfix on Debian Wheezy，使用 OpenDKIM 为 {blog,freeshell,lug}.ustc.edu.cn 外发的邮件添加了 DKIM 验证。

DKIM 的原理是在每封外发的邮件上附加签名，而这个签名所使用的公钥可以通过查询 mail._domainkey.{blog,freeshell,lug}.ustc.edu.cn 域名的 TXT 记录得到。只要 DNS 系统没有问题，经过了签名的邮件就能确保是该域名的所有者发出的。

感谢 常震 的建议。

Tip: 如何测试你的邮件服务器正确配置了 DKIM？可以用邮件服务器向 [email protected] 发送邮件来检测，你会收到一封测试结果的邮件。看 DKIM check 部分就行。如下是 lug.ustc.edu.cn 的测试结果（DomainKeys 是平行于 DKIM 的另一套签名机制，暂不准备添加）。

==========================================================
Summary of Results
==========================================================
SPF check:          pass
DomainKeys check:   neutral
DKIM check:         pass
Sender-ID check:    pass
SpamAssassin check: ham

==========================================================
Details:
==========================================================

HELO hostname:  blog.ustc.edu.cn
Source IP:      128.199.232.134
mail-from:      [email protected]

----------------------------------------------------------
SPF check details:
----------------------------------------------------------
Result:         pass
ID(s) verified: [email protected]
DNS record(s):
    lug.ustc.edu.cn. SPF (no records)
    lug.ustc.edu.cn. 600 IN TXT "v=spf1 mx a a:ip-list.vpn.ustclug.org ~all"
    lug.ustc.edu.cn. 600 IN MX 5 blog.ustc.edu.cn.
    blog.ustc.edu.cn. 600 IN A 202.141.176.99
    lug.ustc.edu.cn. 600 IN A 202.141.162.123
    ip-list.vpn.ustclug.org. 28 IN A 128.199.232.134
    ip-list.vpn.ustclug.org. 28 IN A 202.38.93.95
    ip-list.vpn.ustclug.org. 28 IN A 128.199.170.5
    ip-list.vpn.ustclug.org. 28 IN A 128.199.211.154
    ip-list.vpn.ustclug.org. 28 IN A 202.141.176.99
    ip-list.vpn.ustclug.org. 28 IN A 202.141.160.99
    ip-list.vpn.ustclug.org. 28 IN A 128.199.161.228

----------------------------------------------------------
DomainKeys check details:
----------------------------------------------------------
Result:         neutral (message not signed)
ID(s) verified: [email protected]
DNS record(s):

----------------------------------------------------------
DKIM check details:
----------------------------------------------------------
Result:         pass (matches From: [email protected])
ID(s) verified: header.d=lug.ustc.edu.cn
Canonicalized Headers:
    date:Mon,'20'8'20'Dec'20'2014'20'00:17:17'20'+0800'20'(CST)'0D''0A'
    from:[email protected]'20'(Bojie'20'Li)'0D''0A'
    dkim-signature:v=1;'20'a=rsa-sha256;'20'c=relaxed/simple;'20'd=lug.ustc.edu.cn;'20's=mail;'20't=1417969044;'20'bh=z6TUz85EdYrACGMHYgZhJGvVy5oQI0dooVMKa2ZT7c4=;'20'h=Date:From;'20'b=

Canonicalized Body:
    Hello'20'world!'0D''0A'


DNS record(s):
    mail._domainkey.lug.ustc.edu.cn. 600 IN TXT "v=DKIM1; k=rsa; p=MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQC8MTDW2coES4+fAOLSTBP+0hevVGZwp9ecZnQMpBSBWAcAZ1KiMGmJoM0yDx1Gst4UGz3IXc8uqstSki5mdgpUWONt0zz2Kxr/6zzMu+C8ySiBWPoMdBbXHnfQQ1GisweivhChgxx0MuyL9CylQGcthF9Hu2kMy/4cV3REtg+H3QIDAQAB"

Public key used for verification: mail._domainkey.lug.ustc.edu.cn (1024 bits)

NOTE: DKIM checking has been performed based on the latest DKIM specs
(RFC 4871 or draft-ietf-dkim-base-10) and verification may fail for
older versions.  If you are using Port25's PowerMTA, you need to use
version 3.2r11 or later to get a compatible version of DKIM.

----------------------------------------------------------
Sender-ID check details:
----------------------------------------------------------
Result:         pass
ID(s) verified: [email protected]
DNS record(s):
    lug.ustc.edu.cn. SPF (no records)
    lug.ustc.edu.cn. 600 IN TXT "v=spf1 mx a a:ip-list.vpn.ustclug.org ~all"
    lug.ustc.edu.cn. 600 IN MX 5 blog.ustc.edu.cn.
    blog.ustc.edu.cn. 600 IN A 202.141.176.99
    lug.ustc.edu.cn. 600 IN A 202.141.162.123
    ip-list.vpn.ustclug.org. 28 IN A 128.199.232.134
    ip-list.vpn.ustclug.org. 28 IN A 202.38.93.95
    ip-list.vpn.ustclug.org. 28 IN A 128.199.170.5
    ip-list.vpn.ustclug.org. 28 IN A 128.199.211.154
    ip-list.vpn.ustclug.org. 28 IN A 202.141.176.99
    ip-list.vpn.ustclug.org. 28 IN A 202.141.160.99
    ip-list.vpn.ustclug.org. 28 IN A 128.199.161.228