The burning question, I can hear it now:
Why not just use ChaCha20-poly1305? Its standard, and its good enough for <insert application here>? Why roll your own crypto?
The reason for this gets at a fundamental property of polynomial MACs1 that opens them up to all kinds of fun attacks, like the partitioning oracle2, you didn’t even know you needed to worry about. Polynomial MACs violate intuitive expected properties of cryptosystems, and are so finicky to work with that even the specification for AES-GCM contained invalid proofs of its security3.
There is a not-infrequently important property of cryptosystems, referred to as “commiting to their keys”, or simply “being commiting”4. Simply put, a system has this property when it is impractical to produce a message that will successfully decrypt or verify under multiple chosen keys. As a fundamental result of their construction, polynomial MACs lack this property, and its even worse than simply being able to construct a message that will decrypt under two chosen keys. Even for the more well behaved polynomial MACs, such as Poly1305, it is not only practical to construct garden variety collisions, it is practical to construct multi-collisions that will decrypt under a large number of chosen keys5 .
This lack of even a facade of collision resistance can actually turn into quite a big deal, just as Mega6 famously experienced a loss of security due to their inappropriate use of CBC-MAC (another type of MAC that also lacks collision resistance) polynomial MACs, in practice, worm their way into all kinds of places they shouldn’t be, places where a collision resistant MAC would have been the only appropriate choice. Even in the context of more seemingly sane systems, this lack of collision resistance can cause no end of problems, such as the partitioning oracle attack 1, which was practically exploitable for over the network password recovery in Shadowsocks, and the always interesting Invisible Salamander7.
These issues alone are, personally, enough to turn me off from using poly1305 or any other polynomial MACs in any of my future projects, given that it’s 2021 and hashes are fast now, but, it is my opinion that polynomial MACs being non-collision resistant represents a huge foot gun, and as I am not a fan of handing foot guns to the consumers of my code, I will refrain from offering this one, and provide a committing, collision resistant scheme instead.
This one should be a little bit easier to answer, its not exactly a hot take to say that AES is a lot closer to the bottom of the acceptable ciphers barrel than ChaCha208. Even discounting the standard complaints, like how AES’s construction is somewhat-intrinsically vulnerable to cache based timing attacks when implemented in software9 (though bit-slicing and other tricks mitigate this, they are surprisingly rare to see in the wild, even in software running on devices that lack hardware accelerated AES), and how the block size is too small10, AES still has other concerning factors in its corner, like how the PRP-PRF distinguishing attack reduces the effective security of AES in counter mode.
AES just, all around, isn’t ideal. Sure, ChaCha is naturally vulnerable to electromagnetic side channel attacks11, which have been an item of, ehm, increasing concern for me12, AES is just as vulnerable in that regard, with even side-channel resistant hardware implementations being a rare thing, and for most practical purposes, only existing in the literature13. ChaCha20’s much larger block size (512-bit) and more timing side channel resistant construction just lends itself to fewer foot guns, and while hardware assisted AES is a bit faster, I don’t think there is enough of a speed differential to warrant the loss of misuse resistance in most applications.
This is not intended to be an indictment of any particular protocol, while the issues I’ve brought up are real, and can result in vulnerabilities in the real world, none of them are automatically going to lead to an exploitable vulnerability. The people making the systems you rely on to keep your private data safe by and large understand the state of the research on the matter, and are careful to design systems to avoid the foot guns. Your AES-GCM or ChaCha20-Poly1305 TLS stream is not in immediate peril.
Cryptographic primitives don’t exist in a vacuum, any analysis of vulnerabilities must be done on complete systems, as decisions on any level above the choice of fundamental primitives may sink or save the ship14.
My interest in writing this post is in informing the design of new systems, where I much prefer foot guns be avoided categorically instead of case-by-case, when practical, and I believe the time has come where the use of cryptosystems that categorically avoid these foot guns is practical in all but niche edge case scenarios.
ChaCha20 has a number of benefits over AES, and I don’t really feel the need to go too deep into them, but lets give the overview. ChaCha20 is a stream cipher, but it acts like a block cipher being used in CTR mode. In this respect, it has two major advantages over AES-CTR:
ChaCha20 also has other advantages, like being more energy efficient when dedicated hardware isn’t available, and being naturally resistant to timing side channel attacks. The only real disadvantage is that it is slower than commonly available hardware accelerated AES, but not by an amount that I think makes a critical difference for all but the pickiest of applications.
The choice of XChaCha20 over that of plain ChaCha20 is also easy to explain. ChaCha20 only gives you a 64 bit nonce, which, while enough to encrypt the world with a given key, means that choosing a nonce at random can be incredibly dangerous, even AES-GCM’s 96-bit nonce is much too small, leading to concrete failures in production15, and even the 128-bit nonce of straight AES-CTR leaves me feeling uneasy, the birthday bound there is a lot lower than it feels like it is. XChaCha20’s choice of a 192 bit nonce, however, leaves plenty of headroom to feel safe, even when randomly selecting nonces, taking another foot gun out of the equation.
This is another easy one to answer. The biggest reason, is, well Blake3 is fast, several times faster than ChaCha20 on my machine, and having an HMAC that can out run your cipher several times over is always handy when building HMAC based cryptosystems. In my opinion having a hash, like Blake3, that’s truly fast changes the trade off analysis, making it much less worth the trade off of using a polynomial MAC.
Blake3 also has some other nice properties, its a modern, secure, length-extension resistant hash based on a merkle tree construction, which provides lots of other side benefits, such as the ability to use the hash for verified streaming. It additionally has the nice property of not really having variants, with the HMAC, KDF, hash, and XOF modes all operating in a very similar and consistent manner, easing the cognitive load on the programmer.
Being effectively an HMAC, Blake3 also provides a scheme that commits to its keys, almost for free as a side effect of collision resistance, directly sidestepping my concerns about polynomial MACs.
I believe that using XChaCha20-Blake3 in the encrypt-then-hmac construction provides a more-than-fast-enough base to build more than useable cryptosystems on top of, and acts as a primitive for doing so that presents substantially fewer foot guns than other competing options, while still remaining more than useably fast, even in the absence of dedicated hardware acceleration. Humans are such fallible creatures, and accidental de-footings are much less likely to happen when we are provided fewer chances to do so.
A family of very closely related Message Authentication Codes, including AES-GCM’s GHASH, AES-GCM-SIV’s Polyval, and Poly1305 ↩ ↩2
Julia Len, Paul Grubbs and Thomas Ristenpart: Partitioning Oracle Attacks (USENIX ‘21) ↩
Tetsu Iwata, Keisuke Ohashi, Kazuhiko Minematsu: Breaking and Repairing GCM Security Proofs (CRYPTO 2012) ↩
Kryptos Logic: Faster Poly1305 key multicollisons ↩
Yevgeni Dodis, Paul Grubbs, Thomas Ristenpart, Joanne Woodage: Fast Message Franking: From Invisible Salamanders to Encryptment (CRYPTO 2018) ↩
Source: My Ass ↩
Daniel J. Bernstein: Cache-timing attacks on AES ↩
Even AES256 has a block size of 128 bits, which means you can start expecting collisions after 2^64 encryptions. This sounds like a lot, and the exact details of how this can cause a mode of operation to break down depend on the particular mode, but that’s a not an unachievable amount of data, the internet does that much every few months (assuming 1 block per encryption), and the security can start to break down well before the 2^64 mark. https://www.youtube.com/watch?v=v0IsYNDMV7A ↩
Alexandre Adomnicai, Jacques J.A. Fournier, Laurent Masson: Bricklayer Attack: A Side-Channel Analysis on the ChaCha Quarter Round ↩
Giovanni Camurati, Sebastian Poeplau, Marius Muench, Tom Hayes, Aurélien Francillon: Screaming Channels: When Electromagnetic Side Channels Meet Radio Transceivers ↩
Raghavan Kumar, Vikram Suresh, Monodeep Kar, et al: A 4900-μm2 839-Mb/s Side-Channel Attack-Resistant AES-128 in 14-nm CMOS With Heterogeneous Sboxes, Linear Masked MixColumns, and Dual-Rail Key Addition ↩
One good example of it saving the ship is HMAC-MD5. While the MD5 hash has been blown pretty widely open, the common HMAC construction gets many of its security guarantees from the first-preimage resistance of the underlying hash, and as there are no known practical first preimage attacks against MD5, HMAC-MD5 remains largely secure (please do not use it for new systems though) ↩
Hanno Böck, Aaron Zauner, Sean Devlin, Juraj Somorovsky, Phillip Jovanovic: Nonce-Disrespecting Adversaries: Practical ↩
Installation on my arch linux system was quite simple, but things turned out to be a bit less simple on macOS.
I initially tried to use this guide, but needed to perform several tweaks to get things working.
The gist suggests installing gcc from homebrew, with the following
modification to
/usr/local/Homebrew/Library/Taps/homebrew/homebrew-core/Formula/gcc.rb
:
diff --git a/Formula/gcc.rb b/Formula/gcc.rb
index 1bd636d496..03ad124218 100644
--- a/Formula/gcc.rb
+++ b/Formula/gcc.rb
@@ -53,7 +53,7 @@ class Gcc < Formula
# - Ada, which requires a pre-existing GCC Ada compiler to bootstrap
# - Go, currently not supported on macOS
# - BRIG
- languages = %w[c c++ objc obj-c++ fortran]
+ languages = %w[c c++ objc obj-c++ fortran jit]
osmajor = `uname -r`.split(".").first
pkgversion = "Homebrew GCC #{pkg_version} #{build.used_options*" "}".strip
@@ -73,6 +73,7 @@ class Gcc < Formula
--with-system-zlib
--with-pkgversion=#{pkgversion}
--with-bugurl=https://github.com/Homebrew/homebrew-core/issues
+ --enable-host-shared
]
# Xcode 10 dropped 32-bit support
and then simply running brew install gcc --build-from-source --force
.
This didn't quite work to start with, homebrew, by default, will want
to "update" the gcc formula back to the original, however, this can be
fixed by simply running
env HOMEBREW_NO_AUTO_UPDATE=1 brew install gcc --build-from-source --force
instead.
The guide also neglected to mention that you need jansson
installed
for the --with-json
configure option to work. The version of jansson
provided by brew install jansson
The gist is, additionally, out of date, so I had to edit its references
to gcc 10.1.0 to 10.2.0. It also additionally assumes that CC
is set
to the homebrew-installed gcc. I personally rectified that by manually
pointing my gcc symlink to the homebrew provided gcc.
Attempting to compile with the provided build.sh
also additionally
caused the configure script to scream about AppKit.h
being available,
but not usable. This was rectified by running temporarily setting CC
to /usr/bin/clang
while running the configure script.
After the build and install completed, emacs was complaining about not
being able to find eln
files, which I was able to fix, somewhat
jankily, by copying the native-lisp
directory in the build folder to
/usr/local
.
All-in-all, this is the final version of the modified build.sh
that I
wound up using:
#!/bin/sh
# native-comp optimization
export PATH="/usr/local/opt/gnu-sed/libexec/gnubin:${PATH}"
export CFLAGS="-I/usr/local/Cellar/gcc/10.2.0/include -O2 -march=native"
export LDFLAGS="-L/usr/local/Cellar/gcc/10.2.0/lib/gcc/10 -I/usr/local/Cellar/gcc/10.2.0/include"
export LIBRARY_PATH="/usr/local/Cellar/gcc/10.2.0/lib/gcc/10:${LIBRARY_PATH:-}"
cd emacs || exit
git clean -xfd
echo ""
echo "autogen"
echo ""
./autogen.sh
echo ""
echo "configure"
echo ""
export CC=/usr/bin/clang
./configure \
--disable-silent-rules \
--enable-locallisppath=/usr/local/share/emacs/28.0.50/site-lisp \
--prefix=/usr/local/opt/gccemacs \
--without-dbus \
--without-imagemagick \
--with-mailutils \
--with-ns \
--with-json \
--disable-ns-self-contained \
--with-cairo \
--with-modules \
--with-xml2 \
--with-gnutls \
--with-rsvg \
--with-nativecomp
read -p "Press any key to resume ..."
# Ensure /usr/local/opt/gccemacs exists
rm -rf /usr/local/opt/gccemacs
mkdir /usr/local/opt/gccemacs
# Ensure the directory to which we will dump Emacs exists and has the correct
# permissions set.
libexec=/usr/local/libexec/emacs/28.0.50
if [ ! -d $libexec ]; then
sudo mkdir -p $libexec
sudo chown $USER $libexec
fi
echo ""
echo "make"
echo ""
export CC=gcc
make -j6
make install
rm -rf "/Applications/Gccemacs.app"
mv nextstep/Emacs.app "/Applications/Gccemacs.app"
rm -rf /usr/local/native-lisp/
cp -R native-lisp /usr/local
cd /usr/local/bin || exit
rm gccemacs
rm gccemacsclient
ln -s /usr/local/opt/gccemacs/bin/emacs ./gccemacs
ln -s /usr/local/opt/gccemacs/bin/emacsclient ./gccemacsclient
cd /Applications/Gccemacs.app/Contents || exit
ln -s /usr/local/opt/gccemacs/share/emacs/28.0.50/lisp .
I, personally, decided to name my binaries as gccemacs
, to allow
coexistence with regular emacs.
The first start on Gcc Emacs is, well, brutal.
I don't know exactly how long it took to compile all my packages, as I wound up leaving my desk to do some things around the house, and came back an hour later to find it complete.
After the initial cost of compiling everything is paid, emacs is notably
snappier. My init time has dropped from 5.2 seconds to 4.1 seconds, and
there is no longer a noticeable hangup when a lazy loaded package kicks
in, and commands that were once painful to run, such as
helm-org-rifle
, are now much smoother and I no longer experience a
noticeable hang upon running them. I haven't actually experienced a
momentary hang since switching over.
I have not noticed any behavioral difference (besides speed) over
running regular emacs. straight.el
's native-comp support just
worked.
Gcc Emacs is going excellent so far, and I hope its path to master is short and pleasant.
]]>There is some contention over whether XDG style directories (e.g.
~/.config/
{.verbatim}) or macOS default style directories should be
used (e.g. ~/Library/Application Support
{.verbatim}).
Uses macOS default paths, actively maintained, in the 360 most popular python packages. Does not respect XDG dirs environment variables on macOS
Seems to just always fall back to XDG dirs, without attempting to use macOS default paths. Doesn't even support windows paths. It will, however, respect the XDG environment variables if set.
There is a smaller appdirs that is a direct port of the python version, with the same behavior as the python version, but nobody seems to use it (~7,000,000 vs 102 weekly downloads)
Defaults to macOS default directories, but appears to respect XDG variables when set.
There is also the slightly more popular BurntSushi xdg, which seems to also default to XDG paths.
Not super popular, but most popular one I could find. Defaults to macOS default directories, does not seem to respect XDG variables on macOS.
Seems to always follow the XDG standard.
Summary of behavior on macOS:
|----------+----------+------------+------------------|
| Language | XDG Dirs | macOS Dirs | Respects XDG Env |
|----------+----------+------------+------------------|
| Python | | x | no |
|----------+----------+------------+------------------|
| NodeJS | x | | yes |
|----------+----------+------------+------------------|
| Golang | | x | yes |
|----------+----------+------------+------------------|
| Java | | x | no |
|----------+----------+------------+------------------|
| Ruby | x | | yes |
|----------+----------+------------+------------------|
In my experience, most of the command line applications I have on my mac
(which I essentially only use for testing my software on macOS). All of
my applications appear to be using either XDG base directories, or
traditional ~/.appname
{.verbatim} folders.
directories
Currently, directories
is returning ~/Library/Preferences
{.verbatim}
as the config directory. This is, at best, incorrect, as the apple
documentation (and inspecting the current state of my mac) indicate that
this directory should only be used for macOS managed plists.
Unfortunately, by macOS standards, the config directory
(~/Library/Application Support
{.verbatim}) happens to be the same as
the data directory, so 'fixing' this bug may cause issues with
dependent code assuming that the data and config directories are
separate folders, when on macOS, they will not be. This, at the very
least, needs a callout and some examples in the documentation/readme.
In my opinion, keeping things in line with the apple way, the correct path moving forward is as follows:
~/Library/Application Support
{.verbatim} by defaultIn a practical sense, this release mostly signifies that the core API has settled down enough that I am no longer scared of adding new, more complicated, (and more powerful) features.
That doesn't mean asuran is completely useless though, it already has many of the core features you would come to expect from an archiver. It is actually functional, and can make and restore archives. It even has several knobs for you to tweak!
While I will try to avoid changes that break backwards compatibility, and will likely entirely avoid changes that break the ability to read existing repos until at least 0.2.0, there are currently no promises about breaking API changes. I will try to keep these to a minium, but asuran is still experimental software.
On that note, please do not use this as an excuse to rely solely on asuran. It will eat your laundry if you do so.
Asuran is currently continuously tested on 64bit linux, macos, and windows. All of these are first class support targets, and any test failures on any of them will hold up a release.
Test coverage is decent at the moment but not great. I would like to get automated fuzz testing setup for at least linux in the very near future.
Asuran can create, store, list, and extract archives from within a repository on the local filesystem, with a few other niceties.
Take a look at the output of asuran-cli --help
:
Asuran-CLI 0.1.0-6405ca0 2020-04-23
Nathan McCarty <nathan@mccarty.io>
Indicates which subcommand the user has chosen
USAGE:
asuran-cli [FLAGS] <SUBCOMMAND>
FLAGS:
-h, --help Prints help information
-q, --quiet Squelch non-logging operations
-V, --version Prints version information
SUBCOMMANDS:
bench-crypto Runs benchmarks on all combinations of asuran's supported crypto primitives
contents Lists the contents of an archive, with optional glob filters
extract Extracts an archive from a repository
help Prints this message or the help of the given subcommand(s)
list Provides a listing of the archives in a repository
new Creates a new repository
store Creates a new archive in a repository
Now that I am confident the core repository API isnt going to be changing much, its time to work on the more exciting features.
First in line is the S3 backend, I have begun working on it in earnest. Following that will likely be the ability to push/pull archives between repositories.
Another important step I will be taking in the near future will be to provide hostname/machineid based caching of the last times files were modified, so that reprocessing of files can be avoided when possible.
If you are interested in participating, please do!
Take a look at our CONTRIBUTING.md an issue, or join our matrix chat.
]]>I recently purchased a new air filter from costco, and its actually a pretty nice unit:
Only one problem. For some ungodly reason, its an internet of shit device. It has built in wifi and an android app that both kinda seriously sucks and provides functionality not readily accessible through the physical controls on the air filter. This is unacceptable for a number of reasons.
In this series of posts, I will explore the (ongoing) process of reverse engineering my air filter, and hopefully figuring out how to pop a custom firmware on it.
First, I want to explore what we can learn without cracking it open.
Thankfully, my router runs OpenWrt, so I can do a little bit of packet capture by looking up the air filter's DHCP lease, and pulling one of these boys:
ssh root@router tcpdump -i wlan1 -U -s0 -w - 'not port 22' | sudo wireshark -k -i -
Filtering by the air filter's IP (in this case, 10.0.0.170) in wireshark, then gives me the ability to inspect all of it's traffic:
Curiously enough, the air filter seems to be making repeated dns querys
for www.google.com
at a set interval. Playing around with the air
filter and leaving the packet capture running for long periods of time
have lead me to a few conclusions:
www.google.com
is the only dns query the device
makeswww.google.com
in my dns server (change the dns server to respond with 0.0.0.0
)34.193.163.206
IP that the device is communicating with34.193.163.206
Based on this information, my belief is that the device is using the DNS
query as some sort of janky probe for internet connectivity, and that it
doesn't care what the response is, just that it gets one. I also
believe the 34.193.163.206
IP to be hard coded into the device's
firmware (boo). This leads me to believe that this is probably an
incredibly simple device, and my be using a bare microcontroller rather
than the typical embedded linux.
On the bright side, the device is actually using TLSv1.2 for its communication, though this hinders my RE efforts.
34.193.163.206
This IP belongs to amazon, and the rdns record for it points to
ec2-34-193-163-206.compute-1.amazonaws.com
, which doesn't give us
much to go off of, other than the fact that data is flowing into an
amazon EC2 instance.
My next stop is to throw nmap with OS fingerprinting turned on at it:
$ sudo nmap 10.0.0.170 -O
Starting Nmap 7.80 ( https://nmap.org ) at 2020-04-16 17:02 EDT
Nmap scan report for 10.0.0.170
Host is up (0.015s latency).
Not shown: 999 closed ports
PORT STATE SERVICE
8088/tcp open radan-http
MAC Address: 84:72:07:32:3F:50 (I&C Technology)
Device type: specialized|general purpose|VoIP phone
Running (JUST GUESSING): NodeMCU embedded (98%), lwIP 1.4.X (98%), Espressif embedded (97%), Philips embedded (91%), Cognex embedded (90%), Ocean Signal embedded (89%), Grandstream embedded (89%), Rigol Technologies embedded (89%)
OS CPE: cpe:/o:nodemcu:nodemcu cpe:/a:lwip_project:lwip cpe:/h:philips:hue_bridge cpe:/a:lwip_project:lwip:1.4 cpe:/h:grandstream:gxp1105 cpe:/h:rigol_technologies:dsg3060
Aggressive OS guesses: NodeMCU firmware (lwIP stack) (98%), Espressif esp8266 firmware (lwIP stack) (97%), ESPEasy OS (lwIP stack) (92%), Philips Hue Bridge (lwIP stack v1.4.0) (91%), Cognex DataMan 200 ID reader (lwIP TCP/IP stack) (90%), Ocean Signal E101V emergency beacon (FreeRTOS/lwIP) (89%), Grandstream GXP1105 VoIP phone (89%), lwIP 1.4.0 lightweight TCP/IP stack (89%), Rigol DSG3060 signal generator (89%)
No exact OS matches for host (test conditions non-ideal).
Network Distance: 1 hop
OS detection performed. Please report any incorrect results at https://nmap.org/submit/ .
Nmap done: 1 IP address (1 host up) scanned in 11.85 seconds
This doesn't give us too much to go off of, only one port open and nmap left making an absolute crapshoot at what OS is running on it.
Pulling up a connection to this service on 8088 in a web browser reveals two things, a service gated by http basic auth, and the headers from the response reveal that it is, at least, using lwIP (also this device has no concept of time):
HTTP/1.0 401 Authorization Required
WWW-Authenticate: Basic
Server: lwIP/1.4.1 (http://savannah.nongnu.org/projects/lwip)
Content-type: text/html
Expires: Fri, 10 Apr 2008 14:00:00 GMT
Pragma: no-cache
The standard username/password combos haven't worked, and trying a list of 10 common usernames with 1000 common passwords also failed. I am currently in the process of running a longer brute force attempt, as this thing's only rate limiting is how slow its CPU seems to be, but that's going to take the rest of the day, and I have my doubts that it will work.
I strongly suspect that there is some commodity microcontroller running the show in there, most likely running NodeMCU, but unless I make a breakthrough with the http basic auth on that port 8088 service, I've hit a wall for how far I can go without cracking it open.
Join me next time as I crack it open!
]]>Dotfile management is hard. Most of us are doing it with complicated series of batch scripts, and some of the smarter among us are using ansible to do the job. Ansible is, however, quite heavy weight for this task, and not really well suited to the simple set-and-forget nature of dotfile directories, however well it does work.
dotfile-playbook should DWIM, and should be idempotent (i.e., it should be safe to run as many times as the user wishes without causing issues).
This blog post serves as a rough outline for the project, and may be edited as time goes on.
.
|-- data
| |-- bar
| |-- baz
| `-- foo
|-- files
| |-- Xmodmap
| |-- bashrc
| |-- config
| | |-- fish
| | | `-- config.fish
| | |-- i3
| | | `-- config
| | `-- riot-web
| | `-- config.json
| |-- data
| | `-- exampleapp
| | `-- example.db
| |-- emacs.el
| `-- my-x-profile
|-- packages.yml
`-- recipies.yml
dotfile-playbook should first check packages.yml
and attempt to
install any packages that don't already exist in some sort of state
saving directory.
dotfile-playbook should, mostly automatically based on the structure of the repository, generate symlinks for the user. It should automatically create any parent directories that don't already exist.
It should try to respect the XDG Base Directory environment variables,
but if they aren't set, it should default to the 'normal' locations
(e.g. ~/.config/
{.verbatim} for $XDG_CONFIG_HOME
{.verbatim})
It should generally not create symlinks to directories, instead creating the directory and symlinking to each file in it individually.
XDG dir linking
dotfile-playbook should first iterate through each of the folders in
the files
directory, and associate each one with an XDG base
directory (e.g. files/conifg
would be associated with
$XDG_CONFIG_HOME
).
Each XDG folder would then have its contents 'overlayed' with
symlinks to those contents in their associated folder. (e.g. in the
[above example]{.spurious-link target=”*Layout”}
$XDG_CONFIG_HOME/fish/config.fish
would become a symlink to
files/config/fish/config.fish
).
Direct file linking
Each file inside of files/
, but not inside a subfolder (e.g.
bashrc
) will have a symlink generated to it in the users
homefolder, with the name of that symlink being the name of the
file, but prefixed with a dot (e.g. files/bashrc
becomes
~/.bashrc
{.verbatim}).
dotfile-playbook should parse the recipes.yml
{.verbatim} file, and
look for any applicable linkOverride
{.verbatim} map, with a format
like so
linkOverride:
"my-x-profile": "~/.Xprofile"
"emacs.el":
- "~/.emacs.el"
- "~/.emacs-2.d/init.el"
"config/riot-web": "$XDG_CONFIG_HOME/Riot"
This allows overriding the default link location for files or folders.
Keys that are directories should "overlay" the source onto the
destination, in the same way that the DWIM linking does. Keys that are
files should create a direct link. All keys are paths relative to the
files
{.verbatim} directory. Keys that show up in an applicable
linkOverride
{.verbatim} should not have the DWIM linking applied to
them.
A list may be used as the value, in this case, all the specified links are created.
In this example (assuming .config
{.verbatim} for the xdg config
directory):
~/.Xprofile
{.verbatim} is a symlink to
files/my-x-profile
{.verbatim}~/.emacs.el
{.verbatim} is a symlink to files/emacs.el
{.verbatim}~/.emacs-2.d/init.el
{.verbatim} is a symlink to
files/emacs.el
{.verbatim}~/.config/Riot/config.json
{.verbatim} is a symlink to
files/riot-web/config.json
{.verbatim}The recipes.yml
{.verbatim} can contain link overrides and additional
conditional actions.
The two conditional blocks are as follows, executing the described actions/overrides when the value of the key is equal to the value given, or when they are not equal, as follows. Multiple conditions may be specified, and will be combined with an 'and' operation.
Conditionals may be arbitrarily nested.
When:
when:
(condition): (value)
commands:
- (action1):
- a
- b
- c
- (action2):
- d
- e
- f
- linkOverrides:
"a": "b"
"c": "d"
when-not:
when:
(condition): (value)
commands:
- (action1):
- a
- b
- c
- (action2):
- d
- e
- f
- linkOverrides:
"a": "b"
"c": "d"
distro
{.verbatim}
the distro key will either be the operating system, for general
unixen (e.g. darwin
{.verbatim} for macOS, freebsd
{.verbatim},
netbsd
{.verbatim}, etc.), or the distribution if it is linux
(arch
{.verbatim}, ubuntu
{.verbatim}, fedora
{.verbatim})
linux
{.verbatim}
Will be true
{.verbatim} if dotfile-playbook is being run on linux,
false
{.verbatim} otherwise
release
{.verbatim}
Will be the VERSION_ID
{.verbatim} or equivalent from
/etc/os-release
{.verbatim} if it exists, or the OS version number
on non-linux unixen
hostname
{.verbatim}
Will be the output of hostname -s
{.verbatim}
hostname-full
{.verbatim}
Will be the output of hostname -f
{.verbatim}
domain
{.verbatim}
Will be the output of hostname -d
{.verbatim}
Actions can either be a key in a conditional, or can be free floating in
the recipes.yml
{.verbatim}
Manually links a file or directory. This is independent of the DWIM linking, and, unlike the DWIM linking, will create a symlink to a directory if asked.
It will perform this action for all keys specified under it.
Example:
link:
"data/foo": "some/random/directory/foo"
Unconditionally copies a file, overriding it if it exists.
Will overwrite the file every time dotfile-playbook is ran.
Example:
copy:
"data/foo": "some/random/directory/foo"
Same as copy
{.verbatim}, but will not overwrite the file if it exists.
Example:
copy-once:
"data/bar": "some/random/directory/bar"
The current arsuran repository format is roughly an 'improved' copy of the one borg uses. This format works reasonably well for most purposes, and it is quite possible to squeeze great performance out of it.
One weakness it does have, though, is failing to hide chunk lengths. Since both borg and asuran use content defined chunking, both formats are (theoretically) open to fingerprinting attacked based on stored chunk sizes.
Borg uses a mitigation approach based on randomizing the look table used inside BuzHash for the content defined chunking. Asuran currently employs a similar defense for when operating with BuzHash CDC1, but currently has no such mitigation when operating with the default FastCDC implementation.
Such an approach is theoretically also possible to implement with FastCDC, as it also uses a 'random' lookup table as part of the algorithm. This 'table randomization' technique does provide a countermeasure against stored chunk length fingerprinting, by making it hard or impossible for an attacker to guess what size chunks a known plaintext will be broken up into, however I do not believe this to be a good general approach to fixing this concern with content defined chunking.
Asuran is designed to be the archiver for new/future technologies. While it may be possible to implement a similar kludge for FastCDC, I don't believe it is a good general approach going forward. New algorithms may exist in the future that provide even better performance than FastCDC, and may not be amenable to such a countermeasure. Users should not be expected to pay a performance cost for security when an alternative approach exists.
The borg approach, as a fundamental element of its design, does not produce the same set of chunks for a file when it is inserted into two separate repositories. Under this model, in order to sync an object directly from one repository to another, it either requires manual intervention at the time the repository is initialized, or reconstruction and re-chunking of the object. The former being an error prone and not user friendly solution, and the latter requiring an unnecessary compute overhead, especially in cases where the remote repository already has most of the chunks.
The asuran format specification already does not require a priori knowledge of the length of chunks to be able to pull them out of the backend. Currently, when the repository wants to request a chunk from the backend, it uses the following struct to tell the backend where the chunk it wants is located:
pub struct SegmentDescriptor {
pub segment_id: u64,
pub start: u64,
}
Right now, the start
value is the offset of the first byte of the
chunk within the segment, and a format that includes length information
in its encoding 2 is used. This does not entirely avoid the issue, as
the lengths of the chunks are still encoded in plain text. However, the
asuran API makes no assumptions about what the start value actually is,
and we shall exploit this to change it to a table index that does not
provide any direct information about chunk location or size.
Currently, chunks are described with the following struct:
pub struct Chunk {
/// The data of the chunk, stored as a vec of raw bytes
#[serde(with = "serde_bytes")]
data: Vec<u8>,
/// Compression algorithim used
compression: Compression,
/// Encryption Algorithim used, also stores IV
encryption: Encryption,
/// HMAC algorithim used
hmac: HMAC,
/// HMAC tag of the cyphertext bytes of this chunk
#[serde(with = "serde_bytes")]
mac: Vec<u8>,
/// `ChunkID`, used for indexing in the repository and deduplication
id: ChunkID,
}
Chunks are currently written to disk hole, and segment files are simple concatenations of MessagePacked chunks.
As a step in resolving the issue, I will be splitting the chunk's on disk representation into two new structs, a header containing the metadata:
pub struct ChunkHeader {
compression: Compression,
encryption: Encryption,
hmac: HMAC,
mac: Vec<u8>,
id: ChunkId,
}
and a body, containing the actual data:
pub struct ChunkBody(Vec<u8>)
Currently, each segment consists of only one file, containing a concatenation of serialized chunks.
Each segment will now be split into two files, N.data
and N.header
,
where N is some non-negative base 10 integer.
N.data
will now contain a raw concatenation of the chunks payload/body
bytes, with no attached metadata. As these payloads will be encrypted,
the data fill will be effectively a list of random numbers, and will
provide no insight into where a chunk starts or ends.
The N.header
file will now contain a serialized
Vec<(ChunkHeader,usize,usize)>
, with the two usizes
being the start
and end offsets of the chunk body within the data file. Before being
written to disk, this Vec
will be serialized, packed into a Chunk
3, and that Chunk
will need serialized instead. As the entire thing
will be serialized as one encrypted Chunk
, there will be no leaking of
information inside the ChunkHeaders
, except for a rough guess of how
many chunks are in a given segment, but this can be counteracted by
zero-padding the serialized Vec<(ChunkHeader,usize,usize)>
up to a
length that would allow storing the expected maximum number of chunks in
the file4.
This header file will contain a relatively small amount of information, and with the speed of modern encryption algorithms, can afford to be rewritten in its entirety. In-memory caching and only rewriting the header file when a segment is closed (with modifications) can further reduce this overhead.
The start
field in the SegmentDescriptor
struct will be renamed to
index
, and instead of being the offset of the start of the chunk, will
now be the index of the header inside the array in the header file.
The universal listing API, introduced in last weeks post, is now used by
the Target
api. This increases the flexibility of Archives
and also
comes with a mild performance boost!. This was by far the hardest of the
actual features this week. A lot of work had to be done to undo the bad
assumptions I had made when the listings were just a Vec<String>
.
The Backend
trait family has been updated to talk about Chunk
rather
than Vec<u8>
. This greatly reduces cognitive complexity, by allowing
the backend API to take care care of all the serialization tasks. which
should have been its responsibility all along.
The TaskedSegment
struct in the backend API was no longer used, with
the introduction of the SyncBackend
trait and wrapper, and has been
removed.
The Segment
struct was originally written under the assumption that
the backend API would need a priori knowledge of a chunks size to be
able to read it. This turned out to not be the case, and the API needed
to be updated to reflect this change.
The asuran
crate contained a less-than-ideal semver parser, that was
used to parse the current semver passed by cargo, for use in generating
headers for segment files and what not. The previous parser was
fundamentally too simple to work, and broke down when given the
"x.y.z-dev" type semvers I am now using for "in between" versions.
I have resolved this issue by pulling in the semver
crate, and using
it to parse the semver string passed by cargo. I will also use this
crate in the future for doing comparisons against segment headers to
detect version incompatibilities.
This was, by far, the most time consuming task I underwent this week. I audited all the rustdoc and comments, in all the modules of all the crates, and made sure they were accurate and only contained up to date information. Asuran isn't as well documented as I would like, but still has a lot of comments and external documentation.
As I am writing this (after the completion of the doc audit), the asuran
repository has one line of comment for about every 4 lines of code,
taking a look at this tokei
report:
-------------------------------------------------------------------------------
Language Files Lines Code Comments Blanks
-------------------------------------------------------------------------------
Markdown 5 493 493 0 0
Rust 51 8909 6502 1656 751
TOML 11 195 173 4 18
YAML 1 12 12 0 0
-------------------------------------------------------------------------------
Total 68 9609 7180 1660 769
-------------------------------------------------------------------------------
In addition to this, the Internals document on the Asuran website had drifted out of sync due to changes I have made to the repository format as I worked, and required substantial updating.
There is still a substantial amount of documentation to be written, but at least what does exist is up to date and correct now.
Hole-Punch
Sparse Files are really difficult to handle in a cross platform way.
Sparse files are, fundamentally, a beautiful optimizing hack that make everyone's lives easier, but as hacks in general go, no-one has ever really done it the same way twice, so each platform and filesystem handle sparse files in a slightly different way.
To combat this, I have developed and released the hole-punch crate, which provides a nice, simple, cross platform way of finding which sections of a sparse file are holes and which are data.
Support is currently only provided for Unix-like platforms that support
the SEEK_HOLE
and SEEK_DATA
arguments to leesk
. I need to get my
hands on a Mac in general, but also for testing and future development
on hole-punch
, and Windows support is currently in the works.
The lowest abstraction level asuran
provides is a Content Addressable
Storage interface where the blobs, hereafter refereed to as "Chunks",
are keyed by an HMAC of their plaintext.
An HMAC is used, rather than a plain hash, since these keys may leak out
of the encrypted sections of the repository, and be visible in
plaintext. asuran
thus uses an HMAC with a key (stored securely with
the rest of the repository key material) used only for this purpose, and
as this key is unique to each repository, these HMAC keys end up being
as good as random numbers to any would be attacker, not having access to
the key material.
One might, somewhat correctly, come to the first assumption that leaking a cryptographic hash of the plaintext of your data is no big deal. After all, you shouldn't be able to reverse the contents of a Chunk based on the cryptographic hash of its plaintext, right?
While that is true, an attacker doesn't necessarily need to reverse the hash to extract compromising information.
Asuran's threat model is pretty pessimistic, assuming the repository is located on completely untrusted storage,, meaning an attacker is assumed to have complete, unlimited, unprotected access to the on-disk repository. To understand how an attacker might be able to extract compromising information (under this threat model) if Asuran were to use plain hashing rather than HMAC keys, we are going to come up with a bit of a contrived example.
Lets assume that your local MediaMegaCorp has recently released a new film on your favorite format, and they have been having some trouble with piracy. A major scene group has released a popular rip of the film, and in a futile effort to purge the internet of this rip, they reach out to local storage providers for help.
Meanwhile, you have purchased a physical copy of the movie, and, as you
enjoyed it, dutifully (and, for legalities sake, legally) make a backup
of it, for when the inevitable happens and time has rendered your
physical copy unusable. You notice you happen to have used the same
tools the scene group is know to use for producing their rips, and those
tools happen to be known for producing byte-for-byte reproducible
copies. You are aware of MediaMegaCorp's attempt to wipe the rip off
the net, but pay it no mind, as you use LesserAsuran
for your backups,
an archiver that strives to leak nothing about its repository's
contents.
A few days later, MediaMegaCorp approaches your storage provider about removing copies of the rip from their servers. The storage provider, fearing retribution in the form of a massive copyright lawsuit, complies and asks that MediaMegaCorp provides the hash of the file, which they do, we'll call it #🎞.
Your storage provider, savvy to the existence of such tools as
LesserAsuran
that index encrypted data based on the hash of the
plaintext, not only scans their disks for files whose hashes are #🎞, but
also scans the disk for the value of #🎞 itself. When they stumble upon
your LesserAsuran
repository, and find #🎞 in its index, they don't
need to reverse the hash to know that the film itself is stored in the
repository, as they already have the plaintext the hash is tied to.
Asuran
effectively prevents this attack by using an HMAC instead of a
plain hash function for keying. The use of HMAC ties the output of the
key function to something other than just the plaintext of the object,
in this case, it ties it to secret key material that either never
touches the remote/untrusted storage unencrypted, or never touches it at
all. Even if they have the plaintext they are searching for, the
attacker can not determine1 if your repository contains that blob
without also having your secret key material.
While using an HMAC to generate the content keys has obvious security advantages, the fact that it also serves as the key for deduplication poses an issue to practical use, namely, efficiently synchronizing data between repositories.
If Asuran
used a plain hash instead of an HMAC, synchronizing archives
between repositories would be dead simple. The process would be as
simple as making a list of all the chunk keys in a particular archive on
the local repository, then interrogating the remote repository to see
which chunks it is missing. Once you have that information you could
simple re-encrypt only the missing chunks with the remote's key, as
well as the archive structure itself, and send them over to the remote
repository for direct storage.
Using an HMAC tied to the repository's key material confounds this approach, as each repository will, presumably have different HMAC keys, and prevent the efficient detection and sending of only changed/missing chunks. The local repository would, at a minimum, have to locally decrypt all chunks and reprocess them with the remote's key to determine which ones are missing from the remote, which, while a valid strategy, is not very efficient, epically for the use case of backing up large file systems with infrequent/small changes between snapshots.
One obvious approach would be to allow copying of the entire key material for a repository into a new one, and only allowing direct sync between repositories sharing key material, falling back to the inefficient "just reprocess everything" approach when this is not the case. While this is certainly a valid approach, and we probably will support doing this, it will not be the default behavior, as violates asuran's "leak nothing" policy, by making it trivial to determine if two repositories contain the same information.
One might think that sharing only the HMAC key would be sufficient, and keeping the other components of the key different between repositories, as in this case the encrypted bytes of the chunks themselves would still be different on disk, and you would still need to know the secret HMAC key to be able to determine if a repository contains a particular plaintext.
This approach, however, still leaks that two repositories contain the same information, in a less than obvious way, so I consider it unsafe.
To demonstrate this attack, lets posit a future where people share cool
stuff they want to archive, but also be available to the public, by
hosting public asuran archives, and sharing the password to it amongs't
trusted members of the community (or even having the public repository
be NoEncryption
), and you might pull these files into your own asuran
archives by a direct pull. Say you are subscribed to a historical
document archiving group that you pull from a lot, so as a matter of
convenience you clone your own personal asuran repository's HMAC key
from that groups public repository. It would probably seem like no big
deal, since your local repository is encrypted with a different
encryption key, so it shouldn't leak anything anyway.
Lets go for a slightly more insidious example2 than the last one. Assume you live in a country with a state secrets act that prohibits civilian possession and distribution of certain pieces of information, and there has been an illegal photo of your country's new stealth bomber making the rounds on certain parts of the internet. One of the maintainers of your archiving group (in my opinion, rightly so) decides that the photo of the stealth bomber should be preserved, and sees the best way of doing this as sneaking it into one of their regular archive uploads to the group repository. None the wiser, you conduct your normal weekly pull from the public repository, blissfully unaware of the illegal content that has just been so rudely thrust upon you.
Now lets assume that the original uploader has either been caught in the act, or confessed to his crime, or the government has found out through some other means, and that even though the government knows who uploaded the illegal content and when, they still do not have the key to the repository. Even though they could not positively identify which chunks contained the illegal information3, the government still use time stamp information and other side channels to make a definitive statement beyond a reasonable doubt that if a repository contains all of a specific set of chunk keys, it contains the illegal information. The government could then go to storage providers and require that they scan their disks for the offending sets of chunk ids, and if your repository contains all of them, then congratulations, you are now, at the very least on a list.
Based on the above described attacks, any efficient solution to the problem of synchronizing archives between repositories must satisfy the following properties:
I propose modifying the manifest API such that each archive entry has an optional pointer to a chunk containing the following struct:
struct IDMap {
known_previously: Vec<ChunkID>,
additional: BiMap<ChunkID, ChunkHash>,
}
Where known_previously
is a vector of pointers to the heads of all
other known IDMap trees at the time of archive creation, and
additional
is a bijective
mapping of HMAC keys to the plain hashes of each chunk that was not
previously known. Since the mapping between ChunkID
and ChunkHash
should be globally bijective5, it is trivial to walk the entire tree
and union these together at run time, to construct a complete mapping.
Syncing to a remote repository can be accomplished by either
interrogating the remote repository6 using ChunkHash
rather than
ChunkID
.
This satisfies each property as follows:
Must not leak plain hashes of plaintext
Chunks are encrypted before hitting storage, so the plain hashes are never written in the clear
Must not share any secret key state between repositories
This just straight up is not a requirement here, the ChunkIDs
are
converted to a secret key agnostic format during interrogation.
Must not require any deep inspection of chunks that are shared between the repositories
The bijective map between ChunkID
and ChunkHash
means that
determining if a chunk is present in either repository requires only
a handful of constant time HashMap
lookups on either end.
Must require the secret keys of both repositories to determine if they share information
As the plain hashes themselves are encrypted in storage, an attacker would only have access to the HMACs, which will still be different between repositories.
Must not have any non-optional storage overhead
As the IDMap
pointer will be optional, it will be perfectly valid
for a repository to just not include this information.
Must still allow synchronization between repositories that do not have special features enabled
This information can still be recovered at run time through deep chunk inspection, though at the cost of the I/O and compute that takes.
Due to the current on-disk storage format, there is still the potential for a chunk-length-based fingerprinting attack. We support a modified version of buzhash that partially mitigates this though the use of a randomized lookup table, and I am still looking into ways to erase knowledge of chunk length from the on-disk format. ↩
I am aware that the specifics of this example are grossly implausible, but there are an entire family of just slightly less feasible and a lot more dangerous versions of this attack, and this version just serves as illustration of the basic concepts. ↩
Unless the repository was NoEncryption
, obviously ↩
i.e. complete decryption ↩
Within a single repository ↩
Over some sort of secure connection, either a TLS type connection, or just pulling the chunks from the remote repository down and decrypting locally. ↩
This edition covers work completed between 2020-02-24 to 2020-03-01
The asuran_core
library now has cargo features setup for the
dependencies of the various compression, encryption, and hashing
algorithms. A core
feature has been defined supporting ZStd, AES-CTR,
and Blake3, features have been setup for each encryption family, as well
as features for supporting all the algorithms for each operations (e.g..
all-compression
).
The enums in the core library still have variants for the algorithms whose support has not been compiled in, for compatibility purposes, however, attempting to perform operations on these variants will result in a run time panic.
asuran_core
will also compile-time panic if there are no HMAC
algorithm features selected, as the repository format fundamentally does
not make sense without an HMAC.
The default
feature for asuran_core
includes all the features for
all the supported algorithms.
The core data structure of the universal object listing API has been
written. I chose to implement this as a "flat" tree stored in a
HashMap
. Right now the Node entries are keyed by String
, however it
would probably be wise to change these to some sort of byte string, to
support keying Nodes by arbitrary data. The current model, at the very
least, conflicts with the definition of *nix paths as "any sequence of
non-NUL bytes".
I additionally added a field to the Archive
definition for the listing
associated with that archive.
I am still in the process of porting the Target
interface to use the
new listing API instead of the old strategy of using Vec<String>
.
The asuran-cli
binary crate was, more or less, completely rewritten
from scratch, with much less jank. I used structopt
to simplify
argument and sub-command handling, and created modules for each separate
command, which has generally simplified things. I still need to figure
out how to get structopt
to output the help for global options when
you pass --help
to a sub command.
The rewrite highlighted a pain point in the asuran
API, namely, using
tasks to insert objects into an archive involves lots of unnecessary
cloning. Currently, it looks like this:
let mut repo = repo.clone();
let archive = archive.clone();
let backup_target = backup_target.clone();
task_queue.push(task::spawn(async move {
(
path.clone(),
backup_target
.store_object(&mut repo, chunker.clone(), &archive, path.clone())
.await,
)
}));
In addition to the general code cleanliness improvement, the CLI has also been updated to support the FlatFile backend, as well as the MultiFile backend.
I believe the route forward on this is to have BackupTarget
implementations be required to store a reference to the repository and
archive, and have a method on the BackupTarget
Trait for spawning a
task directly.
Encryption::NoEncryption
The NoEncryption
variant of the Encryption
was updated to have a
non-zero key length. Some other places in the asuran
library assume
that encryption key has a non-zero length, such as the use of argon2
in the key encryption algorithm.
Even fully NoEncryption
repositories still require some key material
for HMAC generation and other tasks, so the best fix for this seemed to
be pretending NoEncryption
behaves like other encryption modes and
giving it a non-zero key length.
FlatFile
constructorThis was mistakenly added as a reflection of the MultFile
interface.
Unlike MultFile
, which requires the key to verify the manifest merkle
tree on open, FlatFile
performs no such operation and thus does not
require the key.
FlatFile
Produces Absurdly Large ArchivesAt the moment, FlatFile
is producing archives that are way too big.
With the test 10GB data set I have been using, and the same general
settings, MultiFile
produces a 3.6GB archive, where FlatFile produces
a 6.8GB archive. I need to do more debugging, but I believe this to be
due to a bug in how I am determining the location to start the next
segment. I believe the route forward is going to be to to have FlatFile
write the segements into an in memory buffer, and use the size of that
buffer to extract length information, rather than the current abuse of
read.seek(SeekFrom::Current(0)
that I am currently using.