Episode 176
Linking your world
January 11th, 2017
1 hr 32 mins 24 secs
Your Hosts
Tags
About this Episode
Another exciting week on BSDNow, we are queued up with LLVM / Linking news, a look at NetBSD’s scheduler,
This episode was brought to you by
alt="DigitalOcean - Simple Cloud Hosting, Built for Developers" />
title="Tarsnap">
Headlines
FreeBSD Kernel and World, and many Ports, can now be linked with lld
- “With this change applied I can link the entirety of the FreeBSD/amd64 base system (userland world and kernel) with LLD.”
- “Rafael's done an initial experimental Poudriere FreeBSD package build with lld head, and found almost 20K out of 26K ports built successfully. I'm now looking at getting CI running to test this on an ongoing basis. But, I think we're at the point where an experimental build makes sense.”
- Such testing will become much easier once llvm 4.0 is imported into -current
- “I suggest that during development we collect patches in a local git repo -- for example, I've started here for my Poudriere run https://github.com/emaste/freebsd-ports/commits/ports-lld”
- “It now looks like libtool is responsible for the majority of my failed / skipped ports. Unless we really think we'll add "not GNU" and other hacks to lld we're going to have to address libtool limitations upstream and in the FreeBSD tree. I did look into libtool a few weeks ago, but unfortunately haven't yet managed to produce a patch suitable for sending upstream.”
- If you are interested in LLVM/Clang/LLD/LLDB etc, check out: A Tourist’s Guide to the LLVM Source Code ***
Documenting NetBSD's scheduler tweaks
- A followup to our previous coverage of improvements to the scheduler in NetBSD
- “NetBSD's scheduler was recently changed to better distribute load of long-running processes on multiple CPUs. So far, the associated sysctl tweaks were not documented, and this was changed now, documenting the kern.sched sysctls.”
- kern.sched.cacheht_time (dynamic): Cache hotness time in which a LWP is kept on one particular CPU and not moved to another CPU. This reduces the overhead of flushing and reloading caches. Defaults to 3ms. Needs to be given in ``hz'' units, see mstohz(9).
- kern.sched.balance_period (dynamic): Interval at which the CPU queues are checked for re-balancing. Defaults to 300ms.
- kern.sched.min_catch (dynamic): Minimum count of migratable (runable) threads for catching (stealing) from another CPU. Defaults to 1 but can be increased to decrease chance of thread migration between CPUs.
- It is important to have good documentation for these tunables, so that users can understand what it is they are adjusting ***
FreeBSD Network Gateway on EdgeRouter Lite
- “EdgeRouter Lite is a great device to run at the edge of a home network. It becomes even better when it's running FreeBSD. This guide documents how to setup such a gateway. There are accompanying git repos to somewhat automate the process as well.”
- “Colin Percival has written a great blog post on the subject, titled FreeBSD on EdgeRouter Lite - no serial port required . In it he provides and describes a shell script to build a bootable image of FreeBSD to be run on ERL, available from GitHub in the freebsd-ERL-build repo. I have built a Vagrant-based workflow to automate the building of the drive image. It's available on GitHub in the freebsd-edgerouterlite-ansible repo. It uses the build script Percival wrote.”
- “Once you've built the disk image it's time to write it to a USB drive. There are two options: overwrite the original drive in the ERL or buy a new drive. I tried the second option first and wrote to a new Sandrive Ultra Fit 32GB USB 3.0 Flash Drive (SDCZ43-032G-GAM46). It did not work and I later found on some blog that those drives do not work. I have not tried another third party drive since.”
- The tutorial covers all of the steps, and the configuration files, including rc.conf, IP configuration, DHCP (and v6), pf, and DNS (unbound)
- “I'm pretty happy with ERL and FreeBSD. There is great community documentation on how to configure all the pieces of software that make a FreeBSD-based home network gateway possible. I can tweak things as needed and upgrade when newer versions become available.”
- “My plan on upgrading the base OS is to get a third party USB drive that works, write a newer FreeBSD image to it, and replace the drive in the ERL enclosure. This way I can keep a bunch of drives in rotation. Upgrades to newer builds or reverts to last known good version are as easy as swapping USB drives.”
- Although something more nanobsd style with 2 partitions on the one drive might be easier.
- “Configuration with Ansible means I don't have to manually do things again and again. As the configs change they'll be tracked in git so I get version control as well. ERL is simply a great piece of network hardware. I'm tempted to try Ubiquiti's WiFi products instead of a mixture of DD-WRT and OpenWRT devices I have now. But that is for another day and perhaps another blog post.” ***
A highly portable build system targeting modern UNIX systems
- An exciting new/old project is up on GitHub that we wanted to bring your attention to.
BSD Owl is a highly portable build-system based around BSD Make that supports a variety of popular (and not so popular) languages, such as:
- C programs, compiled for several targets
- C libraries, static and shared, compiled for several targets
- Shell scripts
- Python scripts
- OCaml programs
- OCaml libraries, with ocamldoc documentation
- OCaml plugins
- TeX documents, prepared for several printing devices
- METAPOST figures, with output as PDF, PS, SVG or PNG, either as part of a TeX document or as standalone documents
What about features you may ask? Well BSD Owl has plenty of those to go around:
- Support of compilation profiles
- Support of the parallel mode (at the directory level)
- Support of separate trees for sources and objects
- Support of architecture-dependant compilation options
- Support GNU autoconf
- Production of GPG-signed tarballs
- Developer subshell, empowered with project-specific scripts
- Literate programming using noweb
- Preprocessing with m4
As far as platform support goes, BSD Owl is tested on OSX / Debian Jesse and FreeBSD > 9. Future support for OpenBSD and NetBSD is planned, once they update their respective BSD Make binaries to more modern versions
News Roundup
find -delete in OpenBSD. Thanks to tedu@ OpenBSD will have this very handy flag to in the future.
- OpenBSD’s find(1) utility will now support the -delete operation
- “This option is not posix (not like that's stopped find accumulating a dozen extensions), but it is in gnu and freebsd (for 20 years). it's also somewhat popular among sysadmins and blogs, etc. and perhaps most importantly, it nicely solves one of the more troublesome caveats of find (which the man page actually covers twice because it's so common and easy to screw up). So I think it makes a good addition.”
- The actual code was borrowed from FreeBSD
- Using the -delete option is much more performant than forking rm once for each file, and safer because there is no risk of mangling path names
- If you encounter a system without a -delete option, your best bet is to use the -print0 option of find, which will print each filename terminated by a null byte, and pipe that into xargs -0 rm
- This avoids any ambiguity caused by files with spaces in the names ***
New version of the Lumina desktop released
- Just in time to kickoff 2017 we have a new release of Lumina Desktop (1.2.0)
Some of the notable changes include fixes to make it easier to port to other platforms, and some features:
New Panel Plugins:
- “audioplayer” (panel version of the desktop plugin with the same name): Allows the user to load/play audio files directly through the desktop itself.
- “jsonmenu” (panel version of the menu plugin with the same name): Allows an external utility/script to be used to generate a menu/contents on demand.
New Menu Plugins:
- “lockdesktop”: Menu option for instantly locking the desktop session.
New Utilities:
lumina-archiver: This is a pure Qt5 front-end to the “tar” utility for managing/creating archives. This can also use theddutility to burn a “*.img” file to a USB device for booting.“
Looks like the news already made its rounds to a few different sites, with Phoronix and Softpedia picking it up as well
TrueOS users running the latest updates are already on the pre-release version of 1.2.1, so nothing has to be done there to get the latest and greatest.
dd is not a disk writing tool
- “If you’ve ever used dd, you’ve probably used it to read or write disk images:” > # Write myfile.iso to a USB drive > dd if=myfile.iso of=/dev/sdb bs=1M
- “Usage of dd in this context is so pervasive that it’s being hailed as the magic gatekeeper of raw devices. Want to read from a raw device? Use dd. Want to write to a raw device? Use dd. This belief can make simple tasks complicated. How do you combine dd with gzip? How do you use pv if the source is raw device? How do you dd over ssh?”
- “The fact of the matter is, dd is not a disk writing tool. Neither “d” is for “disk”, “drive” or “device”. It does not support “low level” reading or writing. It has no special dominion over any kind of device whatsoever.”
- Then a number of alternatives are discussed
- “However, this does not mean that dd is useless! The reason why people started using it in the first place is that it does exactly what it’s told, no more and no less. If an alias specifies -a, cp might try to create a new block device rather than a copy of the file data. If using gzip without redirection, it may try to be helpful and skip the file for not being regular. Neither of them will write out a reassuring status during or after a copy.”
- “dd, meanwhile, has one job*: copy data from one place to another. It doesn’t care about files, safeguards or user convenience. It will not try to second guess your intent, based on trailing slashes or types of files. When this is no longer a convenience, like when combining it with other tools that already read and write files, one should not feel guilty for leaving dd out entirely.”
- “dd is the swiss army knife of the open, read, write and seek syscalls. It’s unique in its ability to issue seeks and reads of specific lengths, which enables a whole world of shell scripts that have no business being shell scripts. Want to simulate a lseek+execve? Use dd! Want to open a file with O_SYNC? Use dd! Want to read groups of three byte pixels from a PPM file? Use dd!”
- “It’s a flexible, unique and useful tool, and I love it. My only issue is that, far too often, this great tool is being relegated to and inappropriately hailed for its most generic and least interesting capability: simply copying a file from start to finish.”
- “dd actually has two jobs: Convert and Copy. Legend has it that the intended name, “cc”, was taken by the C compiler, so the letters were shifted by one to give “dd”. This is also why we ended up with a Window system called X.”
- dd countdown ***
Bhyve setup for tcp testing
- FreeBSD Developer Hiren Panchasara writes about his setup to use bhyve to test changes to the TCP stack in FreeBSD
- “Here is how I test simple FreeBSD tcp changes with dummynet on bhyve. I’ve already wrote down how I do dummynet so I’ll focus on bhyve part.”
- “A few months back when I started looking into improving FreeBSD TCP’s response to packet loss, I looked around for traffic simulators which can do deterministic packet drop for me.”
- “I had used dummynet(4) before so I thought of using it but the problem is that it only provided probabilistic drops. You can specify dropping 10% of the total packets”
- So he wrote a quick hack, hopefully he’ll polish it up and get it committed
- “Setup: I’ll create 3 bhyve guests: client, router and server”
- “Both client and server need their routing tables setup correctly so that they can reach each other. The Dummynet node is the router / traffic shaping node here. We need to enable forwarding between interfaces: sysctl net.inet.ip.forwarding=1”
- “We need to setup links (called ‘pipes’) and their parameters on dummynet node”
- “For simulations, I run a lighttpd web-server on the server which serves different sized objects and I request them via curl or wget from the client. I have tcpdump running on any/all of four interfaces involved to observe traffic and I can see specified packets getting dropped by dummynet. sysctl net.inet.ip.dummynet.io_pkt_drop is incremented with each packet that dummynet drops.”
- “Here, 192.* addresses are for ssh and 10.* are for guests to be able to communicate within themselves.”
- Create 2 tap interfaces for each end point, and 3 from the router. One each for SSH/control, and the others for the test flows. Then create 3 bridges, the first includes all of the control tap interfaces, and your hosts’ real interface. This allows the guests to reach the internet to download packages etc. The other two bridges form the connections between the three VMs
- The creation and configuration of the VMs is documented in detail
- I used a setup very similar to this for teaching the basics of how TCP works when I was teaching at a local community college ***
