April 2011 (#185):

• News Bytes, by Deividson Luiz Okopnik and Howard Dyckoff
• Linux based SMB NAS System Performance Optimization, by Sagar Borikar and Rajesh Palani
• Activae: towards a free and community-based platform for digital asset management, by Manuel Domínguez-Dorado
• Away Mission for April 2011, by Howard Dyckoff
• HAL: Part 3 Functions, by Henry Grebler
• synCRONicity: A riff on perceived limitations of cron, by Henry Grebler
• Applying Yum Package Updates to Multiple Servers Using Fabric, by Ikuya Yamada and Yoshiyasu Takefuji
• Ubuntu, the cloud OS, by Neil Levine
• Back in the Day: Red Hat Linux 3.0.3 - 4.2, 1996-1997, by Anderson Silva
• Taking the Pain Out of Authentication, by Yvo Van Doorn
• HelpDex, by Shane Collinge
• XKCD, by Randall Munroe
• Doomed to Obscurity, by Pete Trbovich

News Bytes

By Deividson Luiz Okopnik and Howard Dyckoff

Contents: News in General Conferences and Events Distro News Software and Product News

Selected and Edited by Deividson Okopnik

Please submit your News Bytes items in plain text; other formats may be rejected without reading. [You have been warned!] A one- or two-paragraph summary plus a URL has a much higher chance of being published than an entire press release. Submit items to bytes@linuxgazette.net. Deividson can also be reached via twitter.

Share

Talkback: Discuss this article with The Answer Gang

Howard Dyckoff is a long term IT professional with primary experience at Fortune 100 and 200 firms. Before his IT career, he worked for Aviation Week and Space Technology magazine and before that used to edit SkyCom, a newsletter for astronomers and rocketeers. He hails from the Republic of Brooklyn [and Polytechnic Institute] and now, after several trips to Himalayan mountain tops, resides in the SF Bay Area with a large book collection and several pet rocks.

Linux based SMB NAS System Performance Optimization

By Sagar Borikar and Rajesh Palani

Introduction

SMB NAS systems range from the low end, targeting small offices (SOHO) that might have 2 to 10 servers, requiring 1 TB to 10 TB of storage, to the high end, targeting the commercial SMB (Small to Medium Business) market. Customers are demanding a rich feature set at a lesser cost, given the relatively cheap solutions available in the form of disk storage, compared to what was available 5 years ago. One of the key factors in deciding the unique selling proposition of the NAS (Network Attached Storage) in the SOHO market is the network and storage performance. Due to extremely competitive pricing, many NAS vendors are focused on reducing the manufacturing costs, so one of the challenges is dealing with the scarcity of system hardware resources without compromising the performance. This article describes the optimizations (code optimizations, parameter tuning, etc.) required to support the performance requirements in a resource-constrained Linux based SMB NAS system.

System Overview

A typical SMB NAS hardware platform contains the following blocks:

1. 500MHz RISC processor
2. 256 MB RAM
3. Integrated Gigabit Ethernet controller
4. PCI 2.0
5. PCI-to-Serial ATA Host Controller

We broadly target the network and storage path in the kernel to improve the performance. While focusing on the network path optimization, we look at device level, driver level and stack level optimization sequentially and with decreasing order of priority. Before starting the optimization, the first thing to measure is the current performance for CIFS, NFS and FTP accesses. We can use tools such as IOzone, NetBench or bonnie++ to characterize the NAS box for network and storage performance. The system level bottlenecks can be identified using tools like Sysstat, LTTng or SystemTap that make use of less intrusive instrumentation techniques. OProfile is another tool which can be helpful to understand the functional bottlenecks.

1. Networking Optimization

1.1. Device level optimization

Many hardware manufacturers typically add a lot of bells and whistles in their hardware for accelerating specific functions. The key ones that have an impact on the NAS system performance are described below.

1.1.1 Interrupt Coalescence

Most of the gigabit Ethernet devices support interrupt coalescence, which when clubbed with NAPI will provide an improved performance. (Please refer to driver level optimization later). Choosing the coalescence value for the best performance is a purely iterative job and highly dependent on the target application. For the systems which are targeted for excessive network load of large packets, specifically jumbo frames, and CPU intensive applications or CPUs with lesser frequency, this value plays a significant role in reducing the frequent interruptions of the CPU. Even though there is no rule of thumb to decide the coalescence value, keeping the following points in mind for reducing the number of interrupts will be helpful:

• For the systems with the memory size of 256-512 MB, it helps to choose the smaller value of the coalescence, clubbed with a smaller budget value (The budget parameter specifies how many packets the driver is allowed to pass to the network stack on a poll() call). Note that the coalescence value also depends upon the type of the interface of the device, viz. Gigabit or Fast Ethernet. For a Gigabit Ethernet device and 256 MB memory size, interrupt coalescence timeout value of 64 will be helpful.
• Driver budget parameter will decide the buffering capacity which needs to be set, keeping in mind the coalescence value, and this could also be kept at 64 for the above configuration.

1.1.2 MAC Filters

Enabling MAC filters in the device would greatly offload CPU processing of non-required packets and help to drop the packets at the device level itself.

1.1.3 Hardware Checksum

Most of the devices support hardware checksum calculation. Setting CHECKSUM_HW as the CRC calculation will greatly offload the CPU from the CRC computation and rejection of packets if there is a mismatch. Note that this is only the physical layer CRC and not the protocol level.

1.1.4 DMA support

The DMA controller tracks the buffer address and ownership bits for the transmit and receive descriptors. Based on the availability of buffers in the memory, the DMA controller will transfer the packets to and from system memory. This is the standard behavior of any Ethernet device IP. The performance can be improved by moving the descriptors to onchip SRAM if available in the processor. As transmit and receive descriptors would be accessed most frequently, moving them to the SRAM will help with faster access to the buffers.

1.2 Driver level optimization

This optimization scope is vast and needs to be treated based on the application requirement. The treatment can be different if the need is to optimize only for packet forwarding performance, rather than TCP level performance. Also, it varies based on the system configuration and resources viz, other important applications making use of the system resources, DMA channels, system memory and whether the Ethernet device is PCI based or directly integrated to the System- on-Chip (SoC) through the system bus. In the DUT, the Ethernet device was hooked up directly to the system bus through the bridge so we'll not focus on PCI based driver optimization for now, although to some extent, this can be applied to PCI devices as well. Most of the NAS systems would be subject to heavy stress when they are up and running for several days and a lot of data transfer is going on over the period. Specifically, with the Linux based NAS systems the free buffer count keeps on reducing over the period and as the pdflush algorithm is not optimized in the kernel for this specific use case, and given that the cache flush mechanism is left to the individual file systems, it depends upon how efficiently the flushing algorithm is implemented in the filesystem. It also depends upon how efficient the underlying hardware is, viz. SATA controller, the disks used etc.Following factors need to be considered when tweaking the Ethernet device driver:

1.2.1 Transmit interrupts

We can mask off transmit interrupts and avoid interrupt jitter occurring because of transmission of the packets. If the transmit gets blocked, there is not much performance impact on the overall network process as such, and what matters is only the receive interrupts.

1.2.2 Error interrupts

Disable the error interrupts in the device unless you really want to take some action based on the error occurred and pass the status up to the application level to let it know that there is some problem in the device.

1.2.3 Memory alignment for descriptors

Even though there are some gaps added through padding in the descriptors, it always helps to have efficient access to the descriptors when they are cache line size aligned and bus width aligned.

1.2.4 Memory allocation for the packets

You may want to make a trade off for memory allocation in the receive path, viz. either choose to pre-allocate all the buffers, preferably fixed sized, fixed number of buffers and use them recursively or make dynamic allocations of buffers on a need basis. Each has its own advantages and disadvantages. In the prior case, you will consume memory and other applications in the system can't make use of the buffers. This may lead to internal fragmentation of the buffers as well as not knowing the exact size of the packet to be received. But it will significantly reduce the memory allocation and deallocation of the buffers in the receive path. As we rely on kernel memory allocation / deallocation routines, there is a possibility that we may end up starving or looping in the receive context to get the free buffers, or if the required memory is not available then it may end up in page allocation failures. Dynamic memory allocation will significantly save on memory in a memory constrained system. This will help other applications to run freely without any issues even when the system is exposed to the heavy stress of IO. But you will have to pay the penalty in the time required to allocate and free the buffers at runtime.

1.2.5 SKB Recycling

Socket buffers(skb) are allocated and freed by the driver on arrival of packets in standard Ethernet drivers. By implementing SKB recycling, the sockets are pre-allocated and would be provided to the driver when requested and would be put back in the recycle queue when freed in kfree implementation.

1.2.6 Cache coherency

Cache coherency is usually not supported in most of the hardware to reduce the BOM, but if supported, it makes the system highly performance driven as software doesn't need to invalidate the cache line and flush the cache. Especially under heavy IO stress, this can have a worse impact on the network performance, which will get propagated to the storage stack as well.

1.2.7 Prefetching

If the underlying CPU supports prefetching, it can help to dramatically improve the system performance under heavy stress. Specifically under heavy stress, this helps to improve the performance by avoiding cache misses. Note that the cache needs to be invalidated if the coherency is not supported in the hardware, else it will greatly hamper the performance.

1.2.8 RCU locks

Spin locks / IRQ locks are expensive compared to light weight RCU locks. The lock/unlock mechanism of RCU locks is much lighter than the spin locks and helps a lot in performance improvement.

1.3 Stack level optimization

TCP/IP stack parameters are defined considering all the supported protocols in the network. Based on the priority of the path followed, we can choose the parameters as per our need. For example, NAS uses CIFS, NFS protocols primarily for the data transfer.

1.3.1 TCP buffer sizes

We could choose the following parameters:

net.core.rmem_max = 64K
net.core.rmem_default = 109568

While this would provide ample space for the TCP packets to get queued up in the memory and yield better performance, it eats up the system memory, and therefore the memory required by other applications in the system to run smoothly needs to be considered. With low system memory, following options can be tried out, to check the performance improvement. Again these changes are highly application dependent and would not necessarily yield a similar performance improvement as observed in our DUT.

1.3.2 Virtual Memory (VM) parameters

1.3.2.1 min_free_kbytes

This parameter decides when pdflush will kick in to flush out the kernel cache to the disk. Setting this value high will spawn the pdflush threads frequently to flush out the data to the disks. This will help with freeing the buffers for the rest of the applications, when the system is exposed to heavy stress. The flip side is that, this may end up causing the system to thrash and consume CPU cycles for freeing the buffer more frequently, and hence dropping packets.

1.3.2.2 pdflush threads

The default number of threads in the system is 2. By increasing the number to 4, we can achieve better performance for NFS. This will help to flush the data promptly and not get queued up for long, thus making space available for free buffers for other applications.

2. Storage Optimization

Storage optimization highly depends upon the filesystem used along with the characteristics of the storage hardware path i.e. the type of the interface, viz. whether it has integrated SATA controller or over PCI. It also depends on whether it has hardware RAID controller or software RAID controller. Hardware RAID controller increases the BOM dramatically. Hence for SOHO solutions, software RAID manager "mdam" is used. In addition to HW support for RAID functionality, the performance depends upon the software IO path which includes the block device layer, device driver and filesystem used. NAS usually has journaling file systems such as ext3 or XFS. XFS is more commonly used for sequential writes or reads due to its extent based architecture, whereas ext3 is well suited for sector wise reads or writes which work on 512 bytes rather than big chunks. We used XFS in our system. The flip side of the XFS is the fragmentation which will be discussed in the next section. The following areas were tweaked for performance improvements in the storage path:

2.1 Software RAID manager

There are two crucial parameters which can decide the performance of the RAID manager.

2.1.1. Chunk size

We should set the chunk size to 64K which will help in mirroring redundancy application, viz. RAID1 or RAID10

2.1.2. Stripe Cache size

Stripe cache size decides the stripe size per disk. Stripe size should be judiciously decided as it will decide the data that needs to be written in per disk of the RAID controller. In case of journaled file system, there will be duplicate copy of the disk and would bog down the IO if the chunk and stripe size value is high.

2.2 Memory fragmentation

Due to its architecture, XFS requires memory in multiples of extents. This leads to severe internal fragmentation if the blocks are of smaller sizes. XFS demands the memory in multiples of 4K and if the buddy allocator doesn't have enough room in order 2 or order 3 hash, then the system may slow down till the other applications release the memory and the kernel can join them back in the buddy allocator. This will specifically be seen under heavy stress, after the system has been running for a couple of days. This can be improved by tweaking the xfs syncd centisecs (see below) to flush the stale data to disks at higher frequency.

2.3 External journal

Journal or log plays an important role in defining XFS performance. Any journaled file system doesn't write the data directly at the destined sectors, but maintains the copy in the log and later writes to the sectors. To achieve better performance, XFS writes the log at the center of the disk whereas the data is stored in the tracks at the outer periphery of the disk. This helps the actuator arm of the disk to have reduced spindle movements, thereby increasing performance. If we remove the dependency of maintaining the log in the same disk where the actual data is stored, it increases performance dramatically. Having NAND flash of 64M to store the log would certainly help reduced spindle movements.

2.4 Filesystem optimization

2.4.1. Mount options

While mounting the filesystem, we can choose the mount options as logsize=64M,noatime,nodiratime. Removing the access time for files and directories would help relieve the CPU from continuously checking the inodes for files and directories.

2.4.2. fs.xfs.xfssyncd_centisecs

This is the frequency at which XFS flushes the data to disk. It will flush the log activity and clean up the unlinked inodes. Setting this parameter to 500 will help increase the frequency of flushing the data. This would be required if the hardware performance is limited. Especially when the system is exposed to constant writes, this helps a lot.

2.4.3. fs.xfs.xfsbufd_centisecs

This is the frequency at which XFS cleans up the metadata. Keeping it low would lead to frequent cleanup of the metadata for extents which are marked as dirty.

2.4.4. fs.xfs.age_buffer_centisecs

This is the frequency at which xfsbufd flushes the dirty metadata to the disk. Setting it to 500 will lead to frequent cleanup of the dirty metadata.

The parameters listed in b,c and d above will lead to significant performance improvement when the log is kept in a separate disk or flash, compared to keeping it in the same media as the data.

2.5 Random disk IO

Simultaneous writes at non-sequential sectors would lead to a lot of actuator arm and spindle movement, inherently leading to the performance drop and impact on the overall throughput of the system. Although there is very little we can do here to avoid random disk IO, what we can control is the fragmentation.

2.6 Defragmentation

Any journaled filesystem will always have this issue when the system is running over a long period of time or exposed to heavy stress. XFS provides a utility named xfs_fsr which can be run periodically to reduce the fragmentation in the disk, although it works only on the dirty extents which are currently not in use. In fact, random disk IO and the fragmentation issue go hand in hand. More the fragmentation more will be the random disk IO and lesser the throughput. It is of utmost importance to keep the fragmentation under control to reduce the random IO.

2.7 Results

We can easily figure out that when the system is under heavy stress, the amount of time taken by the kernel for packet processing is more than the user space processing for the end-to-end path, i.e. from Ethernet device to hard disk.

After implementing the above features, we captured the performance figures on the same DUT.

For RAID creation, use '--chunk=1024'; for RAID5, use

echo 8192 > /sys/block/md0/md/stripe_cache_size

Around 37% degradation in performance was observed after 99% fragmentation of the filesystem, as compared to when the filesystem was 5% fragmented. Note that these figures were captured using the bonnie++ tool.

2.7.2 Having an external log gives around 28% improvement in performance compared to having the log in the same disk where the actual data is stored, on a system that was 99% fragmented.

2.7.3 Running the defragmentating utility xfs_fsr on a system that is 99% fragmented gave an additional 37% improvement in performance.

2.7.4 CIFS write performance for a 1G file on a system that was exposed to 7 days of continuous writes, increased by 133% after implementing the optimizations. Note that this is end to end performance that includes network and storage path.

2.7.5. The NFS write performance was measured using IOzone. NFS write performance for a 1G file on a system that was exposed to 7 days of continuous writes, increased by 180% after implementing the optimizations. Note again that this is the end to end performance that includes network and storage path.

4. Conclusion:

While Linux based commercial SMB NAS systems have the luxury of throwing more powerful hardware to address the performance requirements, SOHO NAS systems have to make do with resource constrained hardware platforms, in order to meet the system cost requirements. Although Linux is a general purpose OS that has a lot of code to run a wide variety of applications, and is not necessarily purpose built for an embedded NAS server application, it is possible to configure and tune the OS to obtain good performance for this application. Linux has a lot of knobs that can be tweaked to improve system performance for a specific application. The challenge is in figuring out the right set of knobs that need to be tweaked. Using various profiling and benchmarking tools, the performance hotspots and the optimizations to address the same, for a NAS server application have been identified and documented.

Test data:

Bonnie++ benchmarks (ODS)
MagellanCIFS/XP (CSV)
MagellanNFS (CSV)

Share

Talkback: Discuss this article with The Answer Gang

Activae: towards a free and community-based platform for digital asset management

By Manuel Domínguez-Dorado

Abstract

Every day, more businesses, organizations, and institutions of all kinds generate media assets that become part of their value chains, their know-how and production processes. We understand a multimedia digital asset not only as a media file (audio, video, or image) but as the combination of the file itself and its associated metadata; this allows, from an organizational perspective, exploiting, analysis, and documentation.

About Activae project

From this perspective, it is essential that organizations have platforms that allow the management of multimedia assets. There are several platforms of this kind on the market [1]: Canto Cumulus, Artesia DAM, Fuzion Media Asset Management, Nuexeo DAM, and others. However, the number of simple and scalable platforms based on open source software is not remarkable and that is why CENATIC [2] (Spanish National Competence Center for the Application of Open Source Technologies) has developed Activae [3]; a platform for digital and multimedia asset management that is highly scalable, open source, vendor-independent, fully developed in Python, and designed for the Ubuntu Server.

If we were to list the features of activae according to their importance, these would be [4]:

• Scalability: Activae consists of four modules: transcoding, data access, web frontend, and an API exported via XML/RPC. Each one of them allows load balancing and high availability, so the scalability possibilities are broad. It can all run on one machine or additional machines can be arranged as required. If the use of the platform requires more storage, you can easily hot-plug it. If you use a lot of transcoding of multimedia formats, machines can be dedicated specifically to this work and even more can be added as you see the need.
• Load balancing: is provided by the internal engine of the platform, the Cherokee web server [5]. Access to the application can be configured in this way to provide service in those cases where visitors to the platform are abundant.
• Transcoding: Activae allows advanced format transcoding in the platform itself. If, for example, a video asset having DivX AVI format is added, it can be transformed seamlessly into a video MPEG H.264, or FLV to web. You can even convert a video to an MP3; Activae is responsible for automatically eliminating the video layer and extracting the audio. It supports transcoding between 200 different formats/containers, and most importantly, the transcoding is done asynchronously - decoupled and detached (in the background) while you can continue working with Activae. It is a non-blocking process.
• 100% AJAX GUI: provided by the CTK library [6] (Cherokee Tool Kit), a jQuery wrapper written in Python that allows the development of advanced, light and professional user interfaces.
• The platform is fully developed in Python 2.5, familiar to many readers: the API that is exported via XML/RPC, the AJAX user interface, the control and business logic, the database access, etc...
• It is open source. Activae is licensed under the New BSD License terms. This allows modification, use, distribution, sale, or relicensing, which is an advantage to any individual, company, or organization that wishes to use their properties for personal use or to provide services to third parties, expanding its portfolio of products and services and ultimately expanding their business opportunities.

As for the technical soundness of the platform, it is ensured by being based on renowned open source components: Xapian, Cherokee, Python, ImageMagick, FFMpeg, etc. The following figure shows more specifically the modules that are used by Activae, how they are interconnected, and how it used each one of them.

Model of community and government

The project is currently housed in the Forge of CENATIC [7] . For now, the CENATIC staff is managing its evolution, although we are completely open to incorporate more participants: the more players, the better. Workers from companies such as Octality [8] and ATOS Origin [9] are involved in the process; various developers and beta testers of other entities are conducting pilot projects to deploy the platform (museums, universities, television networks, associations, media companies, ...)

The development model used is the "rolling release". That is, there is no single development branch constantly being evolved. Stable packages that contain highly tested features are released on a frequent basis. To the extent that the merits of the developers dictate, they go on to have enough permissions to manage their own community around the platform. No one is excluded on grounds of origin, sex, race, age, or ideology. The community is a meritocracy: open, flexible, and dynamic, with the main goal of changing the product for the benefit of all.

Call for participation

The project has been released very recently. This implies that the user and developer community around the platform is still small, although the growth rate is quite high. Therefore, CENATIC calls for the participation in the project to all Python developers of the open source community that want to be part of an innovative and forward-looking project. The way to get started is to sign up in the Forge of CENATIC and request to join the project. But you can also work anonymously downloading the sources via Subversion platform.

svn checkout https://svn.forge.morfeo-project.org/activae/dam

All necessary information for participation can be found in the project Wiki [10]. Any input is welcome: developers, translators, graphic designers, beta testers, distributors... or just providing ideas for improvements or future features. Although the website is currently available only in Spanish, we are working hard to get the English version up as soon as possible. In any case, the internal documentation of the source code as well as the comments for each commit are completely written in English.

About CENATIC

CENATIC is the Spanish National Competence Center for the Application of Open source Technologies. It is a public foundation, promoted by the Ministry of Industry, Tourism, and Commerce of the Spanish Government (through the Secretary of State for Telecommunications and Information Society and the public entity red.es) and the Extremadura Regional Government. Members of the Board are also the Regional Governments of Andalusia, Asturias, Aragon, Cantabria, Catalonia, the Balearic Islands, and the Basque Country, as well as the Atos Origin, Telefonica and GPEX companies.

CENATIC's mission is to promote knowledge and use of open source software in all areas of society, with special attention to government, businesses, the ICT sector - providers or users of open source technologies - and development communities. In addition, the established Spanish regulatory framework corresponds to CENATIC general advice on legal issues, technology, and methodology most appropriate for the release of software and knowledge.

Related links and references

• [1] http://www.opensourcedigitalassetmanagement.org/reviews/available-open-source-dam/
• [2] http://www.cenatic.es
• [3] http://activae.cenatic.es
• [4] Activae, el gestor de activos digitales y multimedia. M. Dominguez-Dorado, Isabel San Jose Pelaez. Todo Linux 121. Pages. 10-14. Iberprensa press (Madrid). DL M-39535-2000. December, 2010.
• [5] http://www.cherokee-project.com
• [6] http://svn.cherokee-project.com/browser/CTK
• [7] http://forja.cenatic.es/projects/activae
• [8] http://www.octality.com
• [9] http://www.es.atosorigin.com/es-es/
• [10] http://activae.cenatic.es/wiki/index.php/Main_Page

Share

Talkback: Discuss this article with The Answer Gang

Away Mission for April 2011

By Howard Dyckoff

There are quite a few April events of interest to the Linux and Open Source professional. The menu includes hard-core Linux development, Android, MySQL, system and network security, and cloud computing.

USENIX NSDI '11

NSDI '11 is actually March 30-April 1 in Boston. It is co-located with and preceeded by LEET '11, the Large-Scale Exploits and Emergent Threats seminar.

NSDI '11 will focus on the design, implementation, and practical evaluation of large-scale networked and distributed systems. It is primarily a forum for researchers and the papers presented there represent the leading research on these topics. Of course, it starts before April 1st. But it is worth noting and, if you are in Boston, you might catch the end of it.

I attended LEET and NSDI in San Francisco 3 years back and was impressed with the technical depth and the level of innovative thinking. But I also have to say that many of the presentations represent advanced grad student work at leading universities. Part of the conversation is about the footnotes and several of the presentations are follow-ups to research presented at earlier NSDI conferences or related events. I can honestly say that none of the presentations were duds but equally you couldn't split your attention with email or twitter... unless you were in that research community.

All of the papers are available after the conference and past papers from past Symposia are available in the NSDI archive: http://www.usenix.org/events/bytopic/nsdi.html

Here is the link to the abstracts for this year's NSDI: http://www.usenix.org/events/nsdi11/tech/techAbstracts.html

Multiple Linux Foundation events

The Linux Foundation Collaboration Summit is in its 3rd year. It has always been a kind of family week for the extended community on the West Coast, especially as several other major Linux events often occur in the Boston area.

The Linux Foundation Collaboration Summit is an invitation-only gathering of folks like core kernel developers, distro maintainers, system vendors, ISVs, and leaders of community organizations who gather for plenary sessions and face-to-face workgroup meetings in order to tackle and solve pressing issues facing Linux. Keynote speakers include the CTO of Red Hat, Brian Stevens, and the head of Watson Supercomputer Project, David Ferrucci.

This year, because the merger of the Foundation with the CE Linux Forum (CELF) in 2010, the Summit is followed by an updated Embedded Linux Conference at the same San Francisco venue, allowing for down time between events and lots of socializing. Co-located with the 2011 Embedded Linux Conference is the day and a half Android Builders Summit, beginning mid-day Wednesday, April 13th and continuing on April 14th with a full day of content.

The ELC conference, now in its 6th year, has the largest collection of sessions dedicated exclusively to embedded Linux and embedded Linux developers. CELF is now a workgroup of the Linux Foundation because the principals believe that by combining resources they can better enable the adoption of Linux in the Consumer Electronics industry.

You can use this link to register for both of the embedded Linux events: http://events.linuxfoundation.org/events/embedded-linux-conference/register

Also, Camp KDE is co-located with the Collaboration Summit and actually precedes it on April 4-5. So the festivities actually run from the 4th until the 14th.

There will be "Ask the Expert" open discussion forums on several technical topics and Summit Conference sessions on the following key topics: Btrfs, SystemTap, KVM Enterprise Development, File and Storage for Cloud Systems and Device Mapper, Scaling with NFS and Tuning Systems for Low Latency.

An annual favorite at the Summit is the round table discussion on the Linux Kernel. Moderated by kernel developer Jon Corbet, the panel will address the technology, the process and the future of Linux. Participants will include James Bottomley, Distinguished Engineer at Novell's SUSE Labs and Linux Kernel maintainer of the SCSI subsystem; Thomas Gleixner, main author of the hrtimer subsystem; and Andrew Morton, co-maintainer of the Ext3 filesystem. In previous Summits, this has been both lively and detailed.

Last year, the Collaboration Summit was where the partnership of Nokia, Intel, and The Linux Foundation on MeeGo was announced. I understand that there should be more such announcements this year, especially as there is now also a mini-Android conference at the end of the 2 week period.

View the full program here: http://events.linuxfoundation.org/events/end-user-summit/schedule

Interested in the 2010 presentations? Use this link: http://events.linuxfoundation.org/archive/2010/collaboration-summit/slides

InfoSec World 2011

InfoSec World will be taking place on April 19-21, 2011 in Orlando, FL. The event features over 70 sessions, dozens of case studies, 9 tracks, 12 in-depth workshops, and 3 co-located summits. The last last of these summits, the Secure Cloud & Virtualization Summit, will take place on the Friday after InfoSec. It features a session on VMware Attacks and How to Prevent Them. This is a serious and professional security conference.

The opening keynote at InfoSec this year is "Cybersecurity Under Fire: Government Mis-Steps and Private Sector Opportunities" by Roger W. Cressey, counterterrorism analysmmmt and former pesidential advisor.

I attended an InfoSec event in 2008 and found the level of presentations to be excellent. For instance, there was a detailed session on detecting and exploiting stack overflows and another 2-part session with a deep dive on Identity Management.

And their "Rock Star" Panel promised and delivered challenging opinions on the then-hot information security topics. In 2008, the panelists discussed hardware hacking and included Greg Hoglund, CEO of the now more famous HBGary, Inc. and founder of www.rootkit.com, and Fyodor Vaskovich, Creator of Nmap and Insecure.org, among other notables. (This is actually the pseudonym of Gordon Lyon.)

My problem with prior security events hosted by MISTI is that it had been very hard to get most of the presentation slides after the event and nothing was available to link to for perspective attendees who might want to sample the content - or to readers of our Away Mission column. This year the management at MISTI is aiming to do better.

Among the sessions presented this year at InfoSec World, these specifically will be accessible by the public after the conference - sort of the opposite of RSA's preview mini sessions:

1. - Cloud Computing: Risks and Mitigation Strategies- Kevin Haynes
2. - Computer Forensics, Fraud Detection and Investigations- Thomas Akin
3. - Using the Internet as an Investigative Tool- Lance Hawk
4. - Social Networking: How to Mitigate Your Organization's Risk- Brent Frampton
5. - Securing Content in SharePoint: A Road Map- David Totten
6. - Cloud and Virtualization Security Threats- Edward Haletky
7. - Hacking and Securing the iPhone, iPad and iPod Touch DEMO- Diana Kelley
8. - Google Hacking: To Infinity and Beyond - A Tool Story DEMO- Francis Brow/ Rob Ragan
9. - Penetration Testing in a Virtualized Environment DEMO- Tim Pierson

For more information about InfoSec World 2011, visit: http://www.misti.com/infosecworld

MySQL User Conference

After Sun's purchase of MySQL a few years ago, and recently Oracle's purchase of Sun, the bloom may be coming off the open source flower that MySQL has been. Many of the original developers are working on the fork of MySQL that is Maria. And O'Reilly has made it somewhat difficult for Linux Gazette to attend this and other events that they host. But there is still a huge user base for MySQL and this is the annual user conference.

Last year's event seemed a bit more compressed -- fewer tracks, fewer rooms in the hotel adjacent to the SC Convention Center, fewer vendors in the expo hall. Also, in a sign of the depressed economy, there was no breakfast in the mornings. There were coffee and tea islands between the session rooms, but the signature Euro breakfast at the MySQL conference was missing, as were many of the Europeans. This is now clearly an American corporate event.

That being said, there were better conference survey prizes last year -- an iPad or a free pass to MySQL 2011 -- and it still has a community presence with BOFs from community projects.

Last year, conference organizers added an Open MySQL unconference room. This was only one space and it seemed to be only lightly used. This year may be different.

Another interesting change were the slides between sessions promoting other sessions organized by show sponsors like Pentahoo and Kickfire, as well as several sponsored by Oracle itself. That seemed a bit heavy handed... your mileage may vary.

The actual presentations emphasized MySQL tuning and admin, plus sessions on alternate DBs such as NoSQL, RethinkDB, Drizzle, and MariaDB (see MariaDB: 20 Slides, 5 Minutes, The Full Monty by Michael Widenius). Here is the OReilly link for materials from the 2010 MySQL conference: http://en.oreilly.com/mysql2010/public/schedule/proceedings

CloudSlam '11

CloudSlam '11 is the 3rd annual CloudSlam conference, covering latest trends and innovations in the world of cloud computing. Conference panels, workshops, and tutorials will cover topics in cloud computing, including public sector cloud initiatives, transitioning traditional IT resources into the cloud, Cloud 2.0, Application Delivery, Private cloud / Public cloud, and Greener Data Centers.

This is a hybrid global event: The first day, April 18th, is a physicial conference, followed by 4 days of virtual conference.

The $99 cost of the regular registration includes access to all Cloud Slam 2011 live online sessions and downloadable conference proceedings plus access to Cloud Slam '09 and Cloud Slam '10 downloadable conference proceedings. But some of the sponsors have sent out discount registration codes. The $299 VIP rate includes face time the first day at the Microsoft Silicon Valley campus, should you want that.

Online access to just the 2010 conference proceedings will set you back $29.95 plus a $1.74 fee for eventbrite.com: http://postcloudslam10.eventbrite.com/?ref=etckt

So attending CloudSlam is a small investment, both in time and cash. We didn't attend last year but this seems to be a good value at low risk.

NSDI '11 USENIX Symposium on Networked Systems Design and Implementation
Sponsored by USENIX with ACM SIGCOMM and SIGOPS
March 30-April 1, 2011, Boston, MA
http://www.usenix.org/events/nsdi11/

Linux Foundation Collaboration Summit 2011
April 6-8, Hotel Kabuki, San Francisco, CA
http://events.linuxfoundation.org/events/collaboration-summit

Embedded Linux Conference and Android Builders Summit
April 11-13, Hotel Kabuki, San Francisco, CA
http://events.linuxfoundation.org/events/embedded-linux-conference

MySQL Conference & Expo
April 11-14, Santa Clara, CA
http://en.oreilly.com/mysql2011/

Cloud Slam - Virtual Conference
April 18-22, 2011, On-line
http://www.cloudslam.org

InfoSec World 2011
April 19-21, 2011, Orlando, FL
http://www.misti.com/infosecworld

Share

Talkback: Discuss this article with The Answer Gang

Howard Dyckoff is a long term IT professional with primary experience at Fortune 100 and 200 firms. Before his IT career, he worked for Aviation Week and Space Technology magazine and before that used to edit SkyCom, a newsletter for astronomers and rocketeers. He hails from the Republic of Brooklyn [and Polytechnic Institute] and now, after several trips to Himalayan mountain tops, resides in the SF Bay Area with a large book collection and several pet rocks.

HAL: Part 3 Functions

By Henry Grebler

I began this series with HAL: Part 1 Preliminaries. I followed this with HAL: Part 2 My Favourite Aliases. This article deals with functions and another will talk about scripts.

In the first article there was a link which would allow the reader to download my HAL. There you will find 63 functions. My use of aliases predates my use of functions, so I would expect fewer functions to be obsolete.

And yet, as I went through my functions while writing this article, I found several that I no longer use; didn't realise I had.

Note that sometimes you will notice a function (for example, 'help'), and another with the same name with 'o' at the end ('helpo'). This is my convention for the older version of a function. It allows me to be experimental in modifying a function, confident that I can easily invoke the previous version which worked well.

In writing this article, I learned a trick. To list all aliases, one simply issues the command alias without arguments. How do I list bash functions? In the past I have used set, but that lists all variables as well as functions. Eventually I discovered the less-than-obvious declare -f.

		* * * * * * * * * * * * *

	BSD_su HP_su Linux_su

The su command must be about the most idiosyncratic command found on Unix-like platforms. In ~/.bashrc, I define functions for each platform. Near the bottom of that file, I set the alias su according to the current platform.

On a FreeBSD machine, I see this:

	type -a su
	su is aliased to `BSD_su'
	su is /usr/bin/su

whereas on a Linux machine I see this:

	type -a su
	su is aliased to `Linux_su'
	su is /bin/su

		* * * * * * * * * * * * *

	alias

The alias function was discussed in part 2. It's probably worth repeating.

Normally, you have to write

        alias name=value

My function allows me to write

        alias name value

This is why in $HOME/.ENV./.cshrc_bash you will see all the aliases defined without the use of an equal sign (=).

I started using c-shell before I discovered bash (which originated about 1989). In c-shell, the syntax for aliases does not use an equal sign:

        alias name wordlist

My function facilitated the transition from c-shell to bash.

		* * * * * * * * * * * * *

	cutf

For many requirements, cut is much more flexible than awk. Syntax like -f 3-7, imbues cut with a convenience which is vastly greater than that of awk. However, cut is much more restrictive in its notion of a field; whereas awk understands fields in a more human way. If a delimiter is not specified to awk, it treats white space as a delimiter. The default delimiter for cut is the tab character.

I think I intended to build up my function cutf over time. In the event, it has served me well in its initial incarnation.

I can (and often do) write something like

	scan | grep -i article | cutf 1

in preference to

	scan | grep -i article | awk '{print $1}'

It's just so much more convenient.

Here's why I might do it. This example is from the way I handle my mail with nmh. I've edited a little for readability.

$ scan | grep -i article
  30 -14Feb2011 Ben Okopnik        Re: Article (perhaps): Life Can be Tough
  33  14Feb2011 Ben Okopnik        Re: Article update: Everything You Always
  52  19Feb2011 Ben Okopnik        Re: Rounding articles 
$ aaa=`scan | grep -i article | cutf 1`
$ refile $aaa +lg

The first command displays a summary of current emails (in my inbox), 1 line per email, for those lines in which the string "article" appears. In the second command, I gather together just the first field of each line (the message numbers). The third command moves the selected messages to a different mailbox.

I could do the above in fewer lines, but that's not my objective here.

		* * * * * * * * * * * * *

	dc

This function panders to my apparent dyslexic typing. I often get the order of my characters wrong. I may type dc $HOME when I meant to type cd $HOME. There is a command dc (an arbitrary precision calculator), but the way I use it, it normally takes no arguments. The dc function checks if there is a single argument, and whether that single argument is a directory, and if so does what I meant rather than what I said. If I type dc without args, it invokes the command dc.

Obviously, I never use the first form deliberately.

		* * * * * * * * * * * * *

	dirs dsd dsdc dth files links

These are all variations on the ls command. dirs shows just directories; files shows just regular files; links shows just symbolic links. dsd shows all entries except for emacs backup files with the result piped through more.

dsdc is like dsd but only shows entries with today's (the current) date. It turned out to be a stepping stone. I think something like dsdc existed on VMS. Nothing similar was available on Linux. However, these days dth has almost completely supplanted dsdc.

The function dth was discussed exhaustively in What Really Matters or The Counting Article.

		* * * * * * * * * * * * *

	gt_doit gttv gttvu gtx

In essence, gttv is a function which lists the table of contents of a tar-zipped file. It performs gunzip or bunzip2 (depending on file type) piped into "tar -tvf -" piped into my pager. gttvu does all of the above preceded by a uudecode (not all that common these days). gtx is like gttv except it unpacks a tar-zipped file.

gt_doit is simply the helper function invoked by the other functions.

		* * * * * * * * * * * * *

	help

This function is discussed in detail in Henry's Techno-Musings: My Help System.

		* * * * * * * * * * * * *

	lesh

One of my most frequently used functions. Think of it as a cross between less and dth. With no arguments, it works like less on the most recent file in the current directory. If the first argument is of the form +n, it displays the first n files in the current directory, or, if there are args after the +n, the first n files which match the remaining args. If args are files, they are the candidates for most recent; if they are directories, the files within those directories are the candidates.

For example, lesh +3 /var/log will display the 3 most recent files in the system log directory.

The section "My 'less' command" in Henry's Techno-Musings: My Help System explains why this allows me to scan through several files conveniently.

		* * * * * * * * * * * * *

	lest

I really should use the concept of this function more extensively. Let's say I have script. It could be mine, or one somewhere on the system; I know its name; it's in my path; but I can't remember where it is. Or I can remember where it is, but I don't really care. I just want to look at it with less. lest scriptname will find it and open it. For example, irb is a ruby script in /usr/local/bin. lest irb opens it in less.

It's a consolidation of my typical behaviour. Often, I found myself going type -a command, then wanting to look in the file in the answer.

Here's a better example. It shows that lest also works on aliases.

type -a tcu
tcu is aliased to `/home/henryg/scripts/tarcompressuuencode.sh'

less tcu makes no sense ("tcu: No such file or directory"). But lest tcu works just fine.

		* * * * * * * * * * * * *

	loc

This is my variation on find. With a single arg, it looks in the current directory tree for an entry with the supplied string somewhere in its name. With 2 args, it takes the first as the directory to search.

These days I think it would be more useful if it were rewritten to do "find | grep" rather than "find -name".

		* * * * * * * * * * * * *

	mangl

Sometimes I get the impression there are man pages everywhere. It's hard to keep up. On most of the systems I manage, I have a pretty good handle on where man pages are. But even on these systems, some man pages pop up in unexpected places. On systems that I visit but don't control, man pages could be anywhere. If I've tried apropos and man and still haven't found what I'm looking for, I might try locate. If this exposes a man page of interest, I can use mangl pathname to get the man page formatted with nroff.

And, yes, the name is a weak pun.

		* * * * * * * * * * * * *

	mcd

Another simple function that often gets a work-out. Basically, mkdir plus cd: make a directory and cd to it.

		* * * * * * * * * * * * *

	ping

I have never seen the point of having ping relentlessly bang on the door of the remote machine. It reminds me of Sheldon in The Big Bang Theory: [knock, knock, knock] "Penny!" [knock, knock, knock] "Penny!" [knock, knock, knock] "Penny!"

/sbin/ping fw
PING fw (192.168.0.1): 56 data bytes
64 bytes from 192.168.0.1: icmp_seq=0 ttl=64 time=0.340 ms
64 bytes from 192.168.0.1: icmp_seq=1 ttl=64 time=0.278 ms
64 bytes from 192.168.0.1: icmp_seq=2 ttl=64 time=0.207 ms
64 bytes from 192.168.0.1: icmp_seq=3 ttl=64 time=0.240 ms
...

Surely once is enough:

ping fw
PING fw (192.168.0.1): 56 data bytes
64 bytes from 192.168.0.1: icmp_seq=0 ttl=64 time=0.277 ms



--- fw ping statistics ---
1 packets transmitted, 1 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.277/0.277/0.277/0.000 ms

On Solaris, by default, all you get is a single line: "fw is alive".

		* * * * * * * * * * * * *

	truss

A very important function. I guess I encountered truss on Solaris. I learned to use and love it. So, of course, when I moved to other platforms, I soon felt the need. Linux flavours have strace; FreeBSD has truss, but the parameters are different.

Once again, I have a function that hides these idiosyncrasies from me and lets me get on with the job.

And That's About It

I don't think there's much more to say about functions. Only 14 descriptions of about 25 functions. The others aren't worth the trouble.

Share

Talkback: Discuss this article with The Answer Gang

synCRONicity: A riff on perceived limitations of cron

By Henry Grebler

	Insanity is hereditary; you get it from your children.
	    Sam Levenson

My younger son (Rob, 22 soon) wants more Internet. Of course. 20GB a month is not enough for him. While he was away, my wife and I were pushing to use 20% of that.

Over the years, I have developed zillions of mechanisms to monitor our Internet usage. Well, perhaps 5. Or maybe just one, which I refined 4 times. And what do I mean by our Internet usage?! Not our usage; not mine, not my wife's; we're well behaved. Just Rob's. If not for him, we wouldn't need to monitor anything.

So, if I'm going to monitor Internet usage at all, I may as well monitor everyone's.

He has returned from 5 months studying in Malaysia. He comes to me with ISP plans that provide 100 GB per month (with a catch or two). Since the subject is on the table, I check what our usage is so far this month.

Anyone who knows me knows I prefer to live in an xterm. I could go to Firefox and navigate half-a-dozen screens to get the information I want, but long ago I wrote an expect script, get_optus_usage.exp, wrapped in a shell script, get_optus_usage.sh, to do the job for me.

I get back something like:

   Plan Data Limit = 20000MB
   Data Used 17% (3517 MB)
   Days Elapsed 7% (2 days)

Am I surprised? No. Annoyed? Sure; but surprised - never.

I decide it's time to run get_optus_usage.sh more frequently. I will run it as a cron job every night at 23:55. Why that particular time? Well, for most days, it will make no difference. As long as it runs every 24 hours (more or less) I'll get an idea of the usage on each day. But on the last day of the month, I'll know the usage for the month. Shortly thereafter, my ISP will reset its counters.

Here's my problem. I know my script works: I just ran it. (I also know that it doesn't always work. Whenever my ISP gets, um, creative and starts rearranging the furniture, the script breaks. I then tweak it, and it works until next time. But that's not my problem du jour.) Will it work when invoked from cron?

My first answer is: not in a way that is useful. When run interactively, it displays its answers on the xterm (stdout). If run from cron, that output will be emailed to me. (Actually, it won't: I don't run a mail daemon. Nevertheless, an attempt will be made to email me.) I don't want the output in an email; I want it in a log file.

Ok, so I will need to redirect stdout. I could do that in the crontab entry, something like this:

55 23 * * * get_optus_usage.sh >> /var/CableUsage/optus_usage.log

(I already have /var/CableUsage from all the other monitoring I've been doing - see above.)

But, in many years of writing cron jobs, I've learnt that it ain't that simple. Jobs that work fine during development become quite recalcitrant when run from cron.

I'm sure I'm not the only one who has this problem. I'll look it up on the Internet. What am I looking for? "cron", of course. Maybe "testing".

I am not alone! "About 1,450,000 results," says Google. But, wait. They're not answers. They're zillions of befuddled folk like me looking for answers.

	How can I test my crontab?
	How can I test a new cron script ?
	is crontab working? how to test run scripts with crontab
	How do I test a cron for functionality?

I spent quite a while reading the questions and the answers. The best that people could come up with is to create a crontab entry to run every minute (or in a few minutes - but you might need a lot of repeats on that idea) and then tweak your job, wait until the next minute, check the answer, etc.

There are just so many things that are wrong with that approach.

So I gave up. I wrote a wrapper script, cron_get_optus_usage.sh:

#! /bin/sh
#       cron_get_optus_usage.sh - interlude for when running from cron

        dir=/home/henryg/projects/TECH/web_capture
        $dir/get_optus_usage.sh >> /var/CableUsage/optus_usage.log

It didn't work.

'...  Your Terminal type is unknown!....  Enter a terminal type: [vt100] '

I added

        TERM=xterm
        export TERM

and then it worked. Pretty lucky. Only took two days.

	synchronicity

	the simultaneous occurrence of events with no discernible causal connection 
	 -- Concise Oxford English Dictionary © 2008 Oxford University Press

Next day, I get an email from my other son, Mark:

	I have a script that when I run from the shell, it works fine
	but when it's run from cron it doesn't work properly.

My response contained all the advice I had acquired over many years of dealing with cron; and it helped him to solve his problems.

But, clearly, the need is there.

The Trouble with Cron

The main trouble with cron is that it is, in effect, a part of production. Even if you are using your personal crontab, reporting still uses the system logging facility, syslog; entries go into log files in /var/log or similar; and communication with the user is typically via the official mail system. So you would like to proceed with caution.

If you're like me, you don't like your system logs filled with dross. At some future date, it may interfere with the diagnosis of real problems.

So you would expect to find a suitable test mechanism. Dream on.

With the default cron program, the degrees of freedom are: running in the foreground, using a non-default mail program, and a couple of other options which are not helpful for testing.

One thing you can't do is specify a different crontab from the default. So if you add entries of your own and get distracted, you may leave the crontab in an undesirable state.

You might try to run as a user other than root and only affect a test-user's crontab. But, if your job needs to run as root (or some other privileged user), that's going to create more problems.

There are two very important requirements for testing: the ability to run the job now (ie on demand); the ability to run in a test environment identical to the environment when run from cron.

Real Sysadmins

Just to be clear: I reject any macho notion of the sort that claims that "real sysadmins" don't need this testing nonsense. We get it right the first time. What's wrong with you? Are you a pussy?

My credo is to minimise risk.

I'm not saying that the use of test features should be made compulsory. If you don't want to test, don't. I hope that you are particularly brilliant or particularly lucky, because, to my mind, your behaviour is unnecessarily risky.

Other cron programs

I investigated a few other cron programs.

Anacron(8) looked promising because you can use -t to specify an alternate anacrontab. There is also the option -f to force immediate invocation of all jobs in the anacrontab. There is also option -n to run jobs now. No, I don't understand the difference between -f and -n either; I just use both. You can run it in the foreground which has the happy effect of displaying "informational messages to standard error". But that's "as well as to syslog."

Sounds pretty good. Unfortunately, anacron reproduces the environment of the interactive user.

I also tried fcron (http://fcron.free.fr/). It turned out to be no more promising than anacron.

In fairness, anacron and fcron don't claim to be useful for testing. As far as I could tell, they do what they are intended to do. They just didn't help me.

So there is no assistance when it comes to testing cron jobs.

Refined

I've refined my script. It now looks like this:

        dir=/home/henryg/projects/TECH/web_capture
        {
                cat <<EOF
------------------------------------------------------------------------
EOF
                date
                $dir/get_optus_usage.sh
        } >> /var/CableUsage/optus_usage.log 2>&1

It draws a line to separate entries, displays the date and time and runs the real program (get_optus_usage.sh); and redirects all the output to the log file.

And here is an edited log entry:

------------------------------------------------------------------------
Wed Feb 23 23:55:01 EST 2011
Billing Period:           01 Feb 11 - 28 Feb 11
Last Update:           23 Feb 11 11:45 PM AET/AEDT
   Plan Data Limit = 20000MB
   Data Used 65% (13142 MB)
   Days Elapsed 79% (22 days)

All very satisfactory for this particular task. But what about an ability to test cron?

A Sufficient Solution

I have a solution of sorts. You could actually invoke it from cron. I might be inclined to do so for the last run before considering a task ready for production.

But I'm reasonably sure that the last step is not necessary.

Here's the script; I call it cronsim.sh:

#! /bin/sh
#       cronsim.sh - simulate the behaviour of invoking a job from cron

#----------------------------------------------------------------------#
# To customise, modify
#
#       CRONHOME
#       LOGDIR          (if required)
#
# for example
#
#       CRONHOME=/usr/local/cronsim
#----------------------------------------------------------------------#

        CRONHOME=/home/henryg/projects/TECH/cronsim

        if [ $# -eq 0 ]
        then
                TGT=$CRONHOME/TEST
        else
                TGT=$1
                if [ ! -e $TGT ]
                then
                        cat <<EOF
Specified target script ($TGT) not found.
EOF
                        exit 1
                fi
        fi

        TODAY=`date '+%Y%b%d'`
        LOGDIR=$CRONHOME/logs
        [ -d $LOGDIR ] || mkdir $LOGDIR || {
                cat <<EOF
Unable to create log directory $LOGDIR
EOF
                exit 1
        }
        LOG=$LOGDIR/log.$TODAY

        {
                cat <<EOF
------------------------------------------------------------------------
EOF
                date
                echo 'Calling args <<<'"$@"'>>>'
                ls -la $TGT
                [ -e $TGT ] && env -i $TGT

                cat <<EOF
------------------------------------------------------------------------
EOF
        } >> $LOG 2>&1

It's not rocket science.

Typically, to invoke it, you provide it with the path to your proposed cron script, for example:

	cronsim.sh get_optus_usage.sh

If cronsim.sh is unable to find or access the script being tested, you'll get an error message.

Basically, it invokes the requested script, logs all output to a date-stamped log file, records its input args, and does a bit of prettifying.

The key fragment is

	env -i $TGT

ENV(1)                           User Commands                          ENV(1)

       env - run a program in a modified environment

       -i, --ignore-environment
              start with an empty environment

Running your cron script with an empty environment is overkill. There are probably some environment variables that are set when a cron job is invoked. But it is sufficient: if your cron job runs (or can be modified to run) from cronsim.sh, I'm pretty sure it will run under any invoking conditions.

You can start with a script (or even part of a script) and satisfy yourself that it does what you want when run interactively (ie like a normal script) - even as a non-root user, if that makes sense. If appropriate, you can then run it under cronsim.sh, still without root privileges to ensure that it still works or to understand the sort of adjustments that will be needed.

Almost certainly you will need to set your PATH variable.

I expect that as you get nearer to the final version of your script, it will only make sense to run it with root privileges. You can still invoke it from cronsim.sh (as long as you invoke cronsim.sh with the relevant privileges).

Does that mean that by the time you put it into production your script will be perfect? Of course not. But I would expect it to be much nearer than if you just set it up to run from cron without testing.

Other Problems with cron

My Internet search indicated that people have other problems with cron. They can't seem to get their jobs to run when they want them to. I've never had that problem. It has been stated that cron syntax is difficult. I suppose there are difficulties.

One difficulty I encountered came in monitoring our Internet usage. I alluded to this earlier. Back then I wanted to run on the last day of the month. There's no way to achieve that with cron. I settled for:

55 23 28-31 * * get_optus_usage.sh

If memory serves, in the past I have written scripts to expect to get invoked as above, but to first perform the calculation as to whether today is the last day of the month. If not, just exit; otherwise, do the work.

There may be a solid argument for a better cron with a better mechanism for specifying when jobs should run, but that's beyond the scope of this article.

Wish List

What are the elements of a variant of cron that would make testing more convenient?

I would like these abilities:

• select an alternate crontab
• run the job on demand
• avoid sending messages to system logs
• avoid sending emails
• run a job under the same conditions as when run normally
• nominate the user to run as

Summary

I cannot claim thousands of hours of flying time with cronsim.sh (not yet, anyway). But I have used it a few times, not so much because I needed to, but to try to give it some testing.

I'm sure that it will undergo refinement.

Eventually I might decide it's a lost cause and go back to the drawing board. That might mean modifying the real cron to do what I want.

I like my script because limited testing has shown it to be platform-independent. I've run it on Linux and FreeBSD. If I start modifying cron, that means C and compiling - and a different executable for different platforms.

I am interested to see whether this is an item with a market of one (me) or whether I strike a responsive chord.

And, of course, if anyone has any suggestions, I'm interested. In particular, if, out in the badlands of the Internet, there is a variant of cron that I've missed and ought to consider, a variant that is a good fit for my criteria above, I would like to hear about it.

Share

Talkback: Discuss this article with The Answer Gang

Applying Yum Package Updates to Multiple Servers Using Fabric

By Ikuya Yamada and Yoshiyasu Takefuji

Introduction

Administrators of multiple servers know that updating software packages can be a tedious task. It requires a log in, followed by the execution of a "yum update" or something similar for each server. Although several tools exist for enabling administrators to update packages automatically, including yum-updatesd and yum-cron, updating the packages without confirmation possibly leads to further issues.

For instance, Yum completely ignores the dependencies of software compiled from the source. If the compiled software has strict version requirements against the update of packages, it may not function properly after the update. Furthermore, even if the package is installed using Source RPM with custom compile options, it is overwritten by a new package and the features specified in the compile options are disabled.

In this article, we introduce a Python-based short script that enables administrators to easily apply command-line package updates to multiple servers. Using this script, the administrator can manually confirm which packages are to be updated. Internally, this script executes a "yum check-update" in sequence in order to verify the presence of package updates and displays a prompt that allows the administrator to select whether the package updates are to be performed.

The proposed script uses a relatively new Python library called Fabric, which allows users to execute commands at multiple remote servers via SSH. Therefore, the script needs to connect to the user's remote servers via SSH.

Although we tested the script on CentOS 5.5 Linux, administrators using other RedHat-based distributions can also use this script.

Example of Error Caused by Yum

In this section, we will provide an actual example for demonstrating the issue that arises in the Yum package update. Postfix, which is a common mail server, implements MySQL support from recent versions. However, the Postfix package provided by the default Yum repository does not include this feature, and hence, many users install Postfix from the Source RPM.

Let us assume that the server has a previous version of Postfix installed from the Source RPM and uses the configuration given in Postfix MySQL Howto .

# sudo rpm -ivh PREVIOUS_POSTFIX_SOURCE_RPM_URL
# cd /usr/src/redhat/
# sudo vim SPEC/postfix.spec
< %define MYSQL 0
> %define MYSQL 1
# rpmbuild -ba SPEC/postfix.spec
# rpm -ivh RPMS/x86_64/postfix-VERSION_NUM.x86_64.rpm

After executing the "yum -y update", Yum detects and installs the newer version of Postfix.

Loaded plugins: downloadonly, fastestmirror
Loading mirror speeds from cached hostfile
(snip)
=======================================================================================================
 Package                   Arch                     Version                     Repository        Size
=======================================================================================================
Updating:
 postfix                   x86_64                   2:2.3.3-2.1.el5_2           base             3.7 M

Transaction Summary
=======================================================================================================
Install       0 Package(s)
Upgrade       1 Package(s)
(snip)
Running Transaction
  Updating       : postfix                                                                         1/2 
warning: /etc/postfix/main.cf created as /etc/postfix/main.cf.rpmnew
  Cleanup        : postfix                                                                         2/2 

Updated:
  postfix.x86_64 2:2.3.3-2.1.el5_2                                                                     

Complete!

Because the installed package does not contain the MySQL support, Postfix starts to output the following errors in /var/log/maillog after it is restarted.

Feb 18 00:31:29 jango postfix/master[29981]: warning: process /usr/libexec/postfix/local pid 6189 exit status 1
Feb 18 00:31:29 jango postfix/master[29981]: warning: /usr/libexec/postfix/local: bad command startup -- throttling
Feb 18 00:32:29 jango postfix/local[6191]: fatal: unsupported dictionary type: mysql

If it is not confirmed whether the updated package works properly, the Yum package update may occasionally result in server failure. Therefore, we recommend the use of our proposed script instead of the automatic package update provided by yum-updatesd and yum-cron .

Installing Fabric

The proposed script requires several software packages. First, the EPEL software repository needs to be installed. Administrators using a Linux distribution based on RedHat Enterprise Linux 5 can install the repository using the following command. Installation instructions can also be found here.

# sudo rpm -Uvh http://download.fedora.redhat.com/pub/epel/5/i386/epel-release-5-4.noarch.rpm

Next, several software packages need to be installed.

# sudo yum -y install gcc python26 python26-devel python26-distribute

Now, Fabric can be installed using easy_install , which also installs the required Python libraries using the following command.

# sudo easy_install-2.6 fabric

Script

The proposed script is given below. As Fabric uses fabfile.py as its default file name, the script should be entered into fabfile.py.

# cd INSTALL_DIR
# vim fabfile.py

#!/usr/bin/env python2.6

from fabric.api import env, run, sudo
from fabric.contrib.console import confirm

env.warn_only = True

def update():
    if run("yum check-update").return_code != 0:
        if confirm("execute yum update?", default=False) == True:
            sudo("yum -y update", pty=True)

As mentioned above, the script executes a yum check-update in each remote server and displays a prompt that enables the user to select whether to execute the yum update. Note that env.warn_only is a necessary component; this is because yum returns a non-zero status code if it finds any package updates, which Fabric incorrectly recognizes as an error and responds by shutting the execution down.

Now, the script can be executed. fab is the command that receives the target Python function name as an argument and runs the function. In this case, the function name is update , and thus, the command is as follows.

# fab -H HOST_NAME1,HOST_NAME2 update

After executing the above command, the script checks for the presence of package updates and outputs the following. As can be seen, if package updates exist, the user can choose whether to perform the updates after confirming the packages that are to be updated.

[qui-gon.ikuya.net] Executing task 'update'
[qui-gon.ikuya.net] run: yum check-update
[qui-gon.ikuya.net] out: Loaded plugins: fastestmirror
[qui-gon.ikuya.net] out:
[qui-gon.ikuya.net] out: bind-libs.x86_64                        30:9.3.6-4.P1.el5_5.3            updates
(snip)
[qui-gon.ikuya.net] out: udev.x86_64                             095-14.21.el5_5.1                updates

Warning: run() encountered an error (return code 100) while executing 'yum check-update'

execute yum update? [y/N]

The user name and password can be optionally specified using -u and -p options, respectively.

# fab -u USER_NAME -p PASSWORD -H HOST_NAME1,HOST_NAME2 update

The target remote server addresses can also be specified using the env.hosts variable in fabfile.py.

#!/usr/bin/env python2.6

from fabric.api import env, run, sudo
from fabric.contrib.console import confirm

env.warn_only = True

env.hosts = [
    "HOST_NAME1",
    "HOST_NAME2"
]

def update():
    if run("yum check-update").return_code != 0:
        if confirm("execute yum update?", default=False) == True:
            sudo("yum -y update", pty=True)

Using the above script, the user can run the update without specifying the remote server addresses.

# fab update

Conclusion

This article proposed a short script that enables server administrators to easily apply package updates to remote servers. The script uses a Python library called Fabric , and executes package updates in sequence.

Note that Fabric is a generic Python library and can be used to automate numerous other server administration tasks. If you want to learn more about Fabric, please refer to the official documentation .

Share

Talkback: Discuss this article with The Answer Gang

Ubuntu, the cloud OS

By Neil Levine

We made a small flurry of announcements recently, all of which were related to cloud computing. I think it is worthwhile to put some context around Ubuntu and the cloud and explain a little more about where we are with this critical strategic strand for our beloved OS.

First of all, the announcements. We announced the release of Ubuntu Enterprise Cloud on Dell servers. This is a hugely significant advance in the realm of internal cloud provision. It's essentially formalising a lot of the bespoke work that Dell has done in huge data centres (based on a variety of OSes) and making similar technology available for smaller deployments. We attended the Dell sales summit in Las Vegas and we were very encouraged to meet with many of the Dell salespeople whose job it will be to deliver this to their customers. This is a big company, backing a leading technology and encouraging businesses to start their investigations of cloud computing in a very real way.

More or less simultaneously, we announced our formal support for the OpenStack project and the inclusion of their Bexar release in our next version of Ubuntu, 11.04. This will be in addition to Eucalyptus, it is worth stating. Eucalyptus is the technology at the core of UEC - and will be in Ubuntu 11.04 - as it has been since 9.04. Including two stacks has caused some raised eyebrows but it is not an unusual position for Ubuntu. While we look to pick one technology for integration into the platform in order to deliver the best user experience possible, we also want to make sure that users have access to the best and most up to date free and open-source software. The increasing speed of innovation that cloud computing is driving has meant that Ubuntu, with its 6 month release cadence, is able to deliver the tools and programs that developers and admins want before any other operating system.

Users will ultimately decide what deployment scenarios each stack best suits. Eucalyptus certainly has the advantage of maturity right now, especially for internal cloud deployments. OpenStack, meanwhile, continue to focus on rapid feature development and, given its heritage, has appeal to service providers looking to stand up their own public clouds. Wherever the technology is deployed, be it in the enterprise or for public clouds, we want Ubuntu to be the underlying infrastructure for all the scenarios and will continue to direct our platform team to deliver the most tightly integrated solution possible.

Finally we saw our partner, Autonomic Resources announce UEC is now available for purchase by Federal US government buyers. This is the first step on a long road the federal deployment, as anyone familiar with the governmental buying cycles will realise. But it is a good example of the built-to-purpose cloud environments that we will see more of - with the common denominator of Ubuntu at the core of it.

Which actually raises an interesting question - why is it that Ubuntu is at the heart of cloud computing? Perhaps we ought to look at more evidence before the theory. In addition to being the OS at the heart of new cloud infrastructures, we are seeing enormous usage of Ubuntu as the guest OS on the big public clouds, such as AWS and Rackspace, for instance. It is probably the most popular OS on those environments and others - contact your vendor to confirm :-)

So why is this OS that most incumbent vendors would dismiss as fringe, seeing such popularity in this new(ish) wave of computing? Well there are a host of technical reasons to do with modularity, footprint, image maintenance etc. But they are better expressed by others.

I think the reason for Ubuntu's prominence is because it is innovation made easy. Getting on and doing things on Ubuntu is a friction-free experience. We meet more and more tech entrepreneurs who tell us how they have built more than one business on Ubuntu on the cloud. Removing licence costs and restrictions allows people to get to the market quickly.

But beyond speed, it is also about reducing risk. With open-source now firmly established in the IT industry, and with the term open used so promiscuously, it is easy to forget that the economic benefits of truly free, open-source software. The combination of cloud computing, where scale matters, and open source is a natural one and this is why Ubuntu is the answer for those who need the reassurance that they can both scale quickly but also avoid vendor lock-in in the long-term.

More specifically, and this brings us back to the announcements, there are now clear scenarios where users can reach a point where even the economics of a licence-free software on a public cloud start to break down. At a certain stage it is simply cheaper to make the hardware investment to run your own cloud infrastructure. Or there might be regulatory, cultural or a host of other reasons for wanting cloud-like efficiencies built on internal servers.

The work we have done with OpenStack and with Eucalyptus means Ubuntu is an ideal infrastructure on which to build a cloud. This will typically be for the internal provision of a cloud environment but equally could be the basis or a new public cloud. It is entirely open as to the type of guest OS and in all cases continues to support the dominant API of Amazon EC2, ensuring portability for those writing applications.

And as we have seen, Ubuntu is the ultimate OS to deploy in a cloud and with which to build a cloud. No-one provides more up-to-date images on the most popular public cloud platforms. Our work to ensure compatibility to the most popular standards means that those guests will run just as well on a UEC cloud however that is deployed - either internally or for cloud provision externally.

So technology moves markets. Economics does too, only more so. Ubuntu has come at the right point in our short IT history to ride both waves. The scale is there, the standards are emerging and the ability to provide an answer to the choice between running a cloud or running on a cloud is more fully realised on Ubuntu than on any other OS - open source or not.

Share

Talkback: Discuss this article with The Answer Gang

Back in the Day: Red Hat Linux 3.0.3 - 4.2, 1996-1997

By Anderson Silva

December 1996, I had just finished my first semester of college. For Winter break, I decide to go to my uncle’s house in Long Island, NY. My uncle, a PhD student at Stony Brook University, hands me a brick-like book that comes with a free operating system called Red Hat Linux, version 3.0.3.

The original Red Hat Linux 3.0.3 CD that came with the book my uncle gave me back in 1996.

February 2011, It’s a cold Winter day in North Carolina, and I am sick with the flu. As I lay around the house trying to find something to do, I run across the one object arguably responsible for my entire 15 year academic and profession career. The thought crosses my mind: “I wonder if I could get this CD to install on a virtual machine?”

And that is what this nostalgic article is all about: a trip down memory lane with Linux. First I tried installing Red Hat Linux 3.0.3 (Picasso), from 1996, which runs on kernel 1.2.13. The boot process for installation is pretty involved as the CD back on those days was not bootable. The installation process involves 3 floppy disks, 1 for the boot image, and 2 ramdisk images. I was able to load all 3 floppy images, but the kernel died before the installation began.

Red Hat Linux 3.0.3: boot image loaded.

Red Hat Linux 3.0.3: failing to install

Unfortunately, I was not able to get version 3.0.3 running on the virtual machine (as you can see above), but I wasn’t ready to give up quite yet. I then decided to go through the next version I owned, Red Hat Linux 4.0. The results:

Version 4.0 was the first official version of Red Hat that was bootable via the CD (that I know of). I was able to boot it up, and even install the OS, but LILO, the bootloader that used to come with Linux distributions by default before GRUB came around, wouldn't install... so, yet again, I decided to move on to the next available version.

Red Hat Linux 4.0 (Colgate,1996).

Red Hat Linux 4.0: Got stuck here... oh well.

Finally, success! The next version available was Red Hat Linux 4.2 (Biltmore, 1997). The installation was pretty straightforward except for getting X server running. Once I figured out how to at least get it running in VGA mode and use the correct mouse protocol, I was able to have some fun with it. Red Hat Linux uses kernel 2.0.30.

Red Hat Linux 4.2, Biltmore. The window manager name was called: FVWM95.

Red Hat Linux 4.2: I had to use the control-panel to start up eth0 using DHCP, but it worked.

Red Hat Linux 4.2: As you can see there wasn’t many applications available out of the box, but it was still more than Windows 95 had to offer. :-)

Red Hat Linux 4.2: GUI for RPM package management. It is worth reminding the reader that 1997 precedes online update tools like up2date, yum and apt-get.

Red Hat Linux 4.2: Installing an application.

Red Hat Linux 4.2: Arena browser with today's Google loaded! :-) Back in those days, you had to download Netscape on your own.

Red Hat Linux 4.2: I really wanted to surf the web with something a bit more advanced than the Arena Browser, after some searching around I found a binary of Netscape 4.08 for 2.0.x Linux kernel. Now, be aware that Netscape Communicator 4.08 only came out in November 1998, which means, by them time Netscape 4.08 came out  Red Hat Linux 5.2 was already out. Anyway, I downloaded and installed it on  Red Hat Linux 4.2 (May 1997), and to my surprise, it worked! Now, compare the Netscape look to the Arena Browser (previous screenshot) to have an idea why Netscape became the powerhouse it became back in those days.

Red Hat Linux 4.2: Today's Google.com through the eyes of 1998's Netscape Communicator.

That weekend with the flu this past February ended up being quite fun as I did not stop after installing Red Hat Linux 4.2. I ended up spending all day installing Red Hat Linux 5.0, 6.1, 7.2, 9 and Fedora Core 1, and I hope to share them with you in future issues of the Linux Gazette.

Share

Talkback: Discuss this article with The Answer Gang

Taking the Pain Out of Authentication

By Yvo Van Doorn

When dealing with authentication, there are a myriad of systems out there that validate user identity and permissions: shadow passwords, Kerberos, NIS and LDAP, just to name a few.

In early Unix and (very early) Linux deployments, the applications would be responsible for directly interacting with whatever security system that was in place for a given server. When it was just the app scanning the /etc/passwd file, that wasn't so hard to manage, but once all those other authentication systems were deployed, then leaving authentication in the application space quickly became unmanageable.

In 1995, Sun Microsystems came up with a better way: a library that would handle all of the communication between applications and whatever authentication system that might be in place. This would take the onus of authentication off application developers' plates, and prove to be much easier to expand. If a new authentication system is put into place, a shared library object can be added to the library, without any changes to the app. This library is known as the pluggable authentication module (PAM), and it's a big part of authentication systems used today.

The biggest advantage to PAM is scalability: the capability to add a new authentication system to any given server without needing to re-configure (or worse, re-compile) a program is a big deal. That's not to say there isn't some sort of payment to be made -- nothing is free. Instead of touching the application, all PAM-aware applications have a text configuration file that must be edited to compensate for any new module within the PAM library.

At first, that may seem like more trouble than it's worth, since managing multiple configuration files could quickly get painful. In truth, individual PAM configuration files enable each application to use any authentication protocol they need, even if they're on the same system. So, one application can use LDAP, while another can rely on shadow passwords, if that's what you need. Better still, with one notable exception, switching to a new protocol is as simple as plugging in the protocol's module to PAM then editing the application(s) configuration files to make note of the change. The application only cares what PAM is telling it, not what the actual authentication solution says.

Another important feature of PAM, which makes it of interest to authentication providers like Likewise, is that it can be used for more than just password management. PAMs can be configured to outline session setup, logging, permissions management, as well as work directly with Active Directory for authentication.

Modules can be combined as well, thus giving apps a layered approach to authentication, if needed. Individual PAM configuration files are comprised of a list, known as a "stack," of all the PAM modules that will govern access for the application in question. Modules are invoked in their stack order and any module failure will usually block access for the app. More lenient policies can be put in place that might circumvent this all-or-nothing approach.

One example of such a policy might be to allow a user to physically access a system at any time of the day, but not grant access remotely during certain times. (Care must be taken, though, to allow for international travelers who may indeed need to login remotely in the middle of the server's night.) Another such module could authenticate a user only if their Bluetooth-enabled device was broadcasting nearby.

Of course, like any system, PAM isn't without its vulnerabilities. A security hole this past summer would allow a malicious user to own the permissions of a vulnerable system's shadow password file. With access to the shadow file, the cracker could easily reset the password for every account on the machine, including root. At that point, the cracker can do just about anything to the box, including resetting the shadow file back to its former state, so that system administrators would be less likely to notice something was amiss. This vulnerability was very quickly patched, but it outlines the need to keep a constant eye on any authentication- or security-based frameworks.

The extreme scalability and flexibility of the PAM framework means it can fit within almost any security policy an organization might come up with. If a module or combination of modules doesn't meet specific security needs, then a new module can be written as a solution. Linux users, for instance, can visit Andrew Morgan's The Linux-PAM Module Writers' Guide to learn more about creating such solutions.

PAM is a major tool in the security world, and one well worth understanding, since it can do far more than just handle passwords.

Share

Talkback: Discuss this article with The Answer Gang

HelpDex

By Shane Collinge

These images are scaled down to minimize horizontal scrolling.

All HelpDex cartoons are at Shane's web site, www.shanecollinge.com.

Talkback: Discuss this article with The Answer Gang

Part computer programmer, part cartoonist, part Mars Bar. At night, he runs around in his brightly-coloured underwear fighting criminals. During the day... well, he just runs around in his brightly-coloured underwear. He eats when he's hungry and sleeps when he's sleepy.

XKCD

By Randall Munroe

Click here to see the full-sized image

Click here to see the full-sized image

Talkback: Discuss this article with The Answer Gang

Doomed to Obscurity

By Pete Trbovich

These images are scaled down to minimize horizontal scrolling.

All "Doomed to Obscurity" cartoons are at Pete Trbovich's site, http://penguinpetes.com/Doomed_to_Obscurity/.

Talkback: Discuss this article with The Answer Gang