William Herrin's RESUME
Facebook (Seattle, WA)
Production Engineer, March 2020 - Present
- Member of the operating systems team responsible for the
fleet of Linux servers.
- Wrote an
automation safety plugin for Red Hat's DNF software
- Automated large-scale software deployment and configuration
in Chef (Ruby).
- Built system management tools in Bash and Python.
Amazon Web Services (Seattle, WA)
Senior Software Engineer, November 2018 - November 2019
- Worked on the data plane team for Global Accelerator, a load
balancer with BGP-anycasted global entry points.
- Wrote high performance Internet packet handling code in C
using Intel's DPDK framework.
- With deep research, solved a TCP throughput degradation that
bedeviled the first release due to an unexpected secondary
effect from network address translation.
- Led design of a core part of the source address preservation
effort, delivering packets to customers without obscuring the
original Internet source.
- Managed devops and software deployment including
participation in a 24/7 operations on-call rotation.
Tesla Government, Inc. (Vienna, VA)
Devops Lead, April 2017 - October 2018
- Transitioned a system from a physical rack environment to
Amazon Web Services' Govcloud, including VPCs, Red Hat Linux and
Windows 2012 EC2 instances, AWS security groups, Route 53 DNS,
MySQL RDS databases, etc.
- Designed the AWS Virtual Private Cloud (VPC) structure to overcome
routing limitations so that all data traffic was forced through
appropriate EC2-based security appliances.
- Built Perl software which interacts with the Amazon IAM APIs
and Microsoft Active Directory to implement password changing
and multifactor authentication compliant with NIST Special
Publications 800-63 revision 3 and 800-53 draft revision 5. This
included collecting and implementing a database of more than 17
million compromised password patterns which are used to reject
- Implemented Microsoft Active Directory on Windows 2012R2 as
the central passwords repository for the system. Built custom
software in C for the Linux servers to rely on Active Directory
via LDAP for passwords only without joining the Linux servers to
- Implemented Linux shell users by creating a Red Hat RPM
package for each. This allows developers to customize their
account via the git repository and have it automatically
installed on every server via the normal "yum udpate" process.
It also guarantees the prompt automated removal of accounts when
- Implemented compliance with DoD's Risk Management Framework
based on NIST Special Publication 800-53 revision 4 with
Identity and Authentication inserts from draft revision 5.
- Shepherded the new system from concept through the full DoD
Authority to Operate (ATO) and Final Operational Capability
(FOC) process including continuous security monitoring with
- Designed a Continuous Integration / Continuous Deployment
(CI/CD) process using Red Hat Packages (RPMS) and the Yum tool
as the primary automation source. Jenkins builds software from
git commits, delivers RPMs to a Yum repository and then triggers
a yum update. Unlike with tools such as Puppet, this approach
does not suffer from the inefficiency of having to recheck the
whole system configuration every 30 minutes or having the
automation tool blindly overwrite configurations.
- Using a combination of Linux iptables logging rules, Snort
and Splunk, implemented a Network Intrusion Detection System
(IDS) based on building a profile of expected data traffic and
alerting on packets which fail to fit the profile.
- Implemented a primary system firewall using Linux and
iptables, with remote access to the system using OpenVPN.
- Implemented automatic backups for Mac OSX laptops using
shell scripts, network drives and Time Machine.
University of Maryland University College. (Adelphi, MD)
Linux Engineer III, April 2016 - March 2017
- Member of a team supporting a large Amazon Web Services
(AWS) cloud deployment.
- Worked with Elastic Compute (EC2) virtual
machines in a private AWS network.
- Worked with AWS security groups
- Worked with Elastic Load Balancers (ELB).
- Worked with Route 53 DNS management and load
- Produced Red Hat RPM packages from a git repository. RPM
packages flowed in
to a puppet-based devops environment.
- Built RPMs with "trigger" scripts which alter the
installation of other RPMs each time they're upgraded. Triggers
let me employ 3rd party RPMs that disobey the RPM standard by
writing unmanaged files to RPM-controlled directories without
having to repackage the 3rd-party RPM for each upgrade.
- Managed Apache Tomcat Java web servlet containers. Wrote a
consolidated health check servlet in Java which checked
operation of multiple servlets in a background thread and
reported a consolidated status to the AWS load balancers.
- Primary telework from home office (typically 4 of 5 days per
week) using VPNs, Webex, Slack, etc.
- Participated in after-hours on-call rotation
Dirtside Systems, Inc. (Falls Church, VA)
Owner, May 2014 - April 2016
- Contractor supporting software development for
unusual computer network applications.
- Work for undisclosed company.
Contractor to Digital Globe Marine Services.
- Major network security work including expansive use of
Linux iptables, ip rules, multiple routing tables and
SELinux for server hardening
- Produced Red Hat RPM packages from a git repository
using a Jenkins continuous build environment. RPM packages flowed in
to a puppet-based devops environment. Built puppet modules,
jenkins scripts, and RPM packaging scripts.
- Supported pure software developers, mentoring to help them
understand operations-level needs and requirements.
- Wrote glue code in Perl and Python, integrating
disparate software systems
- Some work with VMWare ESXi and Amazon EC2 cloud servers
- Some work with Raspberry Pi-based embedded systems
- Some work with Juniper Netscreen firewalls
- Designed specialty VPN solutions based on
OpenVPN and Cisco Anyconnect.
- Worked on the Orb Map fish finder back end Linux system
including cloud virtualization
- Wrote and deployed daily production system using Perl.
- Constructed Red Hat packages (RPMs) for production
- Assisted with debugging of C++ science code.
- Established custom networking and email services connecting
ships with satellite modems to the data source.
ITT Exelis (Dulles, VA)
Principal Engineer, March 2010 - April 2014
- Led the design and implementation of a resilient computer room with standard n+1 power
and air conditioning systems, as well as the structured cabling system for a company
- Built a "distributed responsibility" developer network, facilitating
development of many Internet-connected products. Compartmented systems accessed via VPNs
from engineers' desks permitted parallel development of systems with divergent security
architectures. Included a wifi network with roaming between multiple access
- Built a network of more than 70 virtual machines distributed through a dozen countries
in North America, Europe and Asia.
- Designed and built a system for managing hundreds of millions of
geographic map imagery tiles using C, Perl, some Python and the Linux FUSE
- Defined HTTP-based service oriented architecture APIs for a large
- Built two more BGP-using multi-site continuity of operations networks
- Managed a team of three engineers responsible for around 100 servers, routers and
switches serving multiple government projects.
ITT (Dulles, VA)
Senior Software Engineer, May 2008 - March 2010
- Architected, led the implementation and delivered a geographically diverse (Virginia
& Hawaii) continuity of operations system for an Iridium satellite base-end station.
Use Linux LVS load balancers controlled by custom Perl-based software to
access active-active redundant servers, OSPF and Quagga to attach IP
addresses to the active firewall and BGP to bring packets into the network
at both sites.
- Implemented process control for software packaging and deployment.
- Implemented thorough hardware and software monitoring and reporting system
comScore, Inc (Reston, VA)
Senior Software Engineer, November 2007 - April 2008
- Using C++, C, Perl and shell scripting, developed a Linux-based network appliance that
captured and analyzed HTTP packet traffic (deep packet inspection).
Democratic National Committee (Washington, DC)
Internet Infrastructure Manager, September 2005 - November 2007
- Managed a staff of three engineers responsible for 70+ Linux servers used by three
Democratic Party organizations.
- Designed and built a resilient computing infrastructure including two collocation data
centers, a 56-mile fiber optic ring connecting them with national headquarters, multiple
Internet backbone connections and BGP reciprocal peering.
- Implemented thorough hardware monitoring and reporting to identify and replace ailing
servers before they break.
- Helped specify an Avaya VoIP phone system for 300 users.
- Cleaned up the DNC's split horizon DNS. Implemented consistency checking between the
internal and external servers.
- Performed professional management in the following areas: computer/system operations,
systems administration, communications network administration, software development,
systems software support, hardware support, database administration.
Cambridge Communications Systems, Inc.
Research Analyst - Linux, December 2004 - September 2005
- Contractor for the United States Bureau of the Census, Telecommunications Office.
- Designed and implemented process improvements for a large electronic mail system
combining Linux/Sendmail and IBM/Lotus Domino.
- Designed and built a custom spam control software using C and Perl with individual
customer settings and an intelligent method for handling of false positives.
- Designed and built web security software using Linux, Apache and Perl to permit external
customer access to Lotus iNotes.
Democratic National Committee (Washington, DC)
Senior Information Technology Generalist, August 2004 - December 2004
- Performed complex professional support in the following areas: software development,
computer/systems operations, systems administration, communications network
administration, systems software support, hardware support, database administration, web
- Helped the DNC scale up their Internet operation for the 2004 election.
- Rebuilt the Linux/Apache web application that managed mass-email web bugs and link
handlers. Improved capacity by a factor of twenty while reducing the server count by two
- Constructed and implemented an online UNIX backup solution that backed up multiple
terabytes of data and successfully restored data following several database crashes and an
accidental file deletion. Successfully implemented a differential backup service on the
hundreds of millions of records in the MySQL-based voter database.
- Worked with Linux Virtual Server (LVS) based clustered high availability systems (load
balancers). Helped manage the software and hardware for www.democrats.org
which was spread across 19 servers.
- Member of the team that raised nearly $400 million for the 2004 elections.
- Helped maintain 50+ servers at two sites.
- Troubleshot and rebuilt server hardware, kernel versions, etc.
- Filled in as needed, where needed, when needed.
CrossLink Internet Services (Springfield,
Director of Engineering, August, 2000 - May 2004
- Led a team of three engineers to construct and maintain a large network deploying Cisco,
UNIX, and Windows NT technologies.
- Engineered an Internet Service Provider network consisting of more than 40 sites in 5
states and more than 200 Cisco routers, including a multihomed BGP backbone.
- Developed and deployed an e-mail antivirus system in multi-threaded C and Perl.
- Performed cost analyses on proposed company products. Assist on sales presentations for
- Exercised cost control through careful selection of systems and software, and through
cautious secondary market equipment purchases. Saved the company hundreds of thousands of
dollars on networking equipment.
CrossLink Internet Services (Springfield,
Senior Systems Administrator, November, 1998 - August, 2000
- Developed software including a web based account management application, authentication
and access software, e-mail software, and network monitoring software using C, C++, Perl,
PHP and shell scripts..
- Built and managed UNIX based Internet servers including e-mail servers, web servers,
network monitoring servers, authentication servers, usenet servers, etc.
- Designed and implemented the core computing facility.
netFilter Technologies (Falls Church,
Chief Technical Officer, August 1996 - November, 1998
- Co-Owner of netFilter Technologies, a small
- Lead developer on the Internet Filter, a World Wide Web filtering program used at the
core of the iExalt, SafeSurf, KidzNet
and Library Safe filtering systems.
- Developed Internet Filtering and Security Products.
- Business sold for $300,000 in cash and stock.
Why? Internetworking (Annandale, Virginia)
Owner, October 1995 - August 1998
- Owner/operator of Why? Internetworking, a small
- Developed and operated all World Wide Web services for CrossLink, a regional Internet Service Provider
serving the mid-Atlantic states.
- Developed custom server modifications and accounting software
for the web services.
- Developed World Wide Web and other Internet applications for customers.
- Business sold for $55,000 in cash and services.
Office of Naval Research, Ocean Acoustics Program
Office Automation Clerk, June 1993 - September 1996
- I started working as a summer intern at the Office of
Naval Research in the summer of 1993. I was brought in by Dr. Mohsen Badiey (The
Ocean Acoustics program manager) to run his newly acquired Sun Sparcstation. Dr. Badiey
was so pleased with my performance that at the end of the summer he arranged for me to
continue during the school year, telecommuting from GA Tech via the Internet. I worked
on-site each summer and telecommuting each school year, and continued working for his
replacement, Dr. Jeff Simmen.
- I received an award from the Chief of Naval Research citing
outstanding performance. I detected an intruder who had slipped by the normal computing
staff, breaking in to ONR's computing system via the Internet. I helped the Naval Criminal
Investigative Service begin to trace the perpetrator to his source, and secured the
machine that had been broken in to.
- I was granted a Department of Defense "Secret" level security clearance.
- I managed a network of Sun (Unix) workstations for use by eight people.
- I installed a wide range of services on the Suns, including a customized SMTP (Email)
server, an anonymous FTP server, a World Wide Web server, a Domain Name System (DNS)
server, and a variety of minor servers.
- I converted the Ocean, Atmosphere and Space department's FY 1994 program book for use on
the World Wide Web.
- I configured an Annex terminal server for dial-in/dial-out access.
- I configured ONR's entries in the Internet Domain Name System (DNS) in order to allow
them to interoperate properly with the Internet. I also wrote an application to permit the
network administrators to update the DNS entries using a Microsoft Excel spreadsheet of
host names and IP addresses.
- I installed an Internet Mail gateway for ONR's internal Lan Mail system.
- I assisted with the analysis of data from the AGS '92 Ocean Acoustics experiment at the
Atlantic Generating Station site off the coast of Atlantic City, New Jersey.
- I participated in the SWARM '95 Ocean Acoustics
experiment near the Hudson Canyon, 50 miles off the coast of New York.
- I participated in the AGS '95 Ocean Acoustics experiment at the Atlantic Generating
Station site off the coast of Atlantic City, New Jersey.
Georgia Institute of Technology, Office of
Information Technology (Atlanta, Georgia)
User Assistant, December 1991 - March 1992
- I managed a networked cluster of MSDOS and Macintosh computers for the Georgia Tech Office of Information Technology.
- I provided user support for a broad range of software packages.
- I was selected for the position while a second-quarter freshman. Normally the job is
limited to students who are at least sophomores.
Safeway Food Stores, Inc., Annandale Safeway
Courtesy Clerk, July 1988 - September 1991
- I provided a range of services to customers and staff: bagging groceries, loading
peoples' cars, and locating items.
- I was responsible for end of shift maintenance activities including cleaning, restocking
returned items, supplying rest areas and the outside station, and setting up bagging
stations for the next day.
- I answered a wide variety of questions and requests from customers, often under
difficult circumstances , with the objective of maintaining a positive relationship with
Wednesday, September 02, 2020 05:51 PM.