Freedom, with their exception.

This is mostly for future reference, but maybe you will find it useful. This is on an Ubuntu 10.04.3 LTS box.

Cisco Setup

All I am doing here is trunking one interface (G2/0/18) and restricting access to two vlans (101 and 102).

switch#configure terminal
Enter configuration commands, one per line.  End with CNTL/Z.
switch01(config)#interface G1/0/18
switch01(config-if)#switchport trunk encapsulation dot1q
switch01(config-if)#switchport mode trunk
switch01(config-if)#switchport trunk allowed vlan 101,102
switch01(config-if)#exit
switch01(config)#exit
switch01#show running-config interface G1/0/18
Building configuration...

Current configuration : 167 bytes
!
interface GigabitEthernet1/0/18
 switchport trunk encapsulation dot1q
 switchport trunk allowed vlan 101,102
 switchport mode trunk
end

switch01#

Linux Setup

1. Add the 8021q module to the kernel

   # modprobe 8021q

2. Add the module to the /etc/modules file, so that it loads at next startup

   # echo 8021q >> /etc/modules

3. Install the appropriate package

   # apt-get install vlan

4. Configure the network. In my example, eth0 is connected to a trunk port on a Cisco switch, with vlans 101 and 102 trunked to it. eth1 is connected to a separate network and switch.

   # cat /etc/network/interfaces.
   
   auto eth0.101
   iface eth0.101 inet static
   	address 172.17.101.9
   	netmask 255.255.255.0
   	network 172.17.101.0
   	broadcast 172.17.101.255
   	gateway 172.17.101.1
   	vlan_raw_device eth0
   
   auto eth0.102
   iface eth0.102 inet static
   	address 172.17.102.9
   	netmask 255.255.255.0
   	network 172.17.102.0
   	broadcast 172.17.102.255
   	vlan_raw_device eth0
   
   auto eth1
   iface eth1 inet static
   	address 172.17.100.9
   	netmask 255.255.255.0
   	network 172.17.100.0
   	broadcast 172.17.100.255

5. Restart networking. I came across inconsistent results by using /etc/init.d/networking restart and the ifup and ifdown commands. While your mileage may vary, I just rebooted the box.
6. Make sure things are working. As you can see, eth0 does not have an ip address, though the vlan interfaces, eth0.101 and eth0.102, do.

   kmjohnson@squid:~$ ifconfig
   eth0      Link encap:Ethernet  HWaddr 14:fe:b5:2c:15:86
             inet6 addr: fe80::16fe:b5ff:fe2c:1586/64 Scope:Link
             UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
             RX packets:23763 errors:0 dropped:0 overruns:0 frame:0
             TX packets:10485 errors:0 dropped:0 overruns:0 carrier:0
             collisions:0 txqueuelen:1000
             RX bytes:3481376 (3.4 MB)  TX bytes:2500480 (2.5 MB)
             Interrupt:36 Memory:d6000000-d6012800 
   
   eth1      Link encap:Ethernet  HWaddr 14:fe:b5:2c:15:88
             inet addr:172.17.100.9  Bcast:172.17.100.255  Mask:255.255.255.0
             inet6 addr: fe80::16fe:b5ff:fe2c:1588/64 Scope:Link
             UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
             RX packets:973 errors:0 dropped:0 overruns:0 frame:0
             TX packets:257 errors:0 dropped:0 overruns:0 carrier:0
             collisions:0 txqueuelen:1000
             RX bytes:75737 (75.7 KB)  TX bytes:29086 (29.0 KB)
             Interrupt:48 Memory:d8000000-d8012800 
   
   eth0.101  Link encap:Ethernet  HWaddr 14:fe:b5:2c:15:86
             inet addr:172.17.101.9  Bcast:172.17.101.255  Mask:255.255.255.0
             inet6 addr: fe80::16fe:b5ff:fe2c:1586/64 Scope:Link
             UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
             RX packets:17260 errors:0 dropped:0 overruns:0 frame:0
             TX packets:10350 errors:0 dropped:4 overruns:0 carrier:0
             collisions:0 txqueuelen:0
             RX bytes:2445291 (2.4 MB)  TX bytes:2400114 (2.4 MB)
   
   eth0.102  Link encap:Ethernet  HWaddr 14:fe:b5:2c:15:86
             inet addr:172.17.102.9  Bcast:172.17.102.255  Mask:255.255.255.0
             inet6 addr: fe80::16fe:b5ff:fe2c:1586/64 Scope:Link
             UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
             RX packets:6503 errors:0 dropped:0 overruns:0 frame:0
             TX packets:101 errors:0 dropped:3 overruns:0 carrier:0
             collisions:0 txqueuelen:0
             RX bytes:513299 (513.2 KB)  TX bytes:8438 (8.4 KB)
   
   lo        Link encap:Local Loopback
             inet addr:127.0.0.1  Mask:255.0.0.0
             inet6 addr: ::1/128 Scope:Host
             UP LOOPBACK RUNNING  MTU:16436  Metric:1
             RX packets:277 errors:0 dropped:0 overruns:0 frame:0
             TX packets:277 errors:0 dropped:0 overruns:0 carrier:0
             collisions:0 txqueuelen:0
             RX bytes:25230 (25.2 KB)  TX bytes:25230 (25.2 KB)
   
   kmjohnson@squid:~$ ping 172.17.101.1
   PING 172.17.101.1 (172.17.101.1) 56(84) bytes of data.
   64 bytes from 172.17.101.1: icmp_seq=1 ttl=64 time=0.124 ms
   ^C
   --- 172.17.101.1 ping statistics ---
   1 packets transmitted, 1 received, 0% packet loss, time 0ms
   rtt min/avg/max/mdev = 0.124/0.124/0.124/0.000 ms
   kmjohnson@squid:~$ ping 172.17.102.1
   PING 172.17.102.1 (172.17.102.1) 56(84) bytes of data.
   64 bytes from 172.17.102.1: icmp_seq=1 ttl=64 time=9.39 ms
   ^C
   --- 172.17.102.1 ping statistics ---
   1 packets transmitted, 1 received, 0% packet loss, time 0ms
   rtt min/avg/max/mdev = 9.391/9.391/9.391/0.000 ms
   kmjohnson@squid:~$

Issues

I ran into an issue where when connecting to an interface that did not have a gateway defined (eth0.102), the packet was being dropped by the kernel. To fix this, the following two lines were needed in /etc/sysctl.conf:

net.ipv4.conf.default.rp_filter=0
net.ipv4.conf.all.rp_filter=0

Below is a pretty simple ruby script for parsing files uploaded to an Amazon S3 bucket and inserting the file’s information into a MySQL database.

#!/usr/local/bin/ruby

require 'rubygems'
require 'aws/s3'
require 'mysql'
require 'lockfile'

begin
 Lockfile.new('/tmp/scraper.lock', :retries => 0) do # Setup the lock file

         my = Mysql::new("localhost", "nfssupport", "password", "upload_files") # Setup the MySQL connection
         q = my.query("Select FileName, FileModified from files") # Setup the MySQL query
         db = Array.new # Create the array for the MySQL file list
         q.each_hash do |f| # For each file in the DB
                 db << "#{f['FileName']} - #{f['FileModified']}" # Put it into the array
         end

         AWS::S3::Base.establish_connection!( # Create the S3 connection
                 :access_key_id      => 'access_key_id',
                 :secret_access_key  => 'secret_access_key'
         )
         files = AWS::S3::Bucket.objects( # Put the S3 files into the files array
                 'domain-tld',
                 :prefix => 'uploads/user@domain.tld/uuid'
         )

         files.each do |file| # For each S3 file
                 t = Time.parse(file.about['last-modified'])
                 file_date = "#{t.year}#{sprintf('%02d',t.month)}#{sprintf('%02d',t.day)}"
                 file_name = file.key.split('/',4).last.gsub(/^(\d+\-)/,'')
                 file_modified = file.about['last-modified'].split(' ',5).last

                if !db.include? "#{file_name} - #{file_modified}" # If file not in DB
                         st = my.prepare("insert into files (FileName, FileDate, FileModified, FileLink, FileDescription, FileSize) VALUES (?, ?, ?, ?, ?, ?)
")
                         st.execute(file_name, file_date, file_modified, "https://uploads.domain.com/download/#{file.key.split('/',3).last}", file.metada
ta['x-amz-meta-description'], file.about['content-length']) # Insert it
                         st.close
                end
         end
 end

rescue Lockfile::MaxTriesLockError => e
        puts "Scraper is already running!"
end

After having such a hard time finding (m)any good examples online, this article is going to explain how to use perl to connect to Amazon’s S3 service to download files. I am by no means a perl expert, so take the below code with a pound of salt. This example assumes that you have already a working Amazon S3 service, and the appropriate keys.

This code is used to download files which our customer’s upload to our support site. The files are downloaded and stored in a directory by date (YYYYMMDD). We have a number of different users and as such only want to download the files for a specific user ($prefix).

The first thing that you will need to do is to install the Amazon::S3 CPAN module with a command similar to:

cpan -i Amazon::S3

Now for the code:

#!/usr/bin/perl

use strict;
use warnings;
use Amazon::S3;

## Amazon S3 Stuff ##
my $aws_access_key_id     = 'aws_access_key_id';
my $aws_secret_access_key = 'aws_secret_access_key';
my $bucket = 'bucket-name';  # The bucket name that files are uploaded to.
my $prefix = 'uploads/bucket-name@domain.tld/uuid';  # We only want to get the files that were uploaded for a specific user.
my $http_prefix = 'https://uploads.domain.tld/download/uuid/';  # I use this string to build the URL
## Amazon S3 Stuff ##

## Script Variables ##
my $directory = '/zstore/uploads/';  # Top level of where to save files to
my $logfile = $directory . "scrape.log";  # Where to log the file downloads to.
## Script Variables ##

open(my $fh, "+>>", $logfile);  # Open the file handle for writing logs

my $s3 = Amazon::S3->new({  # This sets up the Amazon S3 connection.
 aws_access_key_id      => $aws_access_key_id,
 aws_secret_access_key  => $aws_secret_access_key
});

print $fh localtime() . ": Getting file list...\n";

my $files = $s3->list_bucket({  # This gets the entire list of files under the $prefix in the $bucket, or dies with an error.  $files is a multidimensional hash.
 bucket => $bucket,
 prefix => $prefix
}) or die $s3->err . ": " . $s3->errstr;

print $fh localtime() . ": Got file list...\n";                 # If we have made it this far, we now have the entire list of files.

foreach my $file ( @{ $files->{keys} } ) {                      # $file is a hash of arrays of hashes, AKA multidimensional hash.
 my $file_name = substr($file->{key}, 69);                      # 20110224110053-file.tar.gz
 my $file_true_name = substr($file->{key}, 84);                 # file.tar.gz
 my $file_date = substr($file->{key}, 69, 8);                   # 20110224
 my $sub_directory = $directory . $file_date;                   # /zstore/uploads/20110224
 my $file_path = $sub_directory . '/' . $file_true_name;        # /zstore/uploads/20110224/file.tar.gz
 my $url = $http_prefix . $file_name;                           # $url is the actual url to the file

 unless (-d $sub_directory){                                    # Unless the directory is already created
  print $fh localtime() . ": Creating $sub_directory...\n";     # Log that we are creating the directory
  mkdir($sub_directory);                                        # And create that directory
 }

 unless (-e $file_path) {                                       # Unless the file has already been downloaded
  print $fh localtime() . ": Fetching $file_name...\n";         # Log that we are fetching the file
  my $line = system("fetch -mq -o '$file_path' '$url'");        # Actually download the file, assigning the return code of `fetch` to $line
  if ($line == 0) {                                             # If the file is successfully downloaded
   print $fh localtime() . ": Succesfully got $file_path...\n"; # Log it
  } else {                                                      # Else
   print $fh localtime() . ": Problem getting $file_path...\n"; # Could not fetch it for whatever reason.  Move on.
  }
 }
}
print $fh localtime() . ": Exiting!\n\n";                       # We've went through the entire file list.  Log it.
close($fh);                                                     # Close the file handle.

If you spot any errors or have any questions, comments or feedback, please post a comment!