37

these are just some notes for myself on how i setup a subversion repository on ubuntu dapper using ssh for the access method. these should probably be combined into a script.

install software

you, obviously, have to have dapper installed and also subversion and an ssh server. on our network, you can just network boot and then type “server” at the ubuntu prompt which should install the os. make sure that you either disable network booting or if in the dmz, configure an ip in the dhcp server for that specific mac address in order to not network boot anymore. after everything’s up:

sudo apt-get install subversion openssh-server

you now should be ready to go.

setup the basic subversion area

1. create a subversion (svn) user:
   

   sudo adduser –disabled-password –home /var/lib/svn svn

2. create a directory for the repositories and backups.
   

   sudo mkdir -m 2770 -p /var/lib/svn/repos /var/lib/svn/repos-backup

3. make sure the owners are correct:
   

   sudo chown -R svn:svn /var/lib/svn

setup a new repository

1. create a group for permissions for the new repository. i would make the group name the same as the name of the repository to reduce confusion.
   

   sudo addgroup [repo-group]

2. create a directory where the repository will be located with the correct permissions. setting these permissions will make sure that all of the directories and files that are created when the repository is created will have the proper group set.
   

   mkdir -m 2770 -p /var/lib/svn/repos/[repo-group]

3. set the owner and group for the new repository:
   

   sudo chown svn:[repo-group] /var/lib/svn/repos/[repo-name]

4. create the repository:
   

   svnadmin create –fs-type fsfs /var/lib/svn/repos/[repo-name]

5. make sure the user and group is create on the files.
6. add users to the group who need access to this repository:
   

   sudo adduser [user] [repo-group]

i think that’s basically it. it seems best to create a separate repository for each project. that way it’s easy to keep permissions separated by project group. it’s also appears to not be a requirement with subversion to keep everything in one repository to be able to include them in other projects using modules as in cvs. i haven’t tested this functionality yet though. see external definitions for more information.

repository backups

once you have your repositories setup, clearly you’re going to want to be backing up your precious data. at the current time, i’m taking the fairly simple approach of just backing up the entire repository every night using svnadmin hotcopy.

simple subversion backup script

i just run this script for each repository every night using cron as the svn user. might should be looping through all repositories in the repos directory, but currently i’m just individually adding each to the crontab. the script doesn’t do any pruning of the backups. that’s the responsibility of the backup machine. what needs to be done is to setup a cronjob on the backup server under the target backup user that uses find to remove old backup files. each new backup has a timestamp in the name.
the script just tars up the hotcopy directory and the pipes it to ssh and stores it on a machine we use for backups. to add a user to the backup machine:

sudo adduser –disabled-password –home /home/backups/$USER $USER

after creating the user, you should create a directory in the new user’s home directory that is the hostname from which you will be backing up. this way you could have backups from multiple machines going to the same user. alternatively, i guess you could create a user for each machine. then do backups as the root user on that machine (necessary for certain directories). this still needs to be worked out a bit.

then you have to generate an ssh key on the subversion machine to use for the backups and put the key on the backup machine under the user you just created. this command (all on one line) would be run as the svn user (or whatever user your doing backups for in the general case).

ssh-keygen -b 4096 -t rsa -C $USER-backup@$HOSTNAME -f $HOME/.ssh/$USER-backup

adding new users

this isn’t really specific to subversion, but is generally how servers should be configured. the only way to log into the server is using ssh with public keys. you can disabled root and password authentication in the /etc/ssh/sshd_config file. the best thing to do seems to be to add a default .ssh/authorized_keys file when the machine is first installed that includes the known ssh keys for admins. then, when a new user is added, these keys will be there by default. it’s then fairly easy for an admin to copy in the real user key and remove any admin keys if desired.

the passwords for all non-admin users are disabled. admins need there password in order to be able to run sudo. therefore, all admin users have to also be in the admin group (sudo adduser [user] admin). the same command for generating an ssh key as mentioned above can be used for generating keys.

this information should probably more flushed out in a different post. oh well. that should do for now.

reference

• subversion ubuntu community documentation

the previous post was babbling about what a waste atsc and hdtv have been. i ran across a post today that includes a graph for when resolutions start to matter. as expected, most people with 20/20 won’t be able to notice unless they have a 30″+ display.

granted, years from now it might matter, but i remain unconvinced that most people even 20 years from now will have a 50″+ display. it’s basically everyone subsidizing the cost of a few people’s home theaters. the bump to 720p would have been sufficient. 576p (pal dvd) isn’t listed, but you would assume it would just move the line a reasonable amount farther out.

intro

for starters, you should probably read this excellent page discussing the advanced tv problem that was most likely in context of around the mid-1990’s. i unquestionably side with the c.i.c.a.t.s. proposal. the same site also has good descriptions of interlacing and compression. i would also point out that the european pal format is superior to ntsc in every way. i am an american.

interlacing

let’s just go ahead and get this one out of the way. interlacing has got to go. unnecessary legacy analog crap. in the absence of digital compression technology it was a brilliant solution. however, those days are supposedly gone so it should no longer be present. i found a good write up about interlacing and digital compression. here’s the important snippet from the conclusion if you don’t want to read the whole thing:

Interlacing is a legacy compression technique developed in the days of vacuum tubes. With MPEG compression, not only is interlacing unnecessary, it’s actually undesirable because MPEG gives worse results with interlaced input for the same bit rate.

atsc’s supposed highest hd format is 1080/60i which is interlaced (that’s what the ‘i’ means). therefore, it should be ignored like the red-headed step-child it is. there are actually a 1080/24p and 1080/30p versions of 1080 which would be fine, but appear to be ignored by the broadcasters.

i should point out that there is a lot of non-sense going on right now about blu-ray and hd dvd and having 1080/60p output. this is totally irrelevant as the majority of content (movies) will have been shot in 24p (24 progressive frames per second) and then full frames encoded into 30 interlaced frames. therefore you will get the full resolution of someting shot on film without having 1080/60p. the xbox or playstation 3 is a different story since they couldn’t theoretically generate full 1920×1080 at 60 frames per second (fps).

the presence of interlacing for non-legacy formats in atsc is the first signs that the committee should have been fired.

high resolution

in their infinite wisdom, atsc thought it was a good idea to have multiple resolutions (clearly having never setup a tv for your average human). this is one of the key components of high definition. there’s 1280×720 and 1920×1080. these are the number of pixels (dots) that make up the screen. the 2 most well known forms are 720p and 1080i which are 60 progressive frames and 60 interlaced frames respectively. pretending for a second that interlaced and progressive scanning are equal (which they’re not), these resolutions are useless for most peoples televisions. supposedly your average home viewer is about 2 or 3 meters away from their display. i wear glasses so my vision should be effectively 20/20. at 2 meters, i question whether or not i can tell the difference between an hd picture and a dvd. for sure, you can tell a difference between hd and analog ntsc, but that is more due to the switch to digital than bumping the resolution. for most everyone with a 42″ display or smaller, they will not be able to tell the difference. the juice is not worth the squeeze.

i found this article from nhk called future prospects of hdtv: technical trends towards 1080p. it’s interesting and claims that the at 2 or 3 meters for a 33″ 1280×720 is sufficient and for 50″ 1920×1080. this is to sustain the appearance of reality. whatever that means. i used to watch dvds and hd on a 6′ wide projected image from approximately 3m using an 800×600 projector. i could make out the pixels, but only if i was looking for it. if you’re actually paying any attention to what you are watching, you don’t notice the pixels.

as a side note, most movies with special effects and all animated movies were generated at 2k resolution (that’s a computer ‘k’, so it’s a 2048 pixels wide image which is very close to 1920×1080). it’s only fairly recently that some movie production is done with 4k (4096 wide) images. that means if you saw something like “finding nemo” in the theater you were seeing an hd image projected at about 30′. don’t you think it’s enough on your 3′ screen? obviously, film has much higher color depth, but that’s a different issue that hd doesn’t even address.

framerates

films have been 24 fps for quite some time and yet we’re still stuck with 30 or 60 fps. wait, my bad, that’s actually 29.97 (30 * (1000/1001)) or 59.94 due to the addition of color to black and white back in the day. another, apparently unnecessary retardedness hoisted upon us by the ntsc. supposedly, it wasn’t even necessary to screw with the frame rate. who selects these government agencies? i digress.

basically, it’s slightly complicated to convert a 24fps movie to 30fps for tv. europe and other pal countries don’t have this problem. pal is 25fps. by speeding up the video and audio by 4%, they can show a movie. as a bonus, the movie is done 4% faster. the downside is that it’s conceivable to notice a slightly higher pitch to the audio. however, i believe i read somewhere that it’s not as big of a problem now using pitch correction software that makes all those pop stars’ cds sound so good. in my experience, it was noticable on older movies, but not on newer ones.

at any rate, the switch to digital was a great opportunity to either get back in sync with the movie industry or get in line with some other countries that have a better choice. unfortunately, atsc chose the same crap. actually, the framerates were apparently originally literal 30 and 60, but later accepted that 29.97 and 59.94 were ok too. show some fucking backbone.

compression

atsc uses mpeg-2 video compression. it’s just not sufficient for the chosen resolutions and bandwidth. especially since most broadcasters around here are sending out an hd channel and a sd channel. so from the already puny ~18Mbps, the alot ~14Mbps for the hd channel. btw, most dvds are ~9Mbps. higher resolutions never should have been attempted with out using better compression. granted, when a lot of this stuff was going on mpeg-4/h.264 or any of the other modern codecs didn’t really exist in any production capacity which is yet another reason why just switching to standard definition digital would have been a good idea.

across the pond

in england, they have digital broadcast television called freeview. the resolution is the same as a pal dvd (720×576 anamorphic widescreen which stretches to 1024×576) although with a slightly lower bandwidth of like 6.5Mbps. the quality was generally fairly good. they made the correct choice.

conclusion

it generally breaks down like this. the first phase of atsc should have been to broadcast the equal of 2 dvd quality channels. by now, everyone could be watching digital television). instead, not. the next phase could including upgrading the image quality and fixing the framerate issues. unfortunately, not. i would have been all for setting 1024×512 @ 25 and 50fps. the worry shouldn’t be with how old stuff is maintained, but looking to the future. we’ll get to that in another post.

i’m sure this is rambling and not very well organized. feel free to suggest edits.

all of this is on ubuntu 6.06 lts dapper drake. it most likely equally applies to all linuxes (possibly other unixes) that have gnu screen installed and use inittab. the runlevels may be different.

i spent a several hours yesterday looking into how to use screen to launch an app from inittab and still maintain a console to it. the initial goal was related to how to start a common lisp server application at boot, but the testing was done using a java app. unfortunately, there’s no console control on the java application, but it does print out some messages at startup and if something goes horribly wrong, the exception is dumped to stderr. it might be worth while to write one.

at any rate, the initial lead came from a post to comp.lang.lisp from like 2004 that i didn’t have the forsight to bookmark. there wasn’t any specifics and searching on google for anything related to screen is practically useless, as one might, due to “screen” being a fairly generic word. hopefully, including “gnu screen” will make it better. probably not.

so, here’s what i found/figured out. i figure the best way is to create a specific screen config file to pass on the command line with “-c”. this makes the command in inittab shorter and possibly makes it easier to keep things separate if you’re starting multiple applications. the main thing to watch out for is that when starting gnu screen with a config file and using the “chdir” command in the file you need to specify the absolute path to the config file. if you don’t, when you try to re-attach, you’ll get an “Unable to open ” message and the detached screen process will exit. not very useful.

here’s the (sort of) contents of my screen config file:

sessionname app

chdir

screen -t ces -L 0 java -jar [full path]/app.jar [args]

the sessionname is whatever you want to use for when you re-attach with “screen -r app”. you may or may not need to use chdir, but it’s probably helpful for most apps if that’s somewhere that the user the application is running as can write to or maybe not. your call. the “screen” command is what starts the application. “-L” does logging to the default file (which is in wherever the current working dir for screen is. in this case, the dir after “chdir”). “-t” is probably unnecessary as it’s just the screen window title. we’re only using one in this case so i doubt it’s important. the rest is the command to launch the application. fairly simple.

the next step is the command to start screen. this is (sort of) what i have in /etc/inittab:

app:23:respawn:/bin/su - [user] -c “screen -D -m -c [fullpath]/app.screenrc”

the “-D -m” will cause the screen process to start detached without forking to a new process. this is important because when the screen process exits, init will respawn it. the app at the beginning of the line can be whatever you want as long as it’s unique to the file and 4 characters or less. currently, we are starting some java applications by just directly calling it from inittab and redirecting the stderr and stdout to files. to get it active, just do a “kill -HUP 1″ as root.

now, to re-attach to the screen session you just use “screen -r app” or whatever you used for sessionname in the config file. you should see any output from your application now and have full terminal access just like if you had started it from a shell. gnu screen has support for password protecting the session, but i haven’t tried it yet. it’s probably less important in this specific case since you can’t input/control anything and you have to be either the user or root to re-attach to the session.

there may be a question of screen stability, but i find it unlikely to be a problem as gnu screen has been around for a while and is only getting bug fixes. what a plus to have an application that is feature complete.

an alternative might be to not use screen at all and directly start the application on an empty console instead of a getty program. the downside is that you would have to have physical access to the console in order to access the program. wasn’t really an option for me.

there’s also a program called detachtty written by a lisper that he thinks works better than screen. supposedly, you can access it over the network using ssh, but it’s not immediately apparent how this is done from my extremely limited scanning of the webpage. it appears to be targeted at use in a real init.d script and doesn’t seem like it would work from within inittab.
hopefully, this post will help me save some time in the future when i finally get around to using this method.

this is using racoon for ike and ipsec tools. i didn’t use the racoon-tool or whatever the debian added config thing is for racoon because it wouldn’t startup properly for me. problems loading modules.

i believe the only extra packages installed passed the server defaults was shorewall firewall, racoon and ipsec-tools.

here’s the notes i took for adding a connection to a new remote host.

• add pre shared key to /etc/racoon/psk.txt
• update /etc/racoon/racoon.conf by adding a remote and sainfo sections similar to an existing entry. sainfo requires encryption_algorithm, authentication_algorithm and compression_algorithm entries.
• add spdadd entry pairs for both local to remote network and remote to local in /etc/ipsec-tools.conf.
• stop racoon: /etc/init.d/racoon stop
• restart ipsec: /etc/init.d/setkey restart
• start racoon: /etc/init.d/racoon start
• if racoon doesn’t start, check /var/log/daemon.log for config errors.
• add an entry to /etc/shorewall/tunnels to define the remote gateway
• add an entry to /etc/shorewall/zones to create a zone to use for rules. the zone name can’t be more than 5 characters long.
• add entries in the /etc/shorewall/hosts to define what hosts are in the zone
• add entires to /etc/shorewall/policy for whatever firewalling rules need to be created for the vpn. at a minimum an entry to allow full access from loc to vpn and vpn to loc would be necessary. everything gets dropped by default.
• restart shorewall: /etc/init.d/shorewall restart