Sugar meeting

All times shown according to UTC.

Time	Nick	Message
16:04	hhardy	hi who is here and awake?
16:04	JasonWoof	here! (and awake)
16:05	hhardy	hi jason
16:05	dogi	hi all
16:05	JasonWoof	hi hhardy
16:05	hhardy	#agenda agenda
16:06		#topic agenda
16:06		#topic agenda
16:07		agenda
16:07		wiki updates/openid
16:07		how to plan and announce planned maintenance and upgrades
16:07		research nice free backup management tools
16:07		dogi servers of VIG
16:07		what needs to be added to big sister monitoring -- new web sites, etc.
16:07		adric rt (old/private/public backup)
16:07		list of databases running
16:07		documentation -- wiki? git?
16:07		other ideas--new business
16:07	adric	laughs.
16:07	hhardy	bad bot! no cookie
16:08		<-- pokes bot
16:08	JasonWoof	lol
16:09		hhardy: got as far as other ideas--new business
16:09	hhardy	one other item is to discuss/clarify the "make the job of the sysadmin easier" purpose statement
16:09		that was all
16:09		we will take that last before new business
16:10		#topic wiki updates/openid
16:10		This went pretty well
16:10		there was about 45 minutes of downtime due to the database password getting silently overridden
16:10		changing the order of two commands in one of the config files fixed
16:11		however there was some negative feedback from people who felt that the mantenence activity should have been better announced in advance
16:11	adric	"some"
16:11	hhardy	I've agreed with Ed that planned maintenence activities we will try to announce at least 24 hours in advance
16:12		he and Holt particularly want to be notified
16:12	kimquirk	i think all of devel might want to be notified
16:12		is that too much?
16:12	hhardy	no I think it is fine
16:12	kimquirk	for wiki maintenance
16:13	hhardy	we did announce to devel, but the lead time was only a few hours
16:13		we will try to do better in future
16:13		on the plus side I really appreciate the 6+ hours of work that Bernie put into this
16:14	kimquirk	great!
16:14	hhardy	Dogi and I don't have yet the level of git knowledge to have made this upgrade as efficiently
16:14	JasonWoof	in some cases (like the wiki upgrade) you can post a notice on the site itself. eg a little banner accross the top of the wiki
16:14	hhardy	that is a good idea
16:14	_bernie	oops I'm late again!
16:14	hhardy	#IDEA in some cases (like the wiki upgrade) you can post a notice on the site itself. eg a little banner accross the top of the wiki
16:15		Bernie: I was just saying how much I appreciate your hard and efficient work on the wiki upgrade
16:15	_bernie	hhardy: thanks, sometimes I break things, sometimes I fix them ;-)
16:16	hhardy	well now is the time for fixing
16:16	_bernie	are we still planning to move the wikis to separate VMs?
16:16	hhardy	so good job on that and we will do better giving advance notice next time
16:16	dogi	bernie and me saved howto install mediawiki from svn in http://laptop.org/internalwiki[…]x.php/Machine:sun
16:16		yes
16:17	hhardy	_bernie: we don't have a "plan" to do this but it is interesting to discuss
16:17	_bernie	I think it would be useful if we could edit the dns ourselves to make such transitions smooth.
16:17	dogi	+1
16:17	hhardy	it would be useful
16:18	_bernie	I mean, we could add a domain testwiki.laptop.org, do all the work there, and then switch it into production
16:18	hhardy	if mb would ever give me control of the domain file
16:18		yes
16:18	dogi	for now i always use my personal dns es for that work ...
16:18	_bernie	hhardy: tell him we'd work better if we could get hold of static IP assignment and domain names.
16:18	adric	+1 using dns for switchover with low TTL
16:19	_bernie	adric: good point, that too.
16:20	adric	that's defintiely how we handle apps and sites at work
16:20	hhardy	I have opened a ticket suggesting ability to edit zone file would facilitate deployment and maintenence
16:20		I will have to discuss with MB there may be a political aspect of this
16:21		#topic how to plan and announce planned maintenance and upgrades
16:22		lfaraone: how about that wiki openid? :)
16:23		#topic research nice free backup management tools
16:23	cjl	hhardy: I haven't forgotten
16:23		I am drafting my researhc on TSM at MIT now and will send to lsit
16:23	hhardy	we are looking for suggestions of a way to centralize and simplify the management and scheduling of backups
16:23		ok great
16:24		issues with tsm: there is no port for ubuntu, however using alien appears to mostly work
16:24	cjl	will go out today.
16:24	hhardy	there is no port for AMD-64, thats a problem
16:24	cjl	hhardy: do you guys have hardware you would use for local backups?
16:24	hhardy	supposedly MIT has an unoficial port for AMD-64 will see if I can track that down
16:25		we have access to the tsm system at MIT
16:25		and we have c. 11TB of raid array for local backups
16:25	cjl	hhardy: The e-mail to ask that question might be TSM-systemsmit.edu
16:25		http://itinfo.mit.edu/article.php?id=6721
16:26	hhardy	#link http://itinfo.mit.edu/article.php?id=6721
16:26		thanks cjl will look forward to your missive
16:27		#topic dogi servers of VIG
16:27	JasonWoof	you can do some pretty cool stuff locally with rsync and hard-links. You can run eg daily and get a seperate directory tree snapshot of your data for each day, but files that have not changed are hard-links so storage is low-ish
16:27	dogi	yes
16:27	hhardy	Dogi: would you like to tell us what VIG virtual servers are set up?
16:27	JasonWoof	that doesn't address scheduling though :)
16:27	dogi	meeting
16:27	adric	JasonWoof: Yep, Mac ships with that now, with a silly GUI.
16:28	dogi	which handles that bot
16:28	JasonWoof	adric: oh, I was wondering what they used on the backend
16:28	dogi	then new-rt
16:28		which is to resize
16:28	adric	JasonWoof: Definitely hardlinks, prob rsync
16:28	hhardy	new-rt is adric's sandbox?
16:28	dogi	yes
16:29	cjl	very happy ot hear tha RT sandboxing is moving forward.
16:29	adric	think so.
16:29	dogi	but that what is set up is too little
16:30		and idea.mogidi.net is the newest ...
16:30		but back to new-rt
16:31		adric I need to know how big this server should be ...
16:31	cjl	If at all possible, a full-size mirror for full-up dry-run of upgrade would be lovely (avoid wiki mishap). Is that too much space to ask for right off?
16:31	dogi	20GB 50?
16:31	lfaraone	dogi: can't it be a dynamic image?
16:31	adric	dogi: Right. Well the database I nee to work on starts at 2.5 GB or so (in var) and we'll need to split/clone/mangle it a bit
16:32		cjl: that's what I'm shooting for, but i can experiment on less
16:32	hhardy	if there is a disk space issue we can add another disk
16:32		adric: how much space would be "ideal"?
16:32	dogi	bingo
16:32	adric	10 .. 20 GB should be good for the sandbox .. for a replacement prod image we'll need to discuss
16:33		10 GB var and comfort on root would be okay, for sandobox
16:33	hhardy	ok adric and dogi please come to an understanding, I propose 20 GB for now
16:33	adric	agree
16:34	dogi	i want to discuss that now ... then later i have only to move the sandbox to production
16:34	adric	I do want to distingish the sandbox from a new rt server image/vm/slice.
16:34		Well, that's one way. ...
16:35		Are we looking at keeping it on vserver or do we need to build a new image for the new public* RT prod?
16:35	hhardy	Here's my proposal on that
16:35	dogi	i can offer 50GB
16:35	adric	I know henry is going to need us to split some Qs out of the old RT to be put elsewhere.
16:35	hhardy	leave the current rt where it is, it becomes "secret rt" for internal OLPC non VIG stuff
16:36		everything public gets reproduced on sandbox
16:36		when we are happy with that, we clone it to one of the puiblic facing machines such as swan to become "VIG rt"
16:36	adric	Right. Will need soem postgres tricks to get just the public stuff out, but this is a good plan.
16:37	hhardy	that way we dont put at risk what we have now while working on a solution for the future
16:39		#topic what needs to be added to big sister monitoring -- new web sites, etc.
16:39	cjl	hhardy: just to tip you off, I am proposing a very minor addition to RT tix (adding a country field). I already have "pre-sale" buy-in from kim, adam, SJ, adric. Will be floating past larger group (SG, VIG) for further comment.
16:39	hhardy	adric: good
16:39		cjl: good :)
16:39		all vig: good :)
16:39	adric	hhardy: , cjl : agree agree
16:40	kimquirk	agreed
16:40	dogi	+1
16:40	hhardy	we have made some infrastructure changes and need to update the monitoring system, which is called "Big Sister"
16:41		#LINK: http://rt.laptop.org/Ticket/Display.html?id=27072
16:41		so that the reverse proxy, and the new g1g1-related sites are monitored
16:42		http://thinker.laptop.org/bigsis/
16:42		right now this isn't passworded but I think I will set a password for login 'olpc'
16:42	dogi	and bigsister.mogidi.net
16:43		bigsister.mogidi.net/bigsis/
16:43	hhardy	dogi: yours is all purples
16:43	dogi	lol
16:43		i told u :
16:44		)
16:44	lfaraone	hhardy: uh... /var/log on crank is full...
16:44	hhardy	if you steal my uxmon-asroot file it should work and you can follow from that I think
16:44		ah let's fix
16:44	lfaraone	hhardy: (oh, and these meetings are logged, so that's a public password)
16:44	dogi	ok
16:44	hhardy	I said login not password :)
16:44	adric	heh
16:45	_bernie	hhardy: for backups, a lot of people recommended duplicity. I've always just used homebrew rsync scripts though.
16:45	lfaraone	/dev/cciss/c0d0p2 957M 957M 0 100% /var/log
16:45		Nasty.
16:45	hhardy	yeah whoever set up the machine with a 1 gb log partition was not thinking right
16:46	adric	uh
16:46		facepalm
16:46	cjl	tHAT'S A TWIG NOT A LOG. .
16:46	adric	nice.
16:46	cjl	switches to inside voice.
16:47	lfaraone	slaps cjl's hand with a ruler.
16:47	hhardy	temporarily moved the old logs to /home/log3
16:47	lfaraone	cjl: There will be no shouting in this class!
16:47	hhardy	that needs to be fixed
16:47	adric	Right, well this meeting is broken. Is it it still under warranty?
16:48	hhardy	sent ticket
16:48	cjl	list of databases running is next on agenda
16:48	lfaraone	hhardy: it's funny, the "free" space is slowly decreasing, currently 900~K
16:50	hhardy	ok going to make a quick fix
16:50		apache going down on crank, restart asap
16:53		moving the apache logfiles to home
16:58	cjl	Intersting to watch the http red-light come on when apache went down on crank. Nice to know that BigSis is working.
16:58	dogi	:)
17:05	hhardy	hmmmm now it doesnt want to restart cause cant write to logs
17:06	edmcnierney	cjl: It would be more interesting to watch it turn off again at this point.
17:06	cjl	edmcnierney: indeed.
17:07	adric	mm. yeah
17:09	hhardy	we are working ont he problem
17:12	adric	hears beeping, dinner is ready.
17:12		gotta go folks, ttys!
17:13	edmcnierney_	notes that he's not the only one who loses control of the meeting agenda.....
17:13	hhardy	sorry we are going to close now to work on the apache issue
17:14	cjl	hhardy: sounds like a good idea.
17:15		hears sound of bugles over hill as cavalry charges in to deal with Apache issue.. .
17:15	JasonWoof	so weird how many things cause apache to not start
17:16	hhardy	ok fixed for now
17:16	cjl	I find quite often that the Apache Tomcat service I use just needs to be restarted twice. No rhyme or reason why it works the second time, but it often does.
17:17	hhardy	the last question was how to improve documentation availability
17:17	dogi	re
17:17	hhardy	#topic documentation -- wiki? git?
17:18		the suggestion has been made to move as much as possible from internalwiki to a designated area of teamwiki
17:18		the alternative suggesiton is check everything into git
17:18		the reason apache was being difficult is that there were still live apache2 process running after the init.d stop
17:18		killall apache2 solved it
17:19		a possible way to combine the two is to use teamwiki for the "live" documentation but make a static copy available via git
17:19	cjl	if it is in git, devels will likely be able to find it. If it is in teamwiki, no one will ever look for it. . which is better? What sort of docs?
17:20	hhardy	there are docs on the setup of most of the machines
17:20	dogi	or to combine both solutions ... wiki and staticwiki/git ...
17:20	hhardy	and some of our proceudres
17:20		well let's take it one step at a time
17:20		first step is get copies of everything we can share with VIG to teamwiki
17:21	cjl	If you want input (edits) on them, then I'd say wiki.
17:21	hhardy	ok sent a ticket
17:24	dogi	maybe it is better for VIG to maintain their own wiki ...?
17:24	hhardy	ok we are going to go back to cleaning up the logs on crank
17:24		dogi: that is an interesting suggestion
17:24		let's talk more on this next meeting
17:24	cjl	dogi: too many wikis.
17:25	hhardy	thanks all
17:25	dogi	in a Virtual machine ... ok :)
17:25	hhardy	#endmeeting

Index | Today Channels | Search | Join

#olpc-admin, 2008-12-09