Web   ·   Wiki   ·   Activities   ·   Blog   ·   Lists   ·   Chat   ·   Meeting   ·   Bugs   ·   Git   ·   Translate   ·   Archive   ·   People   ·   Donate

#olpc-admin, 2008-12-09

Index | Today     Channels | Search | Join

All times shown according to UTC.

Time Nick Message
16:04 hhardy hi who is here and awake?
16:04 JasonWoof here! (and awake)
16:05 hhardy hi jason
16:05 dogi hi all
16:05 JasonWoof hi hhardy
16:05 hhardy #agenda agenda
16:06 #topic agenda
16:06 #topic agenda
16:07 agenda
16:07 wiki updates/openid
16:07 how to plan and announce planned maintenance and upgrades
16:07 research nice free backup management tools
16:07 dogi servers of VIG
16:07 what needs to be added to big sister monitoring -- new web sites, etc.
16:07 adric rt (old/private/public backup)
16:07 list of databases running
16:07 documentation -- wiki? git?
16:07 other ideas--new business
16:07 adric laughs.
16:07 hhardy bad bot! no cookie
16:08 <-- pokes bot
16:08 JasonWoof lol
16:09 hhardy: got as far as other ideas--new business
16:09 hhardy one other item is to discuss/clarify the "make the job of the sysadmin easier" purpose statement
16:09 that was all
16:09 we will take that last before new business
16:10 #topic wiki updates/openid
16:10 This went pretty well
16:10 there was about 45 minutes of downtime due to the database password getting silently overridden
16:10 changing the order of two commands in one of the config files fixed
16:11 however there was some negative feedback from people who felt that the mantenence activity should have been better announced in advance
16:11 adric "some"
16:11 hhardy I've agreed with Ed that planned maintenence activities we will try to announce at least 24 hours in advance
16:12 he and Holt particularly want to be notified
16:12 kimquirk i think all of devel might want to be notified
16:12 is that too much?
16:12 hhardy no I think it is fine
16:12 kimquirk for wiki maintenance
16:13 hhardy we did announce to devel, but the lead time was only a few hours
16:13 we will try to do better in future
16:13 on the plus side I really appreciate the 6+ hours of work that Bernie put into this
16:14 kimquirk great!
16:14 hhardy Dogi and I don't have yet the level of git knowledge to have made this upgrade as efficiently
16:14 JasonWoof in some cases (like the wiki upgrade) you can post a notice on the site itself. eg a little banner accross the top of the wiki
16:14 hhardy that is a good idea
16:14 _bernie oops I'm late again!
16:14 hhardy #IDEA in some cases (like the wiki upgrade) you can post a notice  on the site itself. eg a little banner accross the top of  the wiki
16:15 Bernie: I was just saying how much I appreciate your hard and efficient work on the wiki upgrade
16:15 _bernie hhardy: thanks, sometimes I break things, sometimes I fix them ;-)
16:16 hhardy well now is the time for fixing
16:16 _bernie are we still planning to move the wikis to separate VMs?
16:16 hhardy so good job on that and we will do better giving advance notice next time
16:16 dogi bernie and me saved howto install mediawiki from svn in http://laptop.org/internalwiki[…]x.php/Machine:sun
16:16 yes
16:17 hhardy _bernie: we don't have a "plan" to do this but it is interesting to discuss
16:17 _bernie I think it would be useful if we could edit the dns ourselves to make such transitions smooth.
16:17 dogi +1
16:17 hhardy it would be useful
16:18 _bernie I mean, we could add a domain testwiki.laptop.org, do all the work there, and then switch it into production
16:18 hhardy if mb would ever give me control of the domain file
16:18 yes
16:18 dogi for now i always use my personal dns es for that work ...
16:18 _bernie hhardy: tell him we'd work better if we could get hold of static IP assignment and domain names.
16:18 adric +1 using dns for switchover with low TTL
16:19 _bernie adric: good point, that too.
16:20 adric that's defintiely how we handle apps and sites at work
16:20 hhardy I have opened a ticket suggesting ability to edit zone file would facilitate deployment and maintenence
16:20 I will have to discuss with MB there may be a political aspect of this
16:21 #topic how to plan and announce planned maintenance and upgrades
16:22 lfaraone: how about that wiki openid? :)
16:23 #topic research nice free backup management tools
16:23 cjl hhardy: I haven't forgotten
16:23 I am drafting my researhc on TSM at MIT now and will send to lsit
16:23 hhardy we are looking for suggestions of a way to centralize and simplify the management and scheduling of backups
16:23 ok great
16:24 issues with tsm: there is no port for ubuntu, however using alien appears to mostly work
16:24 cjl will go out today.
16:24 hhardy there is no port for AMD-64, thats a problem
16:24 cjl hhardy: do you guys have hardware you would use for local backups?
16:24 hhardy supposedly MIT has an unoficial port for AMD-64 will see if I can track that down
16:25 we have access to the tsm system at MIT
16:25 and we have c. 11TB of raid array for local backups
16:25 cjl hhardy: The e-mail to ask that question might be TSM-systems@mit.edu
16:25 http://itinfo.mit.edu/article.php?id=6721
16:26 hhardy #link http://itinfo.mit.edu/article.php?id=6721
16:26 thanks cjl will look forward to your missive
16:27 #topic dogi servers of VIG
16:27 JasonWoof you can do some pretty cool stuff locally with rsync and hard-links. You can run eg daily and get a seperate directory tree snapshot of your data for each day, but files that have not changed are hard-links so storage is low-ish
16:27 dogi yes
16:27 hhardy Dogi: would you like to tell us what VIG virtual servers are set up?
16:27 JasonWoof that doesn't address scheduling though :)
16:27 dogi meeting
16:27 adric JasonWoof: Yep, Mac ships with that now, with a silly GUI.
16:28 dogi which handles that bot
16:28 JasonWoof adric: oh, I was wondering what they used on the backend
16:28 dogi then new-rt
16:28 which is to resize
16:28 adric JasonWoof: Definitely hardlinks, prob rsync
16:28 hhardy new-rt is adric's sandbox?
16:28 dogi yes
16:29 cjl very happy ot hear tha RT sandboxing is moving forward.
16:29 adric think so.
16:29 dogi but that what is set up is too little
16:30 and idea.mogidi.net is the newest ...
16:30 but back to new-rt
16:31 adric I need to know how big this server should be ...
16:31 cjl If at all possible, a full-size mirror for full-up dry-run of upgrade would be lovely (avoid wiki mishap).  Is that too much space to ask for right off?
16:31 dogi 20GB 50?
16:31 lfaraone dogi: can't it be a dynamic image?
16:31 adric dogi: Right. Well the database I nee to work on starts at 2.5 GB or so (in var) and we'll need to split/clone/mangle it a bit
16:32 cjl: that's what I'm shooting for, but i can experiment on less
16:32 hhardy if there is a disk space issue we can add another disk
16:32 adric: how much space would be "ideal"?
16:32 dogi bingo
16:32 adric 10 .. 20 GB should be good for the sandbox .. for a replacement prod image we'll need to discuss
16:33 10 GB var and comfort on root would be okay, for sandobox
16:33 hhardy ok adric and dogi please come to an understanding, I propose 20 GB for now
16:33 adric agree
16:34 dogi i want to discuss that now ... then later i have only to move the sandbox to production
16:34 adric I do want to distingish the sandbox from a new rt server image/vm/slice.
16:34 Well, that's one way. ...
16:35 Are we looking at keeping it on vserver or do we need to build a new image for the new public* RT prod?
16:35 hhardy Here's my proposal on that
16:35 dogi i can offer 50GB
16:35 adric I know henry is going to need us to split some Qs out of the old RT to be put elsewhere.
16:35 hhardy leave the current rt where it is, it becomes "secret rt" for internal OLPC non VIG stuff
16:36 everything public gets reproduced on sandbox
16:36 when we are happy with that, we clone it to one of the puiblic facing machines such as swan to become "VIG rt"
16:36 adric Right. Will need soem postgres tricks to get just the public stuff out, but this is a good plan.
16:37 hhardy that way we dont put at risk what we have now while working on a solution for the future
16:39 #topic what needs to be added to big sister monitoring -- new web sites, etc.
16:39 cjl hhardy: just to tip you off, I am proposing a very minor addition to RT tix (adding a country field).  I already have "pre-sale" buy-in from kim, adam, SJ, adric.  Will be floating past larger group (SG, VIG) for further comment.
16:39 hhardy adric: good
16:39 cjl: good :)
16:39 all vig: good :)
16:39 adric hhardy: , cjl : agree agree
16:40 kimquirk agreed
16:40 dogi +1
16:40 hhardy we have made some infrastructure changes and need to update the monitoring system, which is called "Big Sister"
16:41 #LINK: http://rt.laptop.org/Ticket/Display.html?id=27072
16:41 so that the reverse proxy, and the new g1g1-related sites are monitored
16:42 http://thinker.laptop.org/bigsis/
16:42 right now this isn't passworded but I think I will set a password for login 'olpc'
16:42 dogi and bigsister.mogidi.net
16:43 bigsister.mogidi.net/bigsis/
16:43 hhardy dogi: yours is all purples
16:43 dogi lol
16:43 i told u :
16:44 )
16:44 lfaraone hhardy: uh... /var/log on crank is full...
16:44 hhardy if you steal my uxmon-asroot file it should work and you can follow from that I think
16:44 ah let's fix
16:44 lfaraone hhardy: (oh, and these meetings are logged, so that's a public password)
16:44 dogi ok
16:44 hhardy I said login not password :)
16:44 adric heh
16:45 _bernie hhardy: for backups, a lot of people recommended duplicity.  I've always just used homebrew rsync scripts though.
16:45 lfaraone /dev/cciss/c0d0p2     957M  957M     0 100% /var/log
16:45 Nasty.
16:45 hhardy yeah whoever set up the machine with a 1 gb log partition was not thinking right
16:46 adric uh
16:46 *facepalm*
16:46 cjl tHAT'S A TWIG NOT A LOG. .
16:46 adric nice.
16:46 cjl switches to inside voice.
16:47 lfaraone slaps cjl's hand with a ruler.
16:47 hhardy temporarily moved the old logs to /home/log3
16:47 lfaraone cjl: There will be no shouting in this class!
16:47 hhardy that needs to be fixed
16:47 adric Right, well this meeting is broken. Is it it still under warranty?
16:48 hhardy sent ticket
16:48 cjl list of databases running is next on agenda
16:48 lfaraone hhardy: it's funny, the "free" space is slowly decreasing, currently 900~K
16:50 hhardy ok going to make a quick fix
16:50 apache going down on crank, restart asap
16:53 moving the apache logfiles to home
16:58 cjl Intersting to watch the http red-light come on when apache went down on crank.  Nice to know that BigSis is working.
16:58 dogi :)
17:05 hhardy hmmmm now it doesnt want to restart cause cant write to logs
17:06 edmcnierney cjl: It would be more interesting to watch it turn off again at this point.
17:06 cjl edmcnierney: indeed.
17:07 adric mm. yeah
17:09 hhardy we are working ont he problem
17:12 adric hears beeping, dinner is ready.
17:12 gotta go folks, ttys!
17:13 edmcnierney_ notes that he's not the only one who loses control of the meeting agenda.....
17:13 hhardy sorry we are going to close now to work on the apache issue
17:14 cjl hhardy: sounds like a good idea.
17:15 hears sound of bugles over hill as cavalry charges in to deal with Apache issue.. .
17:15 JasonWoof so weird how many things cause apache to not start
17:16 hhardy ok fixed for now
17:16 cjl I find quite often that the Apache Tomcat service I use just needs to be restarted twice.  No rhyme or reason why it works the second time, but it often does.
17:17 hhardy the last question was how to improve documentation availability
17:17 dogi re
17:17 hhardy #topic documentation -- wiki? git?
17:18 the suggestion has been made to move as much as possible from internalwiki to a designated area of teamwiki
17:18 the alternative suggesiton is check everything into git
17:18 the reason apache was being difficult is that there were still live apache2 process running after the init.d stop
17:18 killall apache2 solved it
17:19 a possible way to combine the two is to use teamwiki for the "live" documentation but make a static copy available via git
17:19 cjl if it is in git, devels will likely be able to find it.  If it is in teamwiki, no one will ever look for it. . which is better?  What sort of docs?
17:20 hhardy there are docs on the setup of most of the machines
17:20 dogi or to combine both solutions ... wiki and staticwiki/git ...
17:20 hhardy and some of our proceudres
17:20 well let's take it one step at a time
17:20 first step is get copies of everything we can share with VIG to teamwiki
17:21 cjl If you want input (edits) on them, then I'd say wiki.
17:21 hhardy ok sent a ticket
17:24 dogi maybe it is better for VIG to maintain their own wiki ...?
17:24 hhardy ok we are going to go back to cleaning up the logs on crank
17:24 dogi: that is an interesting suggestion
17:24 let's talk more on this next meeting
17:24 cjl dogi: too many wikis.
17:25 hhardy thanks all
17:25 dogi in a Virtual machine ... ok :)
17:25 hhardy #endmeeting

Index | Today     Channels | Search | Join

Powered by ilbot/Modified.