Time |
Nick |
Message |
16:04 |
hhardy |
hi who is here and awake? |
16:04 |
JasonWoof |
here! (and awake) |
16:05 |
hhardy |
hi jason |
16:05 |
dogi |
hi all |
16:05 |
JasonWoof |
hi hhardy |
16:05 |
hhardy |
#agenda agenda |
16:06 |
|
#topic agenda |
16:06 |
|
#topic agenda |
16:07 |
|
agenda |
16:07 |
|
wiki updates/openid |
16:07 |
|
how to plan and announce planned maintenance and upgrades |
16:07 |
|
research nice free backup management tools |
16:07 |
|
dogi servers of VIG |
16:07 |
|
what needs to be added to big sister monitoring -- new web sites, etc. |
16:07 |
|
adric rt (old/private/public backup) |
16:07 |
|
list of databases running |
16:07 |
|
documentation -- wiki? git? |
16:07 |
|
other ideas--new business |
16:07 |
adric |
laughs. |
16:07 |
hhardy |
bad bot! no cookie |
16:08 |
|
<-- pokes bot |
16:08 |
JasonWoof |
lol |
16:09 |
|
hhardy: got as far as other ideas--new business |
16:09 |
hhardy |
one other item is to discuss/clarify the "make the job of the sysadmin easier" purpose statement |
16:09 |
|
that was all |
16:09 |
|
we will take that last before new business |
16:10 |
|
#topic wiki updates/openid |
16:10 |
|
This went pretty well |
16:10 |
|
there was about 45 minutes of downtime due to the database password getting silently overridden |
16:10 |
|
changing the order of two commands in one of the config files fixed |
16:11 |
|
however there was some negative feedback from people who felt that the mantenence activity should have been better announced in advance |
16:11 |
adric |
"some" |
16:11 |
hhardy |
I've agreed with Ed that planned maintenence activities we will try to announce at least 24 hours in advance |
16:12 |
|
he and Holt particularly want to be notified |
16:12 |
kimquirk |
i think all of devel might want to be notified |
16:12 |
|
is that too much? |
16:12 |
hhardy |
no I think it is fine |
16:12 |
kimquirk |
for wiki maintenance |
16:13 |
hhardy |
we did announce to devel, but the lead time was only a few hours |
16:13 |
|
we will try to do better in future |
16:13 |
|
on the plus side I really appreciate the 6+ hours of work that Bernie put into this |
16:14 |
kimquirk |
great! |
16:14 |
hhardy |
Dogi and I don't have yet the level of git knowledge to have made this upgrade as efficiently |
16:14 |
JasonWoof |
in some cases (like the wiki upgrade) you can post a notice on the site itself. eg a little banner accross the top of the wiki |
16:14 |
hhardy |
that is a good idea |
16:14 |
_bernie |
oops I'm late again! |
16:14 |
hhardy |
#IDEA in some cases (like the wiki upgrade) you can post a notice on the site itself. eg a little banner accross the top of the wiki |
16:15 |
|
Bernie: I was just saying how much I appreciate your hard and efficient work on the wiki upgrade |
16:15 |
_bernie |
hhardy: thanks, sometimes I break things, sometimes I fix them ;-) |
16:16 |
hhardy |
well now is the time for fixing |
16:16 |
_bernie |
are we still planning to move the wikis to separate VMs? |
16:16 |
hhardy |
so good job on that and we will do better giving advance notice next time |
16:16 |
dogi |
bernie and me saved howto install mediawiki from svn in http://laptop.org/internalwiki[…]x.php/Machine:sun |
16:16 |
|
yes |
16:17 |
hhardy |
_bernie: we don't have a "plan" to do this but it is interesting to discuss |
16:17 |
_bernie |
I think it would be useful if we could edit the dns ourselves to make such transitions smooth. |
16:17 |
dogi |
+1 |
16:17 |
hhardy |
it would be useful |
16:18 |
_bernie |
I mean, we could add a domain testwiki.laptop.org, do all the work there, and then switch it into production |
16:18 |
hhardy |
if mb would ever give me control of the domain file |
16:18 |
|
yes |
16:18 |
dogi |
for now i always use my personal dns es for that work ... |
16:18 |
_bernie |
hhardy: tell him we'd work better if we could get hold of static IP assignment and domain names. |
16:18 |
adric |
+1 using dns for switchover with low TTL |
16:19 |
_bernie |
adric: good point, that too. |
16:20 |
adric |
that's defintiely how we handle apps and sites at work |
16:20 |
hhardy |
I have opened a ticket suggesting ability to edit zone file would facilitate deployment and maintenence |
16:20 |
|
I will have to discuss with MB there may be a political aspect of this |
16:21 |
|
#topic how to plan and announce planned maintenance and upgrades |
16:22 |
|
lfaraone: how about that wiki openid? :) |
16:23 |
|
#topic research nice free backup management tools |
16:23 |
cjl |
hhardy: I haven't forgotten |
16:23 |
|
I am drafting my researhc on TSM at MIT now and will send to lsit |
16:23 |
hhardy |
we are looking for suggestions of a way to centralize and simplify the management and scheduling of backups |
16:23 |
|
ok great |
16:24 |
|
issues with tsm: there is no port for ubuntu, however using alien appears to mostly work |
16:24 |
cjl |
will go out today. |
16:24 |
hhardy |
there is no port for AMD-64, thats a problem |
16:24 |
cjl |
hhardy: do you guys have hardware you would use for local backups? |
16:24 |
hhardy |
supposedly MIT has an unoficial port for AMD-64 will see if I can track that down |
16:25 |
|
we have access to the tsm system at MIT |
16:25 |
|
and we have c. 11TB of raid array for local backups |
16:25 |
cjl |
hhardy: The e-mail to ask that question might be TSM-systems mit.edu |
16:25 |
|
http://itinfo.mit.edu/article.php?id=6721 |
16:26 |
hhardy |
#link http://itinfo.mit.edu/article.php?id=6721 |
16:26 |
|
thanks cjl will look forward to your missive |
16:27 |
|
#topic dogi servers of VIG |
16:27 |
JasonWoof |
you can do some pretty cool stuff locally with rsync and hard-links. You can run eg daily and get a seperate directory tree snapshot of your data for each day, but files that have not changed are hard-links so storage is low-ish |
16:27 |
dogi |
yes |
16:27 |
hhardy |
Dogi: would you like to tell us what VIG virtual servers are set up? |
16:27 |
JasonWoof |
that doesn't address scheduling though :) |
16:27 |
dogi |
meeting |
16:27 |
adric |
JasonWoof: Yep, Mac ships with that now, with a silly GUI. |
16:28 |
dogi |
which handles that bot |
16:28 |
JasonWoof |
adric: oh, I was wondering what they used on the backend |
16:28 |
dogi |
then new-rt |
16:28 |
|
which is to resize |
16:28 |
adric |
JasonWoof: Definitely hardlinks, prob rsync |
16:28 |
hhardy |
new-rt is adric's sandbox? |
16:28 |
dogi |
yes |
16:29 |
cjl |
very happy ot hear tha RT sandboxing is moving forward. |
16:29 |
adric |
think so. |
16:29 |
dogi |
but that what is set up is too little |
16:30 |
|
and idea.mogidi.net is the newest ... |
16:30 |
|
but back to new-rt |
16:31 |
|
adric I need to know how big this server should be ... |
16:31 |
cjl |
If at all possible, a full-size mirror for full-up dry-run of upgrade would be lovely (avoid wiki mishap). Is that too much space to ask for right off? |
16:31 |
dogi |
20GB 50? |
16:31 |
lfaraone |
dogi: can't it be a dynamic image? |
16:31 |
adric |
dogi: Right. Well the database I nee to work on starts at 2.5 GB or so (in var) and we'll need to split/clone/mangle it a bit |
16:32 |
|
cjl: that's what I'm shooting for, but i can experiment on less |
16:32 |
hhardy |
if there is a disk space issue we can add another disk |
16:32 |
|
adric: how much space would be "ideal"? |
16:32 |
dogi |
bingo |
16:32 |
adric |
10 .. 20 GB should be good for the sandbox .. for a replacement prod image we'll need to discuss |
16:33 |
|
10 GB var and comfort on root would be okay, for sandobox |
16:33 |
hhardy |
ok adric and dogi please come to an understanding, I propose 20 GB for now |
16:33 |
adric |
agree |
16:34 |
dogi |
i want to discuss that now ... then later i have only to move the sandbox to production |
16:34 |
adric |
I do want to distingish the sandbox from a new rt server image/vm/slice. |
16:34 |
|
Well, that's one way. ... |
16:35 |
|
Are we looking at keeping it on vserver or do we need to build a new image for the new public* RT prod? |
16:35 |
hhardy |
Here's my proposal on that |
16:35 |
dogi |
i can offer 50GB |
16:35 |
adric |
I know henry is going to need us to split some Qs out of the old RT to be put elsewhere. |
16:35 |
hhardy |
leave the current rt where it is, it becomes "secret rt" for internal OLPC non VIG stuff |
16:36 |
|
everything public gets reproduced on sandbox |
16:36 |
|
when we are happy with that, we clone it to one of the puiblic facing machines such as swan to become "VIG rt" |
16:36 |
adric |
Right. Will need soem postgres tricks to get just the public stuff out, but this is a good plan. |
16:37 |
hhardy |
that way we dont put at risk what we have now while working on a solution for the future |
16:39 |
|
#topic what needs to be added to big sister monitoring -- new web sites, etc. |
16:39 |
cjl |
hhardy: just to tip you off, I am proposing a very minor addition to RT tix (adding a country field). I already have "pre-sale" buy-in from kim, adam, SJ, adric. Will be floating past larger group (SG, VIG) for further comment. |
16:39 |
hhardy |
adric: good |
16:39 |
|
cjl: good :) |
16:39 |
|
all vig: good :) |
16:39 |
adric |
hhardy: , cjl : agree agree |
16:40 |
kimquirk |
agreed |
16:40 |
dogi |
+1 |
16:40 |
hhardy |
we have made some infrastructure changes and need to update the monitoring system, which is called "Big Sister" |
16:41 |
|
#LINK: http://rt.laptop.org/Ticket/Display.html?id=27072 |
16:41 |
|
so that the reverse proxy, and the new g1g1-related sites are monitored |
16:42 |
|
http://thinker.laptop.org/bigsis/ |
16:42 |
|
right now this isn't passworded but I think I will set a password for login 'olpc' |
16:42 |
dogi |
and bigsister.mogidi.net |
16:43 |
|
bigsister.mogidi.net/bigsis/ |
16:43 |
hhardy |
dogi: yours is all purples |
16:43 |
dogi |
lol |
16:43 |
|
i told u : |
16:44 |
|
) |
16:44 |
lfaraone |
hhardy: uh... /var/log on crank is full... |
16:44 |
hhardy |
if you steal my uxmon-asroot file it should work and you can follow from that I think |
16:44 |
|
ah let's fix |
16:44 |
lfaraone |
hhardy: (oh, and these meetings are logged, so that's a public password) |
16:44 |
dogi |
ok |
16:44 |
hhardy |
I said login not password :) |
16:44 |
adric |
heh |
16:45 |
_bernie |
hhardy: for backups, a lot of people recommended duplicity. I've always just used homebrew rsync scripts though. |
16:45 |
lfaraone |
/dev/cciss/c0d0p2 957M 957M 0 100% /var/log |
16:45 |
|
Nasty. |
16:45 |
hhardy |
yeah whoever set up the machine with a 1 gb log partition was not thinking right |
16:46 |
adric |
uh |
16:46 |
|
*facepalm* |
16:46 |
cjl |
tHAT'S A TWIG NOT A LOG. . |
16:46 |
adric |
nice. |
16:46 |
cjl |
switches to inside voice. |
16:47 |
lfaraone |
slaps cjl's hand with a ruler. |
16:47 |
hhardy |
temporarily moved the old logs to /home/log3 |
16:47 |
lfaraone |
cjl: There will be no shouting in this class! |
16:47 |
hhardy |
that needs to be fixed |
16:47 |
adric |
Right, well this meeting is broken. Is it it still under warranty? |
16:48 |
hhardy |
sent ticket |
16:48 |
cjl |
list of databases running is next on agenda |
16:48 |
lfaraone |
hhardy: it's funny, the "free" space is slowly decreasing, currently 900~K |
16:50 |
hhardy |
ok going to make a quick fix |
16:50 |
|
apache going down on crank, restart asap |
16:53 |
|
moving the apache logfiles to home |
16:58 |
cjl |
Intersting to watch the http red-light come on when apache went down on crank. Nice to know that BigSis is working. |
16:58 |
dogi |
:) |
17:05 |
hhardy |
hmmmm now it doesnt want to restart cause cant write to logs |
17:06 |
edmcnierney |
cjl: It would be more interesting to watch it turn off again at this point. |
17:06 |
cjl |
edmcnierney: indeed. |
17:07 |
adric |
mm. yeah |
17:09 |
hhardy |
we are working ont he problem |
17:12 |
adric |
hears beeping, dinner is ready. |
17:12 |
|
gotta go folks, ttys! |
17:13 |
edmcnierney_ |
notes that he's not the only one who loses control of the meeting agenda..... |
17:13 |
hhardy |
sorry we are going to close now to work on the apache issue |
17:14 |
cjl |
hhardy: sounds like a good idea. |
17:15 |
|
hears sound of bugles over hill as cavalry charges in to deal with Apache issue.. . |
17:15 |
JasonWoof |
so weird how many things cause apache to not start |
17:16 |
hhardy |
ok fixed for now |
17:16 |
cjl |
I find quite often that the Apache Tomcat service I use just needs to be restarted twice. No rhyme or reason why it works the second time, but it often does. |
17:17 |
hhardy |
the last question was how to improve documentation availability |
17:17 |
dogi |
re |
17:17 |
hhardy |
#topic documentation -- wiki? git? |
17:18 |
|
the suggestion has been made to move as much as possible from internalwiki to a designated area of teamwiki |
17:18 |
|
the alternative suggesiton is check everything into git |
17:18 |
|
the reason apache was being difficult is that there were still live apache2 process running after the init.d stop |
17:18 |
|
killall apache2 solved it |
17:19 |
|
a possible way to combine the two is to use teamwiki for the "live" documentation but make a static copy available via git |
17:19 |
cjl |
if it is in git, devels will likely be able to find it. If it is in teamwiki, no one will ever look for it. . which is better? What sort of docs? |
17:20 |
hhardy |
there are docs on the setup of most of the machines |
17:20 |
dogi |
or to combine both solutions ... wiki and staticwiki/git ... |
17:20 |
hhardy |
and some of our proceudres |
17:20 |
|
well let's take it one step at a time |
17:20 |
|
first step is get copies of everything we can share with VIG to teamwiki |
17:21 |
cjl |
If you want input (edits) on them, then I'd say wiki. |
17:21 |
hhardy |
ok sent a ticket |
17:24 |
dogi |
maybe it is better for VIG to maintain their own wiki ...? |
17:24 |
hhardy |
ok we are going to go back to cleaning up the logs on crank |
17:24 |
|
dogi: that is an interesting suggestion |
17:24 |
|
let's talk more on this next meeting |
17:24 |
cjl |
dogi: too many wikis. |
17:25 |
hhardy |
thanks all |
17:25 |
dogi |
in a Virtual machine ... ok :) |
17:25 |
hhardy |
#endmeeting |