| Time |
Nick |
Message |
| 17:00 |
meeting |
Meeting started Sun Mar 31 17:00:54 2013 UTC. The chair is walterbender. Information about MeetBot at http://wiki.debian.org/MeetBot. |
| 17:00 |
|
Useful Commands: #action #agreed #help #info #idea #link #topic #endmeeting |
| 17:01 |
walterbender |
let's put together a quick agenda |
| 17:01 |
|
feel free to chime in |
| 17:01 |
bernie |
should i send my items? |
| 17:01 |
walterbender |
bernie: just post them here |
| 17:02 |
bernie |
- Self introduction (rralcala) |
| 17:02 |
|
- Status update on SL infra (everyone) |
| 17:02 |
|
- Paraguay Educa hosting requirements (rralcala) |
| 17:02 |
|
- Trac maintenance (bernie) |
| 17:02 |
|
- Pootle maintenance (bernie) |
| 17:03 |
|
ah, and also: |
| 17:03 |
|
- hosting code on GitHub (walter) |
| 17:03 |
walterbender |
ok.. |
| 17:03 |
bernie |
anything else? |
| 17:03 |
walterbender |
#topic Self introduction (rralcala) |
| 17:04 |
bernie |
GA, rralcala |
| 17:04 |
rralcala |
bernie: Ok, I'm not good at this but |
| 17:04 |
walterbender |
@ bernie: we can add more topics at the end |
| 17:04 |
rralcala |
Hi everyone, I'm rralcala, aka Roberto Rodriguez |
| 17:04 |
cjl |
hola |
| 17:05 |
rralcala |
I worked for a year in the Caacupe (Paraguay) implementation of olpc |
| 17:05 |
|
but mostly on the infrastructure side |
| 17:06 |
|
also I've some background maintaining infrastructure from other jobs |
| 17:06 |
|
And, I'd like to give a hand/request some help if possible |
| 17:07 |
|
how long should this be? |
| 17:07 |
|
Also |
| 17:08 |
walterbender |
rralcala: I think we get the idea :) but tell us about Tanzania... |
| 17:08 |
rralcala |
Tanzania is an awesome place |
| 17:08 |
|
really |
| 17:08 |
|
but we don't get more than 1mbps at home |
| 17:08 |
|
The digital breach here is huge |
| 17:08 |
|
mostly because education |
| 17:09 |
|
so in that perspective it might be a huge challenge doing something like paraguayeduca here |
| 17:09 |
|
But I'm working on a telecom company |
| 17:09 |
|
Recommended for safari and nice beaches |
| 17:09 |
walterbender |
rralcala: will bandwidth be an issue for you re helping with infrastructure? |
| 17:10 |
rralcala |
I'm moving to South africa |
| 17:10 |
|
and have 3 backup connections |
| 17:10 |
|
but yes |
| 17:10 |
bernie |
rralcala: what are you going to do in tanzania? |
| 17:10 |
|
ah sorry, i missed that line |
| 17:11 |
rralcala |
I once had none of the 3 connections working because an fishers boat anchor ran over the backbone in egypt |
| 17:11 |
bernie |
rralcala: how much free time can you realistically dedicate to volunteer activities for SL and PyEdu? |
| 17:12 |
rralcala |
but I'm 99% available |
| 17:12 |
|
tch___ <tch___!~webchat jita.sugarlabs.org> has joined #sugar-meeting |
| 17:12 |
rralcala |
bernie: Ok |
| 17:12 |
walterbender |
hi tch___ |
| 17:12 |
rralcala |
bernie: I won't have much |
| 17:12 |
tch___ |
walterbender: hello!, sorry I am late |
| 17:13 |
rralcala |
but enough to maintain 2 services, be available in case of emergency and monitor pyeduca's part |
| 17:13 |
|
tch___: Hola viejo |
| 17:13 |
bernie |
tch___: hello! |
| 17:13 |
rralcala |
bernie: is that useful? |
| 17:13 |
cjl |
hola tch___ |
| 17:13 |
rgs_ |
bernie: yeah maybe discussing in terms of service ownership is better? |
| 17:14 |
bernie |
rgs_: yes, that would be good. let's wait until the third agenda item |
| 17:14 |
rgs_ |
bernie: also, as you said, perhaps some of those services that PyEduca will host can be extended to other deployments later on/ |
| 17:14 |
|
? |
| 17:14 |
tch___ |
hola todos :) |
| 17:15 |
rgs_ |
tch___: are you guys gonna be building new images for Caacupe this year? |
| 17:15 |
|
tch___: if so, is this something we want to do in SL's infra? |
| 17:15 |
|
bernie: ^ |
| 17:16 |
bernie |
rralcala: we're all very busy, we don't expect anyone to have all that much free time. however, system administration is different from software development. In case you realize at some point that you lack the time to maintain something,i ask that you look for a new owner and pass the responsibility to this person. |
| 17:16 |
tch___ |
rgs_: yes, since tomorrow ill be mentoring some FPUNA students to help me with the a new build |
| 17:17 |
bernie |
rralcala: can you accept to do this? |
| 17:17 |
rralcala |
bernie: Sure |
| 17:17 |
bernie |
thanks! |
| 17:18 |
tch___ |
rralcala: welcome! |
| 17:18 |
rralcala |
bernie: Also being in GMT +3 I think helps to be available |
| 17:18 |
bernie |
tch___: sorry for not asking before: do you also want to join the infra team? |
| 17:18 |
|
tch___: initially both you and roberto were called, then only him |
| 17:19 |
|
rralcala: yeah, it's good to have people on the team who are awake when the rest of us are sleeping |
| 17:19 |
walterbender |
bernie: shall we jump to - Status update on SL infra (everyone)? |
| 17:19 |
bernie |
walterbender: sure. i'll try to be brief |
| 17:19 |
walterbender |
As I think we #agree that rralcala is an asset |
| 17:19 |
|
#topic - Status update on SL infra (everyone) |
| 17:19 |
tch___ |
bernie: I could help if needed, but I am probably more useful in other place inside SL hehe |
| 17:20 |
bernie |
tch___: ok no worries. |
| 17:20 |
|
ok, where do i start from? |
| 17:20 |
|
i guess from the bottom |
| 17:20 |
|
== Hosting == |
| 17:21 |
|
SL used to be hosted in very disparate locations, but over time we've consolidated on 2 main locations: |
| 17:21 |
|
1. The FSF hosts our main virtual machine, sunjammer, in their colocation facility in boston downtown |
| 17:22 |
|
2. The Media Lab hosts 2 new servers called freedom and justice, plus dogi's machine housetree which is still running a couple of things for us |
| 17:23 |
|
ah, there's also 3. some old buildbots are still running at Develer, in Italy, but we'll probably move everything over and shut down my ancient machine. |
| 17:23 |
|
== Capacity == |
| 17:23 |
|
We have two brand new machines, and lots of spare disk, ram and cpu |
| 17:24 |
|
creating new VMs shouldn't be a problem. the only thing that is hard to scale up is sysadmin time |
| 17:24 |
dnarvaez |
duh I like the italy machine :) |
| 17:24 |
bernie |
dnarvaez: ah ok, then we can keep it around. it has some historic value for me as well :-) |
| 17:24 |
dnarvaez |
:) |
| 17:25 |
bernie |
Our new production VMs should be mostly concentrated on justice. |
| 17:25 |
|
freedom is meant to be a hot spare box, and currently runs a few buildbots for dnarvaez and a few experimental VMs that belong to dogi |
| 17:26 |
|
in case justice suddenly dies, we simply go over to the ML and swap the drives. this is why we shouldn't host anything important on freedom. |
| 17:26 |
|
== Backups == |
| 17:27 |
|
We do cross-backups between freedom and justice. sunjammer still backs itself up on housetree, but we should fix that |
| 17:28 |
|
all VMs _must_ be backed up one way or another. by design, we do not provide backups at the kvm level (details offline if someone is curious) |
| 17:29 |
|
we have plenty of disk space for backups, but it's important to monitor to avoid filling up the disks due to log spew and other files that grow without bounds |
| 17:29 |
|
== Monitoring == |
| 17:29 |
|
we've been using munin for a long time for graphs and alerts. rgs recently upgraded it to version 2.0 |
| 17:29 |
rgs_ |
(recently == a year ago) |
| 17:30 |
bernie |
by our infrastructure's standards :-) |
| 17:30 |
|
muning isn't perfect, but it served us well. it also sends alerts to the systems-logs@ list AND to my phone |
| 17:31 |
|
if anyone has the guts to deal with the annoyance, make it page your phone too |
| 17:31 |
|
anyway, when something goes wrong people generally complain on irc and by email |
| 17:32 |
cjl |
== Infrastructure Documentation == |
| 17:32 |
bernie |
it would be good if everyone (including newcomers) made their contact info available to get notified |
| 17:32 |
cjl |
http://wiki.sugarlabs.org/go/Infrastructure_Team |
| 17:32 |
bernie |
cjl: oh yeah, good point |
| 17:32 |
cjl |
:-) |
| 17:33 |
bernie |
we used to have really crappy documentation, but more recently we became more diligent in keeping it up to date |
| 17:33 |
cjl |
Let's not let it get stale |
| 17:33 |
bernie |
one secret to fresh documentation is... not too much of it |
| 17:34 |
cjl |
hears the same works for sushi. . . |
| 17:34 |
bernie |
i mainly worked to remove redundant info and standardize how we document services |
| 17:34 |
|
rralcala1 <rralcala1!~rralcala141.222.177.40> has joined #sugar-meeting |
| 17:34 |
bernie |
any new service _*MUST*_ be documented in the wiki |
| 17:34 |
|
if it's not documented, it can't be in production |
| 17:34 |
|
actually, the three requirements before we can add something to *.sugarlabs.org are: |
| 17:34 |
|
1. backups |
| 17:34 |
|
2. monitoring |
| 17:35 |
|
3. documentation |
| 17:35 |
|
http://wiki.sugarlabs.org/go/I[…]re_Team/Resources <- this is the sysadmin playbook for most services and administration tasks |
| 17:35 |
|
== Services == |
| 17:35 |
|
There are many. in fact, a bit too many |
| 17:36 |
|
Some of them are half-abandoned, even |
| 17:36 |
|
we'll talk about Trac and Pootle later, those are my main concern right now |
| 17:36 |
cjl |
:-( |
| 17:36 |
bernie |
sunjammer hosts most of our public-facing infrastructure, while jita hosts most of our development infrastructure |
| 17:37 |
|
rralcala has quit IRC |
| 17:37 |
bernie |
we don't have a clear 1:1 relationship between services and servers |
| 17:38 |
|
this has been discussed several times in the past, but I feel that it's easier to maintain fewer machines running multiple *related* things |
| 17:38 |
|
of course, it's a matter of personal preference. whoever pledges to do the work gets to decide how to split it |
| 17:38 |
|
rralcala1 has quit IRC |
| 17:40 |
bernie |
ay, forgot: almost all our VMs run ubuntu at this time and i'd like to keep it this way. I'm primarily a fedora person, but i think that uniformity across the infrastructure saves us a lot of time |
| 17:40 |
|
rralcala1 <rralcala1!~rralcala141.222.183.62> has joined #sugar-meeting |
| 17:40 |
bernie |
migrating everything over to another distro would be overkill, and ubuntu worked very well for us so far (this might change soon :-) |
| 17:41 |
tch___ |
bernie: rracala seems having trouble getting in IRC, (glod bless logs) |
| 17:41 |
sdanielf |
bernie, BTW, they are running Ubuntu 10.04, should they be updated? |
| 17:41 |
rgs_ |
bernie: why might it changed? |
| 17:41 |
rralcala1 |
glod bless logs |
| 17:41 |
rgs_ |
*change |
| 17:41 |
bernie |
sdanielf: yes, sunjammer badly needs an upgrade |
| 17:41 |
|
== Moving forward == |
| 17:42 |
|
1. upgrade sunjammer to precise (one day i'll get around to do it, and it will be painful) |
| 17:43 |
|
2. finish migrating things from treehouse (pootle is the only SL thing missing... dogi is probably happy to host the vms of PyEdu and rgs, but check with him) |
| 17:44 |
|
Sorry, i meant from housetree. treehouse is dead as far as i know. |
| 17:44 |
|
3. Consolidate user accounts: currently we have a mix of ldap, cas, mediawiki accounts and whatnot |
| 17:45 |
|
4. Membership management. Currently, the poor lfaraone has to process all requests by hand. Something like civicrm with a good registration form would save a lot of time. |
| 17:46 |
|
last point goes with point 3: it's shameful and confusing for users that we have different passwords for different parts of the same infra |
| 17:48 |
|
5. deal with spam. some heroes, including fgrose and cjl, spend a lot of time cleaning up wikis, bug tracker and whatnot. Someone should really help them solve the problem with capchas etc |
| 17:49 |
cjl |
recently Pootle has come under attack from forum spamming bots that have learned the trick of activating accounts by clicking on confirmation links |
| 17:49 |
bernie |
6. New website. this has been on the table for *years* with very little progress. Someone should take ownership and get it done for good. I'd recommend something very simple, maybe just a single page, but very well designed. |
| 17:49 |
|
cjl: :-( |
| 17:50 |
cjl |
I had to turn off self-service registration |
| 17:50 |
bernie |
jeez |
| 17:51 |
|
cjl: recently i had to re-enable registration in mediawiki to help tch___ register his students |
| 17:51 |
|
sdanielf has quit IRC |
| 17:51 |
bernie |
I feel that all these problems could be solved if someone takes (3) in their hands. |
| 17:51 |
|
not an easy task, though |
| 17:51 |
cjl |
true |
| 17:52 |
bernie |
it pretty much affects everything |
| 17:52 |
|
sdanielf <sdanielf!~webchatjita.sugarlabs.org> has joined #sugar-meeting |
| 17:52 |
bernie |
alsroot has been working on it and did good progress with CAS |
| 17:52 |
|
ask him for the details, i feel that he has more status than me |
| 17:52 |
|
ok, enough for this agenda item... let's move on |
| 17:52 |
|
i promised i'd be brief, but i lied :-) |
| 17:53 |
rgs_ |
heh |
| 17:53 |
bernie |
walterbender: GA |
| 17:53 |
walterbender |
OK. I think next up was the specific needs of pyedu |
| 17:53 |
bernie |
rralcala1: are you still tuned? |
| 17:53 |
rgs_ |
rralcala1: tch___ : ^ |
| 17:53 |
walterbender |
#topic - Paraguay Educa hosting requirements (rralcala) |
| 17:53 |
|
sdanielf has quit IRC |
| 17:54 |
rralcala1 |
Yes |
| 17:54 |
tch___ |
rgs_: reading :) |
| 17:54 |
rralcala1 |
Pyeduca basically ran into trouble because of lighting/power/ rat eating wire/vpn and no sysadmin |
| 17:54 |
|
and there are core services that keeps caacupe running |
| 17:55 |
|
what we need |
| 17:55 |
|
and tch___ please correct me if I'm wrong |
| 17:55 |
bernie |
rralcala1: yeah, i remember the server room not being a very stable environment |
| 17:55 |
rralcala1 |
is to move two vms that basically keeps caacupe running |
| 17:55 |
|
to somewhere else |
| 17:56 |
|
and SL is the best option for us, in case possible |
| 17:56 |
|
those are inventario and mothership |
| 17:56 |
|
the key server |
| 17:56 |
bernie |
rralcala1: one question: do you know what's the ping time from caacupe to the ML versus the PyEdu office? |
| 17:56 |
tch___ |
rralcala1: you doing good :) |
| 17:57 |
rgs_ |
bernie: probs ~300ms |
| 17:57 |
|
but tch___ and rralcala1 will tell you the right # |
| 17:58 |
rralcala1 |
bernie: Right now we can't test it as VPN is down |
| 17:58 |
|
bernie: It should be around 50 and 100 ms due to the wimax last miles |
| 17:58 |
bernie |
i guess it doesn't matter for mothership |
| 17:58 |
|
and it matters a little for inventario |
| 17:58 |
rralcala1 |
bernie: I agree |
| 17:59 |
bernie |
it matters the most for the wiki, but that's already hosted at SL :-) |
| 17:59 |
rralcala1 |
bernie: Also that's one of the reasons we want to move both |
| 17:59 |
|
bernie: to ensure low latency between them |
| 17:59 |
bernie |
rralcala1: ah, they intercommunicate? |
| 17:59 |
rralcala1 |
bernie: mothership and inventario yes |
| 18:00 |
tch___ |
bernie: inventario feeds mothership with activation info |
| 18:00 |
bernie |
of course they do... i forgot most of the details |
| 18:00 |
rralcala1 |
bernie: The other reason is because inventario holds all the serials and owners :) |
| 18:01 |
bernie |
that's sort of sensitive data. |
| 18:02 |
|
bah, i guess we can carry it. the olpc activation server is racked just next to freedom and justice |
| 18:02 |
|
we should make sure we have plenty of backups, also on a secure machine |
| 18:02 |
tch___ |
bernie: it is, inventario tells which laptop is in the hand of who and where |
| 18:03 |
bernie |
rralcala1, tch___: who needs to have access to those machines, beside the two of you? |
| 18:04 |
rralcala1 |
bernie: I suggest cgaray as well, but I haven't spoke to him |
| 18:05 |
bernie |
rralcala1: ok. keep in mind that any of the SL core sysadmins can easily access the disk of the machine from the kvm host |
| 18:06 |
rralcala1 |
bernie: Ok tch___ and me in the meantime |
| 18:07 |
bernie |
rralcala1: are you going to install 2 new VMs, or move over the old ones? |
| 18:07 |
|
i think i've already asked you this in the past, but for the record... |
| 18:08 |
rralcala1 |
bernie: I'd like to install two new ones |
| 18:08 |
|
bernie: Bu for inventario I'll probably need the hand of tch___ |
| 18:08 |
|
tch___: ^ |
| 18:09 |
tch___ |
rralcala1: yeah, it is the perfect opportunity to upgrade stuff, dsd did a lot of work to inventario/yaas in the last 2 years |
| 18:10 |
bernie |
rralcala1: tch is of course very welcome. if you need to add new collaborators, even to the VMs you manage personally, please make sure they understand and agree to follow our security policies |
| 18:11 |
|
rralcala1: in the past we had trouble with junior admins doing very dangerous things with lax permissions and open ports |
| 18:12 |
|
tch___: that's very cool. do you think we could host a public instance with fake user data as a demo for otherdeployments? |
| 18:13 |
tch___ |
bernie: of course |
| 18:14 |
bernie |
tch___, rralcala1: ok. i guess we have all the requirements? |
| 18:14 |
tch___ |
bernie: every deployment should have its own instance of inventario and yaas ;) |
| 18:14 |
bernie |
ah, what about wiki.paraguayeduca.org? do we want to move it? |
| 18:15 |
rralcala1 |
bernie: I'd like to do it host by host, and I think it is not on amnesia (inventario) |
| 18:16 |
bernie |
dogi called me up on the phone this morning and said that he's happy to host stuff for PyEdu on treehouse, but talk with him for the details. |