Web   ·   Wiki   ·   Activities   ·   Blog   ·   Lists   ·   Chat   ·   Meeting   ·   Bugs   ·   Git   ·   Translate   ·   Archive   ·   People   ·   Donate

#sugar-meeting meeting, 2013-03-31 17:00:54

Minutes | Index | Today     Channels | Search | Join

All times shown according to UTC.

Time Nick Message
17:00 meeting Meeting started Sun Mar 31 17:00:54 2013 UTC. The chair is walterbender. Information about MeetBot at http://wiki.debian.org/MeetBot.
17:00 Useful Commands: #action #agreed #help #info #idea #link #topic #endmeeting
17:01 walterbender let's put together a  quick agenda
17:01 feel free to chime in
17:01 bernie should i send my items?
17:01 walterbender bernie: just post them here
17:02 bernie  - Self introduction (rralcala)
17:02  - Status update on SL infra (everyone)
17:02  - Paraguay Educa hosting requirements (rralcala)
17:02  - Trac maintenance (bernie)
17:02  - Pootle maintenance (bernie)
17:03 ah, and also:
17:03   - hosting code on GitHub (walter)
17:03 walterbender ok..
17:03 bernie anything else?
17:03 walterbender #topic Self introduction (rralcala)
17:04 bernie GA, rralcala
17:04 rralcala bernie: Ok, I'm not good at this but
17:04 walterbender @ bernie: we can add more topics at the end
17:04 rralcala Hi everyone, I'm rralcala, aka Roberto Rodriguez
17:04 cjl hola
17:05 rralcala I worked for a year in the Caacupe (Paraguay) implementation of olpc
17:05 but mostly on the infrastructure side
17:06 also I've some background maintaining infrastructure from other jobs
17:06 And, I'd like to give a hand/request some help if possible
17:07 how long should this be?
17:07 Also
17:08 walterbender rralcala: I think we get the idea :) but tell us about Tanzania...
17:08 rralcala Tanzania is an awesome place
17:08 really
17:08 but we don't get more than 1mbps at home
17:08 The digital breach here is huge
17:08 mostly because education
17:09 so in that perspective it might be a huge challenge doing something like paraguayeduca here
17:09 But I'm working on a telecom company
17:09 Recommended for safari and nice beaches
17:09 walterbender rralcala: will bandwidth be an issue for you re helping with infrastructure?
17:10 rralcala I'm moving to South africa
17:10 and have 3 backup connections
17:10 but yes
17:10 bernie rralcala: what are you going to do in tanzania?
17:10 ah sorry, i missed that line
17:11 rralcala I once had none of the 3 connections working because an fishers boat anchor ran over the backbone in egypt
17:11 bernie rralcala: how much free time can you realistically dedicate to volunteer activities for SL and PyEdu?
17:12 rralcala but I'm 99% available
17:12 tch___ <tch___!~webchat@jita.sugarlabs.org> has joined #sugar-meeting
17:12 rralcala bernie: Ok
17:12 walterbender hi tch___
17:12 rralcala bernie: I won't have much
17:12 tch___ walterbender: hello!, sorry I am late
17:13 rralcala but enough to maintain 2 services, be available in case of emergency and monitor pyeduca's part
17:13 tch___: Hola viejo
17:13 bernie tch___: hello!
17:13 rralcala bernie: is that useful?
17:13 cjl hola tch___
17:13 rgs_ bernie: yeah maybe discussing in terms of service ownership is better?
17:14 bernie rgs_: yes, that would be good. let's wait until the third agenda item
17:14 rgs_ bernie: also, as you said, perhaps some of those services that PyEduca will host can be extended to other deployments later on/
17:14 ?
17:14 tch___ hola todos :)
17:15 rgs_ tch___: are you guys gonna  be building new images for Caacupe this year?
17:15 tch___: if so, is this something we want to do in SL's infra?
17:15 bernie: ^
17:16 bernie rralcala: we're all very busy, we don't expect anyone to have all that much free time. however, system administration is different from software development. In case you realize at some point that you lack the time to maintain something,i ask that you look for a new owner and pass the responsibility to this person.
17:16 tch___ rgs_: yes, since tomorrow ill be mentoring some FPUNA students to help me with the a new build
17:17 bernie rralcala: can you accept to do this?
17:17 rralcala bernie: Sure
17:17 bernie thanks!
17:18 tch___ rralcala: welcome!
17:18 rralcala bernie: Also being in GMT +3 I think helps to be available
17:18 bernie tch___: sorry for not asking before: do you also want to join the infra team?
17:18 tch___: initially both you and roberto were called, then only him
17:19 rralcala: yeah, it's good to have people on the team who are awake when the rest of us are sleeping
17:19 walterbender bernie: shall we jump to - Status update on SL infra (everyone)?
17:19 bernie walterbender: sure. i'll try to be brief
17:19 walterbender As I think we #agree that rralcala is an asset
17:19 #topic - Status update on SL infra (everyone)
17:19 tch___ bernie: I could help if needed, but I am probably more useful in other place inside SL hehe
17:20 bernie tch___: ok no worries.
17:20 ok, where do i start from?
17:20 i guess from the bottom
17:20 == Hosting ==
17:21 SL used to be hosted in very disparate locations, but over time we've consolidated on 2 main locations:
17:21 1. The FSF hosts our main virtual machine, sunjammer, in their colocation facility in boston downtown
17:22 2. The Media Lab hosts 2 new servers called freedom and justice, plus dogi's machine housetree which is still running a couple of things for us
17:23 ah, there's also 3. some old buildbots are still running at Develer, in Italy, but we'll probably move everything over and shut down my ancient machine.
17:23 == Capacity ==
17:23 We have two brand new machines, and lots of spare disk, ram and cpu
17:24 creating new VMs shouldn't be a problem. the only thing that is hard to scale up is sysadmin time
17:24 dnarvaez duh I like the italy machine :)
17:24 bernie dnarvaez: ah ok, then we can keep it around. it has some historic value for me as well :-)
17:24 dnarvaez :)
17:25 bernie Our new production VMs should be mostly concentrated on justice.
17:25 freedom is meant to be a hot spare box, and currently runs a few buildbots for dnarvaez and a few experimental VMs that belong to dogi
17:26 in case justice suddenly dies, we simply go over to the ML and swap the drives. this is why we shouldn't host anything important on freedom.
17:26 == Backups ==
17:27 We do cross-backups between freedom and justice. sunjammer still backs itself up on housetree, but we should fix that
17:28 all VMs _must_ be backed up one way or another. by design, we do not provide backups at the kvm level (details offline if someone is curious)
17:29 we have plenty of disk space for backups, but it's important to monitor to avoid filling up the disks due to log spew and other files that grow without bounds
17:29 == Monitoring ==
17:29 we've been using munin for a long time for graphs and alerts. rgs recently upgraded it to version 2.0
17:29 rgs_ (recently == a year ago)
17:30 bernie by our infrastructure's standards :-)
17:30 muning isn't perfect, but it served us well. it also sends alerts to the systems-logs@ list AND to my phone
17:31 if anyone has the guts to deal with the annoyance, make it page your phone too
17:31 anyway, when something goes wrong people generally complain on irc and by email
17:32 cjl == Infrastructure Documentation ==
17:32 bernie it would be good if everyone (including newcomers) made their contact info available to get notified
17:32 cjl http://wiki.sugarlabs.org/go/Infrastructure_Team
17:32 bernie cjl: oh yeah, good point
17:32 cjl :-)
17:33 bernie we used to have really crappy documentation, but more recently we became more diligent in keeping it up to date
17:33 cjl Let's not let it get stale
17:33 bernie one secret to fresh documentation is... not too much of it
17:34 cjl hears the same works for sushi. . .
17:34 bernie i mainly worked to remove redundant info and standardize how we document services
17:34 rralcala1 <rralcala1!~rralcala1@> has joined #sugar-meeting
17:34 bernie any new service _*MUST*_ be documented in the wiki
17:34 if it's not documented, it can't be in production
17:34 actually, the three requirements before we can add something to *.sugarlabs.org are:
17:34 1. backups
17:34 2. monitoring
17:35 3. documentation
17:35 http://wiki.sugarlabs.org/go/I[…]re_Team/Resources  <- this is the sysadmin playbook for most services and administration tasks
17:35 == Services ==
17:35 There are many. in fact, a bit too many
17:36 Some of them are half-abandoned, even
17:36 we'll talk about Trac and Pootle later, those are my main concern right now
17:36 cjl :-(
17:36 bernie sunjammer hosts most of our public-facing infrastructure, while jita hosts most of our development infrastructure
17:37 rralcala has quit IRC
17:37 bernie we don't have a clear 1:1 relationship between services and servers
17:38 this has been discussed several times in the past, but I feel that it's easier to maintain fewer machines running multiple *related* things
17:38 of course, it's a matter of personal preference. whoever pledges to do the work gets to decide how to split it
17:38 rralcala1 has quit IRC
17:40 bernie ay, forgot: almost all our VMs run ubuntu at this time and i'd like to keep it this way. I'm primarily a fedora person, but i think that uniformity across the infrastructure saves us a lot of time
17:40 rralcala1 <rralcala1!~rralcala1@> has joined #sugar-meeting
17:40 bernie migrating everything over to another distro would be overkill, and ubuntu worked very well for us so far (this might change soon :-)
17:41 tch___ bernie: rracala seems having trouble getting in IRC, (glod bless logs)
17:41 sdanielf bernie, BTW, they are running Ubuntu 10.04, should they be updated?
17:41 rgs_ bernie: why might it changed?
17:41 rralcala1 glod bless logs
17:41 rgs_ *change
17:41 bernie sdanielf: yes, sunjammer badly needs an upgrade
17:41 == Moving forward ==
17:42 1. upgrade sunjammer to precise (one day i'll get around to do it, and it will be painful)
17:43 2. finish migrating things from treehouse (pootle is the only SL thing missing... dogi is probably happy to host the vms of PyEdu and rgs, but check with him)
17:44 Sorry, i meant from housetree. treehouse is dead as far as i know.
17:44 3. Consolidate user accounts: currently we have a mix of ldap, cas, mediawiki accounts and whatnot
17:45 4. Membership management. Currently, the poor lfaraone has to process all requests by hand. Something like civicrm with a good registration form would save a lot of time.
17:46 last point goes with point 3: it's shameful and confusing for users that we have different passwords for different parts of the same infra
17:48 5. deal with spam. some heroes, including fgrose and cjl, spend a lot of time cleaning up wikis, bug tracker and whatnot. Someone should really help them solve the problem with capchas etc
17:49 cjl recently Pootle has come under attack from forum spamming bots that have learned the trick of activating accounts by clicking on confirmation links
17:49 bernie 6. New website. this has been on the table for *years* with very little progress. Someone should take ownership and get it done for good. I'd recommend something very simple, maybe just a single page, but very well designed.
17:49 cjl: :-(
17:50 cjl I had to turn off self-service registration
17:50 bernie jeez
17:51 cjl: recently i had to re-enable registration in mediawiki to help tch___ register his students
17:51 sdanielf has quit IRC
17:51 bernie I feel that all these problems could be solved if someone takes (3) in their hands.
17:51 not an easy task, though
17:51 cjl true
17:52 bernie it pretty much affects everything
17:52 sdanielf <sdanielf!~webchat@jita.sugarlabs.org> has joined #sugar-meeting
17:52 bernie alsroot has been working on it and did good progress with CAS
17:52 ask him for the details, i feel that he has more status than me
17:52 ok, enough for this agenda item... let's move on
17:52 i promised i'd be brief, but i lied :-)
17:53 rgs_ heh
17:53 bernie walterbender: GA
17:53 walterbender OK. I think next up was the specific needs of pyedu
17:53 bernie rralcala1: are you still tuned?
17:53 rgs_ rralcala1: tch___ : ^
17:53 walterbender #topic - Paraguay Educa hosting requirements (rralcala)
17:53 sdanielf has quit IRC
17:54 rralcala1 Yes
17:54 tch___ rgs_: reading :)
17:54 rralcala1 Pyeduca basically ran into trouble because of lighting/power/ rat eating wire/vpn and no sysadmin
17:54 and there are core services that keeps caacupe running
17:55 what we need
17:55 and tch___ please correct me if I'm wrong
17:55 bernie rralcala1: yeah, i remember the server room not being a very stable environment
17:55 rralcala1 is to move two vms that basically keeps caacupe running
17:55 to somewhere else
17:56 and SL is the best option for us, in case possible
17:56 those are inventario and mothership
17:56 the key server
17:56 bernie rralcala1: one question: do you know what's the ping time from caacupe to the ML versus the PyEdu office?
17:56 tch___ rralcala1: you doing good :)
17:57 rgs_ bernie: probs ~300ms
17:57 but tch___ and rralcala1 will tell you the right #
17:58 rralcala1 bernie: Right now we can't test it as VPN is down
17:58 bernie: It should be around 50 and 100 ms due to the wimax last miles
17:58 bernie i guess it doesn't matter for mothership
17:58 and it matters a little for inventario
17:58 rralcala1 bernie: I agree
17:59 bernie it matters the most for the wiki, but that's already hosted at SL :-)
17:59 rralcala1 bernie: Also that's one of the reasons we want to move both
17:59 bernie: to ensure low latency between them
17:59 bernie rralcala1: ah, they intercommunicate?
17:59 rralcala1 bernie: mothership and inventario yes
18:00 tch___ bernie: inventario feeds mothership with activation info
18:00 bernie of course they do... i forgot most of the details
18:00 rralcala1 bernie: The other reason is because inventario holds all the serials and owners :)
18:01 bernie that's sort of sensitive data.
18:02 bah, i guess we can carry it. the olpc activation server is racked just next to freedom and justice
18:02 we should make sure we have plenty of backups, also on a secure machine
18:02 tch___ bernie: it is, inventario tells which laptop is in the hand of who and where
18:03 bernie rralcala1, tch___: who needs to have access to those machines, beside the two of you?
18:04 rralcala1 bernie: I suggest cgaray as well, but I haven't spoke to him
18:05 bernie rralcala1: ok. keep in mind that any of the SL core sysadmins can easily access the disk of the machine from the kvm host
18:06 rralcala1 bernie: Ok tch___ and me in the meantime
18:07 bernie rralcala1: are you going to install 2 new VMs, or move over the old ones?
18:07 i think i've already asked you this in the past, but for the record...
18:08 rralcala1 bernie: I'd like to install two new ones
18:08 bernie: Bu for inventario I'll  probably need the hand of tch___
18:08 tch___: ^
18:09 tch___ rralcala1: yeah, it is the perfect opportunity to upgrade stuff, dsd did a lot of work to inventario/yaas in the last 2 years
18:10 bernie rralcala1: tch is of course very welcome. if you need to add new collaborators, even to the VMs you manage personally, please make sure they understand and agree to follow our security policies
18:11 rralcala1: in the past we had trouble with junior admins doing very dangerous things with lax permissions and open ports
18:12 tch___: that's very cool. do you think we could host a public instance with fake user data as a demo for otherdeployments?
18:13 tch___ bernie: of course
18:14 bernie tch___, rralcala1: ok. i guess we have all the requirements?
18:14 tch___ bernie: every deployment should have its own instance of inventario and yaas ;)
18:14 bernie ah, what about wiki.paraguayeduca.org? do we want to move it?
18:15 rralcala1 bernie: I'd like to do it host by host, and I think it is not on amnesia (inventario)
18:16 bernie dogi called me up on the phone this morning and said that he's happy to host stuff for PyEdu on treehouse, but talk with him for the details.

Minutes | Index | Today     Channels | Search | Join

Powered by ilbot/Modified.